Improve profile query performance (and most likely performance elsewhere)

[ErikEJ]

Summary of the changes (in less than 80 characters):

• Removes unneeded related table from query
• Adds a useful index for queries against the packages table to avoid expensive key lookups

Addresses #10221 (partly)

fixes #5877
fixes #5659
fixes #10025

[ErikEJ]

Did some informal testing

Baseline: 5078 ms
Final: 1895 ms

With signers:

(20 rows affected)
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'PackageRegistrationRequiredSigners'. Scan count 20, logical reads 0, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'PackageRegistrations'. Scan count 2, logical reads 190, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'PackageRegistrationOwners'. Scan count 22, logical reads 20080, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Packages'. Scan count 10002, logical reads 3109738, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Workfile'. Scan count 0, logical reads 0, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Organizations'. Scan count 1, logical reads 0, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Users'. Scan count 0, logical reads 40, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.

(1 row affected)

 SQL Server Execution Times:
   CPU time = 4547 ms,  elapsed time = 5078 ms.

 SQL Server Execution Times:
   CPU time = 4547 ms,  elapsed time = 5078 ms.

With signers, updated index:

(20 rows affected)
Table 'PackageRegistrationRequiredSigners'. Scan count 20, logical reads 0, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'PackageRegistrations'. Scan count 2, logical reads 190, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'PackageRegistrationOwners'. Scan count 22, logical reads 20080, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Packages'. Scan count 10002, logical reads 83920, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Organizations'. Scan count 1, logical reads 0, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Users'. Scan count 0, logical reads 40, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.

(1 row affected)

 SQL Server Execution Times:
   CPU time = 3141 ms,  elapsed time = 3549 ms.

 SQL Server Execution Times:
   CPU time = 3141 ms,  elapsed time = 3549 ms.

No signers:

(20 rows affected)
Table 'Organizations'. Scan count 1, logical reads 0, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Users'. Scan count 0, logical reads 40, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'PackageRegistrationOwners'. Scan count 21, logical reads 10060, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'PackageRegistrations'. Scan count 1, logical reads 95, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Packages'. Scan count 5001, logical reads 1554869, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.

(1 row affected)

 SQL Server Execution Times:
   CPU time = 2250 ms,  elapsed time = 2525 ms.

 SQL Server Execution Times:
   CPU time = 2250 ms,  elapsed time = 2525 ms.

No signers, updated index:

(20 rows affected)
Table 'Organizations'. Scan count 1, logical reads 0, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Users'. Scan count 0, logical reads 40, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'PackageRegistrationOwners'. Scan count 21, logical reads 10060, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'PackageRegistrations'. Scan count 1, logical reads 95, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Packages'. Scan count 5001, logical reads 41960, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Workfile'. Scan count 0, logical reads 0, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.

(1 row affected)

 SQL Server Execution Times:
   CPU time = 1672 ms,  elapsed time = 1895 ms.

 SQL Server Execution Times:
   CPU time = 1672 ms,  elapsed time = 1895 ms.

[joelverhagen]

This appears to be essentially copying the table to have a new clustered index (effectively). I think this is an option, however we need to maintain this list of columns along with the Packages table itself. This is a maintenance challenge.

I am not sure how effectively SQL Server can add a nullable column to an existing index's INCLUDE. I know adding a nullable column (or a column with a specific default value) to a table is fast. We have done this many times. Based on some web searches it looks like we'll need to recreate the index.

So there are two concerns here:

• How do we keep this list up to data as we add columns to the Packages table. I think we would need remember to do a CREATE INDEX with ONLINE and DROP_EXISTING. We would need to write this SQL each time. If we forget, I think the query engine will stop using the index because all of the needed columns are not present. EF does not help us here.
• We don't have a good way to test the performance impact of the query rebuild and the impact on other endpoints. It is non-trivial to FF this index.

I am thinking more and more that we shouldn't be using SQL for this and instead use our Search Service to populate the Profile page.

[ErikEJ]

I think this index included all columns at the time it was created, and then slipped under the radar, causing it to never be used. I am reviving it. Agree on the up to date challenges, but I have no concerns regarding the performance impact. It can only get better.

[ErikEJ]

Basically, the design of the Packages table is wrong, should have had the PackageRegistrationKey as part of the clustering key from day one.

[joelverhagen]

I think we're getting killed by the download count sort and the fact that package registrations can have multiple owners. I don't know how much time you want to pour into getting SQL to serve use for this query pattern. Our search service would be able to do this very quickly, e.g.

(internal search endpoint, not the same as the V3 search)
https://azuresearch-usnc.nuget.org/search/query?q=owner:soenneker&sortBy=totalDownloads-desc&take=20
We have code that calls into this already. It should be not too bad to produce the list of packages using this (and perhaps keep the summary on SQL, sensible to cache later).

[ErikEJ]

@joelverhagen I implemented random idea 2 (I think?) 😄

256 ms!

(20 rows affected)
Table 'Organizations'. Scan count 1, logical reads 0, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Users'. Scan count 0, logical reads 40, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'PackageRegistrationOwners'. Scan count 20, logical reads 88, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'Packages'. Scan count 20, logical reads 165, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.
Table 'PackageRegistrations'. Scan count 1, logical reads 50, physical reads 0, page server reads 0, read-ahead reads 0, page server read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob page server reads 0, lob read-ahead reads 0, lob page server read-ahead reads 0.

(1 row affected)

 SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 256 ms.

 SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 256 ms.

[ErikEJ]

@joelverhagen Let me do some testing without the "problematic" index changes