CNDB-15508: Query planner metrics

[pkolaczk]

This commit adds new metrics related to the operation of SAI query
planner. The metrics should help checking if the query planner makes
proper decisions by correlating them with the other metrics,
e.g. the metrics of the actual query execution.

Per-query metrics (histograms):

• RowsEstimated: the estimated number of rows to be returned by
  the query
• CostEstimated: the abstract cost of query execution
• InverseSelectivityEstimated: the inverse of query selectivity,
  before applying the query LIMIT
  (1 means the query selects all rows, 10 means it
  selects every 10th row, etc.)
• IndexReferencesInQuery: the number of index references in the
  unoptimized query execution plan (the same index may
  be referenced multiple times and counts separately)
• IndexReferencesInPlan: the number of index references in the
  optimized query execution plan (the same index may
  be referenced multiple times and counts separately)

Per-table:

• TotalRowsEstimated: counts the sum of all row estimates from
  all completed queries
• TotalCostEstimated: counts the sum of all cost estimates from
  all completed queries
• TotalQueriesCompletedInSelectivityGroup{N}, where N in [0, 12):
  counts the number of completed queries with selectivity S:
  10^(-N-1) < S <= 10^(-N) for N < 11,
  S <= 10^(-N) for N = 11
  In other words, the higher the group N, the smaller fraction of rows
  the query is estimated to return. The selectivity calculation
  does not include the final LIMIT of the query.

[github-actions]

Checklist before you submit for review

• This PR adheres to the Definition of Done
• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits
• All new files should contain the DataStax copyright header instead of the Apache License one

[ekaterinadimitrova2]

Not a full review, just started looking into this, just dropped a few quick comments

[ekaterinadimitrova2]

Looks great, left just some questions.
There are some GH problems today and CI fails. You will need to restart it to check the results.

[ekaterinadimitrova2]

Overall looks great, I just left some tiny comments.
There is some issue with CI so it did not run.
Regarding testing - shouldn't we add some test for the new flag added - cassandra.sai.metrics.query_plan.enabled?

[pkolaczk]

@ekaterinadimitrova2

Regarding testing - shouldn't we add some test for the new flag added - cassandra.sai.metrics.query_plan.enabled?

Yes, added.

[adelapena]

Looks great, +1

[ekaterinadimitrova2]

About SAI_QUERY_PLAN_METRICS_ENABLED:

• On the other PR there was a mention the new property may work also dynamically, during runtime. I think we need to add a test for that too, if that will be the expectation.

• I left a comment that we need to reset to default value the property at the end of each test where we toggle it.

• Thinking about enable/disable at runtime - we should think whether that would persist the metrics we already collected (in case we disable) or not. WDYT?

• My main concern is that we report 0s instead of not registering the metrics at all once we disable them. That is something we discussed when we added a setting to be able to disable the IndexMetrics. More in the PR comments.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
95.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[cassci-bot]

❌ Build ds-cassandra-pr-gate/PR-2130 rejected by Butler

---

7 regressions found
See build details here

---

Found 7 new test failures

Test	Explanation	Runs	Upstream
junit.framework.TestSuite.org.apache.cassandra.distributed.test.sai.datamodels.QueryRowDeletionsTest-.jdk11	REGRESSION	🔴🔴	0 / 19
junit.framework.TestSuite.org.apache.cassandra.distributed.test.sai.datamodels.QueryWriteLifecycleTest-.jdk11	NEW	🔴🔴	0 / 19
o.a.c.distributed.test.sai.datamodels.QueryRowDeletionsTest.testRowDeletions[CompositePartitionKeyDataModel]	REGRESSION	🔴🔴	0 / 19
o.a.c.distributed.test.sai.datamodels.QueryWriteLifecycleTest.testWriteLifecycle[CompositePartitionKeyDataModel]	NEW	🔴🔴	0 / 19
o.a.c.distributed.test.sai.datamodels.QueryWriteLifecycleTest.testWriteLifecycle[CompoundKeyWithStaticsDataModel]	NEW	🔴🔴	0 / 19
o.a.c.index.sai.cql.VectorCompaction100dTest.testZeroOrOneToManyCompaction[dc false]	NEW	🔴⚪	0 / 19
o.a.c.metrics.TrieMemtableMetricsTest.testContentionMetrics (compression)	REGRESSION	🔴🔴	2 / 19

---

Found 1 known test failures