CNDB-16146: Reproduce missing rows failure

[michaelsembwever]

What is the issue

https://github.com/riptano/cndb/issues/16146
CC5 UDT test failure.

Not all rows are being returned from the tables with dropped frozen collection/tuple after schema recreation.

Key observations are:

• Only happens in CC5 (but also with test data from Cassandra 4.1 and 5.0)
• Only needs data in sstables (i.e. the commitlogs can flushed and ignored)
• Does not happen if the *-Statistics.db files are deleted
• More than half the rows are missing, there's not yet any discerning pattern to which rows are missing, and which rows are missing changes test run to test run…
• There is only one sstable
• The column types are the same before and after schema recreation.
• The SerializationHeader looks the same as the default used (when the file doesn't exist). (needs further investigation)
• DeserialisationHeader.isDropped(cell) returns the same on failing and passing tests.
• Everything in system_schema tables is the same after schema recreation as it was before.
• Rows are missing from both the writes before the column was dropped and those after the column was dropped.

To reproduce

ant jar -Dant.gen-doc.skip=true

.build/run-tests.sh -a jvm-dtest -t CNDB_16146_Test

[github-actions]

Checklist before you submit for review

• This PR adheres to the Definition of Done
• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits
• All new files should contain the DataStax copyright header instead of the Apache License one

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[cassci-bot]

❌ Build ds-cassandra-pr-gate/PR-2143 rejected by Butler

---

3 regressions found
See build details here

---

Found 3 new test failures

Test	Explanation	Runs	Upstream
o.a.c.cql3.validation.operations.AggregationQueriesTest.testAggregationQueryShouldNotTimeoutWhenItExceedesReadTimeout (compression)	REGRESSION	🔴🔴	2 / 23
o.a.c.distributed.test.CNDB_16146_Test.testMissingRows	NEW	🔴🔴	0 / 23
o.a.c.metrics.TrieMemtableMetricsTest.testContentionMetrics (compression)	REGRESSION	🔴🔴	9 / 23

---

Found 6 known test failures

[michaelsembwever]

apologies, the plan isn't to commit this, the failure is already covered in DropUDTWithRestartTest.
(once it's figured out what the failure is then we might further reduce it and include it)

The purpose of this PR currently is to have the minimal reproducible for other engineers to investigate/understand.