Nightly and unstable tags?

[autata]

Hi @bryanlb and others.

I see the introduction of unstable and nightly tags. Can you elaborate on what these mean and give color on what you all are running at slack? We're running astra as well and want to be up-to-date but not on unstable builds.

Thanks!

For clarity, i've started running HEAD in our staging environment and I'm getting nullpointer exceptions:

Cannot read field "cacheNodeId" because the return value of "com.slack.astra.chunk.ReadOnlyChunkImpl.getCacheNodeAssignment()" is null
t exception.error_root_cause_class_name	   	java.lang.NullPointerException
t exception.error_root_cause_message	   	Cannot read field "cacheNodeId" because the return value of "com.slack.astra.chunk.ReadOnlyChunkImpl.getCacheNodeAssignment()" is null
t exception.error_root_cause_stack_trace	   	java.lang.NullPointerException: Cannot read field "cacheNodeId" because the return value of "com.slack.astra.chunk.ReadOnlyChunkImpl.getCacheNodeAssignment()" is null
	at com.slack.astra.chunk.ReadOnlyChunkImpl.registerSearchMetadata(ReadOnlyChunkImpl.java:474)
	at 

and i think it is due to one of 2 PRs:

• PR that changes the metadata store access: Bburkholder/ng metadata #1341
• PR that changes some cache load logic (and adds that failing line of code): 47c6884#diff-a38d77682a26b4b7ee3aa142674e3328a39a2fdada5a74538fa4b43e59b723dfR461

[kyle-sammons]

👋 Hey @autata! We should probably add some more official documentation around it, but the meaning of the tags are:

• stable - The given commit has been running stably on our clusters for a week to two weeks (excluding dep bumps). This is generally what we run our most important clusters on
• unstable - The most recent commit, which has no guarantee of working or being bug-free. This is generally what we run on our test and low traffic clusters
• nightly - All the commits from that day, which in theory should include reverts/fixes assuming there's a catastrophic issue. We don't use this one much, but it can be used as a bit of a happy-medium between the two

As for the second part of your question, it's a bit tricky to make major jumps between commits in Astra, since we don't support normal semantic versioning. Generally we'd recommend you stay as up to date as possible with whichever tag you choose to stick with to avoid any issues. In this case, and in the case of most deploys, the safest ordering is to deploy:

1. Leaf nodes (cache, indexer, recovery)
2. Preprocessor nodes
3. Manager nodes
4. Query nodes

I'd recommend giving that a go and seeing if it resolves your issue with the metadata

[autata]

Ok, this is helpful. Thanks!

It looks like i got to the root cause of the issue. We have not enabled dynamic chunks yet, but assignment which is null is a dynamic chunk notion (?) and I believe it is null for slot based caching at least on some calls.

it seems that somewhere, probably this PR starts checking it for searchable and that throws a nullptr exception
https://github.com/slackhq/astra/pull/1314/files#diff-a38d77682a26b4b7ee3aa142674e3328a39a2fdada5a74538fa4b43e59b723dfR457-R458

It is possible it is not the above PR but something else could have busted the slot-based caching logic, making assignment null. Considering it makes sense to move onto the latest and we wanted to enable dynamic caching, I'm testing out dynamic caching rather than trying to fix the other code path (when astra.ng.dynamicChunkSizes=false). But that is where my debugging landed me on this specific issue.

I'm assuming none of your clusters have astra.ng.dynamicChunkSizes=false?

[kyle-sammons]

Ahh, yep, that'd do it. And that's correct, we have dynamic chunk sizes enabled for all of our clusters now. I'd highly recommend y'all also enable it as we saw a significant reduction in the amount of cache nodes that we needed to run (which for us at least reduced our cost). I can't remember the exact number, but I believe it was in the region of 20%. In fact we're planning on removing that feature flag and making dynamic chunk sizes as the default eventually