fix: 🚑 make it so is_in_bucket is consistent across same file key records

[TDemeco]

This PR fixes a bug where the is_in_bucket field of files in the indexer DB wasn't being populated correctly when creating new file records for a file that was already previously stored by the MSP.

We allow new storage requests for file keys that have already been fulfilled to allow the user to add redundancy to them (in the form of more BSPs), but since the MSP already has the file from the previous storage request, it can accept the new one with an inclusion proof instead of a non-inclusion proof. This makes it so the runtime doesn't update its bucket root (since it shouldn't, as the bucket already had the file from before), which means the MutationsApplied event is never emitted, and this is the event that the indexer detects to update the is_in_bucket field of a file record.
This made it so the new file records in the indexer DB created by these subsequent storage requests permanently had their is_in_bucket field set to false, and so this could create inconsistencies which could lead to the indexer trying to delete a file record that still has an MSP association, failing to import the block and stalling.

The fix is twofold:

• First, correct the mistake in the current records found in the DB, by executing a migration that checks for file records that have both is_in_bucket set to false and a sibling record (i.e. another file record for the same file key) that has its is_in_bucket set to true (I believe it would have been enough to check the oldest sibling record, but just in case we check all of them), and setting the is_in_bucket field for those files to true.
• Then the more permanent fix is that now when the indexer creates a new file record, instead of defaulting the is_in_bucket field of the new record to false, it checks if any sibling record has its is_in_bucket field set to true and if so, creates the new record with the field set to true as well.

Note

As a comment, I left a function to check MSP associations to file records in the indexer. This is not currently used, but I believe we should add that check before every deletion once we are up and running in an environment where the indexer stalling would end up being critical and we rather keep it working while we work on the fix of whatever caused the inconsistency. Fixing inconsistent information could prove hard though, so we might end up deciding not to use this function at all and maintain the indexer's current behaviour.

⚠️ Breaking Changes ⚠️

• Short description

  There's a new migration for the indexer DB that must be executed for all existing DBs.

• Who is affected

  • Indexer node runners since they'll have to run the new migration.

• Suggested code changes

  No code changes required.

[HermanObst]

Starting 👀

[HermanObst]

So the only way this can happen is if the SR is already accepted by the MSP and the user send other one to increase?

[TDemeco]

Yes, as that's the only way to have more than one file record for the same file key.

[HermanObst]

So when the user ask for more redundancy the MSP needs to "re-accept" it, right?

[TDemeco]

Yes, otherwise the new storage request will expire without an MSP response, so it will be considered rejected.

[HermanObst]

LGTM.
For tracking reasons, it would be cool to be bit more specific with the current bug.
e.g: exactly what was happening with the files with the flag set on false, and why these where triggering a failure in the block import and thus the stall.

Not for this PR, but it would be high priority that we add a test that replicates this specific case, that way we can't re introduce the bug later.
@santikaplan

[TDemeco]

LGTM. For tracking reasons, it would be cool to be bit more specific with the current bug. e.g: exactly what was happening with the files with the flag set on false, and why these where triggering a failure in the block import and thus the stall.

It's a bit convoluted and hard to explain in detail so it wouldn't add much context, but simplifying it a bit the issue we faced was:
The indexer checks after each deletion done by the fisherman whether a file record of that file has any remaining association (BSP or MSP) and tries to delete the file record from the DB if there are none. Since the is_in_bucket for a file was incorrectly set to false even though it had an MSP association, the indexer thought that it was safe to delete it from the DB, tried, but the DB constraint of not allowing a file record deletion if it has any remaining associations made it so the delete returned an error, to which the indexer reacted by erroring out from the block indexing function and retrying for the same block, indefinitely.