feat: implement follower file download flow #653
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
downloadendpoint for the trusted file transfer server that gets a file from storage and returns the(ChunkId, Chunk)pair.TODOS
pg_backend_pid()and veryfing the it is the pid holding the lock (via query topg_lockstable).TrieMutation::Removewhere the file has not been downloaded yet. Not handling this might result in the Follower still having the file and not delete it (if it requests it for the follower before it was deleted and it gets the file after the finality event was processed).ProcessFollowerDownloadswe get a "snapshot" of the files to download an iterate on it. This is not ideal because it does not take into account that the list may change while the files are being downloaded (For example, same as 1) you can be downloading a file that was later deleted). The only solution that came to me was to have a more atomic state machine where you keep in state what file exactly are you downloading and process the case you get TrieMutation::Remove on that file. Also you would never get the snapshot, you would poll the remaining list one by one and get sort of a "download_lock" on a single file.