Ra v3 #13885
Draft
Ra v3 #13885
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kjnilsson
force-pushed
the
ra-v3
branch
2 times, most recently
from
89003b4 to
0720342
Compare
kjnilsson
force-pushed
the
ra-v3
branch
5 times, most recently
from
46c30f4 to
59d889f
Compare
kjnilsson
force-pushed
the
ra-v3
branch
6 times, most recently
from
6aa7ef0 to
45121a9
Compare
And tidy up some backwards compatible code that is no longer needed.
From the main state machine (still used in the dlx module). This can be done as we no longer need to super efficiently query the smallest raft index. Removing it will reduce peak memory use somewhat as well as simplifying the code.
Currently a message references' smallest representation is an improper list with two integer terms; the Raft index and the message size in bytes. This has a heap overhead of 2 words + 2 words for the cons cell in the queue data structure. This commit introduces a new more compact form that is used if the Raft index is less than 44 bits and the message size is less than 32KiB. In this case the Raft index and message size is packed into a 59 bit integer which is the largest unsigned integer that is still an immediate term and thus has 0 heap overhead as the integer value fits into the pointer word completely. This reduces per message overhead to the 2 words for the cons cell and thus halves message memory use for a lot of scenarios. If a message cannot be packed it will fall back to the current list based representation.
Not really a problem but effects passed to checkout/4 would be reversed which could potentially cause future confusion.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR to test Ra v3 changes
Snapshotting
The state machine needs to start tracking the approximate size of all commands that do not need to be kept (anything but enqueues effective). After this it should be as simple as requesting a snapshot every so many reclaimed bytes. This number may be scaled according to the number of messages on queue as this affects the snapshot size.
1M messages creates snapshots in the region of 30MB. We probably dont want to create a snapshot until at least twice the snapshot size can be potentially reclaimed. Num msgs * 64 bytes of reclaimable command data.