feat: finalize change set file format #15
Conversation
Signed-off-by: yihuang <[email protected]>
|
|
||
| last_version = None | ||
| offset = 0 | ||
| output = Path(out_dir) / f"block-{start_version}" |
There was a problem hiding this comment.
not sure if it's a tradeoff to get rid of small files but we might overwrite output with different end_version but with same start_version
There was a problem hiding this comment.
yeah, it's to avoid too many small files, more performant to process and easier to distribute.
we might overwrite output with different end_version but with same start_version
currently it don't override exiting files, but it could be an issue if the chunk files are not continuously or overlapping, but that's an issue of operation, for example, use a fixed granularity.
| ``` | ||
| version: varint | ||
| size: varint # the total size of kv-pairs, so we can skip faster | ||
| kv-pairs: length prefixed proto msg |
There was a problem hiding this comment.
not sure if we need delimiter for this len, msg, len, msg,…
There was a problem hiding this comment.
do you mean: msg, msg, ..., remove the length prefix?
There was a problem hiding this comment.
I think protobuf won't work without an end mark, unless we encode the static fields directly.
There was a problem hiding this comment.
what's the difference?
|
golang version will have more complete support for change set management: crypto-org-chain/cronos#791 |
Closes: #14
The changeset file format is:
Index files:
key -> bitmap of block numbers.Query procedure: