You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Problem:
Today if a chunk record is written corrupted such that a reader loses track of its place relative to records, and a correct EOF index is not written (because of crash, bugs, or configuration), a reader will not be able to parse any subsequent parts of the file, even if subsequent chunks are written uncorrupted.
Potential solution:
For each MCAP recording, generate a random sync marker of sufficient length that it is more-or-less guaranteed never to occur in file content. Put the sync marker in the file header and as the first field of every chunk. This means if your chunk is corrupted, but subsequent uncorrupted chunks are written, a reader may re-synchronize and still extract useful data.
Questions:
We could incorporate this at a regular byte interval as well or in addition, to protect un-chunked files.
Prior art:
Avro incorporates this feature, but documents it mainly in terms of splittability: https://avro.apache.org/docs/current/api/java/org/apache/avro/file/DataFileWriter.html. MCAP files today are splittable as long as the end of file index gets written - however various recording modes or bug conditions could preclude this. Avro files do not contain an index so this is their only mechanism for ready splittability, aside from parsing the file.
Downside: same inputs will no longer produce the same output file hash. Work around this by allowing writers to specify a sync marker if they wish to override the random choice.
Problem:
Today if a chunk record is written corrupted such that a reader loses track of its place relative to records, and a correct EOF index is not written (because of crash, bugs, or configuration), a reader will not be able to parse any subsequent parts of the file, even if subsequent chunks are written uncorrupted.
Potential solution:
For each MCAP recording, generate a random sync marker of sufficient length that it is more-or-less guaranteed never to occur in file content. Put the sync marker in the file header and as the first field of every chunk. This means if your chunk is corrupted, but subsequent uncorrupted chunks are written, a reader may re-synchronize and still extract useful data.
Questions:
We could incorporate this at a regular byte interval as well or in addition, to protect un-chunked files.
Prior art:
The text was updated successfully, but these errors were encountered: