Description
Problem:
Today if a chunk record is written corrupted such that a reader loses track of its place relative to records, and a correct EOF index is not written (because of crash, bugs, or configuration), a reader will not be able to parse any subsequent parts of the file, even if subsequent chunks are written uncorrupted.
Potential solution:
For each MCAP recording, generate a random sync marker of sufficient length that it is more-or-less guaranteed never to occur in file content. Put the sync marker in the file header and as the first field of every chunk. This means if your chunk is corrupted, but subsequent uncorrupted chunks are written, a reader may re-synchronize and still extract useful data.
Questions:
We could incorporate this at a regular byte interval as well or in addition, to protect un-chunked files.
Prior art:
- Avro incorporates this feature, but documents it mainly in terms of splittability: https://avro.apache.org/docs/current/api/java/org/apache/avro/file/DataFileWriter.html. MCAP files today are splittable as long as the end of file index gets written - however various recording modes or bug conditions could preclude this. Avro files do not contain an index so this is their only mechanism for ready splittability, aside from parsing the file.
- Discussion of this strategy as beneficial on deep space links: https://ipnpr.jpl.nasa.gov/progress_report/42-136/136H.pdf