General API for handling sample gaps on rawio

[h-mayorquin]

@zm711, @samuelgarcia, and I had a discussion today about two current Blackrock issues (#1770 and #1755).

The main problem is that the gaps automatically detected by the code are much smaller than what would reasonably qualify as a segment in Neo (for example, someone deliberately pausing the recording). These gaps are more likely artifacts of the system (in #1770, it looks like a case of dropped samples).

Creating format-specific heuristics for each format would be a large maintenance burden. A simpler, more maintainable, and general solution is to provide users (who know their data best) with both information and control. We came up with the following proposal:

1. Default behavior: If gaps are detected, loading should error out. RawIO readers should raise an error when timestamp gaps are found and display a report showing the number, size, and characteristics of the gaps. This way, users can make an informed decision. Examples of this approach can be seen in Blackrock add summary of automatic data segmentation  #1769 and Improve intan reader error message for discontinuities #1484.
2. Opt-in behavior: RawIO readers should provide an argument (e.g. segmentation_threshold, though we can choose a better name) that lets users explicitly load data with gaps if they wish. Gaps smaller than the value will be ignored, gaps larger than the value will be segmented.
3. Timestamps from acquisition system: To make data with gaps more useful, we should provide the original timestamps from the acquisition system when available (see Add utility method to get timestamps on Intan base #1652).

This design would allow us to implement a consistent API across RawIO, so gaps are handled with a common interface. The plan is as follows:

1. Implement the solution first for Blackrock (since there are open issues and users currently cannot read the data). This will also let us to discuss the types and naming of the interface.
2. Extend the interface to other popular readers (e.g. Intan, OpenEphys, SpikeGLX, Plexon) to uncover potential difficulties.
3. Once the API is stable for the main readers, integrate it into the abstract/parent classes as a general RawIO API.
4. If possible, deprecate the current gap-handling mechanisms already in place (e.g. in OpenEphys) to simplify the codebase.

Another important point raised by @samuelgarcia is that integrity checks can be computationally expensive. We discussed adding a general flag (e.g. ignore_integrity_checks) that would let users opt out of all checks, including timestamp checks. This would make data access faster in performance-sensitive environments, while the default behavior would remain safe with integrity checks enabled. To see an example of this interface on Intan check: #1470. Important, note that ignore_integrity_checks is not only for timestamps gaps but covers any safety mechanism to check the integrity of the files (see here for one of such checks: #1740).

[samuelgarcia]

Thanks for the summary.
I would vote for gap_tolerance instead of segmentation_threshold.

[h-mayorquin]

Thanks for the summary. I would vote for gap_tolerance instead of segmentation_threshold.

I think that's a better name but I would like to add s or seconds at the end to make it more specific:

gap_tolerance_s

[zm711]

We voted at the maintenance meeting and we agreed to the general premise of this proposal. We want to:

1. Raises errors with gaps and provide informative errors about the nature of the gaps.
2. Ask the users to define their tolerance for gaps by setting a tolerance value which will then prevent an error from being raised and instead will segment at that tolerance level. gap_tolerance_s vs gap_tolerance_ms to be determined.
3. Add timestamps information for appropriate readers (barring any issues with keeping the rawio lazy)
4. Finally add a feature that skips these checks for users that want a fast reader capability and know their data is okay to not check for any issues (ignore_integrity_checks or potentially another name).

[h-mayorquin]

Great.

I think gap_tolerance_ms is better.

[PeterNSteinmetz]

I think whatever the unit is should be explicit, either in the name or as a parameter. I frankly prefer seconds. While many systems operate with ms as the underlying unit, others do not. For example, Neuralynx raw timestamps are microseconds.

The way I have dealt with this NeuralynxRawIO is by separating out the concept of an 'NcsSection' of contiguous samples with no gaps. There is a factory then to build these from a file. These are then used in the RawIO to present data in the standard way; however, a user can also use this to scan a file to detect when there are gaps. I would suggest a similar separation for the more general case. The options provided in the constructors then control how these sections are used to provide the more general interface.