Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detect corrupt timestamps in OpenEphys >=0.6 binary files, or allow user to choose whether to use timestamps or sample numbers, or both #1438

Open
johnmbarrett opened this issue Mar 19, 2024 · 0 comments
Milestone

Comments

@johnmbarrett
Copy link

Is your feature request related to a problem? Please describe.
For whatever reason, the timestamps in timestamps.npy of an event stream in an OpenEphys >=0.6 binary dataset can become corrupt, resulting in them being non-monotonic and some of them being set to -1. This renders that event stream unusable, including by downstream packages (e.g. spikeinterface), as neo always uses the synchronized timestamps when they are available. When this happens, the sample numbers in the corresponding sample_numbers.npy file seem to be correct (or at least plausible), but are not trivially accessible, especially if not using neo directly but via another package (in my case, spikeinterface).

Describe the solution you'd like
I can see two non-exclusive solutions.

One would be to have neo automatically detect such corrupt timestamps (I think a simple check like timestamps == sorted(timestamps) should suffice) and fall back to using the sample numbers and sample rate to compute the event timestamps and durations instead.

Another would be to allow the user to choose whether to use the synchronized timestamps or the raw sample numbers for reading out event timestamps and calculating the durations. There's a flag for this in the OpenEphysBinaryRawIO constructor, but it gets clobbered on lines 201-204 of openephysbinaryrawio.py. There should really be two flags: does the dataset have synchronized timestamps, and does the user want to use them or the raw sample numbers? Default behaviour should be to use the synchronized timestamps if they exist and are uncorrupt, otherwise do whatever the user says if possible (i.e. still use sample numbers if the user asks for synchronized timestamps from an OpenEphys <0.6 dataset, maybe with a warning).

Describe alternatives you've considered
Right now my workaround is to cut neo/spikeinterface out of the loop entirely and just manually read out the event information from the OpenEphys binary folder.

Additional context
Example dataset exhibiting this behaviour is here: https://drive.google.com/file/d/1Hr3-GFOpOkNc1hgXdVr5HTunchQm4enw/view?usp=drive_link

Corrupt timestamps are in channel 0 ('Rhythm FPGA TTL Events'), which I think is the only one accessible in the release version of neo, but you can access all event streams by checking out the incoming branch from this PR: #1430

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants