You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
For whatever reason, the timestamps in timestamps.npy of an event stream in an OpenEphys >=0.6 binary dataset can become corrupt, resulting in them being non-monotonic and some of them being set to -1. This renders that event stream unusable, including by downstream packages (e.g. spikeinterface), as neo always uses the synchronized timestamps when they are available. When this happens, the sample numbers in the corresponding sample_numbers.npy file seem to be correct (or at least plausible), but are not trivially accessible, especially if not using neo directly but via another package (in my case, spikeinterface).
Describe the solution you'd like
I can see two non-exclusive solutions.
One would be to have neo automatically detect such corrupt timestamps (I think a simple check like timestamps == sorted(timestamps) should suffice) and fall back to using the sample numbers and sample rate to compute the event timestamps and durations instead.
Another would be to allow the user to choose whether to use the synchronized timestamps or the raw sample numbers for reading out event timestamps and calculating the durations. There's a flag for this in the OpenEphysBinaryRawIO constructor, but it gets clobbered on lines 201-204 of openephysbinaryrawio.py. There should really be two flags: does the dataset have synchronized timestamps, and does the user want to use them or the raw sample numbers? Default behaviour should be to use the synchronized timestamps if they exist and are uncorrupt, otherwise do whatever the user says if possible (i.e. still use sample numbers if the user asks for synchronized timestamps from an OpenEphys <0.6 dataset, maybe with a warning).
Describe alternatives you've considered
Right now my workaround is to cut neo/spikeinterface out of the loop entirely and just manually read out the event information from the OpenEphys binary folder.
Corrupt timestamps are in channel 0 ('Rhythm FPGA TTL Events'), which I think is the only one accessible in the release version of neo, but you can access all event streams by checking out the incoming branch from this PR: #1430
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
For whatever reason, the timestamps in
timestamps.npy
of an event stream in an OpenEphys >=0.6 binary dataset can become corrupt, resulting in them being non-monotonic and some of them being set to -1. This renders that event stream unusable, including by downstream packages (e.g. spikeinterface), as neo always uses the synchronized timestamps when they are available. When this happens, the sample numbers in the correspondingsample_numbers.npy
file seem to be correct (or at least plausible), but are not trivially accessible, especially if not using neo directly but via another package (in my case, spikeinterface).Describe the solution you'd like
I can see two non-exclusive solutions.
One would be to have neo automatically detect such corrupt timestamps (I think a simple check like
timestamps == sorted(timestamps)
should suffice) and fall back to using the sample numbers and sample rate to compute the event timestamps and durations instead.Another would be to allow the user to choose whether to use the synchronized timestamps or the raw sample numbers for reading out event timestamps and calculating the durations. There's a flag for this in the
OpenEphysBinaryRawIO
constructor, but it gets clobbered on lines 201-204 ofopenephysbinaryrawio.py
. There should really be two flags: does the dataset have synchronized timestamps, and does the user want to use them or the raw sample numbers? Default behaviour should be to use the synchronized timestamps if they exist and are uncorrupt, otherwise do whatever the user says if possible (i.e. still use sample numbers if the user asks for synchronized timestamps from an OpenEphys <0.6 dataset, maybe with a warning).Describe alternatives you've considered
Right now my workaround is to cut neo/spikeinterface out of the loop entirely and just manually read out the event information from the OpenEphys binary folder.
Additional context
Example dataset exhibiting this behaviour is here: https://drive.google.com/file/d/1Hr3-GFOpOkNc1hgXdVr5HTunchQm4enw/view?usp=drive_link
Corrupt timestamps are in channel 0 ('Rhythm FPGA TTL Events'), which I think is the only one accessible in the release version of neo, but you can access all event streams by checking out the incoming branch from this PR: #1430
The text was updated successfully, but these errors were encountered: