BlackrockIO can't read my nev files.

[layne6689]

Describe the bug
There is an error: "UserWarning: Detected 177 undocumented segments within nev data after timestamps" and "NeoReadWriteError: Inconsistent ns6 and nev file. 178 segments present in .nev file, but 1 in ns6 file," which occurs when I use BlackrockIO to read my .nev and .ns6 files at the same time. This error occurs consistently on our .nev files. But BlackrockIO works fine when I only read the .ns6 file(use load_nev=False). Additionally, it works correctly when using NPMK (https://github.com/BlackrockNeurotech/NPMK) in MATLAB to read the .nev file.

To Reproduce
code:

filepath = 'D:/Desktop/dummy file/test'
reader = BlackrockIO(filename=filepath, load_nev=True)

error:

UserWarning: Detected 177 undocumented segments within nev data after timestamps [   2   19   43   54   86   97  108  128  139  149  161  182  196  209
  226  234  306  315  329  378  401  414  438  453  473  509  559  583
  613  641  653  674  684  742  757  779  802  815  830  842  855  867
  900  910  922  946  982  995 1062 1098 1111 1123 1169 1237 1267 1315
 1327 1338 1369 1379 1416 1425 1469 1529 1548 1560 1582 1595 1683 1715
 1738 1772 1784 1801 1815 1825 1834 1857 1895 1905 1937 1982 1992 2007
 2048 2126 2165 2175 2186 2206 2241 2253 2264 2284 2296 2336 2344 2354
 2392 2438 2452 2486 2495 2506 2529 2545 2557 2568 2592 2647 2671 2689
 2701 2759 2780 2814 2819 2839 2849 2860 2881 2888 2900 2951 2956 2967
 2978 3000 3022 3045 3156 3163 3177 3200 3209 3252 3273 3300 3308 3341
 3364 3376 3396 3420 3432 3450 3484 3496 3508 3540 3551 3588 3682 3694
 3704 3716 3721 3733 3759 3781 3793 3810 3827 3837 3863 3871 3905 3928
 3942 3957 3975 3987 3997 4046 4067 4120 4245].


NeoReadWriteError: Inconsistent ns6 and nev file. 178 segments present in .nev file, but 1 in ns6 file.

I creat a dummy file which have the same error:
dummy file.zip

Can you help me to slove this error? Thank you for your effort.

Environment:

• OS: Windows 11
• Python 3.12.3
• Neo 0.14.2
• NumPy 1.26.4

[zm711]

Thanks @layne6689! And the dummy file will be super useful. Just to help us out do you know if blackrock has had any recent updates or anything just so we can read about it? We were hoping to do a bit more extensive cleanup of blackrockio soon so any additional info would be helpful. @h-mayorquin were you already working on blackrock or are you done with your current work on the IO?

[h-mayorquin]

No, I still have some fixes to do, I can take a look.

[layne6689]

@zm711, thank you for your response. It seems like Blackrock hasn't had recent updates, but I'm not entirely sure. And my .nev file can be read using both the older version of the openNEV function in NPMK from more than 10 years ago and the latest version. I can confirm that the error is not related to the updates.

[zm711]

Thanks @layne6689. Heberto will take a look when he has a moment!

[h-mayorquin]

Hi, @layne6689

I took a look today.

What I see is that you have unclassified spikes of type 0 coming from electrodes 17 and 19.

The data that I see is the folowing:

data["Spikes"][0]["packet_id"][:10]
array([17, 17, 19, 17, 17, 17, 17, 17, 17, 17], dtype=uint16)
timestamps_in_milliseconds[:10]
array([ 0.56666667,  2.3       ,  0.8       ,  3.6       ,  4.5       ,
        6.53333333,  8.2       ,  9.        , 10.53333333, 11.4       ])

Does the data looks correct? is this what you get when you read your data in matlab?

The source of the problem is that the timestamps are decreasing which contradicts the spec:

@samuelgarcia @zm711
Back in #523 the decision was make to split segments when timestamps became negative AND this reset criteria is meet:

python-neo/neo/rawio/blackrockrawio.py

Lines 1316 to 1325 in 7c899cc

	def _get_reset_event_mask(self, raw_event_data, masks, nev_data_masks):
	"""
	Extract mask for reset comment events in 2.3 .nev file
	"""
	restart_mask = np.logical_and(
	masks["Comments"][nev_data_masks["Comments"]],
	raw_event_data["value"] == b"\x00\x00\x00\x00\x00\x00critical load restart",
	)
	# TODO: Fix hardcoded number of bytes
	return restart_mask

Two comments:

1. I could not find documentation for this on the spec. My impression is that comments are something that the user sets but this undocumented functionality look suspect to me. Maybe we overfit with respect to a user annotation? The code is literally matching for a very specific value on a comment to determine if there is a reset on the nev.

2. Independently of that, in this case there are no reset events, should we still divide by segments when there are negative timestamps? Let's get a more clear picture on why this could be from the user to make a decision.

[h-mayorquin]

Yes, I can find documentation about pause and resume functionality (in an another event that is not the comments event) but nothing about resetting the timestamps. Do you guys know some blackrock user that could help us @zm711 @samuelgarcia ?

[h-mayorquin]

Could not find anything here either:
#795

Also, I could not find neither of these strategies on the matlab or blackrock implementation:

• Blackrock Python Utilities Repository
• NPMK - Neural Processing Matlab Kit

Maybe we should re-think our logic for segment nev. I think that if the channel data is interleaved as it is in the user data this should not be segmented, we should only segment if the data within the channel suffers a big jump.

All of that said, I have not find any formal documentation that this should happen (that the data might jump back), if someone has that, it would be good to have those docs.

[h-mayorquin]

@cboulay here at the top of this issue the user shared the file with the spikes in the strange order that we discussed today.

[h-mayorquin]

1. Turns out that this:

python-neo/neo/rawio/blackrockrawio.py

Lines 1316 to 1325 in 7c899cc

	def _get_reset_event_mask(self, raw_event_data, masks, nev_data_masks):
	"""
	Extract mask for reset comment events in 2.3 .nev file
	"""
	restart_mask = np.logical_and(
	masks["Comments"][nev_data_masks["Comments"]],
	raw_event_data["value"] == b"\x00\x00\x00\x00\x00\x00critical load restart",
	)
	# TODO: Fix hardcoded number of bytes
	return restart_mask

Is documented on the central software, so we should add a comment to it.

[layne6689]

@h-mayorquin Sorry, I’m a beginner, and I don’t quite understand what you said. But I can share the matlab file

read_nev_in_NPMK.zip to read the dummy file, maybe you can easily run it and have a check.

[h-mayorquin]

Hi, @layne6689, no worries, I kind of opened too many threads on this conversation.

What I wanted to ask you is how your data looks like. You say that you were able to load the data, how does it look like?

The data that I loaded looked like this. Two units associated to electrodes 17 and 19 each with the following timestamps.

data["Spikes"][0]["packet_id"][:10]
array([17, 17, 19, 17, 17, 17, 17, 17, 17, 17], dtype=uint16)
timestamps_in_milliseconds[:10]
array([ 0.56666667,  2.3       ,  0.8       ,  3.6       ,  4.5       ,
        6.53333333,  8.2       ,  9.        , 10.53333333, 11.4       ])

[cboulay]

FYI, at least myself and one person at Blackrock who I consulted with are stumped by this issue. The way the system works, packets within a 'stream' (raw, filtered, or spk) cannot be out of order. So you should not get a spike at 0.8 msec AFTER a spike at 2.3 msec. We'll investigate if the problem could happen in the file-saving process in Central.

Could you please tell us what version of Central you used to record this?