[Bug]: mismatch between timestamps and frames in miniscope video

[laurelrr]

What happened?

I am working with this repo: https://github.com/catalystneuro/tye-lab-to-nwb/tree/main/src/tye_lab_to_nwb/ast_ophys
and attempting to convert Miniscope calcium recordings but got the same error on 2 subjects.

Steps to Reproduce

python convert_session.py

Traceback

Traceback (most recent call last):
  File "/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/convert_session_carl.py", line 149, in session_to_nwb
    converter.run_conversion(
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/ast_ophysnwbconverter.py", line 59, in run_conversion
    super().run_conversion(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 160, in run_conversion
    self.add_to_nwbfile(nwbfile_out, metadata, conversion_options)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 114, in add_to_nwbfile
    data_interface.add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeconverter.py", line 64, in add_to_nwbfile
    self.data_interface_objects["MiniscopeImaging"].add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeimagingdatainterface.py", line 91, in add_to_nwbfile
    imaging_extractor.set_times(times=miniscope_timestamps)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/roiextractors/imagingextractor.py", line 196, in set_times
    assert len(times) == self.get_num_frames(), "'times' should have the same length of the number of frames!"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'times' should have the same length of the number of frames!

Operating System

Linux

Python Executable

Conda

Python Version

3.11

Package Versions

environment_for_issue.txt

Code of Conduct

Yes

Duplicated Issue Check

Yes

[h-mayorquin]

@weiglszonja. @alessandratrapani (tagging both of you as we have worked on this)

I discussed this with @laurelrr it seems that one of the sessions for the Tye lab has a session where the timestamps do not match the number of frames. So there is an error here:

roiextractors/src/roiextractors/imagingextractor.py

Lines 189 to 198 in 2cf9235

	def set_times(self, times: ArrayType) -> None:
	"""Set the recording times (in seconds) for each frame.
	Parameters
	----------
	times: array-like
	The times in seconds for each frame
	"""
	assert len(times) == self.get_num_frames(), "'times' should have the same length of the number of frames!"
	self._times = np.array(times).astype("float64")

The timestamps are obtained from a csv that is specific to miniscope and aggregated here:

https://github.com/catalystneuro/neuroconv/blob/52cd6aa0ea46c0d143739767da4951ade17c3b74/src/neuroconv/datainterfaces/ophys/miniscope/miniscopeimagingdatainterface.py#L80-L85

And the frames are the sum of the frames of each individual extractor here:

roiextractors/src/roiextractors/multiimagingextractor.py

Lines 40 to 47 in 2cf9235

	self._start_frames, self._end_frames = [], []
	num_frames = 0
	for imaging_extractor in self._imaging_extractors:
	self._start_frames.append(num_frames)
	num_frames = num_frames + imaging_extractor.get_num_frames()
	self._end_frames.append(num_frames)
	self._num_frames = num_frames

Which in turn are obtained from the video directly using OpenCV.

roiextractors/src/roiextractors/extractors/miniscopeimagingextractor/miniscopeimagingextractor.py

Lines 105 to 107 in 2cf9235

	def get_image_size(self) -> Tuple[int, int]:
	return self._image_size[:-1]

So I think that either the timestamps in the csv are wrong or there are some drop frames in some videos. @laurelrr will see how he can share the data with us so we can verify that.

Meanwhile, what should we do in the case there are dropped frames? I think dropping frames is somehow common in acquisition systems that involve video so I think we should handle that case gracefully but I am not sure how.

[alessandratrapani]

The miniscope TimeStamps.csv has a dedicated column "Buffer_Index" to signal problems with frame acquisition, e.g., dropped frames. We can start by looking at this piece of data.

[h-mayorquin]

OK, this happens because there are some files that match the *.avi wildcard but not in the intended way in one of the segments:

├── 13_59_04
│   ├── BehavCam_2
│   │   ├── 0.avi
│   │   ├── 1.avi
│   │   ├── 2.avi
│   │   ├── 3.avi
│   │   ├── 4.avi
│   │   ├── 5.avi
│   │   ├── 6.avi
│   │   ├── 7.avi
│   │   ├── 8.avi
│   │   ├── metaData.json
│   │   └── timeStamps.csv
│   ├── metaData.json
│   ├── Miniscope
│   │   ├── 0.avi
│   │   ├── 0_corrected.avi
│   │   ├── 1.avi
│   │   ├── 2.avi
│   │   ├── 3.avi
│   │   ├── 4.avi
│   │   ├── metaData.json
│   │   └── timeStamps.csv
│   └── notes.csv
└── G2B4_C5-6852_L_yeti_DISC6UNP+TONES_msCamAll_ffd.mat

See the 0_corrected.avi

Can/should we improve the extractor to not match things with letters?

[laurelrr]

OK, my fix for this was to make a copy of the data folder and mv 0_corrected.avi 0.avi for each instance of a mislabeled filename.
I assume I did not need to reprocess any of the other downstream *.mat files that are used in convert_session.py, but please correct me if that seems wrong.
I'm trying it out now and will let you know the results.

[laurelrr]

OK, my idea did not seem to fix the issue. I will confer with my lab mates and see what else might have been affected.

[laurelrr]

After conferring with the team, it looks like we need only the 0.avi behavCam and the 0_corrected.avi miniscope file (which I have renamed to 0.avi). I set this up in my copy of the original folder. it now looks like this:

├── 13_59_04
│   ├── BehavCam_2
│   │   ├── 0.avi
│   │   ├── metaData.json
│   │   └── timeStamps.csv
│   ├── metaData.json
│   ├── Miniscope
│   │   ├── 0.avi   (previously, 0_corrected.avi)
│   │   ├── metaData.json
│   │   └── timeStamps.csv
│   └── notes.csv
└── G2B4_C5-6852_L_yeti_DISC6UNP+TONES_msCamAll_ffd.mat

I am trying to run pulling from the updated miniscope folder, but still see the same error message:

Traceback (most recent call last):
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/convert_session_yeti.py", line 149, in session_to_nwb
    converter.run_conversion(
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/ast_ophysnwbconverter.py", line 59, in run_conversion
    super().run_conversion(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 160, in run_conversion
    self.add_to_nwbfile(nwbfile_out, metadata, conversion_options)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 114, in add_to_nwbfile
    data_interface.add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeconverter.py", line 64, in add_to_nwbfile
    self.data_interface_objects["MiniscopeImaging"].add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeimagingdatainterface.py", line 91, in add_to_nwbfile
    imaging_extractor.set_times(times=miniscope_timestamps)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/roiextractors/imagingextractor.py", line 196, in set_times
    assert len(times) == self.get_num_frames(), "'times' should have the same length of the number of frames!"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'times' should have the same length of the number of frames!

Is there something else that should be updated, beyond just deleting extraneous files?

[laurelrr]

I also tried cropping the timeStamps.csv file to the correct length and tried running again but this also gives the same error:

Traceback (most recent call last):
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/convert_session_yeti.py", line 149, in session_to_nwb
    converter.run_conversion(
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/ast_ophysnwbconverter.py", line 59, in run_conversion
    super().run_conversion(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 160, in run_conversion
    self.add_to_nwbfile(nwbfile_out, metadata, conversion_options)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 114, in add_to_nwbfile
    data_interface.add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeconverter.py", line 64, in add_to_nwbfile
    self.data_interface_objects["MiniscopeImaging"].add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeimagingdatainterface.py", line 91, in add_to_nwbfile
    imaging_extractor.set_times(times=miniscope_timestamps)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/roiextractors/imagingextractor.py", line 196, in set_times
    assert len(times) == self.get_num_frames(), "'times' should have the same length of the number of frames!"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'times' should have the same length of the number of frames!

[laurelrr]

Is your code making use of the metaData.json information when loading in the files?

[weiglszonja]

Is your code making use of the metaData.json information when loading in the files?

Yes, we are extracting the information from "recordingStartTime". But I don't see how that would affect the mismatch in frames. Can you share this session in its current form?

[laurelrr]

OK, I think I have figured out the issue. For some unknown reason, the Miniscope reset its frame rate from 15 fps to 60 fps partway through the recording. To correct this aberrant recording, my colleague used ffmpeg to change frame rate of the video but had not updated the timestamps file. I corrected the timestamps to match the video (and frame rate) and this now processes as expected.

If you wanted to make a more interpretable error message, listing the folder where the first instance of mismatch between the timestamps file and video frame rates would be helpful for your users. As always, thanks for the help!