[Bug]: Miniscope converter - 'samples is not consistent over the files'

[GarrettBlair]

What happened?

Hello, looking into the miniscope converter again as discussed in #356 with @h-mayorquin. I'm attempting to run it on a different data set, though similar configuration (2 miniscopes, no behavior). I get an error concerning the number of frames in the .avi files.

setup the converter and reference the original config file:
converter = MiniscopeConverter( folder_path="E:/", user_configuration_file_path="E:/UserConfig.json", verbose=False)

...
  File "C:\Users\gjb326\AppData\Roaming\Python\Python313\site-packages\roiextractors\multiimagingextractor.py", line 80, in _check_consistency_between_imaging_extractors
    len(unique_values) == 1
AssertionError: The number of samples is not consistent over the files (found {1000, 614}).

Full error here - https://pastebin.com/bjymiFxA

When acquiring, each .avi has 100 frames (or however much is specified by the config.json), but the last file can have fewer than that (and is likely to happen often). In this case, one camera ("HPC_miniscope1") files 0.avi-52.avi have 1000 frames, and 53.avi has 614 frames. For the other camera, "ACC_miniscope2", files 0.avi-52.avi have 1000 frames, and 53.avi has 643 frames.

So the last .avi file should be allowed to have <= the expected frames number.

Steps to Reproduce

from pathlib import Path
from zoneinfo import ZoneInfo
from neuroconv.converters import MiniscopeConverter

converter = MiniscopeConverter( folder_path="E:/", user_configuration_file_path="E:/UserConfig.json", verbose=False)

Traceback

C:\Users\gjb326\AppData\Roaming\Python\Python313\site-packages\pydantic\_internal\_validate_call.py:136: UserWarning: Ignoring 'dataDirectory' field in User Config ('D:'). Using provided folder_path: 'E:\'.
  res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
Traceback (most recent call last):
  File "<python-input-17>", line 1, in <module>
    converter = MiniscopeConverter(
        folder_path="E:/",
        user_configuration_file_path="E:/UserConfig.json",
        verbose=False,
    )
  File "C:\Users\gjb326\AppData\Roaming\Python\Python313\site-packages\pydantic\_internal\_validate_call.py", line 39, in wrapper_function
    return wrapper(*args, **kwargs)
  File "C:\Users\gjb326\AppData\Roaming\Python\Python313\site-packages\pydantic\_internal\_validate_call.py", line 136, in __call__
    res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
  File "C:\Users\gjb326\AppData\Roaming\Python\Python313\site-packages\neuroconv\datainterfaces\ophys\miniscope\miniscopeconverter.py", line 262, in __init__
    interface = MiniscopeImagingInterface(folder_path=device_folder_path)
  File "C:\Users\gjb326\AppData\Roaming\Python\Python313\site-packages\pydantic\_internal\_validate_call.py", line 39, in wrapper_function
    return wrapper(*args, **kwargs)
  File "C:\Users\gjb326\AppData\Roaming\Python\Python313\site-packages\pydantic\_internal\_validate_call.py", line 136, in __call__
    res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
  File "C:\Users\gjb326\AppData\Roaming\Python\Python313\site-packages\neuroconv\datainterfaces\ophys\miniscope\miniscopeimagingdatainterface.py", line 295, in __init__
    super().__init__(
    ~~~~~~~~~~~~~~~~^
        folder_path=folder_path,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        verbose=verbose,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\gjb326\AppData\Roaming\Python\Python313\site-packages\neuroconv\datainterfaces\ophys\baseimagingextractorinterface.py", line 69, in __init__
    super().__init__(**source_data)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "C:\Users\gjb326\AppData\Roaming\Python\Python313\site-packages\neuroconv\baseextractorinterface.py", line 16, in __init__
    self._extractor_instance = self._initialize_extractor(source_data)
                               ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "C:\Users\gjb326\AppData\Roaming\Python\Python313\site-packages\neuroconv\datainterfaces\ophys\baseimagingextractorinterface.py", line 57, in _initialize_extractor
    extractor_instance = extractor_class(**self.extractor_kwargs)
  File "C:\Users\gjb326\AppData\Roaming\Python\Python313\site-packages\roiextractors\extractors\miniscopeimagingextractor\miniscopeimagingextractor.py", line 237, in __init__
    super().__init__(imaging_extractors=imaging_extractors)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\gjb326\AppData\Roaming\Python\Python313\site-packages\roiextractors\multiimagingextractor.py", line 38, in __init__
    self._check_consistency_between_imaging_extractors()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "C:\Users\gjb326\AppData\Roaming\Python\Python313\site-packages\roiextractors\multiimagingextractor.py", line 80, in _check_consistency_between_imaging_extractors
    len(unique_values) == 1
AssertionError: The number of samples is not consistent over the files (found {1000, 614}).

Operating System

Windows

Python Executable

Conda

Python Version

3.10

Package Versions

No response

Code of Conduct

Yes

Duplicated Issue Check

No

[h-mayorquin]

Yes, I don't know why do we have that check:

roiextractors/src/roiextractors/multiimagingextractor.py

Lines 69 to 75 in 551e5c4

	properties_to_check = dict(
	get_sampling_frequency="The sampling frequency",
	get_sample_shape="The shape of a sample",
	get_channel_names="The name of the channels",
	get_dtype="The data type",
	get_num_samples="The number of samples",
	)

I seems perfectly sensible to me to concatenate extractors with different number of samples. @pauladkisson, is there anything that I might be missing?