AssertionError: Not all signals have the same length

[laurelrr]

The good news is that this is the last subject in my dataset... I've managed to convert all the other data to nwb.

(tyenwbenv39) lkeyes@puma:~/Projects/GIT/tye-lab-to-nwb$ python src/tye_lab_to_nwb/neurotensin_valence/neurotensin_valence_convert_session_14.py
Source data is valid!
Continuous files do not have aligned timestamps; clipping to make them aligned.
Traceback (most recent call last):
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/neurotensin_valence/neurotensin_valence_convert_session_14.py", line 265, in <module>
    session_to_nwb(
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/neurotensin_valence/neurotensin_valence_convert_session_14.py", line 149, in session_to_nwb
    converter = NeurotensinValenceNWBConverter(source_data=source_data)
  File "/home/lkeyes/anaconda3/envs/tyenwbenv39/lib/python3.9/site-packages/neuroconv/nwbconverter.py", line 65, in __init__
    self.data_interface_objects = {
  File "/home/lkeyes/anaconda3/envs/tyenwbenv39/lib/python3.9/site-packages/neuroconv/nwbconverter.py", line 66, in <dictcomp>
    name: data_interface(**source_data[name])
  File "/home/lkeyes/anaconda3/envs/tyenwbenv39/lib/python3.9/site-packages/neuroconv/datainterfaces/ecephys/openephys/openephysdatainterface.py", line 45, in __new__
    return OpenEphysLegacyRecordingInterface(
  File "/home/lkeyes/anaconda3/envs/tyenwbenv39/lib/python3.9/site-packages/neuroconv/datainterfaces/ecephys/openephys/openephyslegacydatainterface.py", line 56, in __init__
    available_streams = self.get_stream_names(
  File "/home/lkeyes/anaconda3/envs/tyenwbenv39/lib/python3.9/site-packages/neuroconv/datainterfaces/ecephys/openephys/openephyslegacydatainterface.py", line 17, in get_stream_names
    stream_names, _ = OpenEphysLegacyRecordingExtractor.get_streams(
  File "/home/lkeyes/anaconda3/envs/tyenwbenv39/lib/python3.9/site-packages/spikeinterface/extractors/neoextractors/neobaseextractor.py", line 71, in get_streams
    neo_reader = cls.get_neo_io_reader(cls.NeoRawIOClass, **neo_kwargs)
  File "/home/lkeyes/anaconda3/envs/tyenwbenv39/lib/python3.9/site-packages/spikeinterface/extractors/neoextractors/neobaseextractor.py", line 64, in get_neo_io_reader
    neo_reader.parse_header()
  File "/home/lkeyes/anaconda3/envs/tyenwbenv39/lib/python3.9/site-packages/neo/rawio/baserawio.py", line 178, in parse_header
    self._parse_header()
  File "/home/lkeyes/anaconda3/envs/tyenwbenv39/lib/python3.9/site-packages/neo/rawio/openephysrawio.py", line 164, in _parse_header
    assert all(all_sigs_length[0] == e for e in all_sigs_length),\
AssertionError: Not all signals have the same length

Content of neurotensin_valence_convert_session_14.py:

"""Primary script to run to convert an entire session for of data using the NWBConverter."""
import traceback
from importlib.metadata import version
from pathlib import Path
from typing import Optional, Dict
from warnings import warn

from dateutil import parser

from neuroconv.utils import (
    load_dict_from_file,
    dict_deep_update,
    FilePathType,
    FolderPathType,
)
from nwbinspector import inspect_nwbfile
from nwbinspector.inspector_tools import save_report, format_messages
from packaging.version import Version

from tye_lab_to_nwb.neurotensin_valence import NeurotensinValenceNWBConverter


def session_to_nwb(
    nwbfile_path: FilePathType,
    ecephys_recording_folder_path: Optional[FolderPathType] = None,
    subject_metadata: Optional[Dict[str, str]] = None,
    plexon_file_path: Optional[FilePathType] = None,
    events_file_path: Optional[FilePathType] = None,
    pose_estimation_file_path: Optional[FilePathType] = None,
    pose_estimation_config_file_path: Optional[FilePathType] = None,
    pose_estimation_sampling_rate: Optional[float] = None,
    session_start_time: Optional[str] = None,
    original_video_file_path: Optional[FilePathType] = None,
    labeled_video_file_path: Optional[FilePathType] = None,
    confocal_images_oif_file_path: Optional[FilePathType] = None,
    confocal_images_composite_tif_file_path: Optional[FilePathType] = None,
    stub_test: bool = False,
):
    """
    Converts a single session to NWB.

    Parameters
    ----------
    nwbfile_path : FilePathType
        The file path to the NWB file that will be created.
    ecephys_recording_folder_path: FolderPathType
         The path that points to the folder where the OpenEphys (.continuous) files are located.
    subject_metadata: dict, optional
        The optional metadata for the experimental subject.
    plexon_file_path: FilePathType, optional
        The path that points to the Plexon (.plx) file that contains the spike times.
    events_file_path: FilePathType, optional
        The path that points to the .mat file that contains the event onset and offset times.
    pose_estimation_file_path: FilePathType, optional
        The path that points to the .csv file that contains the DLC output.
    pose_estimation_config_file_path: FilePathType, optional
        The path that points to the .pickle file that contains the DLC configuration settings.
    original_video_file_path: FilePathType, optional
        The path that points to the original behavior movie that was used for pose estimation.
    labeled_video_file_path: FilePathType, optional
        The path that points to the labeled behavior movie.
    confocal_images_oif_file_path: FilePathType, optional
        The path that points to the Olympus Image File (.oif).
    confocal_images_composite_tif_file_path: FilePathType, optional
        The path that points to the TIF image that contains the confocal images aggregated over depth.
    stub_test: bool, optional
        For testing purposes, when stub_test=True only writes a subset of ecephys and plexon data.
        Default is to write the whole ecephys recording and plexon data to the file.
    """

    source_data = dict()
    conversion_options = dict()

    # Add Recording
    if ecephys_recording_folder_path:
        recording_source_data = dict(folder_path=str(ecephys_recording_folder_path), stream_name="Signals CH")
        if Version(version("neo")) > Version("0.12.0"):
            recording_source_data.update(ignore_timestamps_errors=True)

        source_data.update(dict(Recording=recording_source_data))
        conversion_options.update(dict(Recording=dict(stub_test=stub_test)))

    # Add Sorting (optional)
    if plexon_file_path:
        source_data.update(dict(Sorting=dict(file_path=str(plexon_file_path))))
        conversion_options.update(dict(Sorting=dict(stub_test=stub_test)))

    # Add Behavior (optional)
    # Add events
    if events_file_path:
        event_names_mapping = {
            0: "reward_stimulus_presentation",
            1: "phototagging",
            2: "shock_stimulus_presentation",
            3: "reward_delivery",
            4: "shock_relay",
            5: "port_entry",
            6: "neutral_stimulus_presentation",
        }
        read_kwargs = dict(event_names_mapping=event_names_mapping)
        events_source_data = dict(file_path=str(events_file_path), read_kwargs=read_kwargs)
        source_data.update(dict(Events=events_source_data))

        events_column_mappings = dict(onset="start_time", offset="stop_time")
        events_conversion_options = dict(column_name_mapping=events_column_mappings)
        conversion_options.update(
            dict(
                Events=events_conversion_options,
            )
        )

    # Add pose estimation (optional)
    pose_estimation_source_data = dict()
    pose_estimation_conversion_options = dict()
    if pose_estimation_file_path:
        pose_estimation_source_data.update(file_path=str(pose_estimation_file_path))
        if pose_estimation_config_file_path:
            pose_estimation_source_data.update(config_file_path=str(pose_estimation_config_file_path))
        elif pose_estimation_sampling_rate is not None:
            pose_estimation_conversion_options.update(rate=float(pose_estimation_sampling_rate))

        source_data.update(dict(PoseEstimation=pose_estimation_source_data))

    if original_video_file_path:
        pose_estimation_conversion_options.update(original_video_file_path=original_video_file_path)
        source_data.update(
            dict(
                OriginalVideo=dict(file_paths=[str(original_video_file_path)]),
            )
        )
    if labeled_video_file_path:
        pose_estimation_conversion_options.update(labeled_video_file_path=labeled_video_file_path)

    if pose_estimation_source_data:
        # The edges between the nodes (e.g. labeled body parts) defined as array of pairs of indices.
        edges = [(0, 1), (0, 2), (2, 3), (1, 3), (5, 6), (5, 7), (5, 8), (5, 9)]
        pose_estimation_conversion_options.update(edges=edges)
        conversion_options.update(dict(PoseEstimation=pose_estimation_conversion_options))

    # Add confocal images
    images_source_data = dict()
    if confocal_images_oif_file_path:
        images_source_data.update(file_path=str(confocal_images_oif_file_path))
        if confocal_images_composite_tif_file_path:
            images_source_data.update(composite_tif_file_path=str(confocal_images_composite_tif_file_path))

        source_data.update(dict(Images=images_source_data))

    converter = NeurotensinValenceNWBConverter(source_data=source_data)

    # Add datetime to conversion
    metadata = converter.get_metadata()

    # Update default metadata with the editable in the corresponding yaml file
    editable_metadata_path = Path(__file__).parent / "metadata" / "general_metadata.yaml"
    editable_metadata = load_dict_from_file(editable_metadata_path)
    metadata = dict_deep_update(metadata, editable_metadata)

    if subject_metadata:
        metadata = dict_deep_update(metadata, dict(Subject=subject_metadata))

    if "session_id" not in metadata["NWBFile"]:
        if ecephys_recording_folder_path:
            ecephys_recording_folder_path = Path(ecephys_recording_folder_path)
            ecephys_folder_name = ecephys_recording_folder_path.name
            session_id = ecephys_folder_name.replace(" ", "").replace("_", "-")
        elif pose_estimation_file_path:
            session_id = Path(pose_estimation_file_path).name.replace(" ", "").replace("_", "-")
        else:
            session_id = Path(nwbfile_path).stem.replace(" ", "").replace("_", "-")

        metadata["NWBFile"].update(session_id=session_id)

    if "session_start_time" not in metadata["NWBFile"]:
        if session_start_time is None:
            raise ValueError(
                "When ecephys recording is not specified the start time of the session must be provided."
                "Specify session_start_time in YYYY-MM-DDTHH:MM:SS format (e.g. 2023-08-21T15:30:00)."
            )
        session_start_time_dt = parser.parse(session_start_time)
        metadata["NWBFile"].update(session_start_time=session_start_time_dt)

    nwbfile_path = Path(nwbfile_path)
    try:
        # Run conversion
        converter.run_conversion(
            nwbfile_path=str(nwbfile_path), metadata=metadata, conversion_options=conversion_options
        )

        # Run inspection for nwbfile
        results = list(inspect_nwbfile(nwbfile_path=nwbfile_path))
        report_path = nwbfile_path.parent / f"{nwbfile_path.stem}_inspector_result.txt"
        save_report(
            report_file_path=report_path,
            formatted_messages=format_messages(
                results,
                levels=["importance", "file_path"],
            ),
        )
    except Exception as e:
        with open(f"{nwbfile_path.parent}/{nwbfile_path.stem}_error_log.txt", "w") as f:
            f.write(traceback.format_exc())
        warn(f"There was an error during the conversion of {nwbfile_path}. The full traceback: {e}")


if __name__ == "__main__":
    # Parameters for conversion
    # The path that points to the folder where the OpenEphys (.continuous) files are located.
    ecephys_folder_path = Path("/nadata/snlkt/data/hao/Neurotensin/ephys/recordings/forLaurel/14_2019-08-30_09-21-05_Disc4")

    # The path that points to the Plexon file (optional)
    # plexon_file_path = None
    plexon_file_path = Path("/nadata/snlkt/data/hao/Neurotensin/ephys/recordings/forLaurel/14_2019-08-30_09-21-05_Disc4/0014_20190825_Disc4.plx")

    # Parameters for events data (optional)
    # The file path that points to the events.mat file
    # events_mat_file_path = None
    events_mat_file_path = Path(
        "/nadata/snlkt/data/hao/Neurotensin/ephys/recordings/forLaurel/14_2019-08-30_09-21-05_Disc4/0014_20190825_Disc4_events.mat"
    )

    # Parameters for pose estimation data (optional)
    # The file path that points to the DLC output (.CSV file)
    # pose_estimation_file_path = None
    pose_estimation_file_path = (
        "/nadata/snlkt/data/hao/Neurotensin/DLC/DLCresults/14_DiscDLC_resnet50_Hao_MedPC_ephysFeb9shuffle1_800000.csv"
    )
    # The file path that points to the DLC configuration file (.pickle file), optional
    # pose_estimation_config_file_path = None
    pose_estimation_config_file_path = (
        "/snlkt/data/hao/Neurotensin/DLC/DLC_files_from_Aardvark/14_DiscDLC_resnet50_Hao_MedPC_ephysFeb9shuffle1_800000includingmetadata.pickle"
    )
    # If the pickle file is not available the sampling rate in units of Hz for the behavior data must be provided.
    pose_estimation_sampling_rate = None

    # For sessions where only the pose estimation data is available the start time of the session must be provided.
    # The session_start_time in YYYY-MM-DDTHH:MM:SS format (e.g. 2023-08-21T15:30:00).
    session_start_time = None

    # The file path that points to the behavior movie file, optional
    #original_video_file_path = None
    original_video_file_path = r"/nadata/snlkt/data/hao/Neurotensin/ephys/recordings/PVT-BLA_NT_KO_NACproj batch1/Discrimination day 4/video/14_20190830_Disc4.asf"
    # The file path that points to the labeled behavior movie file, optional
    # labeled_video_file_path = None
    labeled_video_file_path = "/nadata/snlkt/data/hao/Neurotensin/DLC/DLCresults/14_DiscDLC_resnet50_Hao_MedPC_ephysFeb9shuffle1_800000_labeled.mp4"

    # Parameters for histology images (optional)
    # The file path to the Olympus Image File (.oif)
    #  confocal_images_oif_file_path = None
    confocal_images_oif_file_path = "/nadata/snlkt/data/hao/Neurotensin/Imaging/H9-14/H14/H14_slidePVT4_slice6_zstack_40x_PVT.oif"
    # The file path to the aggregated confocal images in TIF format.
    confocal_images_composite_tif_file_path = None
    #confocal_images_composite_tif_file_path = r"/snlkt/data/hao/Neurotensin/CellProfiler/NT CRISPR/channel merged/H13_MAX_Composite.tif"

    # The file path where the NWB file will be created.
    nwbfile_path = Path("/nadata/snlkt/data/hao/Neurotensin/NWB/nwbfiles/H14_Disc4.nwb")

    # For faster conversion, stub_test=True would only write a subset of ecephys and plexon data.
    # When running a full conversion, use stub_test=False.
    stub_test = False

    # subject metadata (optional)
    subject_metadata = dict(sex="M",age="P60D",genotype="Wild type",strain="C57BL/6J",subject_id="H14")

    session_to_nwb(
        nwbfile_path=nwbfile_path,
        ecephys_recording_folder_path=ecephys_folder_path,
        subject_metadata=subject_metadata,
        plexon_file_path=plexon_file_path,
        events_file_path=events_mat_file_path,
        pose_estimation_file_path=pose_estimation_file_path,
        pose_estimation_config_file_path=pose_estimation_config_file_path,
        pose_estimation_sampling_rate=pose_estimation_sampling_rate,
        session_start_time=session_start_time,
        original_video_file_path=original_video_file_path,
        labeled_video_file_path=labeled_video_file_path,
        confocal_images_oif_file_path=confocal_images_oif_file_path,
        confocal_images_composite_tif_file_path=confocal_images_composite_tif_file_path,
        stub_test=stub_test,
    )

[weiglszonja]

@laurelrr Great news! Can you share this openephys session with us? I'll take a closer look.

[laurelrr]

@weiglszonja You got it! I am attempting to transfer now. I created a new folder called hao_subject14 in the Tye_data_share drive with all the ephys data. Thanks

[weiglszonja]

Thank you @laurelrr for sharing this data. It looks like something we need to look into, so I opened an issue for this at neo. I'll let you know once I know more about this, or learn about a more immediate solution.

[weiglszonja]

@laurelrr do you have any background information about this dataset in particular? There are gaps in this data but they are not consistent across channels.

@samuelgarcia suggested we should check how many files would have failed without using ignore_timestamps_errors=True flag. Can you run this code snippet and send us what is printed on the console?

import pandas as pd
from neo.rawio import OpenEphysRawIO

# Changes this file path to the master excel file that was used for Hao's conversion
excel_file_path = "/Volumes/t7-ssd/Hao_NWB/session_config.xlsx"
config = pd.read_excel(excel_file_path)
# We only need the ecephys_folder_path column to collect the list of folders
openephys_folder_paths = config["ecephys_folder_path"].dropna().tolist()

affected_files = []
for folder_path in openephys_folder_paths:
    io = OpenEphysRawIO(dirname=folder_path, ignore_timestamps_errors=False)
    try:
        io.parse_header()
    except ValueError:
        affected_files.append(folder_path)

print(len(affected_files))
print(affected_files)

[laurelrr]

print(len(affected_files))
10
print(affected_files)
['/nadata/snlkt/data/hao/Neurotensin/ephys/recordings/forLaurel/1_2019-07-13_11-59-42_DiscS', '/nadata/snlkt/data/hao/Neurotensin/ephys/recordings/forLaurel/8_2019-07-14_18-11-14_DiscS', '/nadata/snlkt/data/hao/Neurotensin/ephys/recordings/forLaurel/11_2019-08-30_12-58-04_Disc4', '/nadata/snlkt/data/hao/Neurotensin/ephys/recordings/forLaurel/13_2019-08-30_09-21-02_Disc4', '/nadata/snlkt/data/hao/Neurotensin/ephys/recordings/forLaurel/14_2019-08-30_09-21-05_Disc4', '/nadata/snlkt/data/hao/Neurotensin/ephys/recordings/forLaurel/H24_2019-12-27_10-25-11_Disc3', '/nadata/snlkt/data/hao/Neurotensin/ephys/recordings/forLaurel/H29_2020-02-20_13-42-59_Disc4_20k', '/nadata/snlkt/data/hao/Neurotensin/ephys/recordings/forLaurel/H35_2020-04-30_09-57-13_Disc7_8,22,23', '/nadata/snlkt/data/hao/Neurotensin/ephys/recordings/forLaurel/H39_2020-04-29_13-33-40_Disc5', '/nadata/snlkt/data/hao/Neurotensin/ephys/recordings/forLaurel/H41_2020-04-29_16-55-26_Disc5_9']

[laurelrr]

@weiglszonja I'm unfamiliar with the details of the dataset but I can ask Hao if there is anything unusual about these subjects. I'm not entirely clear on what the error is -- are the different channels in the recording are stopping at different times? I'll follow up with you and Hao over email.

[laurelrr]

@weiglszonja Is it possible that there is just interference getting picked up on some channels that cuts the recording?

[samuelgarcia]

Hi @laurelrr.
Removing channel on the OpenEphysRawIO is not implemented but this could be done but not straight now.

How many files on total do you have (is 10 files a small parts) ?

[laurelrr]

For this dataset (e.g., what went into our journal article) I have 32 files in total, but it seems only 10 are having this issue.
However, we do have supplementary data (not in the paper) that I planned to convert and share next.

[weiglszonja]

Thank you for the info @laurelrr. I took this issue to OpenEphys team (here) to ask what could be the solution.

[laurelrr]

Hello,
I went ahead with updating neo using the suggestion in Szonja's email from 1/31. I prepared a master config file for the 10 remaining subjects (listed above) and ran these steps:

conda activate tyenwbenv39
pip install -r src/tye_lab_to_nwb/neurotensin_valence/neurotensin_valence_requirements.txt
pip install git+https://github.com/NeuralEnsemble/python-neo.git@4158fbbe2f31339b8909bd3c0672e74b152424d2
python ~/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/neurotensin_valence/neurotensin_valence_convert_all_sessions.py

where the excel_file_path was updated to point to the newest config file.
Though 7 seem to have processed correctly, I received an error log on 3 of the subjects and am attaching them here.
H14_Disc4_error_log.txt
H29_Disc4_error_log.txt
H41_Disc5_error_log.txt

Let me know if you need anything else.

[CodyCBakerPhD]

Though 7 seem to have processed correctly,

Well, at least it's nice to have some good news!

I received an error log on 3 of the subjects and am attaching them here.

From the logs, it looks like yet another deviation from expected structure; could you share one of those (or even all 3) remaining files over the data share and we'll send it back for another round of fixes on neo side?