SpikeInterface 0.103.0

[alejoe91]

• update to SI 0.103.0
• job_dispatch: rename concatenate --> split_segments parameter
• remove aind-subject-nwb: NWBFile is created on-the-fly by the ecephys process

[vncntprvst]

Hi Alessio,
Just FYI, I was curious to test this job-dispatch-update branch with SI 0.103.0 and gave it a try. I ran on a open-ephys NPx recording. I'm getting this error, which I didn't have with the main branch:

WARN: Killing running tasks (4)

executor >  slurm (6)
[9c/e4a03b] process > job_dispatch (job-dispatch)   [100%] 1 of 1 ✔
[-        ] process > preprocessing (preprocessing) -
[-        ] process > spikesort_kilosort4           -
[-        ] process > postprocessing                -
[-        ] process > curation                      -
[-        ] process > visualization                 -
[-        ] process > results_collector             -
[-        ] process > quality_control               -
[-        ] process > quality_control_collector     -
[18/b5b934] process > nwb_ecephys (nwb-ecephys)     [100%] 1 of 1, failed: 1 ✘
[-        ] process > nwb_units                     -
Error executing process > 'nwb_ecephys (nwb-ecephys)'

Caused by:
  Process `nwb_ecephys (nwb-ecephys)` terminated with an error exit status (1)

Command executed:

  #!/usr/bin/env bash
  set -e
  
  mkdir -p capsule
  mkdir -p capsule/data
  mkdir -p capsule/results
  mkdir -p capsule/scratch
  
  if [[ slurm == "slurm" ]]; then
      echo "[nwb-ecephys] allocated task time: 42m"
  fi
  
  echo "[nwb-ecephys] cloning git repo..."
  git clone "https://github.com/AllenNeuralDynamics/aind-ecephys-nwb.git" capsule-repo
  git -C capsule-repo -c core.fileMode=false checkout 27e4b1a0a830c86245427a3dd50d505bfa358d0b --quiet
  mv capsule-repo/code capsule/code
  rm -rf capsule-repo
  
  echo "[nwb-ecephys] running capsule..."
  cd capsule/code
  chmod +x run
  ./run 
  
  echo "[nwb-ecephys] completed!"

Command exit status:
  1

Command output:
  [nwb-ecephys] allocated task time: 42m
  [nwb-ecephys] cloning git repo...
  [nwb-ecephys] running capsule...
  
  
  NWB EXPORT ECEPHYS
  Running NWB conversion with the following parameters:
  Stub test: False
  Stub seconds: 10.0
  Write LFP: True
  Write RAW: False
  Temporal subsampling factor: 2
  Spatial subsampling factor: 4
  Highpass filter frequency: 0.1
  Surface channel indices for agar probes: None
  NWB backend: zarr
  Found 4 JSON job files
  Session: ecephys_session
  Number of NWB files to write: 1
  Number of streams to write for each file: 1
  Failed to create base NWB file from metadata.
  Creating NWB file with info.

Command error:
  INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  INFO:    Environment variable SINGULARITYENV_KACHERY_ZONE is set, but APPTAINERENV_KACHERY_ZONE is preferred
  INFO:    Environment variable SINGULARITYENV_KACHERY_API_KEY is set, but APPTAINERENV_KACHERY_API_KEY is preferred
  Cloning into 'capsule-repo'...
  + python -u run_capsule.py
  Traceback (most recent call last):
    File "capsule/code/run_capsule.py", line 379, in <module>
      ecephys_session_folder, segment_index=segment_index
      ^^^^^^^^^^^^^^^^^^^^^^
  NameError: name 'ecephys_session_folder' is not defined

I know you're actively working on this, so it's not an issue, just wanted to let you know.

[alejoe91]

Hi Alessio, Just FYI, I was curious to test this job-dispatch-update branch with SI 0.103.0 and gave it a try. I ran on a open-ephys NPx recording. I'm getting this error, which I didn't have with the main branch:

I know you're actively working on this, so it's not an issue, just wanted to let you know.

Thank you so much for testing and finding this bug! I forgot to update one parameter of that function. Should be fixed in the last commit :)

[vncntprvst]

Should be fixed in the last commit :)

[7d/7fccc9] process > nwb_ecephys (nwb-ecephys) [100%] 1 of 1 ✔

Yay, it works now ┌|ﾟзﾟ|┘

[alejoe91]

@vncntprvst I tested it again for all supported inputs on SLURM and it works well on our end.

Thanks again for beta testing! Will merge into main and finally release v1.0.0