[ENH] microelectrode electrophysiology specification (BEP032)

[yarikoptic]

Replaces #1352 submitted from a fork outside of bids-specification.

Add specification for microelectrode electrohpysiology datasets based on the BEP032 proposal. old google doc

Note

We meet regularly and everyone is welcome

Next meeting: insert date on URL to join

Communication channel: https://framalistes.org/sympa/info/neuroscience-data-structure

Tip

• HTML preview of this BEP
• Overleaf manuscript draft repo

---

• To use this WiP schema on sample datasets, see https://deno.land/x/[email protected]#modifying-and-building-a-new-schema on how to use "stock" bids-validator with a custom schema. (attn @TheChymera)

DONEs

• Add (at least one) .md with the section for the added new modality/datatype
• [js] Update all npm dependencies (2023-09-10) legacy-validator#1800

TODOs

• Please ensure your name is credited on our Contributors appendix.
  To add your name, please edit our Contributors wiki and add your name with the type of contribution.
  For assistance, please tag @bids-standard/maintainers.

• After opening the PR, our continuous integration services will automatically check your contribution for formatting errors and render a preview of the BIDS specification with your changes.
  To see the checks and preview, scroll down and click on the show all checks link.
  From the list, select the Details link of the ci/circleci: build_docs artifact check to see the preview of the BIDS specification.

• Add instructions here on how to run new bids-validator using schema in this PR

  • Currently there is ongoing effort in Add "master-deno" flavor of validator to use as well bids-examples#435 to assess the state of deno based validator against examples, then chore(deps): bump bids-validator/tests/data/bids-examples from c557d1f to 1c30c6e legacy-validator#1798 is the first one trying it on whitelisted set of packages, and I think we should create a helper action for that : https://github.com/bids-standard/bids-validator/issues/1931

• bids-validator changes needed

  • Add tricky use cases where validator should fail under https://github.com/bids-standard/bids-validator/tree/main/tests/data

• Populate schema with specifications from the google doc ...

  • The majority of changes: [ENH] Schema changes related to BEP032 #1826
  • Following @ree-gupta failed validation - tune up check for multiple electrode files to apply only to EEG, and iEEG modalities
  • Extend schema/rules/checks with checks specific to this BEP032 (from experiences with data conversion eg by @CodyCBakerPhD on @mvmdmlab data)
  • Similarly to iEEG specify <extension> and get a table of extensions (nwb and nix) and check if schema encodes that only one is allowed

• Further markdown description: @Peyman-N is working on a PR

• Define enums of coord spaces to be added -- some image based, some ad-hoc

• Add and reference here PR on bids-examples adding sample dandisets : Draft examples for BEP032 on animal electrophysiology bids-examples#430 (@robertoostenveld )

• Add CI action (likely github) to run bids-validator on sample datasets and this modified schema (@yarikoptic)

• Decide on "contours" specification

  • decided to postpone
  • attempt to potentially generalize across modalities: Formalize specification of shape(s) (AKA contour(s)) #2013

• type column for electrodes

  • seek to replace hardcoded table in iEEG with a macro (@yarikoptic)
  • add our new enums to src/schema/objects/enums.yaml and then within src/schema/objects/columns.yaml (@ree-gupta )
  • depending on decision in iEEG potentially add similar table in our [ie]cephys/ section

• Share/use examples of real datasets

  • WiP: https://dandiarchive.org/dandiset/001470 AKA https://github.com/dandisets/001470 by @CodyCBakerPhD from @mvdm lab

Issues this PR would likely to address

• Fixes run node tests in github actions legacy-validator#1375

Issues to see being addressed while working on this BEP (likely to move above) or not (moved below):

• bump min node version to 14 legacy-validator#1634
• Add summary generation to schema-prototype legacy-validator#1481

Other issues which relate but not in scope here and provided for reference/backreference

• Bundle CLI entrypoint with main() call legacy-validator#1669
• Bump date-and-time from 0.14.1 to 0.14.2 in /bids-validator legacy-validator#1133
• CLI --config flag does not find relative paths? legacy-validator#1165
• spit out more informative/meaningful msg whenever some file is not actually accessible legacy-validator#197
• Make it possible to specify folders layout to be other than sub-{label}/[ses-{label}/] bids-2-devel#54

Spreadsheet with correspondence to ProbeInterface: https://docs.google.com/spreadsheets/d/1O0bZzD-n4MjR68r1GlcH3d2JLXBLAU1PfsDceD3IPeo/edit?usp=sharing

[TheChymera]

Also relevant if you'd like to comment. → https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAABIzHGpUU

[TheChymera]

Also I forgot to link to this when I wrote it → https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAABIGPAMOw

[bendichter]

I took a look at how filtering is represented in iEEG vs the current microephys draft, and I think it would be beneficial to align microephys with the iEEG approach.

What iEEG does today

• The canonical, detailed filter specification lives in the sidecar JSON:
  • SoftwareFilters is REQUIRED in *_ieeg.json
  • HardwareFilters is RECOMMENDED in *_ieeg.json
• In addition, iEEG also captures the effective per-channel frequency response in channels.tsv:
  • low_cutoff and high_cutoff are REQUIRED
  • notch is optional

This gives both (a) structured provenance/parameters and (b) an immediately usable per-channel summary for typical users.

What microephys currently does

• The sidecar JSON is already very close to iEEG:
  • SoftwareFilters required
  • HardwareFilters recommended
• But *_channels.tsv doesn’t currently expose the iEEG-style per-channel low_cutoff/high_cutoff. Instead, it has optional hardware_filters / software_filters string columns intended as references back to the JSON.

Why I think aligning makes sense
In practice, downstream users often need the per-channel cutoff values immediately to interpret microephys signals (especially when multiple streams exist: broadband/LFP/high-pass/spikes, etc.). Requiring/recommending low_cutoff + high_cutoff in channels.tsv makes microephys more self-describing in the same way iEEG already is.

Proposed change

• Add low_cutoff and high_cutoff to microephys *_channels.tsv (at least RECOMMENDED; possibly REQUIRED if we want strict alignment with iEEG).
• Keep SoftwareFilters (required) and HardwareFilters (recommended) in the datatype sidecars as the place where full filter definitions/parameters are recorded.
• Optionally keep hardware_filters / software_filters TSV columns as optional “filter chain identifiers” for more complex/multi-stage setups, but make the cutoff fields the primary per-channel summary.

If folks agree, I can open a small follow-up PR adjusting src/schema/rules/tabular_data/microephys.yaml (and any docs/examples) to implement this.

[lzehl]

@bendichter sounds like you are combining both approaches and nothing is lost so I think you are right by aligning to iEEG with the filters.

[bendichter]

Suggested change

	description: |
	A link to a photo or drawing of the microelectrode probe system.
	type: string
	format: uri
	description: |
	Path to the photo or image file defining the coordinate system when
	`MicroephysCoordinateUnits` is `"pixels"`.
	Should be a [BIDS URI](SPEC_ROOT/common-principles.md#bids-uri).
	type: string
	format: bids_uri

Suggestion: Update MicroephysCoordinateSystemPhoto to use BIDS URI format

Context

The current implementation of MicroephysCoordinateSystemPhoto in src/schema/objects/metadata.yaml uses format: uri, which only supports external URLs. However, in practice, the photo defining the coordinate system (used when MicroephysCoordinateUnits is "pixels") is typically stored within the dataset itself (e.g., operative photos, histology images).

Comparison with iEEG

iEEG handles a similar use case but takes a different approach:

• iEEG doesn't have a dedicated photo field
• Instead, it uses IntendedFor__ds_relative which can point to any reference image (including photos)
• This field supports both BIDS URIs and dataset-relative paths

Proposed Change

Update MicroephysCoordinateSystemPhoto to use format: bids_uri:

MicroephysCoordinateSystemPhoto:
  name: MicroephysCoordinateSystemPhoto
  display_name: Microephys Coordinate System Photo
  description: |
    Path to the photo or image file defining the coordinate system when
    `MicroephysCoordinateUnits` is `"pixels"`.
    Should be a [BIDS URI](SPEC_ROOT/common-principles.md#bids-uri).
  type: string
  format: bids_uri

Rationale

1. BIDS URIs are the modern standard - The deprecated participant_relative and dataset_relative formats are being phased out in favor of BIDS URIs
2. Typical use case is subject-specific - Photos are almost always subject-specific (operative photos, histology), so they're stored within the subject's directory
3. Consistency - Aligns with how other file references are handled across BIDS

Questions for the team

1. Should we support external URIs as well? Under what circumstances? If so, we could use anyOf to allow both bids_uri and uri formats.
2. Is a dedicated field (MicroephysCoordinateSystemPhoto) preferable to reusing IntendedFor like iEEG does?