[ENH] microelectrode electrophysiology specification (BEP032)

[yarikoptic]

Superseded by squashed

• [ENH]: BEP032 formal review #2307

---

Replaces #1352 submitted from a fork outside of bids-specification.

Add specification for microelectrode electrohpysiology datasets based on the BEP032 proposal. old google doc

Note

We meet regularly and everyone is welcome

Next meeting: insert date on URL to join

Communication channel: https://framalistes.org/sympa/info/neuroscience-data-structure

Tip

• HTML preview of this BEP
• Overleaf manuscript draft repo
• Example dataset

---

• To use this WiP schema on sample datasets, see https://deno.land/x/[email protected]#modifying-and-building-a-new-schema on how to use "stock" bids-validator with a custom schema. (attn @TheChymera)

DONEs

• Add (at least one) .md with the section for the added new modality/datatype
• [js] Update all npm dependencies (2023-09-10) legacy-validator#1800

TODOs

• Please ensure your name is credited on our Contributors appendix.
  To add your name, please edit our Contributors wiki and add your name with the type of contribution.
  For assistance, please tag @bids-standard/maintainers.

• After opening the PR, our continuous integration services will automatically check your contribution for formatting errors and render a preview of the BIDS specification with your changes.
  To see the checks and preview, scroll down and click on the show all checks link.
  From the list, select the Details link of the ci/circleci: build_docs artifact check to see the preview of the BIDS specification.

• Add instructions here on how to run new bids-validator using schema in this PR

  • Currently there is ongoing effort in Add "master-deno" flavor of validator to use as well bids-examples#435 to assess the state of deno based validator against examples, then chore(deps): bump bids-validator/tests/data/bids-examples from c557d1f to 1c30c6e legacy-validator#1798 is the first one trying it on whitelisted set of packages, and I think we should create a helper action for that : https://github.com/bids-standard/bids-validator/issues/1931

• bids-validator changes needed

  • Add tricky use cases where validator should fail under https://github.com/bids-standard/bids-validator/tree/main/tests/data

• Populate schema with specifications from the google doc ...

  • The majority of changes: [ENH] Schema changes related to BEP032 #1826
  • Following @ree-gupta failed validation - tune up check for multiple electrode files to apply only to EEG, and iEEG modalities
  • Extend schema/rules/checks with checks specific to this BEP032 (from experiences with data conversion eg by @CodyCBakerPhD on @mvmdmlab data)
  • Similarly to iEEG specify <extension> and get a table of extensions (nwb and nix) and check if schema encodes that only one is allowed

• Further markdown description: @Peyman-N is working on a PR

• Define enums of coord spaces to be added -- some image based, some ad-hoc

• Add and reference here PR on bids-examples adding sample dandisets : Draft examples for BEP032 on animal electrophysiology bids-examples#430 (@robertoostenveld )

• Add CI action (likely github) to run bids-validator on sample datasets and this modified schema (@yarikoptic)

• Decide on "contours" specification

  • decided to postpone
  • attempt to potentially generalize across modalities: Formalize specification of shape(s) (AKA contour(s)) #2013

• type column for electrodes

  • seek to replace hardcoded table in iEEG with a macro (@yarikoptic)
  • add our new enums to src/schema/objects/enums.yaml and then within src/schema/objects/columns.yaml (@ree-gupta )
  • depending on decision in iEEG potentially add similar table in our [ie]cephys/ section

• Share/use examples of real datasets

  • WiP: https://dandiarchive.org/dandiset/001470 AKA https://github.com/dandisets/001470 by @CodyCBakerPhD from @mvdm lab

Issues this PR would likely to address

• Fixes run node tests in github actions legacy-validator#1375

Issues to see being addressed while working on this BEP (likely to move above) or not (moved below):

• bump min node version to 14 legacy-validator#1634
• Add summary generation to schema-prototype legacy-validator#1481

Other issues which relate but not in scope here and provided for reference/backreference

• Bundle CLI entrypoint with main() call legacy-validator#1669
• Bump date-and-time from 0.14.1 to 0.14.2 in /bids-validator legacy-validator#1133
• CLI --config flag does not find relative paths? legacy-validator#1165
• spit out more informative/meaningful msg whenever some file is not actually accessible legacy-validator#197
• Make it possible to specify folders layout to be other than sub-{label}/[ses-{label}/] bids-2-devel#54

Spreadsheet with correspondence to ProbeInterface: https://docs.google.com/spreadsheets/d/1O0bZzD-n4MjR68r1GlcH3d2JLXBLAU1PfsDceD3IPeo/edit?usp=sharing

[TheChymera]

Also relevant if you'd like to comment. → https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAABIzHGpUU

[TheChymera]

Also I forgot to link to this when I wrote it → https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAABIGPAMOw

[ree-gupta]

@yarikoptic I fixed the linter errors but the build is still failing for some reason. Could you please help me here?

edit: nevermind, found the issue!

[CodyCBakerPhD]

@ree-gupta Are we still waiting on the final items from the internal TODO list from the other day?

Please let me know when ready to proceed

[ree-gupta]

@ree-gupta Are we still waiting on the final items from the internal TODO list from the other day?

The schema TODOs are done (607a042 and b46c11f).

Please let me know when ready to proceed

@bendichter Do we need some more time for the markdown?

[bendichter]

We also need to adjust the markdown to reflect @ree-gupta 's changes

[CodyCBakerPhD]

@ree-gupta @bendichter It has been 1 week now - is everything good and ready to submit?

[ree-gupta]

@ree-gupta @bendichter It has been 1 week now - is everything good and ready to submit?

I'm all done on the schema end, the tests pass too so good to go for me.

[bendichter]

I see some issues, hold on

[bendichter]

@CodyCBakerPhD ok, ready!

[CodyCBakerPhD]

Formally opened in squashed form: #2307

I once again strongly recommend that if we ever do this again in the future, we should start it as an external fork (possibly on a specifically created organization) before moving to the main repository, so that the scale of pull requests, issues, and contributions can scale to the large number of contributors. This proved to be a significant pain point over the process on this branch and its predecessors

Are we all good to close this one then?

Note I have backed up this branch's history (though will not include all the history of GitHub comments for this PR) on https://github.com/CodyCBakerPhD/bids-specification/tree/backup-bep032-work

[yarikoptic]

yes, @bids-standard/wg-bep-process is to formalize the process and I am of the same opinion to operate primarily in the fork until "ready for review". Let's let this RiP.