[FIX] schemapath resolution in dev mode install of bidsschematools

[CodyCBakerPhD]

I ran into some issues when trying to achieve a local dev install to thoroughly test the bep032 branch on some real datasets

Investigation revealed that this file: https://github.com/bids-standard/bids-specification/blob/9540a823ed70fa65dcf914e992adabd11c7285fc/tools/schemacode/src/bidsschematools/data/schema, which effectively seems to be a symlink, was being passed diectly into the Namespace.from_json as a raw string, instead of being recognized as a path to a directory

This small fix resolved my problems and allowed local installation to occur smoothly as expected

Speculation: possibly related to #2120? A collaborator (@yarikoptic) who showed me how to do this weeks ago was able to do it themselves without any issues

TODO

• Add self to contributing
• Increment patch version

[CodyCBakerPhD]

Before doing so, I wanted to confirm that https://github.com/bids-standard/bids-specification/blob/fix_dev_install/src/schema/BIDS_VERSION is the right place to change to bump the version for this type of fix?

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Project coverage is 82.03%. Comparing base (daad867) to head (11611bd).

Files with missing lines	Patch %	Lines
tools/schemacode/src/bidsschematools/schema.py	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2129      +/-   ##
==========================================
- Coverage   82.19%   82.03%   -0.17%     
==========================================
  Files          17       17              
  Lines        1528     1531       +3     
==========================================
  Hits         1256     1256              
- Misses        272      275       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[effigies]

Is this a Windows issue? It would be good to test this.

[CodyCBakerPhD]

Is this a Windows issue?

I did not confirm if it is Windows-specific, or anything else environmental specific, but it is quite possible

Steps to reproduce locally or in CI on master:

• Navigate to the bidsschematools
• pip install -e .

My environment: fresh conda with Python 3.13

[effigies]

Okay, so you are on Windows, which would explain it. Do you have symlinks enabled for your filesystem? I seem to recall reading that it's a system setting. And does this occur when you use the WSL?

[CodyCBakerPhD]

Do you have symlinks enabled for your filesystem?

Didn't know it was something you could turn off 😅 I use them prolifically, but unsure what changes under the hood with the file metadata across platforms (I usually create/follow them via things like pathlib.Path(...).symlink_to(...))

I did try pathlib.Path(".../schema").is_symlink() here, but it returns False

And does this occur when you use the WSL?

Just tried this, and yes it does 🤔

git clone https://github.com/bids-standard/bids-specification
cd bids-specification/tools/schemacode
pip install -e .

Error

Obtaining file:///mnt/e/GitHub/bids-specification/tools/schemacode Installing build dependencies ... done Checking if build backend supports build_editable ... done Getting requirements to build editable ... done Preparing editable metadata (pyproject.toml) ... error error: subprocess-exited-with-error

× Preparing editable metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [38 lines of output]
Traceback (most recent call last):
File "/home/theac/anaconda3/envs/linux_bst_install_issue_6_5_25/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in
main()
~~~~^^
File "/home/theac/anaconda3/envs/linux_bst_install_issue_6_5_25/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
json_out["return_val"] = hook(**hook_input["kwargs"])
~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/theac/anaconda3/envs/linux_bst_install_issue_6_5_25/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 209, in prepare_metadata_for_build_editable
return hook(metadata_directory, config_settings)
File "/tmp/pip-build-env-syq086vn/overlay/lib/python3.13/site-packages/pdm/backend/init.py", line 90, in prepare_metadata_for_build_editable
return builder.prepare_metadata(metadata_directory).name
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-syq086vn/overlay/lib/python3.13/site-packages/pdm/backend/wheel.py", line 102, in prepare_metadata
self.initialize(context)
~~~~~~~~~~~~~~~^^^^^^^^^
File "/tmp/pip-build-env-syq086vn/overlay/lib/python3.13/site-packages/pdm/backend/base.py", line 193, in initialize
self.call_hook("pdm_build_initialize", context)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-syq086vn/overlay/lib/python3.13/site-packages/pdm/backend/base.py", line 151, in call_hook
getattr(hook, hook_name)(context, *args, **kwargs)
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/e/GitHub/bids-specification/tools/schemacode/pdm_build.py", line 12, in pdm_build_initialize
schema = bidsschematools.schema.load_schema()
File "/mnt/e/GitHub/bids-specification/tools/schemacode/src/bidsschematools/schema.py", line 227, in load_schema
return Namespace.from_json(schema_path.read_text())
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/e/GitHub/bids-specification/tools/schemacode/src/bidsschematools/types/namespace.py", line 268, in from_json
return cls.build(json.loads(jsonstr))
~~~~~~~~~~^^^^^^^^^
File "/home/theac/anaconda3/envs/linux_bst_install_issue_6_5_25/lib/python3.13/json/init.py", line 346, in loads
return _default_decoder.decode(s)
~~~~~~~~~~~~~~~~~~~~~~~^^^
File "/home/theac/anaconda3/envs/linux_bst_install_issue_6_5_25/lib/python3.13/json/decoder.py", line 345, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/home/theac/anaconda3/envs/linux_bst_install_issue_6_5_25/lib/python3.13/json/decoder.py", line 363, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

I also confirmed that the fix here applies to WSL as well

Sorry for not raising this as an issue beforehand

[effigies]

No worries. Just want to understand. Before merging this, I want to add CI tests for Linux/Mac/Windows to verify that loading the schema from full/editable installs works as expected. If you're up to exploring that, feel free. I'm currently working on a different PR on this repo, so will fiddle around with this later.

[CodyCBakerPhD]

If you're up to exploring that, feel free.

Sure, I can take a crack at it

Does an extra (short) step before primary build stage in https://github.com/bids-standard/bids-specification/blob/fix_dev_install/.github/workflows/schemacode_ci.yml sound like a good strategy?

[effigies]

I think I would add an install_type: [wheel] line to the build matrix:

bids-specification/.github/workflows/schemacode_ci.yml

Lines 57 to 62 in 11611bd

	python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
	include:
	- os: macos-latest
	python-version: 3
	- os: windows-latest
	python-version: 3

         python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
+        install-type: [wheel]
         include:
           - os: macos-latest
             python-version: 3
+            install-type: wheel
           - os: windows-latest
             python-version: 3
+            install-type: wheel

Then add some inclusions for install-type: dev. Then you could duplicate this step, making one conditional on wheel and one on dev.

bids-specification/.github/workflows/schemacode_ci.yml

Lines 82 to 84 in 11611bd

	- name: "Install package"
	run: |
	pip install $( ls dist/*.whl )[all]

[effigies]

One last thing: What is the actual command / Python code you run to induce the error?

[CodyCBakerPhD]

One last thing: What is the actual command / Python code you run to induce the error?

git clone https://github.com/bids-standard/bids-specification
pip install -e bids-specification/tools/schemacode[all]

Error

Obtaining file:///mnt/e/GitHub/bids-specification/tools/schemacode Installing build dependencies ... done Checking if build backend supports build_editable ... done Getting requirements to build editable ... done Preparing editable metadata (pyproject.toml) ... error error: subprocess-exited-with-error

× Preparing editable metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [38 lines of output]
Traceback (most recent call last):
File "/home/theac/anaconda3/envs/linux_bst_install_issue_6_5_25/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in
main()
~~~~^^
File "/home/theac/anaconda3/envs/linux_bst_install_issue_6_5_25/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
json_out["return_val"] = hook(**hook_input["kwargs"])
~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/theac/anaconda3/envs/linux_bst_install_issue_6_5_25/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 209, in prepare_metadata_for_build_editable
return hook(metadata_directory, config_settings)
File "/tmp/pip-build-env-syq086vn/overlay/lib/python3.13/site-packages/pdm/backend/init.py", line 90, in prepare_metadata_for_build_editable
return builder.prepare_metadata(metadata_directory).name
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-syq086vn/overlay/lib/python3.13/site-packages/pdm/backend/wheel.py", line 102, in prepare_metadata
self.initialize(context)
~~~~~~~~~~~~~~~^^^^^^^^^
File "/tmp/pip-build-env-syq086vn/overlay/lib/python3.13/site-packages/pdm/backend/base.py", line 193, in initialize
self.call_hook("pdm_build_initialize", context)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-syq086vn/overlay/lib/python3.13/site-packages/pdm/backend/base.py", line 151, in call_hook
getattr(hook, hook_name)(context, *args, **kwargs)
~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/e/GitHub/bids-specification/tools/schemacode/pdm_build.py", line 12, in pdm_build_initialize
schema = bidsschematools.schema.load_schema()
File "/mnt/e/GitHub/bids-specification/tools/schemacode/src/bidsschematools/schema.py", line 227, in load_schema
return Namespace.from_json(schema_path.read_text())
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/e/GitHub/bids-specification/tools/schemacode/src/bidsschematools/types/namespace.py", line 268, in from_json
return cls.build(json.loads(jsonstr))
~~~~~~~~~~^^^^^^^^^
File "/home/theac/anaconda3/envs/linux_bst_install_issue_6_5_25/lib/python3.13/json/init.py", line 346, in loads
return _default_decoder.decode(s)
~~~~~~~~~~~~~~~~~~~~~~~^^^
File "/home/theac/anaconda3/envs/linux_bst_install_issue_6_5_25/lib/python3.13/json/decoder.py", line 345, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/home/theac/anaconda3/envs/linux_bst_install_issue_6_5_25/lib/python3.13/json/decoder.py", line 363, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[CodyCBakerPhD]

So far as I can tell, it is simply that one file which git clones as a text file containing the symlink target, and is not established as an actual symlink

Speculation: using a more magical built-in pathlib logic to generate the symlink target (or other form of navigation) in-place might be a more robust solution overall than using a static file (symlink or no)

But I feel like I am only one with a very rare particular problem 🤷‍♂️ so maybe no need for over-engineering...

[effigies]

Oh, it's the pip install itself. That is definitely more painful, and makes sense. Since the installation requires loading the schema, if you can install it, you can load it.

[effigies]

Could you try this patch?

diff --git a/tools/schemacode/src/bidsschematools/schema.py b/tools/schemacode/src/bidsschematools/schema.py
index 397e4919..5c318b70 100644
--- a/tools/schemacode/src/bidsschematools/schema.py
+++ b/tools/schemacode/src/bidsschematools/schema.py
@@ -218,6 +218,10 @@ def load_schema(schema_path=None):
         schema_path = data.load.readable("schema.json")
         if not schema_path.is_file():
             schema_path = data.load.readable("schema")
+            # Probably a Windows checkout with a git link. Resolve first.
+            if schema_path.is_file() and (content := schema_path.read_text()).startswith("../"):
+                schema_path = Path.resolve(schema_path.parent / content)
+
         lgr.info("No schema path specified, defaulting to the bundled schema, `%s`.", schema_path)
     elif isinstance(schema_path, str):
         schema_path = Path(schema_path)

This more narrowly targets your case. The current proposal will read any text file passed to load_schema() containing ../* and attempt to resolve it relative to the <env>/site-packages/bidsschematools/schema.py. I think in most such cases, people will prefer a JSONDecodeError if they pass a non-JSON file.

[CodyCBakerPhD]

@effigies Sure thing - re-implemented that and it works fine

Last question is how you want to handle version bump?

[effigies]

There's no need. It'll be in the next release. I'll backport this onto maint/1.10.0 so that it will be in any future releases we make there, also.

Also, can you add yourself to https://github.com/bids-standard/bids-specification/wiki/Recent-Contributors? The change to contributors.md would get wiped out at the next release, otherwise.