Merge Berkeley schema (berkeley-schema-fy24/main) into Legacy schema (nmdc-schema/main)

.github/pull_request_template.md

@@ -2,20 +2,33 @@
 
 ## _Soft_ Schema Freeze
 
-The `nmdc-schema` and `berkeley-schema-fy24` schemas are under a soft freeze, which means changes **should not** be made that have any downstream implications. To ensure this, all PRs created creating during the freeze will be closely reviewed with **every** component of the NMDC system in mind.
+The `nmdc-schema` and `berkeley-schema-fy24` schemas are under a soft freeze, which means changes **should not** be made that have any downstream implications. To ensure this, all PRs created during the freeze will be closely reviewed with **every** component of the NMDC system in mind.
 
 ## Reviewers
 
-To ensure no changes are made unexpectedly, PR creators will request reviews from _all_ [Berkeley Schema Roll Out task coordinators](https://docs.google.com/document/d/1XXN1YuaBuSkxPXeiLKm5YxYzXTamBPQrzzeLhlh7PWs/edit#heading=h.u52g8v319adh).
+To ensure no changes are made unexpectedly, PR creators will use this PR template to tag and notify all task coordinators. Review should be specifically requested from _all_ [Berkeley Schema Roll Out task coordinators](https://docs.google.com/document/d/1XXN1YuaBuSkxPXeiLKm5YxYzXTamBPQrzzeLhlh7PWs/edit#heading=h.u52g8v319adh) that you expect to be affected by this PR.
 
 We expect task coordinators to review PRs and provide feedback/approval within 1 week of when they are identified as reviewers. 
 
-PRs will **NOT** be merged until all task coordinators (or one of their delegates) have approved. 
+PRs will **NOT** be merged until all task coordinators (or their delegates) have approved it; either here on GitHub (via "`Review changes` > `Approve`" or an equivalent comment) or verbally.
 
 Expedition, questions, and discussion can happen at any meeting.
 
 Delays in review & merging should be addressed in meetings or with NMDC leadership.
 
+| If you expect the changes to<br>impact this component... | ...[request a review](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/requesting-a-pull-request-review)<br>from this person |
+| -- | -- |
+| Metadata<br>Schema | @mslarae13 |
+| Runtime<br>Mongo database<br>Database migrations | @eecavanna,<br>who will pull in<br>@shreddd as needed |
+| Postgres<br>Ingest | @naglepuff |
+| Data Portal | @aclum |
+| Workflows: MG & MT | @mbthornton-lbl |
+| Workflows: MetaB & NOM | @corilo |
+| Workflows: LipidO | @kheal |
+| Workflows: MetaP | @SamuelPurvine |
+| ETL code | @sujaypatil96 |
+| Jupyter notebooks | @brynnz22 |
+
 # PR Information
 
 ## What type of PR is this? (check all applicable)
@@ -80,4 +93,4 @@ Delays in review & merging should be addressed in meetings or with NMDC leadersh
 > Examples: any change here that requires a change to workflows, workflow automation, the Mongo-to-Postgres ingest process, Jupyter notebooks, the Runtime, etc.
 
 - [ ] Yes _(Explain below)_
-- [ ] No
+- [ ] No

.github/workflows/main.yaml

@@ -4,15 +4,15 @@
 
 name: Build and test nmdc-schema
 
-on: [pull_request]
+on: [ pull_request ]
 
 jobs:
   test:
 
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.9", "3.10"]
+        python-version: [ "3.9", "3.10" ]
 
     steps:
 
@@ -49,5 +49,5 @@ jobs:
       #              run test suite   
       #----------------------------------------------
       - name: Run tests
-        run: make test
+        run: make squeaky-clean all test
 

.github/workflows/test_pages_build.yaml

@@ -14,7 +14,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v3.1.0
+        uses: actions/checkout@v4
         with:
           fetch-depth: 0
 

.pylintrc

@@ -0,0 +1,5 @@
+[MASTER]
+ignore-paths=nmdc_schema/migrators/*,nmdc_schema/nmdc_data.py,nmdc_schema/nmdc_version.py,nmdc_schema/validate_nmdc_json.py
+
+[MESSAGES CONTROL]
+disable=line-too-long

CONTRIBUTING.md

@@ -109,6 +109,10 @@ Core developers should read the material on the [LinkML site](https://linkml.io/
     - Follow the naming conventions of the parent class
     - Descriptions of child classes may reference parent classes in a genus-differentia definition structure (e.g. "A workflow execution activity that...")
     - Inheritance should be monotonic: `slot_usage` should refine rather than override
+- ID patterning and checks
+    - ID patterns for new classes should follow conventions found [here](https://microbiomedata.github.io/nmdc-schema/identifiers/)
+    - In the _rare_ case that NMCD records must support legacy typecodes, typecodes can be declared on new classes with multiple typecodes (i.e. `syntax: "{id_nmdc_prefix}:(dgns|omprc)-{id_shoulder}-{id_blade}$"`). In this case, the _first_ typecode is the one the NMDC Runtime's [minter](https://github.com/microbiomedata/nmdc-runtime/tree/main/nmdc_runtime/minter) will use when generating new ids for the class.
+    - Class-linking slots (i.e. `has_input`) should have a `slot_usage` declared that limits the slot's values to ids of instances of _only_ the specific classes you want to allow the slot to link to (e.g. using `syntax: "{id_nmdc_prefix}:chrcon-{id_shoulder}-{id_blade}$"` on the `structured_pattern` will make it so only ids having the typecode `chrcon` can fill that slot)
 
 ### Testing Changes Locally
 

Makefile

@@ -21,6 +21,7 @@ TEMPLATEDIR = doc-templates
 
 .PHONY: all clean examples-clean install site site-clean site-copy squeaky-clean test test-python test-with-examples
 
+
 # note: "help" MUST be the first target in the file,
 # when the user types "make" they should get help info
 
@@ -83,15 +84,19 @@ update-linkml:
 create-data-harmonizer:
 	npm init data-harmonizer $(SOURCE_SCHEMA_PATH)
 
+# Note: `all` is an alias for `site`.
 all: site
-site: clean site-clean gen-project gendoc migration-doctests nmdc_schema/gold-to-mixs.sssom.tsv
+site: clean site-clean gen-project gendoc \
+nmdc_schema/gold-to-mixs.sssom.tsv \
+nmdc_schema/nmdc_materialized_patterns.schema.json nmdc_schema/nmdc_materialized_patterns.yaml \
+migration-doctests
 
 %.yaml: gen-project
 
 # was deploy: all mkd-gh-deploy
 deploy: gendoc mkd-gh-deploy
 
-gen-project: $(PYMODEL) src/schema/mixs.yaml
+gen-project: $(PYMODEL) # depends on src/schema/mixs.yaml # can be nuked with mixs-yaml-clean
 	$(RUN) gen-project \
 		--exclude excel \
 		--exclude graphql \
@@ -110,9 +115,8 @@ gen-project: $(PYMODEL) src/schema/mixs.yaml
 		-d $(DEST) $(SOURCE_SCHEMA_PATH) && mv $(DEST)/*.py $(PYMODEL)
 		cp project/jsonschema/nmdc.schema.json  $(PYMODEL)
 
-
-test: examples-clean site test-python examples/output
-only-test: examples-clean test-python examples/output
+test: examples-clean site test-python migration-doctests examples/output
+only-test: examples-clean test-python migration-doctests examples/output
 
 test-schema:
 	$(RUN) gen-project \
@@ -155,7 +159,7 @@ gendoc: $(DOCDIR)
 	# added copying of images and renaming of TEMP.md
 	cp $(SRC)/docs/*md $(DOCDIR) ; \
 	cp -r $(SRC)/docs/images $(DOCDIR) ; \
-	$(RUN) gen-doc -d $(DOCDIR) --template-directory $(SRC)/$(TEMPLATEDIR) $(SOURCE_SCHEMA_PATH)
+	$(RUN) gen-doc -d $(DOCDIR) --template-directory $(SRC)/$(TEMPLATEDIR) --include src/schema/deprecated.yaml $(SOURCE_SCHEMA_PATH)
 	mkdir -p $(DOCDIR)/javascripts
 	$(RUN) cp $(SRC)/scripts/*.js $(DOCDIR)/javascripts/
 
@@ -222,7 +226,9 @@ site-clean: clean
 	rm -rf nmdc_schema/*.tsv
 	rm -rf nmdc_schema/*.yaml
 
+
 squeaky-clean: clean examples-clean rdf-clean shuttle-clean site-clean # does not include mixs-yaml-clean
+	mkdir project
 	rm -rf local/biosample_slots_ranges_report.tsv
 
 nmdc_schema/nmdc_materialized_patterns.yaml:
@@ -238,12 +244,28 @@ nmdc_schema/nmdc_materialized_patterns.schema.json: nmdc_schema/nmdc_materialize
 		--include-range-class-descendants \
 		--top-class Database $< > $@
 
-# todo this target makes a lot of prerequisites if necessary, but they aren't part of the copying prpocess
 # the sssom/ files should be double checked too... they're probably not all SSSSOM files
-nmdc_schema/gold-to-mixs.sssom.tsv: sssom/gold-to-mixs.sssom.tsv nmdc_schema/nmdc_materialized_patterns.schema.json \
-nmdc_schema/nmdc_materialized_patterns.yaml
+nmdc_schema/gold-to-mixs.sssom.tsv: sssom/gold-to-mixs.sssom.tsv
 	# just can't seem to tell pyproject.toml to bundle artifacts like these
 	#   so reverting to copying into the module
 	cp $< $@
 
 
+nmdc_schema/nmdc_schema_merged.yaml: project/nmdc_schema_merged.yaml
+	cp $< $@
+
+####
+
+.PHONY: check-invalids-for-single-failure
+
+# 		echo "Running command: $$cmd"; \
+
+check-invalids-for-single-failure:
+	for file in src/data/invalid/*.yaml; do \
+		echo "$$file:"; \
+		target_class=$$(basename $$file | cut -d'-' -f1); \
+		cmd="poetry run linkml-validate --schema nmdc_schema/nmdc_materialized_patterns.yaml --target-class $$target_class $$file"; \
+		output=$$($$cmd 2>&1 || true); \
+		echo "$$output" | sort | uniq; \
+	done
+

assets/blanklines.txt

@@ -0,0 +1,2 @@
+
+