Prior art

[ml-evs]

The purpose of this thread is collect prior art in the areas of:

• established libraries that extract information from raw data files into a standard representation (I think primarily experimental to start with, though this can be discussed at the meeting)
• frameworks/libraries that have plugin or scripting ecosystems that support the development of new extractors
• registries or lists of such extractors/parsers

To kick things off, here's a list off the top of my head (Python + inorgnaic + XRD/NMR/Echem bias...). I will try to reach out to the developers of many of these projects before the kick-off.

• https://github.com/bluesky/bluesky / https://github.com/bluesky/databroker
• https://github.com/kjappelbaum/chemical-files-registry
• https://github.com/cheminfo
• https://github.com/jjhelmus/nmrglue
• https://github.com/echemdata/galvani
• https://github.com/TheELNConsortium/extractors
• https://github.com/EMMC-ASBL/oteapi-core
• https://github.com/dgbowl/yadg
• https://github.com/materials-data-facility/MaterialsIO
• https://linkml.io
• https://github.com/nomad-coe/atomistic-parsers
• https://github.com/materials-data-facility/battery-data-toolkit

---

• https://github.com/TRI-AMDD/beep
• https://github.com/pycroscopy/SciFiReaders
• https://github.com/wojdyr/xylib
• https://github.com/clowder-framework/pyclowder
• https://github.com/nexpy/nexusformat
• https://github.com/scikit-beam/scikit-beam
• https://github.com/xraypy/xraylarch
• https://github.com/deepanshs/csdmpy

---

Related lists:

• https://github.com/neo-chem/awesome-chemical-data

[PeterKraus]

• https://github.com/deepanshs/csdmpy: Python implementation of the Core Scientific Data Model (CSDM)

[ml-evs]

• https://github.com/galaxyproject/iwc & https://docs.galaxyproject.org/en/master/dev/schema.html

[ml-evs]

https://github.com/nomad-coe/atomistic-parsers

[ml-evs]

https://linkml.io/

[ml-evs]

https://github.com/materials-data-facility/battery-data-toolkit

[ml-evs]

https://github.com/materials-data-facility/MaterialsIO

[SteffenBrinckmann]

https://github.com/NFDI4Chem/formaTAPIRest

[ml-evs]

https://github.com/openmsi/openmsistream from @eminizer

[CSSFrancis]

https://github.com/hyperspy/rosettasciio is also relevant

[ml-evs]

Hi @CSSFrancis, this looks great! We are looking for established libraries such as these to test the feasibility of wrapping all of the supported extractors in the machinery we are developing here, to aid discoverability of the underlying package and provide a unified interface. If you are involved with rosettasciio (or equally, if you are not!), would you be interested in attending on of our meetings (#16) or office hours (#6 (comment)) to discuss this?

[CSSFrancis]

@ml-evs I would be happy to join a meeting! We recently talked about standardizing metadata hyperspy/rosettasciio#89 and it would be good to see where the larger community lands on this before moving forward.

As far as wrapping extractors. I think that we would be very interested in this. Part of the motivation to pulling out the i-o code from hyperspy was to make things more interoperable, reduce the dependency tree and hopefully increase the number of file formats supported.

[kjappelbaum]

filetype detection https://github.com/google/magika

[ml-evs]

Sadly for our kind of data it only returns txt or zip (for any binary); either we don't have enough magic bytes or magicka hasn't seen them yet for our niche file types

[ml-evs]

Just came across https://github.com/intake/intake, "A general python package for describing, loading and processing data" with a catalogue of pluggable "drivers" that can be run automatically datasets https://intake.readthedocs.io/en/latest/ They are about to do a big v2 migration so might be a good time to get involved. Most relevant part of the docs (how to define extractors and data types): https://intake.readthedocs.io/en/latest/tour2.html

Potential tie-ins with us:

• Define a general MaRDA extractor plugin for intake, to give access to all our weird file types
• Try to borrow from some of the output specification tech they have (including e.g., chaining extractors)

It's more focused at ML/data science so the examples are mostly CSV files with special data types (e.g., a column of SMILES). Domain-specific plugins like https://github.com/greglandrum/intake-rdkit exist but don't seem to be catalogued.