Integration of new technologies (AI/ML)

[samikanza]

MADICES 2024 Recap
Several members of the MADICES team attended Future Labs Live 2024 a few months after MADICES 4. Emerging technologies, such as Machine Learning and AI, for enhancing data processing and interoperability were presented at this event, including video-based auto-documentation of lab operations, LLM-driven semantic annotation of metadata, and AI-driven interoperability across lab instruments.

MADICES 2025 Focus Area
A new focus for MADICES 3 will be encouraging representatives from initiatives involved in the application of AI to the challenge of interoperability and semantic data to share their expertise and participate in the integration of such technologies in represented RDM platforms.

Please include links, comments, and discuss plans for this focus area below.

[ml-evs]

This is something we're actively working on for datalab as "codename" yellowhammer. The main focuses for now are

• how we can document our API (both raw REST API and Python SDK) to make it more traversable for LLM code gen (see e.g., https://llmstxt.org/ for more inspiration)
• providing bespoke user interfaces that guide non-technical users through use of AI for data management, e.g., a Jupyter "magic" that has a bundled system prompt (implemented in repo above), and "wizards" to expose functionality that we know works well, like uploading a lab notebook image and adding as structured data. The final use case we want to build towards is using LLMs to enhance search, e.g., if we expose a generic search interface (either raw MongoDB queries, or something like OPTIMADE) and "teach" the LLM our query language, and allow it to do post-processing to restrict results in cases where search is not possible, so that users can ask more complicated questions of their data more easily.

[simontaurus]

• Regarding you first point: For a REST-API OpenAPI-spec may serve already this purpose (see also Simon Stier - OpenSemanticLab, OO-LD and Agentic AI #10) but for the tools/libs in general that are just dumped somewhere on github, etc. additional metadata would be a good approach. But how about re-using schema.org annotated JSON-LD embedding (see https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data?hl=de)?
• Totally share your vision about llm-assisted user interfaces. Would love to build together a common toolset for toolcalling (some early work here: https://github.com/opensemanticworld/osw-chatbot)

[ml-evs]

OpenAPI is necessary but not sufficient, and can easily fill a context window of smaller/local models (especially with microservices each with their own APIs etc) -- we've found we still need the fuzzy "big picture" docs for the LLM to orient itself. Ultimately its not so much the format but just agreeing where to host it/find it for your libraries/APIs so that agentic tools can use it without the user having to tell them where to look. Will definitely check out osw-chatbot!