Provide model for capturing the source of records

[mrshll1001]

Based on the discussion at the June Technical Committee Meeting.

Implemenenters often need to determine the ultimate source of a record (service, organisation, etc) for various reasons

• They might need to provide a link, or method, for a user to contact the original source of information to support amendments, redactions etc
• They might need to implement transforms for specific sources of data based on knowledge about the source system or location.
• They need to understand what transforms have already been performed on the data, based on knowledge about various sources
• Funders want to see who is responsible for the data, people who are doing the work need to see that their data is appearing in various locations so that they can evidence the need for their services.

[mrshll1001]

I've been investigating this, and I think the most effective way to address this would be integration with the OpenLineage data standard.

• OpenLineage's website

OpenLineage is a model + API standard which provides a standardised way to model concepts in Data Lineage which covers much of what I understand to be our needs in this domain. It might not cover all of the stated use-cases above, but I think it's a big step forward vs implementing our own models without being explicit experts in data lineage.

For us, I think we're more interested in the models. They're defined in a JSON Schema, so "compatible" with HSDS by their nature (they even use the same version of JSON Schema).

• OpenLineage JSON Schema

For me the questions to answer next are:

• to what extent do OpenLineage's models support our needs as stated above
• what are the ways we should be integrating OpenLineage with HSDS (e.g. importing the structures, providing objects which reference external OpenLineage datasets etc.)
• what are the governance implications for our standard if we "officially" support OpenLineage in some way.

I plan to mock some of this up in a branch, to try out various solutions.

[mrshll1001]

Additional consideration about complexity. If OpenLineage covers what we need to, but does so in a very complex way that is too detailed/nested for our use cases; we need to evaluate whether this is something we want to integrate/interoperate with.