Implement Support for Relation-Based Properties for DCAT profiles

[jadzlnds]

In the file dcat_us_full.yaml and dcat_ap_full.yaml there are TODO comments at line 356 and line 299 :

# TODO: relation-based properties are not yet included (e.g. is_version_of, source, sample, etc)

These properties are essential for establishing relationships between datasets, such as:

• Specifying that a dataset is a version of another dataset (dct:isVersionOf).
• Indicating that a dataset has a newer version (dct:hasVersion).
• Linking parts of datasets (dct:hasPart, dct:isPartOf).
• Referencing source datasets (dct:source).
• And more (e.g., dct:relation, dct:replaces).

Proposed Solution

Add support for the following relation-based properties to the schema, dataset harvesting, and DCAT export functionality:

Relation-Based Properties to Include

1. dct:hasVersion
2. dct:isVersionOf
3. dct:source
4. dct:relation
5. dct:hasPart
6. dct:isPartOf
7. dct:conformsTo
8. dct:hasFormat
9. dct:isFormatOf
10. dct:isReferencedBy
11. dct:references
12. dct:isReplacedBy
13. dct:replaces
14. dct:isRequiredBy
15. dct:requires

Functionality Overview

• Schema Update: Extend the dcat_us_full.yaml and dcat_ap_full.yaml schema to support these properties. Each property should allow a list of URIs to accommodate multiple relationships of the same type.
• RDF Harvesting: Update the RDF parsing logic to extract these properties from incoming RDF metadata.
• Storage: Save these relationships as CKAN dataset extras(additional attributes) to ensure they are accessible for use within CKAN.
• DCAT Export: Include these relationships in the DCAT RDF export, allowing users to share relationship metadata via RDF.

[brunopacheco1]

Hello @amercader, I hope you still remember this topic.

[amercader]

I do remember it @brunopacheco1 I had many open fronts but next week I'll be able to focus on this and we can work out an initial spec, thanks.

[brunopacheco1]

Great, @amercader thanks for your prompt reply. Sure, no problem, please advise us the best moment.

[amercader]

Thanks for writing the initial issue details @jadzlnds . I've been thinking about this but sadly been also pulled in different directions so haven't managed to finish a proper spec. I think most of the functionality and changes are clear, certainly once we have relations encoded in some form in the dataset dict the DCAT serializer can expose them as the DCAT spec mandates (and covnersely the parser used in the harvesters can create dataset dicts with the necessary fields.

What I'm still not convinced of is on how to store the relation information internally, namely:

1. Store it as dataset fields
2. Store in a dedicated table and include the information somehow in the dataset dict (essentially resurrect the old package relationships feature Fix, document and rewrite tests for dataset relationships (or remove) ckan#4212)

Having a dedicated custom field per relation (e.g. source, has_part, etc) that stores multiple string values (either a dataset id or an external URI) is tempting because of its simplicity but I think it might make difficult to extend the feature in the future. Things like generate the inverse properties, update all linked datasets when a dataset is deleted, etc

On the other hand having a table is obviously more complex as we need to deal with model setup, some CRUD actions for relations, etc but we have tons of examples of similar features so it shouldn't take long to develop. It might be difficult to support the case of linking to an external URI though.

@wardi @smotornyuk do you have any thoughts here?

[amercader]

I've been thinking about this a bit and created a small POC exploring the approach of storing relations as datasets fields:

#335

There are some things to iron out but I think it might be a good way forward, at least the serialization side of things seems clear (CKAN -> DCAT).

One thing I'm really not sure about is the parsing and harvesting (DCAT -> CKAN). The harvesters (not just the DCAT one) create datasets in the order they receive them and by definition, we might be creating a dataset that has a relation defined to another dataset that doesn't exist yet in CKAN. Perhaps relations need to be created separately once a harvest job is finished by a separate process.

In any case let me know what you think @brunopacheco1 @jadzlnds and if you think it aligns with your use case.

[brunopacheco1]

Hey @amercader , thanks again for working on it.

We want display human readable data in our portal. In that sense, we would need at least dct:identifier or ckan_id and title, when calling package_show.

As you are saying, there are two options, weak and strong relationship, but I don't know exactly how it would impact CKAN query performance, nor how this could be visible in package_show. Any thoughts on it?

[brunopacheco1]

The best scenario we query only once package_show, with all data we need. That may require strong relationship management, for performance.

The worst scenario we would have to query two times in CKAN, once to get package details, with package_show, and another to fetch relations titles, with package_search, filtering by relations id. That can be managed with weak relationship managed.

[brunopacheco1]

Btw, harvesting is not the only concern, right? Dataset input form have also to be considered.