More closely follow file system in RO-Crate metadata json

[HLWeil]

connected to nfdi4plants/arc-export#54

Background

In ARC Scaffold Annotation Tables, Data objects can be used as Inputs or Outputs. In the actual ARC Filesystem, these Data annotations can refer to three different entities:

• File
• File Fragment
• Folder

All of these make sense and there is currently no intention to change this annotation in the ARC Scaffold.

The problem arises, when mapping the ARC Scaffold metadata to ARC RO-Crate metadata json. The RO-Crate is meant to be used for various tasks and should be as semantically sound as possible. This is not the case at the moment, as the Folder objects in the Annotation Table are just mapped to Files in RO-Crate, losing both knowledge about its actual type as well as the files it contains (see nfdi4plants/arc-export#54 for why this is problematic).

Implemented Solution

We don't want to access the actual Filesystem when mapping the metadata from Scaffold to RO-Crate, so, in this PR, I implemented logic to make use of the In-Memory Filesystem stored as a field in the ARC object. From there we can check, whether an object annotated as Data is actually a file or a folder and then map it accordingly.

If the object is a folder, I currently add a second type Dataset in addition to File and create and reference all subfiles it contains via the hasPart property.

E.g. when we have an annotation table that references the folder ABC.D as Input [Data], which in turn contains the two files SubFile.txt and SubFolder/SubSubFile.txt, the output RO-Crate metadata will contain the following objects:

[
    {
      "@id": "assays/MyAssay/dataset/ABC.D/SubFile.txt",
      "@type": "File",
      "name": "assays/MyAssay/dataset/ABC.D/SubFile.txt"
    },
    {
      "@id": "assays/MyAssay/dataset/ABC.D/SubFolder/SubSubFile.txt",
      "@type": "File",
      "name": "assays/MyAssay/dataset/ABC.D/SubFolder/SubSubFile.txt"
    },
    {
      "@id": "assays/MyAssay/dataset/ABC.D",
      "@type": [
        "File",
        "Dataset"
      ],
      "name": "assays/MyAssay/dataset/ABC.D",
      "hasPart": [
        {
          "@id": "assays/MyAssay/dataset/ABC.D/SubFile.txt"
        },
        {
          "@id": "assays/MyAssay/dataset/ABC.D/SubFolder/SubSubFile.txt"
        }
      ]
    },
    {
      "@id": "#Process_A_MyAssay_MyTable_1",
      "@type": "LabProcess",
      "name": "MyTable_1",
      "object": {
        "@id": "assays/MyAssay/dataset/ABC.D"
      },
      "result": [],
      "executesLabProtocol": {
        "@id": "#Protocol_MyTable"
      }
    }
 ]

To go full circle, the file and folder objects referenced in the ARC RO-Crate will also be put into the Filesystem of the ARC Scaffold object.

[HLWeil]

Little note on why I used the double type File and Dataset on these folder objects:

The same object satisifies two distinct use-cases, which according to semantics and the profiles each require their own type:

• Dataset or git perspective: The .D object itself is not tracked by git or git-lfs, making it just a container for the actually tracked files
• File or ISA perspective: The .D folder is a singular research entity referenced by the provenance graph, we type these objects as Mediaobjects

[Freymaurer]

this is still an ftestCase? i would assume this throws on CI. If not we can pass pyxpecto an argument to throw if ftests exist.

[Freymaurer]

i assume this is critical as it blocks nearly 1900 tests from running

[HLWeil]

Thanks for noticing! That's quite a big whoopsie on my side. And yes, could you add the argument in a commit to this PR after I fix this specific case?