[Discussion] Dataset Abstraction

[GabrielKP]

When working with multiple documents, it possibly would be great to include some sort of abstraction for multiple documents forming a Dataset.

The Dataset would then allow:

• Caching of Datasets after doing transformations on them (which I am currently requiring), e.g. in a Pipelined workflow
• Saving and loading Datasets efficiently.

@ArneBinder and I were discussing using HuggingFace Datasets as base class, which would bring the hole power of HuggingFace Datasets to pytorch-ie.

[ArneBinder]

@GabrielKP Thanks for opening this. This is related to or depends on #120.

[ChristophAlt]

@GabrielKP @ArneBinder Thanks for kicking of the discussion.

@GabrielKP : What kind of transformations do you currently require? Are these applied before you turn the HF dataset into a collection of PIE documents?

[ChristophAlt]

I've also investigated the use of Huggingface's Dataset class as the base class before going on vacation. The main challenge is how to represent a document, specifically the annotations, as an arrow table.

Let's assume that we have a document with text, an id, metadata, and an arbitrary number of annotations of the types described in #120.

The main questions are:

• How to represent the annotation types as arrow schemas / types?
• Is it necessary to support mapping of these annotations (when represented as an arrow table)?
• How to represent the difference between groundtruth and annotation / prediction?
• How does this affect the operations currently supported by HF Dataset?
• What kind of operations do we want to support?
• How does this work with the inference / evaluation pipeline? E.g. how costly / feasible is it to iteratively add predictions to an instance during inference.

[ChristophAlt]

After spending some thoughts on this and related topics, I think the initial step would be to create a dataset for development that reflects all the aspects we want to implement initially. The dataset should probably contain 3-5 example documents with relevant annotations in a common data format. Based on that we can go through all the steps necessary to go from loading a dataset to training, evaluation, and inference.

Each document should:

• contain text (at least two with multiple sentences)
• contain a document id
• contain some arbitrary metadata
• sentence annotations
• named entity annotations (where applicable)
• relation annotations (where applicable)

Important edge cases:

• sentences without entity annotations (and thus without relation annotations)
• sentences without relation annotations
• sentences without relation annotations but more than two entities
• single sentence per document
• single sentence per document without annotations
• annotations that span multiple sentences

Other considerations:

• the file format should probably be JSON (maybe the dygiepp format)
• the documents should be as short as possible

The steps to be prototyped are:

• loading the dataset as a HF dataset / dataset dict
• converting the HF dataset / dataset dict to PIE dataset / dataset dict (document abstraction)
• preparing the model inputs (dict of tensors and other non-tensor metadata)
  • consider the special case of splitting a document into multiple model inputs
  • consider the special case of windowing
• prepare targets
• caching of model inputs and targets (?)
• creating and running a pipeline (one stage and two stage) (that adds predictions to a document)
  • with gold labels (groundtruth) required to create an input (e.g. relation extraction)
  • with predictions to create an input (e.g. relation extraction as the second stage)

[ArneBinder]

Thanks a lot for the write up! Some remarks for the dummy dataset:

• Can we also go with a predifined test split?
• additional edge cases:
  • relation that spans two sentences

I'm quite curious about the pipeline setup. Do we have to distinguish between an evaluation and a prediction pipeline? I don't think it is useful to answer that now, just collecting further directions of thinking.

Regarding the data format: https://github.com/dwadden/dygiepp looks to be already tokenized. I think this is not favorable. However, we should keep also this case in mind.

[ChristophAlt]

I added annotations span multiple sentences as an edge case.

• Could you elaborate on the predefined test split?
• I'm honestly not sure yet if we have to distinguish evaluation and prediction pipeline -- I agree to postpone that part and see as we progress.
• You're right regarding the data format, tokenized is not what we want. I'd suggest to use a dataset of the following simple format then (the file contains a list of these json objects):

[
  {
    "text": "A works at B.",
    "id": "ABC1234",
    "metadata": { "key1": "value1", "key2": 12345 },
    "sentences": [ {"start": 0, "end": 13} ]
    "entities": [ {"start": 0, "end": 1, "label": "PER"}, {"start": 11, "end": 12, "label": "ORG"} ],
    "relations": [ {"head": 0, "tail": 1, "label": "per:employee_of"} ]
  },
  ...
]

[ArneBinder]

This looks quite good!

Could you elaborate on the predefined test split?

I think it is a common case that a dataset contains a test set and often also a validation set. The idea was just to model this in our dummy data, e.g. by having a single document dedicated to it, to increase visibility of that and to recognize any edge cases related to having multiple splits early.

[ChristophAlt]

Please have a look at the dataset.

• I added examples for all of the usecases (hopefully).
• I added two validation examples to the training data, which can be used when simulating splitting the train dataset into train and validation split.
• I added a dedicated validation as well.
• I added a test set.

[ArneBinder]

I created #129 to document our decisions and to visualize how individual concepts are related and connected. @ChristophAlt please have a look.

[ArneBinder]

Please have a look at the dataset.

I created a PR for that: #130

[GabrielKP]

What kind of transformations do you currently require? Are these applied before you turn the HF dataset into a collection of PIE documents

I am marking my dataset with NER-tags and dependency trees, e.g. with Stanza. After that I mark specific pairings of certain entities found beforehand.

Although this could be done before turning the dataset into a PIE documents, I implemented it (and think it may be better) to happen after the conversion, as I can imagine setting up the project so that I can later enhance these annotations without deleting the old ones by using a more sophisticated method.

Sorry if I am not completely following you on the details of this, but you can ask me about my opinion anytime!

[ChristophAlt]

I have an initial version of a Document that is JSON-serializable and de-serializable, and thus can be persisted (and cached) as an Arrow table. See here.

It still looks a little rough around the edges but the changes can be summarized as follows:

• users can now define their own Documents (and annotations) in a dataclass-style fashion
• annotations are now dataclasses to simplify things like hash computation and enforcing frozen representations
• a document can be serialized as a dictionary of primitive types, references are replaced by object hashes
• a document can be deserialized from such a dictionary, the dependency chain of annotations is resolved too,
  e.g. entities must be deserialized before relations, because the latter depend on the former (this is inferred from the target graph, i.e. the graph that contains the dependencies between annotations, e.g. entities --> text, sentences --> text, relations --> entities).

There are still a few things missing:

• Get datasets to transparently convert from Document to dict and vice versa (this is possible, HF datasets does this for Image and Audio features, however the current implementation doesn't allow for an easy extension)
• The current assumption that annotation fields are always lists of annotations (e.g. a list of spans) but there may be fields that only contain a single annotation, e.g. a document-level label.
• How to continue with the groundtruth / predictions concept for an annotation field? This only concerns evaluation and inference (when operating directly on documents), but has no impact on training.