implement DocumentStatistics

[ArneBinder]

This add DocumentStatistics, an abstract class derived from DocumentMetrics to easily collect statistics over datasets. It requires to implement a method _collect(doc: Document) -> Any. It should calculate any values over the document, can return naive types (such as int, float, str), dicts, or lists.

The _compute() will aggregate the results of _collect() in the following way : naive types are arranged in a list, lists are concatenated, and dictionaries are flattened and their values are handled such as naive types or lists, respectively, and finally unflattened again (this is because calling the DocumentMetric on a DatasetDict will produce a dictionary with the split names as keys and the metric results as values, so this would result in a semi-flattened dict, if we would not unflatten again).

This also implements the following document statistics:

• TokenCountCollector (formerly DocumentTokenCounter): Collects the token count of a field when tokenizing its content with a Huggingface tokenizer.
• FieldLengthCollector (formerly DocumentFieldLengthCounter): Collects the length of a field, e.g. to collect the number the characters in the input text.
• SubFieldLengthCollector (formerly DocumentSubFieldLengthCounter): Collects the length of a subfield in a field, e.g. to collect the number of arguments of N-ary relations.
• LabeledSpanLengthCollector (formerly DocumentSpanLengthCounter): Counts the length of spans in a field per label, e.g. to collect the length of entities per type.
• DummyCollector (formerly DummyCounter): A dummy collector that always returns 1, e.g. to count the number of documents.
• LabelCountCollector (formerly LabelCounter): Collects the number of field entries per label, e.g. to collect the number of entities per type.