Tips with labeling functions - Skweak

[goonhoon]

I am currently annotating a large amount of data and wonder whether anyone has any tips on how to implement a labelling function that would, e.g., find all sentences containing 'X' and label them into a span category. I have about 700 txt files from which I would like to do this. Since I am annotating whole sentences, I do not need to be precise the same way I would with NER, and hence some sort of a function like this would help.

Am I right that Skweak could help me with this? Are there any further tips/tutorials on its implementation?

Many thanks :)

[thomashacker]

Hey, thanks for your question.
If your goal is to label spans based on a set of patterns/rules, using the SpanRuler / EntityRuler would be more helpful for your use case.

[goonhoon]

Thanks a lot, I will have a proper look. So far I have been using Sentencizer which did the job for me but I couldn't manage to extend the actual annotation onto the whole sentence.

[thomashacker]

After you run the SpanRuler all found spans can be found in doc.spans.
A possible approach could be to adjust the boundaries of the spans respective to the sentence after the SpanRuler was run.

May I ask why you want to have the annotations over the whole sentence? Are you aiming to conduct Text Classification or Span Categorization?

[goonhoon]

Thanks a lot, I missed the fact that the nlp object automatically creates a doc, which then needs to be converted to the spacy format.

As for why sentences, I want annotate a whole sentence instead of just the respective pattern from SpanRuler. In which case, I may indeed want to use spancat. I am annotating key contract provisions, and while in more than half of the cases the provisions may match one of my handful of patterns, I want the model to train on these and predict the label even if the wording would differ from all of my patterns (hence why I would prefer NER/SpanCat instead of just pure rule-based categorization). Some other contractual provisions have less rigid wording and so rule-based approach would not work there either.

A possible approach could be to adjust the boundaries of the spans respective to the sentence after the SpanRuler was run

My follow-up question on this point: So this means that the boundaries can be adjusted only after the patterns are matched and saved to Doc.spans. How would I work with Doc.spans to edit the boundaries? This would be my first attempt at further adjusting the doc object after the spans/ents are annotated, so thank you for being patient :D.

[thomashacker]

If the other key contract provisions that you want to catch are similar in terms of context, the Spancat should be able to generalize your annotations in such a way that it can also find the other key spans. I'd recommend trying out training the spancat first with the matched annotations and looking at the performance.

However, in terms of the implementation of extending the spans to sentence level, here is a suggestion:

import spacy
from spacy.tokens import SpanGroup, Span

nlp = spacy.load("en_core_web_sm")
doc = nlp("This is a simple sentence. This is the second sentence.")
doc.spans["spans"] = SpanGroup(
    doc,
    spans=[
        Span(doc, start=3, end=4, label="LABEL_A"),
        Span(doc, start=9, end=10, label="LABEL_B"),
    ],
)
print(doc.spans["spans"])

doc.spans["sentences"] = SpanGroup(doc)
for sentence in doc.sents:
    for span in doc.spans["spans"]:
        if span.start >= sentence.start and span.end <= sentence.end:
            doc.spans["sentences"] += [
                Span(doc, start=sentence.start, end=span.end, label=span.label_)
            ]

print(doc.spans["sentences"])

The first part creates two new spans ([simple, second]), and the other part iterates through all the sentences and checks whether a span lies within that sentence. If yes, it creates a new span with the sentence boundaries and the span label to a new SpanGroup, called sentences. If you manipulate and save your docs this way, you will have two SpanGroups, one with the spans from the SpanRuler and one with the sentence extended spans. When you train a Spancat component, you can specify which SpanGroup you want to use for training in the config; this is a great way to test both approaches and compare their performance.

You can read more about the Span object and SpanGroup object in our docs!

I hope this was helpful!

[goonhoon]

Thanks a lot, this is extremely helpful. Definitely hundred times more elegant than using a regex.

Will play around with the patterns and try to train a model using the below. Again, thanks!

import spacy
from spacy.tokens import SpanGroup, Span

nlp = spacy.blank("en")
ruler = nlp.add_pipe("span_ruler")
nlp.add_pipe("sentencizer")

patterns = [{"label": "Governing Law", "pattern": "shall be governed"}]
ruler.add_patterns(patterns)

doc = nlp("This Agreement shall be governed by the laws of the State of Texas.  All payments hereunder shall be made at Company's offices at Frisco, Texas.")

print([(span.text, span.label_) for span in doc.spans["ruler"]])

doc.spans["sentences"] = SpanGroup(doc)
for sentence in doc.sents:
    for span in doc.spans["ruler"]:
        if span.start >= sentence.start and span.end <= sentence.end:
            doc.spans["sentences"] += [
                Span(doc, start=sentence.start, end=sentence.end, label=span.label_)
            ]

print(doc.spans["sentences"])