Anyone able to use the spacy-experimental.ngram_sentence_suggester.v1 with spancat?

[lio-data]

Hi,
I am having a very hard time using the ngram_sentence_suggester. I have a model which works. The model is standard, generated from the spacy online cfg generator and I train it on a marked dataset. The pipeline = ["tok2vec","ner","sentencizer","spancat","parser","morphologizer","tagger"] and this is initialized with vectors = "fr_core_news_lg"
There are 2 custom elements in my cfg a callback and the tokenizer. All works just fine. I am able to load my model in python and it predicts what it is supposed to predict. This looks like a good baseline 👍

I have then tried to switch from :
@misc = "spacy-experimental.ngram_sentence_suggester.v1" to
@misc = "spacy.ngram_suggester.v1"
because my spancat should mark sentences and ngrams. This is the only change introduced.

With the ngram_sentence_suggester I keep on getting this error message :
ValueError: [E030] Sentence boundaries unset. You can add the 'sentencizer' component to the pipeline with: nlp.add_pipe('sentencizer'). Alternatively, add the dependency parser or sentence recognizer, or set sentence boundaries by setting doc[i].is_sent_start

The problem is that my model marks the is_sent_start . I checked it : I went back to the spacy.ngram_suggester.v1, trained and saved the model, loaded it into python. I have disabled "parser","morphologizer","tagger" and ran the model on dummy text data then looked at the output. Each token had is_sent_start to either True or False.

I just don't seem to get it working. I have even reinstalled from scratch python, spacy, spacy-experimental, ... yesterday but I still cannot make it work and the error message seems wrong ?

Anyone having an idea ?

[kadarakos]

Hey lio-data,

Seems pretty frustrating indeed, may I ask you to post the config?

[lio-data]

Thank you for looking into it :-) ! Sure !

`[paths]
train = ".\marked_data\50.spacy"
dev = ".\marked_data\50.spacy"
vectors = "fr_core_news_lg"
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "fr"
pipeline = ["tok2vec","ner","sentencizer","spancat","parser","morphologizer","tagger"]
batch_size = 1000

tokenizer = {"@Tokenizers":"spacy.Tokenizer.v1"}

tokenizer = {"@Tokenizers":"custom_tokenizerv1"}
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null

[components]

[components.morphologizer]
factory = "morphologizer"
extend = false
overwrite = true
scorer = {"@scorers":"spacy.morphologizer_scorer.v1"}

[components.morphologizer.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false

[components.morphologizer.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.parser]
factory = "parser"
learn_tokens = false
min_action_freq = 30
moves = null
scorer = {"@scorers":"spacy.parser_scorer.v1"}
update_with_oracle_cut_size = 100

[components.parser.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "parser"
extra_state_tokens = false
hidden_width = 128
maxout_pieces = 3
use_upper = true
nO = null

[components.parser.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.sentencizer]
factory = "sentencizer"
overwrite = false
punct_chars = null
scorer = {"@scorers":"spacy.senter_scorer.v1"}

[components.spancat]
factory = "spancat"
max_positive = null
scorer = {"@scorers":"spacy.spancat_scorer.v1"}
spans_key = "attributes"
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 128

[components.spancat.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = null
nI = null

[components.spancat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.spancat.suggester]

@misc = "spacy-experimental.ngram_sentence_suggester.v1"

@misc = "spacy.ngram_suggester.v1"
sizes = [1,2,3]

[components.tagger]
factory = "tagger"
neg_prefix = "!"
overwrite = false
scorer = {"@scorers":"spacy.tagger_scorer.v1"}

[components.tagger.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false

[components.tagger.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,1000,2500,2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 1
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null
before_update = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@Loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_f = 0.2
ents_p = 0.0
ents_r = 0.0
ents_per_type = null
sents_f = 0.0
sents_p = null
sents_r = null
spans_sc_f = 0.2
spans_sc_p = 0.0
spans_sc_r = 0.0
dep_uas = 0.1
dep_las = 0.1
dep_las_per_type = null
pos_acc = 0.1
morph_acc = 0.1
morph_per_feat = null
tag_acc = 0.2

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null

[initialize.after_init]
@callbacks = "feed_vocabulary"
[initialize.components]

[initialize.tokenizer]`

[lio-data]

and sorry the chat interprets the # as markdown ! So the pieces of text that are capitalized are in fact comments !

[kadarakos]

Could you try putting the config text into a

code block
like this

(I'm reviewing it in the meantime just mentioning it here for others to also see)

[lio-data]

[paths]
train = ".\\marked_data\\50.spacy"
dev = ".\\marked_data\\50.spacy"
vectors = "fr_core_news_lg"
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "fr"
pipeline = ["tok2vec","ner","sentencizer","spancat","parser","morphologizer","tagger"]
batch_size = 1000
# tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
tokenizer = {"@tokenizers":"custom_tokenizerv1"}
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null

[components]

[components.morphologizer]
factory = "morphologizer"
extend = false
overwrite = true
scorer = {"@scorers":"spacy.morphologizer_scorer.v1"}

[components.morphologizer.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false

[components.morphologizer.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.parser]
factory = "parser"
learn_tokens = false
min_action_freq = 30
moves = null
scorer = {"@scorers":"spacy.parser_scorer.v1"}
update_with_oracle_cut_size = 100

[components.parser.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "parser"
extra_state_tokens = false
hidden_width = 128
maxout_pieces = 3
use_upper = true
nO = null

[components.parser.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.sentencizer]
factory = "sentencizer"
overwrite = false
punct_chars = null
scorer = {"@scorers":"spacy.senter_scorer.v1"}

[components.spancat]
factory = "spancat"
max_positive = null
scorer = {"@scorers":"spacy.spancat_scorer.v1"}
spans_key = "attributes"
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 128

[components.spancat.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = null
nI = null

[components.spancat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.spancat.suggester]
# @misc = "spacy-experimental.ngram_sentence_suggester.v1"
@misc = "spacy.ngram_suggester.v1"
sizes = [1,2,3]

[components.tagger]
factory = "tagger"
neg_prefix = "!"
overwrite = false
scorer = {"@scorers":"spacy.tagger_scorer.v1"}

[components.tagger.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false

[components.tagger.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,1000,2500,2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 1
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null
before_update = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_f = 0.2
ents_p = 0.0
ents_r = 0.0
ents_per_type = null
sents_f = 0.0
sents_p = null
sents_r = null
spans_sc_f = 0.2
spans_sc_p = 0.0
spans_sc_r = 0.0
dep_uas = 0.1
dep_las = 0.1
dep_las_per_type = null
pos_acc = 0.1
morph_acc = 0.1
morph_per_feat = null
tag_acc = 0.2

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null

[initialize.after_init]
@callbacks = "feed_vocabulary"
[initialize.components]

[initialize.tokenizer]

[kadarakos]

My first question is whether you are training all these components, because they do not seem to be sourced from pre-trained pipelines.

Perhaps it would be the easiest to debug if you'd create a new .cfg for now just with the sentencizer and spancat and your custom tokenizer and callback. Make sure to add the sentencizer to annotating_components during training so that the sentence-boundaries are actually annotated during the process.

[lio-data]

👍 Thank you ! I will try to train a shorter model with only the sentencizer, spancat and my custom tokenizer and callback.

I am currently doing a proof of concept. I am using spacy to try to work on a catalog of tools (hammer, drillers and the like). Today all is done by hand and I want to show it can be much better via ML. We receive raw data from manufacturer in text and I am trying to make spacy learn to identify in the text brands, attributes (like driller has percussions) and measures (2 kg or 1500 Watts)...

My process is the following:

1. I mark data with spacy in python. I start with the "fr_core_news_lg" model and that add multiple custom elements (custom tokenizer, custom rule based NER, custom rule based span, ...). My aim is get documents with the right ents (brands, units and complex numbers) and spans (attributes). I save my marked data via a DocBin on my hard drive (file called 50.spacy). The output is already quite good.
2. I then try to build a new model from that data via cfg. I need to reuse my tokenizer (that's why we have it in the cfg) but all the rest used in 1 should go. I want to move away from rules. The callback function is because the label I have given to certain spans are not French words (things like cn_fraction) and were out of vocabulary for the new model and cause the training to crash (the label was no in the docstring).

Once my POC is ok, I will use prodigy to manual improve the annotation made by my rule based model (described in 1).

[lio-data]

Thank you kadarakos. Adding the sentencizer in the list of annotating_components was the trick ! I had totally forgotten about that. It now works 😄

[kadarakos]

Bumm, there we go! Nice!

To make sure you POC is coming along well let me ask why are you are adding all these components? You would like to use the pre-trained parser and tagger and ner to annotate the Docs because your rules use these annotations?

[lio-data]

I progress step by step with a bit of trial/error/success approach. I learn Spacy while building my POC. So not all is well thought 😆 !

My rules are indeed using partially certain of these annotations (mostly POS) but this is not the reason for including all in the model I am training. My 2 reasons are:

• I am expecting that in the future I will need POS, vectors and potentially others for multiple reasons (which are not yet in my POC). One reason which is easy to explain is that manufacturers give different names to the same measure (and other things I am trying to detect). As an example some will talk about dimensions, some will talk about size, some will talk about length, some will jsut write 50x60 (50 cm by 60 cm)... but this is always the same feature ie how big the object is ! I will try in a future step to find a way to rationalize that and I am sure vectors and others will help !
• I believe that the accuracy of my POC (which is mostly about ner and spancat) will be better if parser, tagger, ... have annotated the data. I wanted jsut to try with and without these parser, tagger, ... and see how good they would do ! Though I'd better add them to list of annotating_components !!

[kadarakos]

If you are trying to use the tagger and parser to produce annotations they need to be sourced from pre-trained pipelines: https://spacy.io/usage/processing-pipelines#sourced-components.