Behaviour of is_oov for en_core_web_trf pipeline

[cbjrobertson]

Hi-- I'm trying to understand the is_oov token attribute. My understanding is that it is a Boolean indicator of whether the token has an associated vector. However, I don't understand it's behaviour when used with the "en_core_web_trf" pipeline.

For example:

nlp = spacy.load("en_core_web_trf")
[tok.is_oov for tok in nlp("Is this out: alksjfad")]
>>> [True, True, True, True, True]

Compare this with:

nlp = spacy.load("en_core_web_lg")
[tok.is_oov for tok in nlp("Is this out: alksjfad")]
>>> [False, False, False, False, True]

Am I to understand that the "en_core_web_trf" pipeline doesn't include such common words as "is" and "this" in its vocabulary? I imaging this has to do with RoBERTa not representing tokens with 1:1 token: vector mappings. Is there a equivalent token.is_oov attribute for "en_core_web_lg"?

[polm]

is_oov indicates whether a token has a (static) word vector; the transformer pipelines don't include word vectors, so everything is OOV in that sense.

[cbjrobertson]

Yeah, that's what I thought. Makes sense, but maybe it isn't that useful. I'm trying to build a textcat pipeline which passes a transformer layer and also oov information forward to a textcat layer. I basically need either a flag for whether a token has been split into sub word pieces, or to build a pipeline with both a transformer and standard tok2vec with multihash embedder. I'm new to spaCy training. Any recommendations?

Background on this is I've experimented using spaCy standard fasttext vectors, floret vectors, and RoBERTa, and performance is ordered as follows: best: fasttext; middle: floret; worst: RoBERTa.

My data is from people who suffer from a condition which causes pathological syllable-level mispronounciation and letter substitutions. My hypothesis is that subword vectors in this case are not meaningful semantic units and the all-zero oov vector in en_core_web_lg is acting as a oov flag. I want to test whether adding an oov flag improves RoBERTa.

[adrianeboyd]

You can always add vectors to a trf pipeline if you want both for some reason. Just:

nlp_trf.vocab.vectors = nlp_md.vocab.vectors

The transformer component and the vectors don't interact.

However, if you're just using the vectors as a word list, the vector data is a lot of large, unnecessary data, and it's also not a very good word list because it's the most frequent terms from mostly web crawl data and it's really noisy.

I'd recommend using just a word list for this, and a word list that's more suitable than the vector entries.

[cbjrobertson]

That sounds like good advice, how would you recommend passing forward oov information to the textcat component of the pipeline, whilst still using a transformer? I'm specifically hoping for any guidance on how to write a config.cfg which does this. I think a possible approach is to use spacy.MultiHashEmbed.v2 as well as a transformer, and feed these forward into my textcat layer, but I'm struggling with how to define this concretely in my config.cfg file.

[adrianeboyd]

While I think it's not entirely impossible to implement, we wouldn't really recommend trying to do this. I think it would be difficult to combine MultiHashEmbed with a transformer and you would need* a custom embedding architecture because MultiHashEmbed doesn't support an IS_OOV feature. The supported features are here: https://spacy.io/api/attributes

*Actually, just to test the idea, I'd recommend first trying it out by hijacking a lexical attribute you don't use otherwise use so you can test it more easily with MultiHashEmbed (just with tok2vec) to see if this helps at all. You could create a custom language that redefines IS_STOP to mean IS_OOV (with a simple word list, not through vectors), and then add IS_STOP as an attr with ~100 rows in MultiHashEmbed along with NORM/PREFIX/SUFFIX/SHAPE. Custom languages with custom stop word lists can be defined like this: https://spacy.io/usage/linguistic-features#language-subclass. In case it matters, be aware that stop words are lowercased by the default is_stop method before checking whether they're in the set.

My guess is that it's not going to help in addition to the existing default lexical tok2vec features, but you'd only know for sure after trying it out.