Question: is it possible to remove words from a `fastText` model (`.bin` file)?

[jchwenger]

Hi there,

I'm testing the pre-trained models from Meta, and it's nice to have the option of subword search for OOV words. However, for French at least, data cleaning leaves to be desired, leading to many duplicates and badly tokenised words.

I was thinking of testing this nevertheless, but filtering out all the words that are not in a separate dictionary file (I have pretty comprehensive lists of words). I could do that after computing the similarity, but it would be neater to remove the words and vectors from the model instead, so that there is less computation waste, and I wouldn't need to implement checks for topn (to make sure I actually obtain the n neighbours at each request).

Is this something that can be done using gensim (I would have asked on fastText first, but the repo is read-only) by any chance?

Thanks for this!
Best,
Jeremie

[gojomo]

If you've loaded a full model as a Gensim FastText type, into a variable ft_model, then the lookup map of known 'in-vocabulary' words is in ft_model.wv.key_to_index.

So there's one easy potential 1st step towards what you seek (but which likely won't deliver the full computation/storage savings you hope for): deleting keys from this map. After that, attempting to look-up the deleted keys won't find the known precomputed vector - but treat the word as if it were OOV, synthesizing the vector from n-grams.

However, such a quick bit of tampering doesn't remove the rows from the big dense array holding the raw vectors for all known words. So the .most_similar() operations, doing (vector-operation-optimized) bulk calculations against the whole array, will still calculate candidate similarities against those (now-phantom) words with deleted keys, and sort those results into the top-n results. And, since the names of those rows still exist in the ft_model.wv.index_to_key list, even report such words by name accurately. (You could replace their slots in this list with nulls/plug-values, but those plug values would still be reported.)

To really save the space, & skip the calcs, would require more-extensive & careful surgery on the model that carefully fixes any inconsistencies: remove the key_to_index keys, the associated slots in index_to_key, and the associated rows of the backing array – copying the later rows earlier, moving the final result to a smaller array, ensuring any remaining old assumptions/invariants corrected. It's certainly possible (and wouldn't even be that much code) but I don't know any existing tested code to do it. The big array move/compact means you'd probably want to do it in one batch (rather than word-by-word in a loop), or perhaps alternatively copy just the desired words to a new model. (If you wound up writing or finding good code for this, it'd be a valuable snippet to contribute to library docs or wiki – or if well-structured, even a utility method in the code.) After doing this once to a loaded model, you could re-save the full model with the edits.

Another approach that might approximate the same benefits with an existing supported option: use the restrict_vocab parameter of most_similar() to limit your nearest-neighbor searches to just the 1st n rows of the whole known-vocabulary – which are also by convention in descending frequency order. The junkiest maybe-shoulda-been-cleaned tokens are typically rarer than good results, so limiting results to some subset of more-common tokens often ensures more-reasonable results. And, the underlying bulk vector-optimized calcs will still be efficient on a contiguous prefix of the storage array.

[jchwenger]

That sounds all very lovely, thanks @gojomo for the detailed explanation!

I did try restrict_vocab, and unfortunately that seems to work pretty well, haha!

However, w.r.t. fiddling with the array, do you mean something like this? From that small experiment it looks like the results are correct, but really a shot in the dark, I might have missed one of those variables...

[gojomo]

That's the general idea, but from a quick look-over:

• pretty sure you don't want to alter vectors_ngrams in any way: those rows are the (collision-oblivious) ngram bucket vectors, their indices are unrelated to those of specific known words
• I think you could optionally avoid messing with .vectors (which can be rebuilt if missing via .adjust_vectors); and same (after .vectors is in place) with.norms which can be recalculated with .fill_norms(). (unsure if this would be faster than your abbreviated copies, but it might be less code, or make it easier to do destructively in-place in a single model.)
• if any model used hs mode (rather than negative sampling) the process might be more complicated
• it's been a while since I was familiar with this code; there might be some other assumptions/cached-state elsewhere needing updates. but if you run enough sanity checks in your usages that (eg) similarities & relative neighbor orders remain the same, and none of your usual triggers new exceptions, you're probably good enough

[jchwenger]

Thanks for this! vectors_ngrams were also the same shape as the others, (2000000, 300), looked too suspiciously the same not to be related, but ok. Indeed it all might require more checks and a bit of study of the code base (no idea what that hs mode is, for instance) on my part. Will test the .adjust_vectors method, I corrected .norms because it had been computed after I did one test with model2, but it's None on a newly instantiated model. Typically I don't know what the vectors are (as opposed to e.g. syn1neg), so I will have to see if I pursue that further or if I just limit the vocab.

[jchwenger]

Update. Looking at this bit It looks like actually vectors are the vocabulary word vectors adjusted by adding the relevant subword vector information to them (geometrically), which means that vectors_vocab contains zero or more vectors (subword decomposition if there is one) for each word in the vocab. Therefore, when filtering things out, rows ought to be removed there as well.

[jchwenger]

Additional update (sorry for pestering!): given that in my application removing those vectors would be a quite regular process (by applying some constraint on the search, say if you only want the top 10 most similar words to a query, but only words that have five letters, or start with something, or correspond to a regex, then reverting to the original vocab, etc.), it is probably not that smart to create these big copies of the vector matrix. Another line of inquiry could be to modify most_similar so as to do the similarity computation only on the valid vectors. Fun times ahead!

[jchwenger]

Hello once more @gojomo, most likely my last update: in the end I modded most_similar, removing the excluded indices from the computation, but leaving the full matrix intact (see the notebook). Probably not quite optimised, and not extensively tested, but just now it looks like it's working. Unclear to me that this would interest many people, but I thought I'd leave it there, in case.