Using external stemmers in tantivy-py

[yoav-orca]

I want to use the stemmers from https://github.com/testuj-to/tantivy-stemmers, but it doesn't seem possible with the available API
Would you be open to extend the API to support external stemmers?

[cjrh]

We are certainly open to it. But it's a bit of a hornet's nest. See this note in the tantivy-stemmers repo:

We don't have any kind of plugin system yet that allows a user to install a separate python package or wheel, and have that somehow link dynamically to the main tantivy-py binary. There are no good plugin solutions in general right now in the rust ecosystem. It's an area of active development.

For us (tantivy-py) to use tantivy-stemmers would mean either making a build of tantivy-py with all algortihms, that everybody must pay the cost of carrying, or we must make separate builds tantivy-py for each extra stemmer. Neither of these are particularly attractive.

We already have this issue with external tokenizers, which I mentioned in this PR and also in #358 which proposes to investigate what the polars people are doing with runtime-installed plugins.

For your specific use-case, if you know which stemmers you want to support, you could modify your checkout of the tantivy-py source, include the stemmers you want, and build your own wheel (it's not hard) which you can then install into your python environment. It's a lot more difficult for me to solve in the one tantivy-py wheel build that everybody will use.

[yoav-orca]

That's what I ended up doing, but I thought there might be a better path (which I could also share with other people)
Thanks for the thoughtful response