Deploying a project with spaCy model dependency [ en_core_web_sm for example] to PyPi.

[hrshdhgd]

So we're trying to release our project to PyPi and have a dependency on the en_core_web_sm model. When I try to publish, I get the error:

  HTTP Error 400: Invalid value for requires_dist. Error: Can't have direct dependency: 'en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.2.0/en_core_web_sm-3.2.0.tar.gz'

We're using poetry as our package management tool and the dependency is declared as follows in pyproject.toml:

[tool.poetry.dependencies.en_core_web_sm]
url = "https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.2.0/en_core_web_sm-3.2.0.tar.gz"

Is there a way around this? Any ideas would be greatly appreciated!

[polm]

The best way to handle non-PyPI dependencies is to have a download step that needs to be executed after your project is installed, similar to how spaCy has the download command. I'm not familiar with Poetry, but basically you would remove the model as a dependency, check if it's installed at runtime, and if not, install it or fail with an error and tell the user how to install it.

[AntonOfTheWoods]

At runtime??? I thought spaCy, unlike corenlp & co, was intended for real production usecases. And you are advising downloading large dependencies to production servers when an instance spins up? Hmmm. Sounds like a data science convert from physics designing a devops workflow or something... Almost anything is possible with computers - lots of things aren't advisable though...

[polm]

This question is specifically about putting a project on PyPI, which doesn't offer a way to include large files or specify non-PyPI dependencies. Having a separate download step, or downloading a model the first time it's used, is a pretty standard approach taken by Huggingface Transformers, NLTK, and other machine learning libraries.

If you're hosting a production application, as you note, this is not the approach you want, and you can specify the pipeline as a dependency from a URL or download it in advance and keep it as a local package, as covered in the docs.