Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set PYTHONPATH environment variable for Python script scrapers #4372

Merged
merged 3 commits into from
Dec 20, 2023

Conversation

Maista6969
Copy link
Contributor

With the introduction of the new scraper and plugin management system in #4242 we've seen several scrapers failing because of their dependency on other packages such as py_common. Since the new management system puts these scrapers in separate folders they need to hack around this by manually adding their parent to their sys.path

To solve this in Stash itself we can set the PYTHONPATH environment variable when running Python scrapers to add the base scraper directory to the list of paths where Python will search for module imports

Notes:

  • I know the variable names aren't great, but my imagination is lacking and they're entirely local.
  • I would have loved to have had access to the local path configured for this scraper source, but I do not know the code base well enough to include it. Currently the scraper knows its absolute path so I simply used that to get the parent of its parent folder, but this will fail if scrapers have nested folder structures 🤷‍♂️
  • I'm not familiar with any plugins that will need this so I have only added this to the scraper package

@WithoutPants WithoutPants added this to the Version 0.24.0 milestone Dec 19, 2023
@WithoutPants WithoutPants added the improvement Something needed tweaking. label Dec 19, 2023
@WithoutPants WithoutPants merged commit e8af3c8 into stashapp:develop Dec 20, 2023
2 checks passed
@Maista6969 Maista6969 deleted the pythonpath-env-variable branch December 20, 2023 02:39
halkeye pushed a commit to halkeye/stash that referenced this pull request Sep 1, 2024
…app#4372)

* Set PYTHONPATH environment variable for Python script scrapers
* Convert PYTHONPATH to absolute
* Generalise and apply to plugins

---------

Co-authored-by: WithoutPants <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Something needed tweaking.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants