-
Notifications
You must be signed in to change notification settings - Fork 101
Description
Considering that Spidermon was created at Zyte (former Scrapinghub), it is understandable that we have monitors and actions that are related to services provided by it. However, considering that Spidermon can be used by users that are not running their spiders at Zyte infrastructure, it would be a good refactoring to keep every code that is Zyte-related only inside a separated module.
My first suggestion is to move everything (actions, monitors, mixins and utils) that are spread in the project into a separated spidermon.contrib.zyte.*
module, so it will be clear to everyone that anything imported from this module is related to Zyte and may require that the extension is being executed inside Zyte infrastructure.
I found Zyte-only related code in the following files:
- spidermon/spidermon/utils/zyte.py
- spidermon/contrib/stats/statscollectors/sc_collections.py
- spidermon/contrib/scrapy/monitors.py
- spidermon/contrib/scrapy/monitors/mixins/job.py
- spidermon/contrib/actions/jobs/tags.py
Tasks to accomplish
- Create
spidermon.contrib.zyte
module - Move all related code to this module
- Avoid backward incompatibility adding a deprecated warning if we try to import in old locations (but allows it)
- Update documentation