Skip to content

Separate Zyte related code from core #382

@rennerocha

Description

@rennerocha

Considering that Spidermon was created at Zyte (former Scrapinghub), it is understandable that we have monitors and actions that are related to services provided by it. However, considering that Spidermon can be used by users that are not running their spiders at Zyte infrastructure, it would be a good refactoring to keep every code that is Zyte-related only inside a separated module.

My first suggestion is to move everything (actions, monitors, mixins and utils) that are spread in the project into a separated spidermon.contrib.zyte.* module, so it will be clear to everyone that anything imported from this module is related to Zyte and may require that the extension is being executed inside Zyte infrastructure.

I found Zyte-only related code in the following files:

  • spidermon/spidermon/utils/zyte.py
  • spidermon/contrib/stats/statscollectors/sc_collections.py
  • spidermon/contrib/scrapy/monitors.py
  • spidermon/contrib/scrapy/monitors/mixins/job.py
  • spidermon/contrib/actions/jobs/tags.py

Tasks to accomplish

  • Create spidermon.contrib.zyte module
  • Move all related code to this module
  • Avoid backward incompatibility adding a deprecated warning if we try to import in old locations (but allows it)
  • Update documentation

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions