Skip to content

ingest: Split monolith transform rule #44

@joverlee521

Description

@joverlee521

Context

Currently in the ingest pipeline, there is a single transform rule that runs a shell pipeline of multiple Python scripts. This works in the automated ingest pipeline, but may be tedious to debug when developing or when there's an error in the pipeline.

Description

We can split up the single rule into multiple rules by using Snakemake's piped outputs feature. I don't think anyone in the group has used this feature, so we don't know the pitfalls.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions