Skip to content

Process-level container runtime directive to support hybrid operation #6227

Open
@pinin4fjords

Description

@pinin4fjords

New feature

Currently, Nextflow supports a process-level executor directive, which is useful for [hybrid executions](https://www.nextflow.io/docs/latest/aws.html#hybrid-workloads) in scenarios where Docker is used across all executors. However, this approach has limitations when mixing container runtimes, for example, using Singularity locally and Docker on AWS Batch.

To address this, could we introduce a process-level containerRuntime directive? This would allow explicit pairing of the executor and container runtime, enabling more flexible and robust hybrid execution workflows.

Use case

In hybrid execution scenarios, changing the executor for a process often requires switching the container runtime accordingly. Having a dedicated directive to specify the container runtime per process would streamline this configuration and avoid potential mismatches.

Suggested implementation

process {
    executor = 'singularity'
    singularity.enabled = true

    withName: 'someprocess' {
        executor = 'awsbatch'
        containerRuntime = 'docker'
    }
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions