Description
Bug report
Expected behavior and actual behavior
I'm currently using a custom pipeline with Nextflow v24.04.2 that includes topic channels using three of the .command files.
For example:
output:
tuple val(task.process), val(task.tag), path('.command.err'), emit: nf_stderr, topic: 'command_errors'
tuple val(task.process), val(task.tag), path('.command.sh'), emit: nf_script, topic: 'command_script'
tuple val(task.process), val(task.tag), path('.command.out'), emit: nf_stdout, topic: 'command_stdout'
This code works fine in v24, but when updating to v25, I get the following error:
***** Error report *****
Error executing process > 'HELLO_WORLD_DOCKER (Test)'
Caused by:
java.lang.RuntimeException:
Command executed:
echo "hello world"
Command exit status:
1
Command output:
hello worldmv: '.command.sh' and '/mnt/disks/nextflow-workspace./.command.sh' are the same filecp: cannot stat '.command.out': No such file or directorycp: cannot stat '.command.err': No such file or directory
Steps to reproduce the problem
An example of this process:
process HELLO_WORLD_DOCKER {
tag "$meta.id"
label 'process_single'
container "test_docker"
input:
tuple val(meta), path(r1), path(r2)
output:
tuple val(task.process), val(task.tag), path('.command.err'), emit: nf_stderr, topic: 'command_errors'
tuple val(task.process), val(task.tag), path('.command.sh'), emit: nf_script, topic: 'command_script'
tuple val(task.process), val(task.tag), path('.command.out'), emit: nf_stdout, topic: 'command_stdout'
when:
task.ext.when == null || task.ext.when
script:
"""
echo "hello world"
"""
}
Environment
The runtime environment:
- Local: Ubuntu 24.04.2 LTS
- Running local Nextflow head job with gcp-batch profile
- Ubuntu 22.04 docker container
Have tried the following versions which result in the same error:
- Nextflow: 25.04.2, 25.04.4 and 25.05.0-edge
- nf-google plugin: 1.21.0 and 1.22.0
Additional context
To narrow the issue down, I created a mock local example, which does run.
nextflow.enable.dsl=2
process DEMO_TOPICS {
container 'nextflow-test:latest'
output:
tuple val(task.process), val(task.tag), path('.command.err'), emit: nf_stderr, topic: 'command_errors'
tuple val(task.process), val(task.tag), path('.command.sh'), emit: nf_script, topic: 'command_script'
tuple val(task.process), val(task.tag), path('.command.out'), emit: nf_stdout, topic: 'command_stdout'
script:
"""
echo "Hello world"
"""
}
workflow {
// Collect from the topic channels
Channel.topic('command_errors').view { process, tag, file -> "ERROR from $process: ${file.text}" }
Channel.topic('command_script').view { process, tag, file -> "SCRIPT from $process: ${file.text.trim().split('\n')[0]}" }
Channel.topic('command_stdout').view { process, tag, file -> "OUTPUT from $process: ${file.text}" }
DEMO_TOPICS()
}
So the issue would appear to be specifically linked to using GCP and accessing the .command files.
Suspecting this, I tried using afterScript, but the same issue occurred.
I've tried a bunch of somewhat related flags and options, including the following - but didn't resolve the issue:
- Making the outputs optional:
path('.command.err', optional = true) - Adding wildcards:
??.command.err - Different stage-outs:
StageOutMode 'copy', 'move' - Don't follow symlinks when copying:
mode: 'copyNoFollow' - Relative/absolute symlinks:
mode: 'rellink', 'symlink' - Enabled scratch:
scratch true
Possible cause:
It appears that in Nextflow 25.x, there's a race condition or change in how .command.* files are handled when used with topic channels in the GCP Batch environment. The error suggests that the files are being accessed simultaneously by different parts of the Nextflow execution system, or that there's a timing issue between when the files are created/staged and when the topic channels try to access them.
Workaround
There is a workaround for anyone else in the same situation, but it would be nice to be able to use the .command files directly.
output:
tuple val(task.process), val(task.tag), path('cmd.err'), emit: nf_stderr, topic: 'command_errors'
tuple val(task.process), val(task.tag), path('cmd.sh'), emit: nf_script, topic: 'command_script'
tuple val(task.process), val(task.tag), path('cmd.out'), emit: nf_stdout, topic: 'command_stdout'
script:
"""
echo "hello world" > cmd.out 2> cmd.err
cp .command.sh cmd.sh || echo "Script not available" > cmd.sh
"""