Description
Bug report
Expected behavior and actual behavior
Using the storeDir
directive in combination with the rsync
stageOutMode
is unable to create a valid storage directory which causes the pipeline to fail. Everything works fine when using the directive with a copy
stageOutMode
Expected behaviour (copy
with storeDir 'example'
):
$ tree example/
example/
├── 1.txt
├── 2.txt
├── 3.txt
├── 4.txt
└── 5.txt
Actual behaviour (rsync
with storeDir 'example'
):
$ ll
...
-rw-rw-r-- 1 nvnieuwk nvnieuwk 42 Jun 25 10:44 example
...
A file has been created with the name of the storeDir
instead of a directory containing the correct files
Steps to reproduce the problem
main.nf:
workflow {
def numbers = Channel.of(1, 2, 3, 4, 5)
EXAMPLE(numbers)
}
process EXAMPLE {
storeDir 'example'
tag "$number"
input:
val number
output:
path('*.txt')
script:
"""
echo "This is an example process with number: ${number}" > ${number}.txt
"""
}
nextflow.config:
params.mode = "rsync"
process.stageOutMode = params.mode
The issue can be replicated by running the following command in a directory with the files presented above:
rm -rf example && nextflow run main.nf --mode rsync
Note that this method does work when the storeDir
is already filled with the correct files, that's why you should remove it every time you run this command
The expected behaviour can be replicated by running the following command
rm -rf example && nextflow run main.nf --mode copy
Program output
ERROR ~ Error executing process > 'EXAMPLE (3)'
Caused by:
Missing output file(s) `*.txt` expected by process `EXAMPLE (3)`
Command executed:
echo "This is an example process with number: 3" > 3.txt
Command exit status:
0
Command output:
(empty)
Work dir:
/home/nvnieuwk/Documents/nextflow/issues/rsync-storeDir/work/93/2feb18d4f824a9d9c247c773b31e15
Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
-- Check '.nextflow.log' file for details
Environment
- Nextflow version: 25.04.2 (but also noticed in older versions)
- Java version: 17.0.3
- Operating system: Linux
- Bash version: 5.0.17