Skip to content

Using storeDir with process.stageOutMode = "rsync" creates an invalid cache dir #6214

Open
@nvnieuwk

Description

@nvnieuwk

Bug report

Expected behavior and actual behavior

Using the storeDir directive in combination with the rsync stageOutMode is unable to create a valid storage directory which causes the pipeline to fail. Everything works fine when using the directive with a copy stageOutMode

Expected behaviour (copy with storeDir 'example'):

$ tree example/
example/
├── 1.txt
├── 2.txt
├── 3.txt
├── 4.txt
└── 5.txt

Actual behaviour (rsync with storeDir 'example'):

$ ll
...
-rw-rw-r--  1 nvnieuwk nvnieuwk    42 Jun 25 10:44 example
...

A file has been created with the name of the storeDir instead of a directory containing the correct files

Steps to reproduce the problem

main.nf:

workflow {
    def numbers = Channel.of(1, 2, 3, 4, 5)
    EXAMPLE(numbers)

}

process EXAMPLE {
    storeDir 'example'
    tag "$number"

    input:
    val number

    output:
    path('*.txt')

    script:
    """
    echo "This is an example process with number: ${number}" > ${number}.txt
    """
}

nextflow.config:

params.mode = "rsync"
process.stageOutMode = params.mode

The issue can be replicated by running the following command in a directory with the files presented above:

rm -rf example && nextflow run main.nf --mode rsync

Note that this method does work when the storeDir is already filled with the correct files, that's why you should remove it every time you run this command

The expected behaviour can be replicated by running the following command

rm -rf example && nextflow run main.nf --mode copy

Program output

ERROR ~ Error executing process > 'EXAMPLE (3)'

Caused by:
  Missing output file(s) `*.txt` expected by process `EXAMPLE (3)`


Command executed:

  echo "This is an example process with number: 3" > 3.txt

Command exit status:
  0

Command output:
  (empty)

Work dir:
  /home/nvnieuwk/Documents/nextflow/issues/rsync-storeDir/work/93/2feb18d4f824a9d9c247c773b31e15

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details

Environment

  • Nextflow version: 25.04.2 (but also noticed in older versions)
  • Java version: 17.0.3
  • Operating system: Linux
  • Bash version: 5.0.17

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions