Unable to use basic groovy operations on process `path()` input

[jfy133]

Bug report

(Please follow this template by replacing the text between parentheses with the requested information)

Expected behavior and actual behavior

When investigating this bug report of find not working with CIFS mounted symlinks, @mahesh-panchal @prototaxites and I tried to test replacing the common nf-core find pattern of finding a filename prefix inside an input directory with a native groovy system.

i.e., we tried to replace

script:
INDEX=`find -L ./ -name "*.rev.1.bt2" | sed "s/\\.rev.1.bt2\$//"`

We tried this with different variations of:

def index_prefix = file(index).listFiles().find { it.name.endsWith(".rev.1.bt2") }.toString() - ".rev.1.bt2"

script:
INDEX=${index_prefix}

What we discovered was that this works fine if the input defining index is a raw file(),

process test {

    input:
    path dir
    
    output:
    val var
    
    shell:
    var = file(dir).listFiles().find { file(it).extension == "c" }
    """
    echo $var
    """
}

workflow {
    test(file("./files"))
    test.out.view()
}

However, when the input to the process is a path() output from a previous process (in this case taking the output directory emitted by BOWTIE2_BUILD as input to BOWTIE2_ALIGN), the listFiles() and subsequent operators do not work, as listFiles() always resulting in null.

When we println(index.getProperties()) we saw that the input path(index) appears to be a special Nextflow class:

  [target:/home/james/git/nf-core/modules/.nf-test/tests/6efa8cd7f5a46a16a3241dcd8a20baf0/work/6c/a3068651f4f8926503437b12cdbc1c/bowtie2, alias:bowtie2, absolute:false, baseName:bowtie2, class:class nextflow.processor.TaskPath, extension:, fileName:bowtie2, fileSystem:sun.nio.fs.LinuxFileSystem@231c78f2, hidden:false, link:true, name:bowtie2, nameCount:1, parent:null, permissions:rwxr-xr-x, root:null, simpleName:bowtie2, uri:file:///home/james/git/nf-core/modules/.nf-test/tests/6efa8cd7f5a46a16a3241dcd8a20baf0/bowtie2]

However we feel that the path() class should(?) still respect Groovy functions.

Steps to reproduce the problem

1. Clone https://github.com/nf-core/modules/
2. Modify https://github.com/nf-core/modules/blob/master/modules/nf-core/bowtie2/build/main.nf as described above
3. Run nf-test test modules/nf-core/bowtie2/align/tests/main.nf.test --profile +docker --update-snapshot --verbose

Program output

When running the file(index).listFiles()... pattern above we get:

java.lang.NullPointerException: Cannot invoke method getAt() on null object

and

Command executed:
  
    INDEX=null
    
    bowtie2 \
        -x $INDEX \
        -1 test_1.fastq.gz -2 test_2.fastq.gz \
        --threads 2 \
         \
         \
        2>| >(tee test.bowtie2.log >&2) \
        | samtools view  --threads 2  -o test.bam -

Environment

• Nextflow version: 25.04.7
• Java version: 17.0.13-internal (openjdk)
• Operating system:Linux
• Bash version: GNU bash, version 5.2.21(1)-release (x86_64-pc-linux-gnu)

Additional context

(Add any other context about the problem here)

[jfy133]

Please @mahesh-panchal @prototaxites correct me wherever I've got this wrong 😅

[pontus]

My first thought here is whatever listFiles is guaranteed to be implemented for other file drivers (e.g. supporting s3:// work directories).

[bentsherman]

I'm having trouble reproducing your error. Try to provide a self-contained script, not something that requires modifying nf-core modules and running nf-test

I am getting a different error with this example:

process test {
    input:
    path dir

    script:
    println dir.listFiles()
    '''
    '''
}

workflow {
    test(file("files"))
}

$ nextflow run main.nf
ERROR ~ Error executing process > 'test'

Caused by:
  java.nio.file.ProviderMismatchException -- Check script 'main.nf' at line: 7


Source block:
  println dir.listFiles()
  '''
  '''

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details

This error is also coming from listFiles().

I think in your case listFiles() is returning null:

nextflow/modules/nf-commons/src/main/nextflow/extension/FilesEx.groovy

Lines 639 to 642 in 253666d

	static Path[] listFiles(Path self, @ClosureParams(value = FromString.class, options = ["java.nio.file.Path", "java.nio.file.Path,java.nio.file.attribute.BasicFileAttributes"]) Closure<Boolean> filter=null) {
	if( !self.isDirectory() )
	return null

In both cases it seems like the TaskPath class is missing a few things. I have included a potential fix in #6581

[jfy133]

Sorry about the lack of a reprex @bentsherman this was 3 errors deep at the time so wanted to get it recorded some how.

I will see if I reconstruct the exact case in a minimal context, and get back to you.