Native process resubmitting even though inputs are the same.

[mahesh-panchal]

Bug report

I have a native process that resubmits itself on -resume even though the inputs don't change, but only if I've cleaned previous executions.

Expected behavior and actual behavior

I expect the process to be cached. Nextflow only uses a cached version if I don't clean previous runs.

Steps to reproduce the problem

https://github.com/NBISweden/Earth-Biogenome-Project-pilot

nextflow run main.nf -profile docker,test -resume
nextflow clean -f -before $( nextflow log -q | tail -n 1 ) && find work/ -type d -empty -delete
nextflow run main.nf -profile docker,test -resume

If I don't clean, then the process uses a cached hit.

Program output

I posted the question in the community forum initially:
https://community.seqera.io/t/why-would-native-processes-exec-be-resubmitted-even-though-the-hash-inputs-dont-change/2259

hashes from run1

Jul-03 14:03:28.903 [Actor Thread 5] INFO  nextflow.processor.TaskProcessor - [ASSEMBLY_REPORT:TOL_SEARCH (1)] cache hash: 96e3c94a530ae88fb7647046ec705b3e; mode: STANDARD; entries: 
  76ea2021cc55a873477f77be45e8a1fc [java.util.UUID] 8e358b4c-7f8d-4421-8742-0a77a3f8c4d6 
  bdf226854332f21139e1ddac952e22ba [java.lang.String] ASSEMBLY_REPORT:TOL_SEARCH 
  f13fc1b539c963cfd9f7a5eda57f24b3 [java.lang.String]     def args = task.ext.args ?: ''
    def response = new URL("https://id.tol.sanger.ac.uk/api/v2/species?taxonomyId=$taxid").text
    def lazy_json = new groovy.json.JsonSlurper().parseText(response)
    json = [
        tol_id:  lazy_json.species[0]['tolIds'][0]?.tolId?: "No ToL ID",
        species: lazy_json.species[0]['scientificName'],
        class:   lazy_json.species[0]['taxaClass'],
        order:   lazy_json.species[0]['order'],
    ]
 
  16cb95a8f5388c13899a651695911162 [java.lang.String] taxid 
  dec56b9915c1138e7cd4677f712b5833 [java.lang.Integer] 7227 
  4f9d4b0d22865056c37fb6d9c2a04a67 [java.lang.String] $ 
  16fe7483905cce7a85670e43e4678877 [java.lang.Boolean] true 
  eab359affdb9334848cc02803d7db724 [java.util.HashMap$EntrySet] [task.ext.args=null] 

Jul-03 14:03:28.922 [Task submitter] INFO  nextflow.Session - [c1/d3eb41] Submitted process > ASSEMBLY_REPORT:TOL_SEARCH (Taxid: 7227)

and hashes from run2

Jul-03 14:04:33.476 [Actor Thread 2] INFO  nextflow.processor.TaskProcessor - [ASSEMBLY_REPORT:TOL_SEARCH (1)] cache hash: 96e3c94a530ae88fb7647046ec705b3e; mode: STANDARD; entries: 
  76ea2021cc55a873477f77be45e8a1fc [java.util.UUID] 8e358b4c-7f8d-4421-8742-0a77a3f8c4d6 
  bdf226854332f21139e1ddac952e22ba [java.lang.String] ASSEMBLY_REPORT:TOL_SEARCH 
  f13fc1b539c963cfd9f7a5eda57f24b3 [java.lang.String]     def args = task.ext.args ?: ''
    def response = new URL("https://id.tol.sanger.ac.uk/api/v2/species?taxonomyId=$taxid").text
    def lazy_json = new groovy.json.JsonSlurper().parseText(response)
    json = [
        tol_id:  lazy_json.species[0]['tolIds'][0]?.tolId?: "No ToL ID",
        species: lazy_json.species[0]['scientificName'],
        class:   lazy_json.species[0]['taxaClass'],
        order:   lazy_json.species[0]['order'],
    ]
 
  16cb95a8f5388c13899a651695911162 [java.lang.String] taxid 
  dec56b9915c1138e7cd4677f712b5833 [java.lang.Integer] 7227 
  4f9d4b0d22865056c37fb6d9c2a04a67 [java.lang.String] $ 
  16fe7483905cce7a85670e43e4678877 [java.lang.Boolean] true 
  eab359affdb9334848cc02803d7db724 [java.util.HashMap$EntrySet] [task.ext.args=null] 

Jul-03 14:04:33.497 [Task submitter] INFO  nextflow.Session - [c1/d3eb41] Submitted process > ASSEMBLY_REPORT:TOL_SEARCH (Taxid: 7227)

Environment

• Nextflow version: 25.04.4
• Java version: [?]
• Operating system: [macOS, Linux, etc]
• Bash version: (use the command $SHELL --version)

Additional context

(Add any other context about the problem here)

[bentsherman]

If you suspect that nextflow clean is removing the native process executions, you should be able to see that reflected in nextflow log. Are the tasks not in the log after cleanup?

[mahesh-panchal]

So it seems nextflow clean is cleaning it up:

[c1/d3eb41] ASSEMBLY_REPORT:TOL_SEARCH (Taxid: 7227)                                                                 [100%] 1 of 1 ✔

Then running nextflow clean -f -before $(nextflow log -q | tail -n 1) on it's own.

Removed /Users/mahpa906/Documents/Projects/Earth-Biogenome-Project-pilot/work/c1/d3eb4139d7c041f71e9080b7aa1554
...

If it hadn't been this, then it would have been the find command deleting it since the directory is empty (no hidden files).

[mahesh-panchal]

I can't recreate it with a toy though.

workflow {
    TASK(1).view()
}

process TASK {
    input:
    val thing

    exec:
    out = thing+1

    output:
    val out
}

nextflow run main.nf -resume
nextflow run main.nf -resume # shows cached
nextflow clean -f -before $( nextflow log -q | tail -n 1 ) # Doesn't remove anything

[mahesh-panchal]

Wait, now it recreates after cleaning everything with find.

nextflow clean -f -before $(nextflow log -q | tail -n 1) && find work -type d -empty -delete
nextflow run main.nf -resume
nextflow clean -f -before $( nextflow log -q | tail -n 1 ) # Delete's task

$ nextflow run main.nf -resume
Nextflow 25.04.6 is available - Please consider updating your version to it

 N E X T F L O W   ~  version 25.04.5

Launching `main.nf` [golden_franklin] DSL2 - revision: e1f22fac32

executor >  local (1)
[51/268687] TASK [100%] 1 of 1 ✔
2

$ nextflow clean -f -before $( nextflow log -q | tail -n 1 )
Removed /Users/mahpa906/Sandbox/Nextflow_sandbox/work/51/268687ae95c958a3ce05e7ca57839d

[bentsherman]

Finally had some time to play with your example. It looks like the latest finding is that native tasks aren't resumed if you delete their work directory.

This is currently the expected behavior because the existence of the task directory is one of the pre-conditions to resuming a cached task:

nextflow/modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy

Lines 815 to 817 in 064ef34

	final cached = shouldTryCache && exists && entry.trace.isCompleted() && checkCachedOutput(task.clone(), resumeDir, hash, entry)
	if( cached )
	break

Native tasks are a bit of a corner case here since 99% of the time a cached task will have outputs that need to be collected In the future I'd like to improve the cache by relying on the work directory as little as possible, including cases like this.

But for now , this doesn't seem like a bug?

[mahesh-panchal]

The issue is nextflow clean. It's cleaning the directory when it should stay. (I think. Putting kids to bed so not fully there)

[mahesh-panchal]

Writing a file to the directory is a simple fix, but still it would be good that nextflow clean didn't remove a directory if its empty and was used in the selected run ( well, whatever the flag settings are since I'm normally using -before).

[bentsherman]

But in your toy example you had to explicitly delete empty directories with find ? So doesn't that mean nextflow clean wasn't deleting them?

[mahesh-panchal]

$ nextflow run main.nf -resume
Nextflow 25.04.6 is available - Please consider updating your version to it

N E X T F L O W ~ version 25.04.5

Launching main.nf [golden_franklin] DSL2 - revision: e1f22fac32

executor > local (1)
[51/268687] TASK [100%] 1 of 1 ✔
2

$ nextflow clean -f -before $( nextflow log -q | tail -n 1 )
Removed /Users/mahpa906/Sandbox/Nextflow_sandbox/work/51/268687ae95c958a3ce05e7ca57839d

This was the example showing nextflow clean removing the directory, not find, but it's only happening if find was run to delete the initial directory. So there's something with the update of the workdir database that's not reflecting the new workdir that's written.

[mahesh-panchal]

I guess no one is really going to notice this, because in order not to break resume one needs to either output a file, or find would remove it even if nextflow clean didn't.