Metapipeline job silently failed during recalibrate BAM - identify_outputs

[nwiltsie]

Describe the issue

@WuSelina's job silently failed while reporting success:

Submission Script output:
/hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/output/P004_P006_paired_metaDNA.log:

[fa/549b6a] process > create_config_metapipeline_... [100%] 2 of 2 ✔
[0b/b645b4] process > call_metapipeline_DNA (2)      [100%] 2 of 2 ✔
[40/9e021f] process > check_process_status (2)       [100%] 2 of 2 ✔

Completed at: 05-Apr-2024 12:56:33
Duration    : 3d 17h 13m 14s
CPU hours   : 89.2
Succeeded   : 8

Step output:
/hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/workdir/bb/ff891798967bb1bbcebd760873cd7a/.command.log:

[16/a88725] process > recalibrate_BAM:create_YAML... [100%] 1 of 1 ✔
[-        ] process > recalibrate_BAM:run_recalib... -
[03/ff7666] process > recalibrate_BAM:run_recalib... [  0%] 0 of 1
[-        ] process > call_gSNP:create_YAML_call_... -
[-        ] process > call_gSNP:run_call_gSNP        -
[-        ] process > call_sSNV:create_YAML_call_... -
[-        ] process > call_sSNV:run_call_sSNV        -
[-        ] process > call_mtSNV:create_YAML_call... -
[-        ] process > call_mtSNV:run_call_mtSNV      -
[-        ] process > call_gSV:create_YAML_call_gSV  -
[-        ] process > call_gSV:run_call_gSV          -
[-        ] process > call_sSV:create_YAML_call_sSV  -
[-        ] process > call_sSV:run_call_sSV          -
[-        ] process > call_sCNA:create_YAML_call_... -
[-        ] process > call_sCNA:run_call_sCNA        -
ERROR ~ Unexpected error [InvocationTargetException]

 -- Check script '/hot/software/pipeline/metapipeline-DNA/Nextflow/release/6.0.0-rc.2/module/recalibrate_BAM/./identify_outputs.nf' at line: 25 or see '.nextflow.log' file for more details

executor >  local (2)
[16/a88725] process > recalibrate_BAM:create_YAML... [100%] 1 of 1 ✔
[-        ] process > recalibrate_BAM:run_recalib... -
[03/ff7666] process > recalibrate_BAM:run_recalib... [100%] 1 of 1 ✔
[-        ] process > call_gSNP:create_YAML_call_... -
[-        ] process > call_gSNP:run_call_gSNP        -
[-        ] process > call_sSNV:create_YAML_call_... -
[-        ] process > call_sSNV:run_call_sSNV        -
[-        ] process > call_mtSNV:create_YAML_call... -
[-        ] process > call_mtSNV:run_call_mtSNV      -
[-        ] process > call_gSV:create_YAML_call_gSV  -
[-        ] process > call_gSV:run_call_gSV          -
[-        ] process > call_sSV:create_YAML_call_sSV  -
[-        ] process > call_sSV:run_call_sSV          -
[-        ] process > call_sCNA:create_YAML_call_... -
[-        ] process > call_sCNA:run_call_sCNA        -
ERROR ~ Unexpected error [InvocationTargetException]

 -- Check script '/hot/software/pipeline/metapipeline-DNA/Nextflow/release/6.0.0-rc.2/module/recalibrate_BAM/./identify_outputs.nf' at line: 25 or see '.nextflow.log' file for more details

Step .nextflow.log file:
/hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/workdir/bb/ff891798967bb1bbcebd760873cd7a/.nextflow.log:

~> TaskHandler[id: 2; name: recalibrate_BAM:run_recalibrate_BAM_delete_all (1); status: RUNNING; exit: -; error: -; workDir: /scratch/110525/03/ff76662bfcb03f435899f47ba4b4dd]
Apr-05 12:54:59.923 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: recalibrate_BAM:run_recalibrate_BAM_delete_all (1); status: COMPLETED; exit: 0; error: -; workDir: /scratch/110525/03/ff76662bfcb03f435899f47ba4b4dd]
Apr-05 12:55:00.004 [Actor Thread 71] ERROR nextflow.extension.OperatorImpl - @unknown
java.lang.reflect.InvocationTargetException: null
>---at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>---at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
... snip ...
>---at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>---at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.codehaus.groovy.runtime.powerassert.PowerAssertionError: assert file_found.size() == 1
       |          |
       []         0
>---at org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:432)
>---at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:670)
>---at Script_c0b80539.identify_file(Script_c0b80539:20)
>---... 76 common frames omitted
Apr-05 12:55:00.038 [Actor Thread 71] DEBUG nextflow.Session - Session aborted -- Cause: java.lang.reflect.InvocationTargetException

• Pipeline release version: 6.0.0-rc.2
• Cluster you are using: Main cluster
• Node type: F72
• Submission method: submission script
• Actual submission script: /hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/run_metapipeline-DNA_batch2_paired_P004_P006.sh
• Sbatch or qsub command and logs if applicable
• Config files
• Path to the working directory
• Any logs produced by the pipeline

/hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/workdir/bb/ff891798967bb1bbcebd760873cd7a
/hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/batch2_paired/output
/hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/input/batch2_aligned_BAMs_input_P004_P006.csv

[nwiltsie]

So there are really two issues here:

1. The pipeline died for some unknown reason
2. The wrapper job reported success despite the failure

$ sacct -j 110520,110525 --parsable --format=JobID,JobName,State,Start,End | xsv table -d'|'
JobID          JobName                                                                     State      Start                End
110520         P004_P006_paired_metaDNA                                                    COMPLETED  2024-04-01T19:43:05  2024-04-05T12:56:33
110520.batch   batch                                                                       COMPLETED  2024-04-01T19:43:05  2024-04-05T12:56:33
110520.extern  extern                                                                      COMPLETED  2024-04-01T19:43:05  2024-04-05T12:56:33
110525         wgs_call_metapipeline_DNA_STGHKGFH000004_bb_ff891798967bb1bbcebd760873cd7a  FAILED     2024-04-01T23:19:59  2024-04-05T12:55:01
110525.batch   batch                                                                       FAILED     2024-04-01T23:19:59  2024-04-05T12:55:01
110525.extern  extern                                                                      COMPLETED  2024-04-01T23:19:59  2024-04-05T12:55:01

[WuSelina]

I ran metapipeline of the same patient (STGHKGFH000004) in 'paired' mode using only the first 1000 lines in the tumor/normal BAMs (excluding the header). I have set the pipeilne_work_dir = leading_work_dir instead of /scratch to save logs/output that might otherwise be lost for this run. Perhaps we can see some errors logs/outputs by setting it up this way while keeping the inputs small.
Update: It ran successfully.

Here is the path to the metapipeline run: /hot/project/disease/KidneyTumor/GHLR-000108-SubclonalArchitecture/metapipeline-DNA/debug_recalibrate-BAM