Description
Hi Team
Since upgrading to version 3.5.1 of the sarek pipeline, I have been encountering a series of new errors when running on my university's PBS HPC cluster with Singularity. Despite extensive troubleshooting and following the recommendation in different threads/channels , I have not been able to resolve these issues.
System and Pipeline Setup
Pipeline version: (e.g. sarek 3.5.1)
Nextflow version: (e.g. 24.04.2)
Cluster type: PBS
Script that I am trying to run
`#!/bin/bash
#PBS -l select=1:ncpus=64:mem=256g
#PBS -l walltime=1:00:00
#PBS -P 11003581
#PBS -N new-sarek-mash-TO
#PBS -j oe
cd $PBS_O_WORKDIR
module load java/17.0.6-jdk
module load singularity
export NXF_SINGULARITY_CACHEDIR=/home/project/11003581/Tools/singularity-cache/
export SINGULARITY_CACHEDIR=/home/project/11003581/Tools/singularity-cache/
/home/project/Tools/nextflow run nf-core/sarek -r 3.5.1
-profile singularity
--input /path/to/csv/samplesheet.csv
--outdir /path/to/outdir/
--genome GATK.GRCh38
--tools mutect2
--joint_mutect2`
Error that I am facing
ERROR ~ Error executing process > 'NFCORE_SAREK:SAREK:FASTP (WHT463-6)'
Caused by:
Missing output file(s) *.json expected by process NFCORE_SAREK:SAREK:FASTP (WHT463-6)
Command executed:
[ ! -f WHT463-6_1.fastq.gz ] && ln -sf WHT463_R1.fastq.gz WHT463-6_1.fastq.gz
[ ! -f WHT463-6_2.fastq.gz ] && ln -sf WHT463_R2.fastq.gz WHT463-6_2.fastq.gz
fastp
--in1 WHT463-6_1.fastq.gz
--in2 WHT463-6_2.fastq.gz
--out1 WHT463-6_1.fastp.fastq.gz
--out2 WHT463-6_2.fastp.fastq.gz
--json WHT463-6.fastp.json
--html WHT463-6.fastp.html
--thread 12
--detect_adapter_for_pe
--disable_adapter_trimming --split_by_lines 200000000 --length_required 15
2> >(tee WHT463-6.fastp.log >&2)
cat <<-END_VERSIONS > versions.yml
"NFCORE_SAREK:SAREK:FASTP":
fastp: $(fastp --version 2>&1 | sed -e "s/fastp //g")
END_VERSIONS
Command exit status:
0
Command output:
(empty)
Work dir:
/scratch/users/nus/ash.ps/NCCS-MASH/analysis/tumor/analysis/work/d6/0c0bcbee9ce1274cdffe653dfde842
Container:
/home/project/11003581/Tools/singularity-cache/depot.galaxyproject.org-singularity-fastp-0.23.4--h5f740d0_0.img
Debugging
When I go into the process work dir and run the command.sh manually it runs and is able to generate outputs, However, it throws error when I m running it through nextflow.
I have spent few days trying to understand this error but I am unable to figure out what this is.
IMPORTANT NOTE:
If this was an HPC/ permissions error I wouldn't be able to run nf-core\sarek pipeline previously. I have been using the pipeline for almost more than 6 months. I recently updated it to 3.5.1 and I have been facing the error ever since.
Any help would be greatly appreciated.
I am attaching few necessary files for debugging.
Thanks in advance.