rm: cannot remove `/scratch`: permission denied

[tgebo]

Describe the issue
A clear and concise description of what the issue is. Please include the following in your issue report along with any explicit errors observed

• Pipeline release version 5.3.1
• Cluster you are using (SGE/Slurm-Dev/Slurm-Test) Slurmdev
• Node type (F2s (lowmem) / F72s (midmem) / M64s (execute)) F72
• Submission method (interactive/submission script)

python3 /hot/software/package/tool-submit-nf/Python/release/2.5.1/submit_nextflow_pipeline.py \
    --nextflow_script /hot/software/pipeline/metapipeline-DNA/Nextflow/release/5.3.1/main.nf \
    --nextflow_config /hot/user/alfgonzalez/project/project-LAND-SARC/project-SARC-LAND/pipeline/16.Brohl-Gonzalez-SARC-ERSM/metapipeline-DNA/config/WGS_BGSAERSM_metapipeline-DNA.config  \
    --pipeline_run_name WGS_BGSAERSM_run_metapipeline-DNA-5.3.1 \
    --partition_type F2

• Actual submission script (python submission script, "nextflow run ...", etc.)
• Sbatch or qsub command and logs if applicable
• Config files /hot/user/alfgonzalez/project/project-LAND-SARC/project-SARC-LAND/pipeline/16.Brohl-Gonzalez-SARC-ERSM/metapipeline-DNA/config/WGS_BGSAERSM_metapipeline-DNA.config
• Path to the working directory /hot/user/tgebo/Brohl-Gonzalez-SARC-ERSM_metapipeline/WGS/workdir
• Any logs produced by the pipeline /hot/user/tgebo/Brohl-Gonzalez-SARC-ERSM_metapipeline/WGS/workdir/86/b5d3d08dff9f2054d10c08c67778ab/.command.log

Both the working_dir and output_dir are my user directory. I've tried submitting using both the python submission script (command above) as well as this script /hot/user/alfgonzalez/project/project-LAND-SARC/project-SARC-LAND/pipeline/16.Brohl-Gonzalez-SARC-ERSM/metapipeline-DNA/run_script/WGS_launch_metapipeline-DNA-5.3.1_BGSAERSM.sh and see the same error time after time.

[tgebo]

job is still currently running 158093

[yashpatel6]

This seems to be related a bug with the directory being created by Spark and then having issues with removing it; the fix is likely with updating the GATK version used in align-DNA for Spark to include bugfixes since the version we currently use. In the meantime, a workaround can be to use Picard markduplicates instead of Spark at the cost of runtime

[tgebo]

okay, I just relaunched it changing the Spark setting under alignDNA. Thanks for looking into it.

[tgebo]

@yashpatel6 I got another error after changing the Spark setting^.
/hot/user/tgebo/Brohl-Gonzalez-SARC-ERSM_metapipeline/WGS/workdir/6b/d2dd9f269098c398387f51a7aacb36/.command.log

Looks like it just got Killed during MarkDuplicates. Please help.