Error while loading shared library for BWA in Sarek pipeline with Singularity

[Simonassaf]

Description of the bug

Running the Sarek pipeline while using Singularity results in the following error: bwa: error while loading shared lib. This issue occurs occasionally when running bwa to get the version information, producing the message: bwa: failed to load shared library. This error string is included instead of the version in the intermediate version.yml file generated by the module, leading to a corrupt YAML file (missing quotes around the string due to the presence of :). Consequently, the Nextflow job, that combines all these version.yml files into one, fails. Note that this error is occurring after 40 subsamples.

Command used and terminal output

./nextflow run nf-core/sarek -r 3.4.2 -profile singularity --input ./input-sarek.csv --outdir /home/sarek-nf-results --genome null --igenomes_ignore --fasta ./GCF_002263795.3_ARS-UCD2.0_genomic.fna  --skip_tools baserecalibrator --tools deepvariant

[b9/3b06bd] process > NFCORE_SAREK:SAREK:FASTQC (zr8425-99-lane_99)                                                              [ 98%] 259 of 263, cached: 259
[d2/7b9351] process > NFCORE_SAREK:SAREK:FASTP (zr8425-595-lane_595)                                                             [ 56%] 149 of 263, cached: 149
[1f/2990a2] process > NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:BWAMEM1_MEM (zr8425-105-lane_105)              [  2%] 40 of 1787
[-        ] process > NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:BWAMEM2_MEM                                    -
[-        ] process > NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:DRAGMAP_ALIGN                                  -
[-        ] process > NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:SENTIEON_BWAMEM                                -
[-        ] process > NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES                                                 -
[-        ] process > NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:CRAM_QC_MOSDEPTH_SAMTOOLS:SAMTOOLS_STATS                             -
[-        ] process > NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:CRAM_QC_MOSDEPTH_SAMTOOLS:MOSDEPTH                                   -
[-        ] process > NFCORE_SAREK:SAREK:CRAM_TO_BAM                                                                             -
[-        ] process > NFCORE_SAREK:SAREK:CRAM_SAMPLEQC:BAM_NGSCHECKMATE:BCFTOOLS_MPILEUP                                         -
Plus 9 more processes waiting for tasks…
ERROR ~ mapping values are not allowed here
 in 'reader', line 2, column 13:
        bwa: bwa: error while loading shared lib ...
                ^

Relevant files

No response

System information

No response

[asp8200]

Discussion on nf-core/sarek Slack about a seemingly very similar issue:

https://nfcore.slack.com/archives/CGFUX04HZ/p1718984599684549

[FriederikeHanssen]

Hey! Something seems to be wrong in a weird way.I see that at least 40 tasks finished successfully. So it's not generally wrong, but borks in some cases. Are you able to retrieve on which compute nodes the jobs ran? is there a particular node that has issues?

[fbdtemme]

I have been working with @Simonassaf on investigating this.
There were 2 runs done on the head node of an SGE cluster. Both of them failed exactly after 40 samples.

As a workaround suggested by @asp8200 on slack I have edited the pipeline and just removed the call to extract the version from bwa and hardcoded it instead.
Weirdly enough that seems to be working. It is really strange that only calls to get the version fail and not the actual command itself.

We will do some test runs soon both running everything on a single compute node and using the sge executor to see if it this failure pops as well.

[asp8200]

I have been working with @Simonassaf on investigating this. There were 2 runs done on the head node of an SGE cluster. Both of them failed exactly after 40 samples.

As a workaround suggested by @asp8200 on slack I have edited the pipeline and just removed the call to extract the version from bwa and hardcoded it instead. Weirdly enough that seems to be working. It is really strange that only calls to get the version fail and not the actual command itself.

We will do some test runs soon both running everything on a single compute node and using the sge executor to see if it this failure pops as well.

It is indeed very weird. I don't think we ever got to the bottom of the problem, but I'll ask my colleagues.

[fbdtemme]

When running the pipeline over multiple nodes using the sge executor this issue does not occur.
So maybe it is triggered by running with singularity on a single node?

[asp8200]

I got that problem again. This time with Sarek/3.4.4 (test_full_germline with singularity):

ERROR ~ mapping values are not allowed here
 in 'reader', line 2, column 13:
        bwa: bwa: error while loading shared lib ...
                ^
 -- Check script '/ngc/tools/ngctools/sarek/3.4.4/assets/nf-core/sarek/./workflows/sarek/../../subworkflows/local/utils_nfcore_sarek_pipeline/../../nf-core/utils_nfcore_pipeline/main.nf' at line: 106 or see '.nextflow.log' file for more details

It used to be CUSTOM_DUMPSOFTWAREVERSIONS would fail due to garbage (error messages) in some versions.yml-files, and now I believe something similar might be happening with the functionality that replaced CUSTOM_DUMPSOFTWAREVERSIONS.

Most of the versions.yml-files for the BWA-jobs look like this:

"NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:BWAMEM1_MEM":
    bwa: 0.7.18-r1243-dirty
    samtools: 1.20

which seems fine to me, but I found one that look like this:

"NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:BWAMEM1_MEM":
    bwa: bwa: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory
    samtools: samtools: error while loading shared libraries: libhts.so.3: cannot open shared object file: No such file or directory

And that versions.yml-file is from a BWA-job which Nextflow claims finished successfully 😬

@maxulysse : I thought the new "versions"-functionality was immune to those kinds of malformed versions.yml-files?

[asp8200]

@pontus and @adamrtalbot : I'm thinking this issue might also be of interest to you.

I'm slightly worried about the fact that the calls to bwa and samtools fail in the "version section" of the bwa-module while the corresponding bwa-job still completes successfully (at least according to Nextflow). The pipeline only fails because the versions.yml-file is malformed - does that mean that bwa and samtools could have somehow failed "silently" in other ways (not affecting the "versions section" but perhaps the alignment itself 🫣 ) in other runs?

[pontus]

Can anyone trigger this at will? In that case it would be interesting to see. I recall some rather weird instances of this happening but generally, my first guess as to why would be some cluster where singularity is unable to mount the image directly but rather unpacks it onto some file system that behaves oddly.

[asp8200]

Can anyone trigger this at will? In that case it would be interesting to see. I recall some rather weird instances of this happening but generally, my first guess as to why would be some cluster where singularity is unable to mount the image directly but rather unpacks it onto some file system that behaves oddly.

The error pops up once in a while on our HPC. Today it wasn't BWA but VEP that triggered the error:

ERROR ~ mapping values are not allowed here
in 'reader', line 2, column 21:
       ensemblvep: perl:
                       ^

-- Check script '/ngc/tools/ngctools/sarek/3.4.4/assets/nf-core/sarek/./workflows/sarek/../../subworkflows/local/utils_nfcore_sarek_pipeline/../../nf-core/utils_nfcore_pipeline/main.nf' at line: 106 or see '.nextflow.log' file for more details 

The versions.yml for VEP is malformed and looks like this:

"NFCORE_SAREK:SAREK:VCF_ANNOTATE_ALL:VCF_ANNOTATE_ENSEMBLVEP:ENSEMBLVEP_VEP":
    ensemblvep: perl:

I'm running test_full_germline on sarek/3.4.4, jdk/21.0.1, NF 24.04.3, and singularity/4.0.2.

We thought it might have something to do with too many jobs being submitted at the same time to pbs/torque, but now it occurred with queueSize = 5. This is the executor-config:

executor {
    name = 'pbs'
    queueSize = 5
    pollInterval = '30 sec'
    submitRateLimit = '3 sec'
    exitReadTimeout = '600 sec'
}

[asp8200]

The same job can produce a valid versions.yml on qsub .command.run 🤔

[asp8200]

@pontus : The problem with error messages getting written in the versions.yml-files goes away if I (1) hardcode the version-numbers in the versions.yml-generating code in the modules or (2) move the versions.yml-generating code in the very top of the script-section of the modules.

I think this whole issue is due to instability on our HPC, but it would be nice if Sarek was somehow setup to handle such instability in a better way. If a malformed versions.yml is written, then the run of the pipeline does eventually fail, but it is the Nextflow-job that is crashing with a msg like

ERROR ~ mapping values are not allowed here
 in 'reader', line 2, column 13:
        bwa: bwa: error while loading shared lib ...

(I think the module-job itself should have crashed.)

Restarting the pipeline with -resume doesn't help; the pipeline will fail with the same message, because NF thinks the module was executed correctly and so it doesn't re-run the module-job.

The instability on our HPC also results in a lot of errors like the following which is from the execution of the nf-modules in Sarek:

samtools: error while loading shared libraries: libhts.so.3: cannot open shared object file: No such file or directory

Those kinds of errors cause the execution of the module-job to fail, and so the module-job gets "retries". I'd like to see the same behaviour from NF/Sarek if the error occurs during the execution of the versions.yml-generating code-snippet of the modules.

[pontus]

It's likely is the case that the versions handling code doesn't do the right thing, exactly visavi-errors.

On the other hand, there's only so much that can be done to manage broken environments and getting it fixed for all modules and making everything behave seems non-trivial.

Instead, my recommendation would be to as far as possible make you suffer less from when your cluster is unstable. I've already mention that this looks to me like a setup where the images are unpacked rather than mounted, is that the case? If so, I'd recommend fixing that (apptainer has an installer for a non-privileged version that will do the right thing for most current systems).

[asp8200]

I've already mention that this looks to me like a setup where the images are unpacked rather than mounted, is that the case? If so, I'd recommend fixing that (apptainer has an installer for a non-privileged version that will do the right thing for most current systems).

How do I tell if "images are unpacked rather than mounted"?

From our custom.config:

singularity {
    enabled = true
    autoMounts = true
}

I am guess that that means the images are mounted, no?