Follow up to PR #418 #430

SHuang-Broad · 2023-10-13T01:01:51Z

Following up #418 with more optimizations, and two new pipelines for

ingesting single demultiplexed PacBio Hifi (u)BAM (aka a readgroup uBAM)

tedsharpe

Just a few suggestions. Might want to omit "/cromwell_root" wherever it appears, as I suggested in the last PR.

tedsharpe · 2023-10-23T14:46:29Z

.dockstore.yml

@@ -1,17 +1,11 @@
 version: 1.2


I haven't reviewed this. Going to assume it's OK.

tedsharpe · 2023-10-23T14:47:24Z

docker/lr-gcloud-samtools/Dockerfile

@@ -0,0 +1,92 @@
+# incorporate samtools and bcftools, plus some additional but most common genomics tools


Not reviewed here -- appears to be the same as the version in sh_ingest_high_pass_inputs, which I've already reviewed.

tedsharpe · 2023-10-23T14:48:01Z

docker/lr-gcloud-samtools/Makefile

@@ -0,0 +1,12 @@
+VERSION = 0.1.2


Not reviewed here -- appears to be the same as the version in sh_ingest_high_pass_inputs, which I've already reviewed.

tedsharpe · 2023-10-23T14:48:06Z

docker/lr-gcloud-samtools/re-auth.sh

@@ -0,0 +1,6 @@
+#!/bin/bash


Not reviewed here -- appears to be the same as the version in sh_ingest_high_pass_inputs, which I've already reviewed.

tedsharpe · 2023-10-23T20:01:52Z

wdl/tasks/QC/AlignedMetrics.wdl

+        time gcloud storage cp ~{bam} ~{local_bam}
+
+        mv ~{bai} "~{local_bam}.bai"
+        mosdepth -x -n -Q1 ~{prefix} ~{local_bam} || echo "mosdepth failed somehow"


Is it really your intention to swallow mosdepth failures with just a message to the log?

tedsharpe · 2023-10-24T15:20:38Z

wdl/tasks/Utility/BAMutils.wdl

+        samtools view --no-PG -H ~{local_bam} > header.txt
+        grep -v "^@SQ" header.txt
+
+        # fix SM in the RG lines
+        grep "^@RG" header.txt > rg_lines.txt
+        if ! grep -qF "SM:" rg_lines.txt; then
+            sed -i "s/$/SM:tbd/" rg_lines.txt
+        fi
+        awk -v sm="~{sample_name}" -F '\t' 'BEGIN {OFS="\t"} { for (i=1; i<=NF; ++i) { if ($i ~ "SM:") $i="SM:"sm } print}' \
+            rg_lines.txt \
+            > fixed_rg_lines.txt
+        cat fixed_rg_lines.txt
+
+        # paste things back
+        grep -v "^@RG" header.txt > otherlines.txt
+        cat otherlines.txt fixed_rg_lines.txt > fixed_header.txt


All this could be replaced with a 1-liner:

samtools view --no-PG -H "~{local_bam}" | \ sed -E "/^@RG/s/( SM:[^ ]*|$)/ SM:~{sample_name}/" > fixed_header.txt #these are tabs --- ^ ^

tedsharpe · 2023-10-24T18:30:44Z

wdl/tasks/Utility/BAMutils.wdl

@@ -43,3 +52,64 @@ task GetReadGroupInfo {
        docker: "us.gcr.io/broad-dsp-lrma/lr-basic:0.1.1"
    }
 }
+
+task ResetSamplename {


This seems like an awfully dangerous thing to do (arbitrary replacement of sample name). The task doesn't appear to be in use, currently. I hope we'll use it judiciously.

tedsharpe · 2023-10-24T18:33:33Z

wdl/pipelines/PacBio/Utility/ProcessOnInstrumentDemuxedChunk.wdl

+    ###################################################################################
+    # generate PBI
+    call PBUtils.PBIndex as Index {input: bam = uBAM}
+    call BU.GetReadGroupInfo as RG {input: uBAM = uBAM, keys = ['SM', 'LB', 'PU']}


doesn't look like we actually need the SM value

* update Hifiasm to version 0.19.5 * update how Hifiasm outputs are compressed (bgz replacing gz), also * monitor hifiasm resources usage

* update docker used in PBSV tasks to the version coming with official SMRTLink releases (2.9.0) * change how the 2-step PBSV process is done (following the recommended way now)

* to version 2.0.7 * using TRF bed * conditionally phase sv (requires phased bam) * generates its own vcf.gz and tbi

Overhaul how small variants are called in the WG pipelines * default to use DV to call small variants, Clair3 analysis needs to be requested explicitly * retire the Pepper toolchain completely from the CCS pipeline, using DV directly * for R10.4+ ONT data, also use DV directly * older ONT data would still use the PEPPER-DV-Margin pipeline * offers GPU version (though based on, it's not worth it yet) * update how bam haplotagging is done Cleanup structural variants calling * experiment with SNF2 phasing SV calls (implicitly depends on small variants calling now) * tune PBSV calling - discover now supports --hifi - output vcf.gz and tbi - less verbose logging by default Misc.: * optimizations to BAM merging and metrics workflow * updates coverage collection step * new R script to visualize log from vm_monitoring_script.sh

* organize dockstore.yml file a bit * make WDL validation shell script more usable * update pbmm2 and pbindex to versions in SMRTLink * update GeneralUtils.wdl - two bash-like new tasks [CoerceMapToArrayOfPairs, CoerceArrayOfPairsToMap] - cleanup task CollapseArrayOfStrings * update resource allocations to tasks - NanoplotFromBam (also changes docker) - MosDepthWGS

* incorporates gcloud cli (not just gsutil) * integrate libdeflate for more speedups

…sorted ONT bam

incorporate new tasks and optimize them * [CountMethylCallReads, GatherReadsWithoutMethylCalls] from sh_beans * [GetPileup, BamToRelevantPileup] from sh_more_atomic_qc * [GetReadGroupLines, GetSortOrder, SplitNameSortedUbam] from sh_ont_fc * [SamtoolsFlagStats, ParseFlagStatsJson] from sh_trvial_stats * [FilterBamByLen, InferSampleName] from sh_seqkit * [CountAlignmentRecords, StreamingBamErrored, CountAlignmentRecordsByFlag] from sh_maha_aln_metrics * [ResetSamplename] from sh_ingest_singlerg * [MergeBamsWithSamtools] from sh_ont_fc.Utils.wdl * [BamToFastq] from sh_more_bam_qcs and optimize it with sh_ingest_singlerg.Utils.wdl delete * GetSortOrder as that's now implemented in GatherBamMetadata * Drop2304Alignments as that's no longer used update dockers to the latest

CHERRY-PICK FROM VARIOUS QC/METRICS BRANCHES: * collect information about ML/MM tags in a long-read BAM (sh_beans) * a heuristic way to find peaks in a distribution (using dyst) (sh_dyst_peaker) * filter reads by length in a BAM * collect some read quality stats from (length-filtered) FASTQ/BAM (sh_seq_kit) * VerifyBamID2 (for contamination estimation) * naive sex-concordance check (sh_more_atomic_qc) * check fingerprint of a single BAM file (sh_sample_fp) * collect SAM flag stats (sh_trivial_stats)

* make BeanCounter finalization optional (wdl/pipelines/TechAgnostic/Utility/CountTheBeans.wdl) * custom struct for sub-workflow config using a JSON (wdl/pipelines/TechAgnostic/Utility/LongReadsContaminationEstimation.wdl) * make fingerprint checking subworkflow control size filtering (wdl/tasks/QC/FPCheckAoU.wdl) (wdl/pipelines/TechAgnostic/Utility/VerifyBamFingerprint.wdl) * fix a warning by IDE/miniwdl complaining WDL stdlib function length only applies to Array (wdl/tasks/Utility/BAMutils.wdl) * various updates to Finalize (wdl/tasks/Utility/Finalize.wdl) New tasks in (wdl/tasks/Utility/GeneralUtils.wdl) to * correctly convert Map to TSV * concatenate files

* AlignAndCheckFingerprintCCS.wdl * CollectPacBioAlignedMetrics.wdl * CollectSMRTCellUnalignedMetrics.wdl

(CHRRY-PICK & follow up to PR 406)

* SampleLevelAlignedMetrics.wdl * PBCLRWholeGenome.wdl

* new struct in AlignedBamQCandMetrics.wdl to facilicate as-sub-workflow calling * change parameters name for fingerprint workflows

* make saving of reads without methylation SAM tags optional * better parameter naming

(affects contamination estimation)

handle (ref-indepedent, alignment) a single piece of on-instrument demultiplexed CCS bam

SHuang-Broad requested a review from tedsharpe October 13, 2023 01:01

tedsharpe approved these changes Oct 24, 2023

View reviewed changes

bshifaw assigned SHuang-Broad Oct 25, 2023

SHuang-Broad added 11 commits November 8, 2023 12:08

UPDATES TO THE Hifiasm pipeline:

e09dc65

* update Hifiasm to version 0.19.5 * update how Hifiasm outputs are compressed (bgz replacing gz), also * monitor hifiasm resources usage

For both CCS/ONT, update PBSV

d3afc4b

* update docker used in PBSV tasks to the version coming with official SMRTLink releases (2.9.0) * change how the 2-step PBSV process is done (following the recommended way now)

For both CCS/ONT, update Sniffles-2

3a2ac5c

* to version 2.0.7 * using TRF bed * conditionally phase sv (requires phased bam) * generates its own vcf.gz and tbi

CRITICAL: updating pbindex to that from the SMRTLink 12 release

29aa964

New docker that's intended to replace lr-basic:

1ff0912

* incorporates gcloud cli (not just gsutil) * integrate libdeflate for more speedups

New docker that offers a new mode remove duplicates from a queryname-…

4224ab4

…sorted ONT bam

New docker that updates the samtools in GATK to the latest (1.18)

a13a541

update some legacy code due to new interfaces in updated BAMutils.wdl

b16c619

SHuang-Broad force-pushed the sh_ingest_singlerg branch 2 times, most recently from c905a5a to a9ae6c3 Compare December 4, 2023 23:57

SHuang-Broad added 6 commits December 19, 2023 17:07

New workflow to save files (avoid multiple calls to Finalize)

cbfed4e

New workflow to unify QC and metrics collection on aligned BAM

8039c88

Update grouping of pipelines in .dockstore.yml

b16155f

update WDL validation bash script

5bf0b2f

SHuang-Broad force-pushed the sh_ingest_singlerg branch from a9ae6c3 to 57dcfaf Compare December 27, 2023 19:18

DEPRECATION

0d61935

* AlignAndCheckFingerprintCCS.wdl * CollectPacBioAlignedMetrics.wdl * CollectSMRTCellUnalignedMetrics.wdl

SHuang-Broad force-pushed the sh_ingest_singlerg branch from 57dcfaf to be67190 Compare December 28, 2023 18:51

SHuang-Broad added 3 commits January 2, 2024 10:38

Fix bug in deduplicating aligned ONT BAM

989ead1

(CHRRY-PICK & follow up to PR 406)

DEPRECATION:

b8e67ff

* SampleLevelAlignedMetrics.wdl * PBCLRWholeGenome.wdl

Update utility code:

e5a79c2

* new struct in AlignedBamQCandMetrics.wdl to facilicate as-sub-workflow calling * change parameters name for fingerprint workflows

SHuang-Broad force-pushed the sh_ingest_singlerg branch from be67190 to 295be09 Compare January 3, 2024 03:28

a few tweaks to to AlignedBamQCandMetrics:

b5dc978

* make saving of reads without methylation SAM tags optional * better parameter naming

SHuang-Broad force-pushed the sh_ingest_singlerg branch from 295be09 to 74388b3 Compare January 15, 2024 18:24

correct two mistakes: 1. a typo; 2. should append not replace

1abcc7a

SHuang-Broad force-pushed the sh_ingest_singlerg branch 3 times, most recently from ea29e7c to 29c570b Compare January 22, 2024 02:14

make read-length util tasks pre-emptible

da5bfb0

SHuang-Broad force-pushed the sh_ingest_singlerg branch from 29c570b to 50d7235 Compare January 22, 2024 02:55

SHuang-Broad added 4 commits January 23, 2024 00:20

make fingerprint reads extraction more efficient/resilient

7975d6d

fix a stupid logical typo error (in saving files)

5620d74

option to force run fingerprintcheck on small bams

e910bac

make pbindex task more frugal

04b82f8

SHuang-Broad force-pushed the sh_ingest_singlerg branch from 34d754d to c4b670f Compare February 2, 2024 17:19

SHuang-Broad added 4 commits March 1, 2024 17:09

Safer and more efficient way to do targetted pileup conversion

14be4e2

(affects contamination estimation)

Update Pacbio alignment task

136091a

New subworkflow to align PacBio uBAM

d95be34

Two new workflows:

1e26de7

handle (ref-indepedent, alignment) a single piece of on-instrument demultiplexed CCS bam

SHuang-Broad force-pushed the sh_ingest_singlerg branch from c4b670f to 1e26de7 Compare March 1, 2024 22:19

SHuang-Broad added 3 commits March 7, 2024 10:24

bug fix for bam2fastq

44b7330

optimize alignment further

526765e

test reducing resources for pbmm2 further

3dd95fa

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Follow up to PR #418 #430

Follow up to PR #418 #430

SHuang-Broad commented Oct 13, 2023

tedsharpe left a comment

tedsharpe Oct 23, 2023

tedsharpe Oct 23, 2023

tedsharpe Oct 23, 2023

tedsharpe Oct 23, 2023

tedsharpe Oct 23, 2023

tedsharpe Oct 24, 2023

tedsharpe Oct 24, 2023

tedsharpe Oct 24, 2023

		@@ -0,0 +1,92 @@
		# incorporate samtools and bcftools, plus some additional but most common genomics tools

Follow up to PR #418 #430

Are you sure you want to change the base?

Follow up to PR #418 #430

Conversation

SHuang-Broad commented Oct 13, 2023

tedsharpe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment