Crash upon no variants found

[SergeWielhouwer]

Hi,

Thank you for developing ClairS-TO.

I’m currently encountering an issue where the pipeline crashes without a clear error message. It appears to happen when no variants are reported during intermediate steps. I'm running ClairS-TO on some rAAV samples, while I realize the tool isn’t specifically designed for this use case, I’ve successfully applied it to other small genomes before. Clair3 also ran well on this dataset, but since I suspect low-frequency variants in a small subpopulation (i.e., not at 50% or 100% VAF), I wanted to try somatic variant calling.

To adapt the pipeline, I disabled phasing and non-somatic tagging from the panel of normals, and included all contigs.

Here’s the command used within our Snakemake pipeline:

The command I use to run in our Snakemake pipeline is:
mkdir -p TEMP/106306-001-001/clairs_to_tmp TEMP/106306-001-001/clairs_to_home variants/106306-001-001/clairs_to && singularity exec -B /mnt --home TEMP/106306-001-001/clairs_to_home:/home --env TMPDIR=$PWD/TEMP/106306-001-001/clairs_to_tmp /mnt/shared/tools/ClairS-TO-0.3.0/container/clairs-to-0.3.0.sif /opt/bin/run_clairs_to --tumor_bam_fn alignments/106306-001-001.bam --ref_fn /mnt/flashblade01/scratch/106306/references/samples001_003.fasta --threads 64 --platform "ont_r10_dorado_sup_5khz_ssrs" --include_all_ctgs --disable_nonsomatic_tagging --disable_intermediate_phasing --output_dir variants/106306-001-001/clairs_to 2>>logs/clair_somatic_tumor_only_snp_indel.106306-001-001.log && rm -rf "TEMP/106306-001-001/clairs_to_tmp" "TEMP/106306-001-001/clairs_to_home

The error I get is:

Traceback (most recent call last):
File "/opt/bin/clairs_to.py", line 110, in
main()
File "/opt/bin/clairs_to.py", line 104, in main
submodule.main()
File "/opt/bin/src/postprocess_vcf.py", line 259, in main
merge_vcf(args)
File "/opt/bin/src/postprocess_vcf.py", line 111, in merge_vcf
pileup_vcf_reader.read_vcf()
File "/opt/bin/shared/vcf.py", line 242, in read_vcf
if columns[0][0] == "#":
IndexError: list index out of range

run_clairs_to.log

It seems the script attempts to process an empty or malformed VCF file. Possibly this happens when no variants are detected at an intermediate step.

Let me know if you need more information. I am on version 0.3.0.

Best regards,
Serge

[tamuanand]

Hi @JasonCLEI

I am getting a similar error as above. I am using version 0.4.0 with singularity.

Thanks in advance for your time and help.

singularity exec docker.io-hkubal-clairs-to-v0.4.0.img run_clairs_to --tumor_bam_fn ./dorado_basecalls.sorted.bam --ref_fn ./ref.fasta --threads 1 --platform ont_r10_dorado_sup_5khz_ssrs --output_dir SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO --include_all_ctgs --snv_min_af 0.01 --indel_min_af 0.01 --conda_prefix /opt/micromamba/envs/clairs-to --disable_verdict --disable_nonsomatic_tagging 2>&1 | tee -a SSRS_STDERR_DISABLE_NON_SOMATIC_TAGGING.txt

[INFO] Create folder SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO

[COMMAND] /opt/bin/run_clairs_to --tumor_bam_fn ./dorado_basecalls.sorted.bam --ref_fn ./ref.fasta --threads 1 --platform ont_r10_dorado_sup_5khz_ssrs --output_dir <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO --include_all_ctgs --snv_min_af 0.01 --indel_min_af 0.01 --conda_prefix /opt/micromamba/envs/clairs-to --disable_verdict --disable_nonsomatic_tagging

[INFO] Create folder <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs
[INFO] Create folder <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp
[INFO] Create folder <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/split_beds
[INFO] Create folder <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/split_indel_beds
[INFO] Create folder <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/candidates
[INFO] Create folder <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/pileup_tensor_can_affirmative
[INFO] Create folder <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/pileup_tensor_can_negational
[INFO] Create folder <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output
[INFO] Create folder <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/phasing_log
[INFO] Create folder <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/phased_vcf_output
[INFO] Create folder <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/phased_bam_output
[INFO] --include_all_ctgs enabled
[INFO] Call variants in contigs: sample_anonymized
[INFO] Number of chunks for each contig: 1
[WARNING] The length of the longest contig 13687 is more than five times smaller than the default chunk size 5000000, consider setting a smaller --chunk_size= instead for better parallelism.

[INFO] CALLER VERSION: 0.4.0
[INFO] TUMOR BAM FILE PATH: <path_to>/dorado_basecalls.sorted.bam
[INFO] REFERENCE FILE PATH: <path_to>/ref.fasta
[INFO] PLATFORM: ont_r10_dorado_sup_5khz_ssrs
[INFO] THREADS: 1
[INFO] OUTPUT FOLDER: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO
[INFO] SNV OUTPUT VCF PATH: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/snv.vcf.gz
[INFO] INDEL OUTPUT VCF PATH: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/indel.vcf.gz
[INFO] SNV MINIMUM AF: 0.01
[INFO] INDEL MINIMUM AF: 0.01
[INFO] SNV PILEUP AFFIRMATIVE MODEL PATH: /opt/micromamba/envs/clairs-to/bin/clairs-to_models/ont_r10_dorado_sup_5khz_ssrs/pileup_affirmative.pkl
[INFO] SNV PILEUP NEGATIONAL MODEL PATH: /opt/micromamba/envs/clairs-to/bin/clairs-to_models/ont_r10_dorado_sup_5khz_ssrs/pileup_negational.pkl
[INFO] INDEL PILEUP AFFIRMATIVE MODEL PATH: /opt/micromamba/envs/clairs-to/bin/clairs-to_models/ont_r10_dorado_sup_5khz_ssrs/indel/pileup_affirmative.pkl
[INFO] INDEL PILEUP NEGATIONAL MODEL PATH: /opt/micromamba/envs/clairs-to/bin/clairs-to_models/ont_r10_dorado_sup_5khz_ssrs/indel/pileup_negational.pkl
[INFO] BED FILE PATH: None
[INFO] SPECIFIED REGIONS FOR CALLING: None
[INFO] CONTIGS FOR CALLING: None
[INFO] ENABLE INCLUDING ALL CTGS FOR CALLING: True
[INFO] GENOTYPING MODE VCF FILE PATH: None
[INFO] HYBRID MODE VCF FILE PATH: None
[INFO] PANEL OF NORMALS: /opt/micromamba/envs/clairs-to/bin/clairs-to_databases/gnomad.r2.1.af-ge-0.001.sites.vcf.gz,/opt/micromamba/envs/clairs-to/bin/clairs-to_databases/dbsnp.b138.non-somatic.sites.vcf.gz,/opt/micromamba/envs/clairs-to/bin/clairs-to_databases/1000g-pon.sites.vcf.gz,/opt/micromamba/envs/clairs-to/bin/clairs-to_databases/CoLoRSdb.GRCh38.v1.1.0.deepvariant.glnexus.af-ge-0.001.vcf.gz
[INFO] PANEL OF NORMALS REQUIRE ALLELE MATCHING: True,True,False,False
[INFO] CHUNK SIZE: 5000000
[INFO] CONDA BINARY PREFIX: /opt/micromamba/envs/clairs-to
[INFO] SAMTOOLS BINARY PATH: samtools
[INFO] PYTHON BINARY PATH: python3
[INFO] PYPY BINARY PATH: pypy3
[INFO] PARALLEL BINARY PATH: parallel
[INFO] LONGPHASE BINARY PATH: /opt/micromamba/envs/clairs-to/bin/longphase
[INFO] WHATSHAP BINARY PATH: /opt/micromamba/envs/clairs-to/bin/whatshap
[INFO] ENABLE DRY RUN: False
[INFO] ENABLE REMOVING INTERMEDIATE FILES: False
[INFO] DISABLE INTERMEDIATE PHASING: False
[INFO] DISABLE INDEL CALLING: False
[INFO] ENABLE PRINTING REFERENCE CALLS: False
[INFO] ENABLE APPLYING REALIGNMENT: True
[INFO] ENABLE APPLYING POSTFILTERING: True
[INFO] ENABLE APPLYING HAPLOTYPE FILTERING: True
[INFO] DISABLE APPLYING VERDICT: True
[INFO] DISABLE APPLYING NON-SOMATIC TAGGING: True
[INFO] DISABLE PRINTING NON-SOMATIC CALLS: False

[INFO] STEP 1: Extract Variant Candidates from Tumor BAM
[INFO] RUN THE FOLLOWING COMMAND:
( parallel --joblog <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/parallel_1_extract_tumor_candidates.log -C " " -j 1 pypy3 /opt/bin/clairs_to.py extract_candidates_calling --tumor_bam_fn <path_to>/dorado_basecalls.sorted.bam --ref_fn <path_to>/ref.fasta --samtools samtools --snv_min_af 0.01 --indel_min_af 0.01 --chunk_id {2}  --chunk_num {3}  --ctg_name {1}  --platform ont --min_coverage 4 --min_bq 20 --bed_fn_source None --bed_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/split_beds/{1} --call_indels_only_in_these_regions <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/split_indel_beds/{1} --select_indel_candidates True --candidates_folder <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/candidates --output_depth True  --genotyping_mode_vcf_fn None --hybrid_mode_vcf_fn None :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/CHUNK_LIST ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/1_EC.log && pypy3 /opt/bin/clairs_to.py concat_files --input_dir <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/candidates --input_prefix SNV_CANDIDATES_FILE_ --output_fn SNV_CANDIDATES_FILES

[INFO] sample_anonymized chunk 0/1: Total SNV candidates found: 549, total Indel candidates found: 509
[faidx] Truncated sequence: sample_anonymized:1-14687

[INFO] STEP 2: SNV Pileup Model Calling
[INFO] Create Tensors for SNV Pileup Affirmative Model
[INFO] RUN THE FOLLOWING COMMAND:
( parallel --joblog <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/parallel_2-1_create_tensor_affirmative_snv.log -j 1 pypy3 /opt/bin/clairs_to.py create_tensor_pileup_calling --tumor_bam_fn <path_to>/dorado_basecalls.sorted.bam --ref_fn <path_to>/ref.fasta --ctg_name {1/.} --min_bq 20 --samtools samtools --candidates_bed_regions {1} --tensor_can_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/pileup_tensor_can_affirmative/{1/}  --platform ont :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/candidates/SNV_CANDIDATES_FILES ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/2-1_CPT_Affirmative_SNV.log

[INFO] sample_anonymized  Tensors generated: 549
[faidx] Truncated sequence: sample_anonymized:1432-14418

[INFO] Create Tensors for SNV Pileup Negational Model
[INFO] RUN THE FOLLOWING COMMAND:
( parallel --joblog <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/parallel_2-1_create_tensor_negational_snv.log -j 1 pypy3 /opt/bin/clairs_to.py create_tensor_pileup_calling --tumor_bam_fn <path_to>/dorado_basecalls.sorted.bam --ref_fn <path_to>/ref.fasta --ctg_name {1/.} --samtools samtools --candidates_bed_regions {1} --min_bq 0 --tensor_can_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/pileup_tensor_can_negational/{1/}  --platform ont :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/candidates/SNV_CANDIDATES_FILES ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/2-1_CPT_Negational_SNV.log

[INFO] sample_anonymized  Tensors generated: 549
[faidx] Truncated sequence: sample_anonymized:1432-14418

[INFO] SNV Pileup Model Prediction along with Affirmative & Negational Model
[INFO] RUN THE FOLLOWING COMMAND:
( parallel --joblog <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/parallel_2-2_predict_snv.log -j 1 python3 /opt/bin/clairs_to.py predict --tensor_fn_acgt <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/pileup_tensor_can_affirmative/{1/}  --tensor_fn_nacgt <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/pileup_tensor_can_negational/{1/}  --predict_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/predict/{1/}  --chkpnt_fn_acgt /opt/micromamba/envs/clairs-to/bin/clairs-to_models/ont_r10_dorado_sup_5khz_ssrs/pileup_affirmative.pkl --chkpnt_fn_nacgt /opt/micromamba/envs/clairs-to/bin/clairs-to_models/ont_r10_dorado_sup_5khz_ssrs/pileup_negational.pkl --use_gpu False --platform ont --ctg_name {1/.} --pileup  --disable_indel_calling True :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/candidates/SNV_CANDIDATES_FILES ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/2-2_PREDICT_SNV.log

[INFO] sample_anonymized total processed positions: 549, time elapsed: 3.3s

[INFO] SNV Pileup Model Calling Variants
[INFO] RUN THE FOLLOWING COMMAND:
( parallel --joblog <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/parallel_2-3_call_variants_snv.log -j 1 python3 /opt/bin/clairs_to.py call_variants --predict_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/predict/{1/}  --call_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/p_{1/}.vcf --ref_fn <path_to>/ref.fasta --platform ont --likelihood_matrix_data /opt/micromamba/envs/clairs-to/bin/clairs-to_models/ont_r10_dorado_sup_5khz/likelihood_matrix.txt  --disable_indel_calling True :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/candidates/SNV_CANDIDATES_FILES ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/2-3_CALL_VARIANTS_SNV.log

[INFO] Calling tumor-only somatic variants from sample_anonymized.0_0_1_snv ...
[INFO] Total time elapsed: 0.02 s
[INFO] No vcf output in file <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/p_sample_anonymized.0_0_1_snv.vcf, remove.

[INFO] Merge SNV VCFs
[INFO] RUN THE FOLLOWING COMMAND:
pypy3 /opt/bin/clairs_to.py sort_vcf --ref_fn <path_to>/ref.fasta --contigs_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/CONTIGS --input_dir <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output --vcf_fn_suffix snv.vcf --output_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/snv_pileup.vcf

[INFO] Sorting VCFs...
[WARNING] No vcf file found with suffix:<path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/None, output empty vcf file

[INFO] STEP 3: Non-somatic Tagging for SNV Variants
[INFO] RUN THE FOLLOWING COMMAND:
ln -sf <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/snv_pileup.vcf <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/snv_pileup_nonsomatic_tagging.vcf


[INFO] Select Heterozygous SNP for Phasing
[INFO] RUN THE FOLLOWING COMMAND:
( parallel --joblog <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/phasing_log/parallel_1_select_hetero_snp_for_phasing.log -j 1 pypy3 /opt/bin/clairs_to.py select_hetero_snp_for_phasing --tumor_vcf_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/snv_pileup.vcf --output_folder <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/vcf --ctg_name {1} :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/CONTIGS ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/phasing_log/1_select_hetero_snp_for_phasing.log

[INFO] Total HET SNP calls selected: sample_anonymized: 0, not found:0, not match:0, low_qual_count:0. Total tumor:0, pro: 0.0

[INFO] Phase the Tumor BAM
[INFO] RUN THE FOLLOWING COMMAND:
( parallel --joblog <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/phasing_log/parallel_2_phase_tumor.log -j 1 /opt/micromamba/envs/clairs-to/bin/longphase phase  -s <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/vcf/{1}.vcf -b <path_to>/dorado_basecalls.sorted.bam -r <path_to>/ref.fasta -t 1 -o <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/phased_vcf_output/tumor_phased_{1} --ont :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/CONTIGS && parallel -j 1 bgzip -f <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/phased_vcf_output/tumor_phased_{1}.vcf :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/CONTIGS ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/phasing_log/2_phase_tumor.log && parallel -j 1 tabix -f -p vcf <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/phased_vcf_output/tumor_phased_{1}.vcf.gz :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/CONTIGS

LongPhase Ver 1.7

--- File Parameter ---
SNP File      : <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/vcf/sample_anonymized.vcf
SV  File      :
MOD File      :
REF File      : <path_to>/ref.fasta
Output Prefix : <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/phased_vcf_output/tumor_phased_sample_anonymized
Generate Dot  : False
BAM File      : <path_to>/dorado_basecalls.sorted.bam

--- Phasing Parameter ---
Seq Platform       : ONT
Phase Indel        : False
Distance Threshold : 300000
Connect Adjacent   : 20
Edge Threshold     : 0.7
Mapping Quality    : 1
Variant Confidence : 0.75
ReadTag Confidence : 0.65

parsing VCF ... 0s
parsing SV VCF ... 0s
parsing Meth VCF ... 0s
reading reference ... 0s

parsing total:  0s
merge results ... 0s
writeResult SNP ... 0s

total process: 0s

[INFO] Haplotag the Tumor BAM
[INFO] RUN THE FOLLOWING COMMAND:
( parallel --joblog <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/phasing_log/parallel_3_haplotag_tumor.log -j 1 /opt/micromamba/envs/clairs-to/bin/longphase haplotag -o <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/phased_bam_output/tumor_{1} --reference <path_to>/ref.fasta --region {1}  -s <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/phased_vcf_output/tumor_phased_{1}.vcf.gz -b <path_to>/dorado_basecalls.sorted.bam :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/CONTIGS ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/phasing_log/3_tumor_haplotag.log && parallel -j 1 samtools index  -@1 <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/phased_bam_output/tumor_{1}.bam :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/CONTIGS

phased SNP file:   <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/phased_vcf_output/tumor_phased_sample_anonymized.vcf.gz
phased SV file:
phased MOD file:
input bam file:    <path_to>/dorado_basecalls.sorted.bam
input ref file:    <path_to>/ref.fasta
output bam file:   <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/phased_bam_output/tumor_sample_anonymized.bam
number of threads: 1
write log file:    false
log file:
-------------------------------------------
tag region:                    sample_anonymized
filter mapping quality below:  1
percentage threshold:          0.6
tag supplementary:             false
-------------------------------------------
parsing SNP VCF ... 0s
tag read start ...
chr: sample_anonymized ... 316s
tag read 316s
-------------------------------------------
total process time:  316s
total alignment:     6059894
total supplementary: 1292014
total secondary:     0
total unmapped:      0
total tag alignment: 0
total untagged:      6059894

[INFO] STEP 4: Post Filtering for SNV Variants
[INFO] RUN THE FOLLOWING COMMAND:
( pypy3 /opt/bin/clairs_to.py haplotype_filtering --tumor_bam_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/phased_bam_output/tumor_ --ref_fn <path_to>/ref.fasta --germline_vcf_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/snv_pileup.vcf --pileup_vcf_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/snv_pileup_nonsomatic_tagging.vcf --output_dir <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output --output_vcf_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/snv_pileup_filtering.vcf --samtools samtools --pypy3 pypy3 --parallel parallel --threads 1 --apply_haplotype_filtering True --disable_read_start_end_filtering False ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/4_HAP_FILTER_SNV.log

[INFO] Total input calls: 0, filtered by all hard filers: 0
[INFO] Total input calls: 0, filtered by low alt bq: 0
[INFO] Total input calls: 0, filtered by low alt mq: 0
[INFO] Total input calls: 0, filtered by read start and end: 0
[INFO] Total input calls: 0, filtered by variant cluster: 0
[INFO] Total input calls: 0, filtered by no ancestry: 0
[INFO] Total input calls: 0, filtered by multi haplotypes: 0
[INFO] Total input calls: 0, filtered by strand bias: 0
[INFO] Total input calls: 0, filtered by low sequence entropy: 0

[INFO] STEP 5: SNV VCF Post-processing
[INFO] RUN THE FOLLOWING COMMAND:
( pypy3 /opt/bin/clairs_to.py postprocess_vcf --ref_fn <path_to>/ref.fasta --pileup_vcf_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/snv_pileup_filtering.vcf --output_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/snv.vcf --platform ont --qual 4 --qual_cutoff_phaseable_region 4 --qual_cutoff_unphaseable_region 4 --sample_name SAMPLE --disable_indel_calling True --cmdline <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/CMD ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/5_PV_SNV.log

Traceback (most recent call last):
  File "/opt/bin/clairs_to.py", line 111, in <module>
    main()
  File "/opt/bin/clairs_to.py", line 105, in main
    submodule.main()
  File "/opt/bin/src/postprocess_vcf.py", line 259, in main
    merge_vcf(args)
  File "/opt/bin/src/postprocess_vcf.py", line 111, in merge_vcf
    pileup_vcf_reader.read_vcf()
  File "/opt/bin/shared/vcf.py", line 242, in read_vcf
    if columns[0][0] == "#":
IndexError: list index out of range

gzip: stdout: Broken pipe

[INFO] STEP 6: Indel Pileup Model Calling
[INFO] Create Tensors for Indel Pileup Affirmative Model
[INFO] RUN THE FOLLOWING COMMAND:
pypy3 /opt/bin/clairs_to.py concat_files --input_dir <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/candidates --input_prefix INDEL_CANDIDATES_FILE_ --output_fn INDEL_CANDIDATES_FILES  && ( parallel --joblog <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/parallel_6-1_create_tensor_affirmative_indel.log -j 1 pypy3 /opt/bin/clairs_to.py create_tensor_pileup_calling --tumor_bam_fn <path_to>/dorado_basecalls.sorted.bam --ref_fn <path_to>/ref.fasta --ctg_name {1/.} --min_bq 20 --samtools samtools --candidates_bed_regions {1} --tensor_can_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/pileup_tensor_can_affirmative/{1/}  --platform ont :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/candidates/INDEL_CANDIDATES_FILES ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/6-1_CPT_Affirmative_INDEL.log

[INFO] sample_anonymized  Tensors generated: 509
[faidx] Truncated sequence: sample_anonymized:710-14477

[INFO] Create Tensors for Indel Pileup Negational Model
[INFO] RUN THE FOLLOWING COMMAND:
( parallel --joblog <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/parallel_6-1_create_tensor_negational_indel.log -j 1 pypy3 /opt/bin/clairs_to.py create_tensor_pileup_calling --tumor_bam_fn <path_to>/dorado_basecalls.sorted.bam --ref_fn <path_to>/ref.fasta --ctg_name {1/.} --samtools samtools --candidates_bed_regions {1} --min_bq 0 --tensor_can_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/pileup_tensor_can_negational/{1/}  --platform ont :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/candidates/INDEL_CANDIDATES_FILES ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/6-1_CPT_Negational_INDEL.log

[INFO] sample_anonymized  Tensors generated: 509
[faidx] Truncated sequence: sample_anonymized:710-14477

[INFO] Indel Pileup Model Prediction along with Affirmative & Negational Model
[INFO] RUN THE FOLLOWING COMMAND:
( parallel --joblog <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/parallel_6-2_predict_indel.log -j 1 python3 /opt/bin/clairs_to.py predict --tensor_fn_acgt <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/pileup_tensor_can_affirmative/{1/}  --tensor_fn_nacgt <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/pileup_tensor_can_negational/{1/}  --predict_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/predict/{1/}  --chkpnt_fn_acgt /opt/micromamba/envs/clairs-to/bin/clairs-to_models/ont_r10_dorado_sup_5khz_ssrs/indel/pileup_affirmative.pkl --chkpnt_fn_nacgt /opt/micromamba/envs/clairs-to/bin/clairs-to_models/ont_r10_dorado_sup_5khz_ssrs/indel/pileup_negational.pkl --use_gpu False --platform ont --ctg_name {1/.} --pileup  --disable_indel_calling False  :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/candidates/INDEL_CANDIDATES_FILES ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/6-2_PREDICT_INDEL.log

[INFO] sample_anonymized total processed positions: 509, time elapsed: 2.6s

[INFO] Indel Pileup Model Calling Variants
[INFO] RUN THE FOLLOWING COMMAND:
( parallel --joblog <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/parallel_6-3_call_variants_indel.log -j 1 python3 /opt/bin/clairs_to.py call_variants --predict_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/predict/{1/}  --call_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/p_{1/}.vcf --platform ont --likelihood_matrix_data /opt/micromamba/envs/clairs-to/bin/clairs-to_models/ont_r10_dorado_sup_5khz/indel/likelihood_matrix.txt  --disable_indel_calling False  :::: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/candidates/INDEL_CANDIDATES_FILES ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/6-3_CALL_VARIANTS_INDEL.log

[INFO] Calling tumor-only somatic variants from sample_anonymized.0_0_1_indel ...
[INFO] Total time elapsed: 0.03 s
[INFO] No vcf output in file <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/p_sample_anonymized.0_0_1_indel.vcf, remove.

[INFO] Merge Indel VCFs
[INFO] RUN THE FOLLOWING COMMAND:
pypy3 /opt/bin/clairs_to.py sort_vcf --ref_fn <path_to>/ref.fasta --contigs_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/CONTIGS --input_dir <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output --vcf_fn_suffix indel.vcf --output_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/indel_pileup.vcf

[INFO] Sorting VCFs...
[WARNING] No vcf file found with suffix:<path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/None, output empty vcf file

[INFO] STEP 7: Non-somatic Tagging for SNV Variants
[INFO] RUN THE FOLLOWING COMMAND:
ln -sf <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/indel_pileup.vcf <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/indel_pileup_nonsomatic_tagging.vcf


[INFO] STEP 8: Post Filtering for Indel Variants
[INFO] RUN THE FOLLOWING COMMAND:
( pypy3 /opt/bin/clairs_to.py haplotype_filtering --tumor_bam_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/phasing_output/phased_bam_output/tumor_ --ref_fn <path_to>/ref.fasta --germline_vcf_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/snv_pileup.vcf --pileup_vcf_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/indel_pileup_nonsomatic_tagging.vcf --output_dir <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output --output_vcf_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/indel_pileup_filtering.vcf --samtools samtools --pypy3 pypy3 --parallel parallel --threads 1 --is_indel  --apply_haplotype_filtering True --disable_read_start_end_filtering False ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/8_HAP_FILTER_INDEL.log

[INFO] Total input calls: 0, filtered by all hard filers: 0
[INFO] Total input calls: 0, filtered by low alt bq: 0
[INFO] Total input calls: 0, filtered by low alt mq: 0
[INFO] Total input calls: 0, filtered by read start and end: 0
[INFO] Total input calls: 0, filtered by variant cluster: 0
[INFO] Total input calls: 0, filtered by no ancestry: 0
[INFO] Total input calls: 0, filtered by multi haplotypes: 0
[INFO] Total input calls: 0, filtered by strand bias: 0
[INFO] Total input calls: 0, filtered by low sequence entropy: 0

[INFO] STEP 9: Indel VCF Post-processing
[INFO] RUN THE FOLLOWING COMMAND:
( pypy3 /opt/bin/clairs_to.py postprocess_vcf --ref_fn <path_to>/ref.fasta --pileup_vcf_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/vcf_output/indel_pileup_filtering.vcf --output_fn <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/indel.vcf --platform ont --qual 2 --qual_cutoff_phaseable_region 2 --qual_cutoff_unphaseable_region 2 --sample_name SAMPLE --disable_indel_calling False --indel_calling  --cmdline <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/tmp/CMD ) 2>&1 | tee <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/logs/9_PV_INDEL.log

Traceback (most recent call last):
  File "/opt/bin/clairs_to.py", line 111, in <module>
    main()
  File "/opt/bin/clairs_to.py", line 105, in main
    submodule.main()
  File "/opt/bin/src/postprocess_vcf.py", line 259, in main
    merge_vcf(args)
  File "/opt/bin/src/postprocess_vcf.py", line 111, in merge_vcf
    pileup_vcf_reader.read_vcf()
  File "/opt/bin/shared/vcf.py", line 242, in read_vcf
    if columns[0][0] == "#":
IndexError: list index out of range

gzip: stdout: Broken pipe

[INFO] Total time elapsed: 10m44.00s

[INFO] Finish calling, SNV output VCF file: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/snv.vcf.gz

[INFO] Finish calling, Indel output VCF file: <path_to>/SSRS_DISABLE_NON_SOMATIC_TAGGING_CLAIRS-TO/indel.vcf.gz

[JasonCLEI]

Hi @SergeWielhouwer, @tamuanand,

Thanks a lot for your interest in ClairS-TO. As for the issue you encountered, it seems that ClairS-TO did not detect any valid somatic variants in your dataset. I suggest you add an option --print_ref_calls to print out reference calls or relax the minimum VAF limit by adjusting the option --snv_min_af.

Lei

[SergeWielhouwer]

Hi @JasonCLEI ,

Thank you for your suggestions. The print_ref_calls works as intended and now I'm getting a gVCF-style file.
That said, I’d ideally still like to produce a standard VCF output, even if it contains only a few variants or none at all. Are there any additional parameters or settings I could adjust to make that possible?

For instance, print_ref_calls outputs the following two entries:
ClairS-TO

PL020	6295	.	T	.	20.0268	RefCall	FAU=0;FCU=0;FGU=0;FTU=484;RAU=2;RCU=0;RGU=0;RTU=291	GT:GQ:DP:AF:AD:AU:CU:GU:TU	0/0:20:4883:0.1587:775:2:0:0:775
PL020	6298	.	G	.	15.1808	RefCall	FAU=0;FCU=1;FGU=183;FTU=3;RAU=0;RCU=0;RGU=288;RTU=0	GT:GQ:DP:AF:AD:AU:CU:GU:TU	0/0:15:3513:0.1341:471:0:1:471:3

While Clair3 reports them as:

PL020	6295	.	TC	T	10.21	PASS	F	GT:GQ:DP:AD:AF	1/1:10:51270:1723,48552:0.9470
PL020	6298	.	GC	G	14.39	PASS	P	GT:GQ:DP:AD:AF	1/1:14:51267:1684,48022:0.9367

Is there a way to tweak the settings so that these indels are still reported, even if they’re likely just noise?

Thanks again!

[JasonCLEI]

Hi @SergeWielhouwer,

The cases you reported seem to be germline deletions, and they should appear in the file indel.vcf.gz when using ClairS-TO. Notably, there is currently no VCF output when ClairS-TO does not detect any valid somatic variants, and you can obtain a standard VCF output by adding the option --print_ref_calls. We will fix this issue in the next release.

Lei