grenepipe v0.15.0

Notable Changes

• Upgrade freebayes 1.3.8 → 1.3.9, to fix freebayes/freebayes#764
• Add min-contig-size and max-contigs-per-group options
• Add script to collect and plot benchmarks
• Clean up log and benchmark paths
• Refine generate table script to accept accession numbers

grenepipe v0.14.0

This update restructures some intermediate file names of the calling steps. Furthermore, we now request dummy done files for almost all rules, which are created once a rule finishes successfully. Usually, Snakemake should not need those, and be able to figure out if a rule was executed successfully or failed. However, there are some circumstances where this mechanism does not work, and Snakemake fails us. Hence, we now manually fix this via those dummy files.

As a consequence of these changes, note that when you upgrade to this version of grenepipe, any existing analysis run will likely want to re-compute most of its files, as the restructured file names and the dummy files will not be present.

Major Changes

• Add GenomicsDBImport as new default for merging GATK HaplotypeCaller gvcfs #55
• Switch to new picard command line syntax with dashes
• Split java options from memory resources in config (with backwards compatibility)
• Restructure calling intermediate files
• Request done files for (almost) all rules
• Upgrade tools:
  • picard 3.2.0 → 3.3.0
  • gatk4 4.5.0.0 → 4.6.1.0
  • freebayes 1.3.7 → 1.3.8, to fix freebayes/freebayes#796

Bugfixes

• Fix picard CollectMultipleMetrics java memory issue #60
• Fix freebayes issue freebayes/freebayes#796
• Make bowtie indexing a non-local rule
• Fix slurm config number of threads for variant calling

grenepipe v0.13.4

Notable Changes

• Refine default resource config to work around snakemake/snakemake#2997 and snakemake/snakemake#3191
• Fix HAF-pipe merge samples script, and refine success checks for HAF-pipe rules

grenepipe v0.13.3

Notable Changes

• Update slurm cluster config files for new snakemake executor, fix #51, fix #54
• Add --shallow (no directory recursion) option to generate-table script @meixilin
• Add F1/R2 naming scheme (Genome Sequence Archive) to generate-table script

grenepipe v0.13.2

Notable Changes

• Fix conda setup for samtools rules, which broke with recent pandas/numpy versions (again)

grenepipe v0.13.1

Minor update for some quality of life after the last big release of v0.13.0

Notable Changes

• Add extra snakemake logging to work around Snakemake bug (snakemake/snakemake#2974)
• Add backwards compatibility with the config.yaml of grenedalf pre-v0.13.0

grenepipe v0.13.0

This is a major update of grenepipe, as conda suddenly broke backwards compatibility, and so nothing was working any more. We used this opportunity to updated almost all tools in the pipeline to their most recent versions, and also added some new features. Furthermore, we restructured the internal files for compliance with the Snakemake workflow catalog, and restructured the output files of the pipeline for more convenience and clarity.

Major Changes

• Upgrade from Snakemake v6.0.5 to v8.15.2 by default
  • python 3.7.10 → 3.12
  • pandas 1.3.1 → 2.2.2
  • numpy 1.21.2 → 2.0.0
• Upgrade tools:
  • adapterremoval 2.3.1 → 2.3.3
  • bcftools 1.16 → 1.20
  • bowtie2 2.4.1 → 2.5.4
  • bwa 0.7.17 → 0.7.18
  • cutadapt 2.10 → 4.9
  • fastqc 0.11.9 → 0.12.1
  • freebayes 1.3.1 → 1.3.7
  • gatk4 4.1.4.1 → 4.5.0.0
  • mapdamage2 2.2.1 → 2.2.2.2
  • multiqc 1.10.1 → 1.22.3
  • picard 2.27.4 → 3.2.0
  • qualimap 2.2.2d → 2.3.0
  • samtools 1.16.1 → 1.20
  • seqkit 2.2.0 → 2.8.2
  • snpeff 4.3.1t → 5.2
  • vep ensembl 104 → 112
• Restructure all output files for more user convenience. See wiki for details.

New Features

• Add variant calling from bam files workflow (instead of starting from fastq files) #47
• Add automatic download of reference genome and known references #41
• Make trimming tool optional #35

Bugfixes

• Fix conda env for bcftools #37
• Deactivate bcftools stats by default
• Add picard-compatible default to bwa mem
• Add reference genome name check #44
• Update ref genome extensions for new GATK

grenepipe v0.12.2

Notable Changes

• Fix snakemake wrapper utils, which suddenly broke backwards compatibility (see here)
• Fix vcftools, which also suddenly broke, despite fixed version (see here)

grenepipe v0.12.1

Notable Changes

• Add single end mode to generate table script
• Add java options to picard tools
• Replace GATK by Picard for CreateSequenceDictionary rule
• Fix minor platform issue with empty cells in samples table

grenepipe v0.12.0

This release restructures the merging of mapped bam files prior to variant calling, see Processing of the mapped reads for details. It has breaking changes in the config file (e.g., renaming the entry for the samples table to data: samples-table, see below), so you will need to use the new config file to start an analysis.

Furthermore, some tool versions were updated (although those should be non-breaking changes), and the overall robustness of the conda environments has been greatly increased, in particular for running on MacOS. All environments work with conda and with mamba now, but we still strongly recommend to use mamba; in our tests, conda needs 4h, and mamba 15min to install all environments.

Notable Changes

• Change data: samples to data: samples-table in the config file, and change some tool parameter keys
• Rework mapped read merging to occur before all other bam processing
• Fix bam read group ID tag to match sample units. The ID was equal to the sample name before; now with the reworked merging of units, we also use the unit in the read group ID
• Add further config validations of the samples table
• Improved tests and CI, testing on Ubuntu and MacOS now, with conda and mamba

Bug Fixes

• Fix and update several conda environments, in particular for picard and qualimap, which were not solving with conda before. This fixes #25 and #11. As a result, some tools have been upgraded:
  • bcftools 1.13 → 1.16
  • picard 2.20.1 → 2.27.4
  • samtools 1.12 → 1.16.1
  • vcflib 1.0.2 → 1.0.3
• Fix #29, correctly set bwa mem2 threads