Merge pull request #525 from nf-core/dev

v1.2 - Bouncy Basenji Release PR

Author: LilyAnderssonLee

.github/workflows/ci.yml

@@ -36,7 +36,7 @@ jobs:
           - "test_motus"
           - "test_falco"
           - "test_fastp"
-          - "test_adapterremoval"
+          - "test_alternativepreprocessing"
           - "test_bbduk"
           - "test_prinseqplusplus"
 
@@ -65,8 +65,10 @@ jobs:
           if [[ "${{ matrix.tags }}" == "test_motus" ]]; then
             wget https://raw.githubusercontent.com/motu-tool/mOTUs/master/motus/downloadDB.py
             python downloadDB.py --no-download-progress
-            echo 'tool,db_name,db_params,db_path' > 'database_motus.csv'
-            echo "motus,db_mOTU,,db_mOTU" >> 'database_motus.csv'
+            echo 'tool,db_name,db_params,db_type,db_path' > 'database_motus.csv'
+            echo "motus,db1_mOTU,,short,db_mOTU" >> 'database_motus.csv'
+            echo "motus,db2_mOTU,,long,db_mOTU" >> 'database_motus.csv'
+            echo "motus,db3_mOTU,,short;long,db_mOTU" >> 'database_motus.csv'
             nextflow run ${GITHUB_WORKSPACE} -profile docker,${{ matrix.tags }} --databases ./database_motus.csv --outdir ./results_${{ matrix.tags }};
           else
             nextflow run ${GITHUB_WORKSPACE} -profile docker,${{ matrix.tags }} --outdir ./results_${{ matrix.tags }};

CHANGELOG.md

@@ -3,6 +3,47 @@
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## v1.2 - Bouncy Basenji [2024-10-03]
+
+### `Added`
+
+- [#417](https://github.com/nf-core/taxprofiler/pull/417) Added reference-free metagenome complexity/coverage estimation with Nonpareil (added by @jfy133)
+- [#466](https://github.com/nf-core/taxprofiler/pull/466) Input database sheets can specify a `db_type` column to distinguish between short- and long-read databases (added by @LilyAnderssonLee)
+- [#505](https://github.com/nf-core/taxprofiler/pull/505) Add small files to the file `tower.yml` (added by @LilyAnderssonLee)
+- [#508](https://github.com/nf-core/taxprofiler/pull/508) Add `nanoq` as a filtering tool for nanopore reads (added by @LilyAnderssonLee)
+- [#511](https://github.com/nf-core/taxprofiler/pull/511) Add `porechop_abi` as an alternative adapter removal tool for long reads nanopore data (added by @LilyAnderssonLee)
+- [#512](https://github.com/nf-core/taxprofiler/pull/512) Update all tools to the latest version and include nf-test (updated by @LilyAnderssonLee & @jfy133)
+- [#537](https://github.com/nf-core/taxprofiler/pull/537) Update the module `motus/merge` to the latest version (Updated by @sofstam & @LilyAnderssonLee)
+
+### `Fixed`
+
+- [#518](https://github.com/nf-core/taxprofiler/pull/518) Fixed a bug where Oxford Nanopore FASTA input files would not be processed (❤️ to @ikarls for reporting, fixed by @jfy133)
+- [#523](https://github.com/nf-core/taxprofiler/pull/523) Removed hardcoded `-m lca` from GANON_CLASSIFY due to more options in new version of ganon (fixed by @LilyAnderssonLee & @jfy133)
+- [#531](https://github.com/nf-core/taxprofiler/pull/531) Fix FASTA input validation in schema allowing FASTQ extension, expand allowed FASTA extensions (fixed by @jfy133)
+- [#512](https://github.com/nf-core/taxprofiler/pull/532) Minor formatting and ordering improvements in MultiQC report (by @jfy133)
+- [#532](https://github.com/nf-core/taxprofiler/pull/532) - Added missing documentation behind the 'ignore' BRACKEN_BRACKEN error strategy (❤️ to @Mavti for reporting, fixed by @jfy133)
+- [#536](https://github.com/nf-core/taxprofiler/pull/536) - Redefine `contents_re` for filtlong to fix its missing from the MultiQC report (fixed by @LilyAnderssonLee)
+
+### `Dependencies`
+
+| Tool      | Previous version | New version |
+| --------- | ---------------- | ----------- |
+| bbmap     | 39.01            | 39.06       |
+| bowtie2   | 2.4.4            | 2.5.2       |
+| bracken   | 2.7              | 2.9         |
+| diamond   | 2.0.15           | 2.1.8       |
+| ganon     | 1.5.1            | 2.0.0       |
+| kraken2   | 2.1.2            | 2.1.3       |
+| krona     | 2.8              | 2.8.1       |
+| megan     | 6.24.20          | 6.25.9      |
+| metaphlan | 4.0.6            | 4.1.1       |
+| minimap2  | 2.24             | 2.28        |
+| motus     | 3.0.3            | 3.1.0       |
+| multiqc   | 1.21             | 1.25        |
+| samtools  | 1.17             | 1.20        |
+
+### `Deprecated`
+
 ## v1.1.8 - Augmented Akita Patch [2024-06-20]
 
 ### `Added`

CITATIONS.md

@@ -30,14 +30,26 @@
 
   > Schubert, M., Lindgreen, S., & Orlando, L. (2016). AdapterRemoval v2: rapid adapter trimming, identification, and read merging. BMC Research Notes, 9, 88. https://doi.org/10.1186/s13104-016-1900-2
 
+- [Nonpareil](https://doi.org/10.1128/mSystems.00039-18)
+
+  - Rodriguez-R, L. M., Gunturu, S., Tiedje, J. M., Cole, J. R., & Konstantinidis, K. T. (2018). Nonpareil 3: Fast Estimation of Metagenomic Coverage and Sequence Diversity. mSystems, 3(3). https://doi.org/10.1128/mSystems.00039-18
+
 - [Porechop](https://github.com/rrwick/Porechop)
 
   > Wick, R. R., Judd, L. M., Gorrie, C. L., & Holt, K. E. (2017). Completing bacterial genome assemblies with multiplex MinION sequencing. Microbial Genomics, 3(10), e000132. https://doi.org/10.1099/mgen.0.000132
 
+- [Porechop_ABI](https://github.com/bonsai-team/Porechop_ABI)
+
+  > Bonenfant, Q., Noé, L., & Touzet, H. (2023). Porechop_ABI: discovering unknown adapters in Oxford Nanopore Technology sequencing reads for downstream trimming. Bioinformatics Advances, 3(1):vbac085. https://10.1093/bioadv/vbac085
+
 - [Filtlong](https://github.com/rrwick/Filtlong)
 
   > Wick R (2021) Filtlong, URL: https://github.com/rrwick/Filtlong
 
+- [nanoq](https://github.com/esteinig/nanoq)
+
+  > Steinig, E., & Coin, L. (2022). Nanoq: ultra-fast quality control for nanopore reads. Journal of Open Source Software, 7(69). https://doi.org/10.21105/joss.02991
+
 - [BBTools](http://sourceforge.net/projects/bbmap/)
 
   > Bushnell B. (2022) BBMap, URL: http://sourceforge.net/projects/bbmap/

README.md

@@ -29,11 +29,11 @@
 
 1. Read QC ([`FastQC`](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) or [`falco`](https://github.com/smithlabcode/falco) as an alternative option)
 2. Performs optional read pre-processing
-   - Adapter clipping and merging (short-read: [fastp](https://github.com/OpenGene/fastp), [AdapterRemoval2](https://github.com/MikkelSchubert/adapterremoval); long-read: [porechop](https://github.com/rrwick/Porechop))
-   - Low complexity and quality filtering (short-read: [bbduk](https://jgi.doe.gov/data-and-tools/software-tools/bbtools/), [PRINSEQ++](https://github.com/Adrian-Cantu/PRINSEQ-plus-plus); long-read: [Filtlong](https://github.com/rrwick/Filtlong))
+   - Adapter clipping and merging (short-read: [fastp](https://github.com/OpenGene/fastp), [AdapterRemoval2](https://github.com/MikkelSchubert/adapterremoval); long-read: [porechop](https://github.com/rrwick/Porechop), [Porechop_ABI](https://github.com/bonsai-team/Porechop_ABI))
+   - Low complexity and quality filtering (short-read: [bbduk](https://jgi.doe.gov/data-and-tools/software-tools/bbtools/), [PRINSEQ++](https://github.com/Adrian-Cantu/PRINSEQ-plus-plus); long-read: [Filtlong](https://github.com/rrwick/Filtlong)), [Nanoq](https://github.com/esteinig/nanoq)
    - Host-read removal (short-read: [BowTie2](http://bowtie-bio.sourceforge.net/bowtie2/); long-read: [Minimap2](https://github.com/lh3/minimap2))
    - Run merging
-3. Supports statistics for host-read removal ([Samtools](http://www.htslib.org/))
+3. Supports statistics metagenome coverage estimation ([Nonpareil](https://nonpareil.readthedocs.io/en/latest/)) and for host-read removal ([Samtools](http://www.htslib.org/))
 4. Performs taxonomic classification and/or profiling using one or more of:
    - [Kraken2](https://ccb.jhu.edu/software/kraken2/)
    - [MetaPhlAn](https://huttenhower.sph.harvard.edu/metaphlan/)
@@ -73,7 +73,7 @@ Additionally, you will need a database sheet that looks as follows:
 
 `databases.csv`:
 
-```
+```csv
 tool,db_name,db_params,db_path
 kraken2,db2,--quick,/<path>/<to>/kraken2/testdb-kraken2.tar.gz
 metaphlan,db1,,/<path>/<to>/metaphlan/metaphlan_database/

assets/multiqc_config.yml

@@ -1,8 +1,8 @@
 report_comment: >
 
-  This report has been generated by the <a href="https://github.com/nf-core/taxprofiler/releases/tag/1.1.8" target="_blank">nf-core/taxprofiler</a>
+  This report has been generated by the <a href="https://github.com/nf-core/taxprofiler/releases/tag/1.2" target="_blank">nf-core/taxprofiler</a>
   analysis pipeline. For information about how to interpret these results, please see the
-  <a href="https://nf-co.re/taxprofiler/1.1.8/docs/output" target="_blank">documentation</a>.
+  <a href="https://nf-co.re/taxprofiler/1.2/docs/output" target="_blank">documentation</a>.
 
 report_section_order:
   "nf-core-taxprofiler-methods-description":
@@ -11,6 +11,48 @@ report_section_order:
     order: -1001
   "nf-core-taxprofiler-summary":
     order: -1002
+  general_stats":
+    order: 1000
+  fastqc:
+    order: 900
+  fastqc-1:
+    order: 800
+  fastp:
+    order: 750
+  adapterremoval:
+    order: 700
+  nonpareil:
+    order: 600
+  bbduk:
+    order: 550
+  prinseqplusplus:
+    order: 500
+  porechop:
+    order: 450
+  porechop_abi:
+    order: 400
+  filtlong:
+    order: 350
+  nanoq:
+    order: 300
+  bowtie2:
+    order: 200
+  samtools:
+    order: 100
+  kraken:
+    order: 90
+  bracken:
+    order: 80
+  centrifuge:
+    order: 70
+  malt:
+    order: 60
+  diamond:
+    order: 50
+  kaiju:
+    order: 40
+  motus:
+    order: 30
 
 export_plots: true
 
@@ -21,14 +63,15 @@ custom_logo_title: "nf-core/taxprofiler"
 
 run_modules:
   - fastqc
-  - adapterRemoval
+  - adapterremoval
   - fastp
+  - nonpareil
   - bbduk
   - prinseqplusplus
   - porechop
   - filtlong
+  - nanoq
   - bowtie2
-  - minimap2
   - samtools
   - kraken
   - kaiju
@@ -39,11 +82,16 @@ run_modules:
 
 sp:
   diamond:
-    fn_re: ".*.diamond.log$"
+    fn: "*.diamond.log"
   fastqc/data:
     fn_re: ".*(fastqc|falco)_data.txt$"
   fastqc/zip:
     fn: "*_fastqc.zip"
+  nonpareil:
+    fn: "nonpareil_all_samples.json"
+  filtlong:
+    contents: Scoring long reads
+    contents_re: " "
 
 top_modules:
   - "fastqc":
@@ -60,13 +108,23 @@ top_modules:
       path_filters_exclude:
         - "*raw*"
       extra: "If used in this run, Falco is a drop-in replacement for FastQC producing the same output, written by Guilherme de Sena Brandine and Andrew D. Smith."
-  - "fastp"
-  - "adapterRemoval"
+  - nonpareil
   - "porechop":
+      name: "Porechop"
+      anchor: "porechop"
+      target: "Porechop"
+      path_filters:
+        - "*porechop.log"
       extra: "ℹ️: if you get the error message 'Error - was not able to plot data.' this means that porechop did not detect any adapters and therefore no statistics generated."
-  - "bbduk"
-  - "prinseqplusplus"
-  - "filtlong"
+  - "porechop":
+      name: "Porechop_ABI"
+      anchor: "porechop_abi"
+      target: "Porechop_ABI"
+      doi: "10.1093/bioadv/vbac085"
+      info: "find and remove adapters from Oxford Nanopore reads."
+      path_filters:
+        - "*porechop_abi.log"
+      extra: "ℹ️: if you get the error message 'Error - was not able to plot data.' this means that porechop_abi did not detect any adapters and therefore no statistics generated."
   - "bowtie2":
       name: "bowtie2"
   - "samtools":
@@ -95,12 +153,11 @@ top_modules:
         - "*.centrifuge.txt"
   - "malt":
       name: "MALT"
-  - "diamond"
   - "kaiju":
       name: "Kaiju"
-  - "motus"
 
-#It is not possible to set placement for custom kraken and centrifuge columns.
+# It is not possible to set placement for custom kraken
+# and centrifuge columns.
 
 table_columns_placement:
   FastQC / Falco (pre-Trimming):
@@ -130,16 +187,33 @@ table_columns_placement:
     percent_aligned: 370
     percent_collapsed: 380
     percent_discarded: 390
+  nonpareil:
+    nonpareil_R: 400
+    nonpareil_LR: 410
+    nonpareil_kappa: 420
+    nonpareil_C: 430
+    nonpareil_diversity: 440
   Porechop:
-    Input Reads: 400
-    Start Trimmed: 410
-    Start Trimmed Percent: 420
-    End Trimmed: 430
-    End Trimmed Percent: 440
-    Middle Split: 450
-    Middle Split Percent: 460
+    Input Reads: 500
+    Start Trimmed: 510
+    Start Trimmed Percent: 520
+    End Trimmed: 530
+    End Trimmed Percent: 540
+    Middle Split: 550
+    Middle Split Percent: 560
+  Porechop_ABI:
+    Input Reads: 500
+    Start Trimmed: 510
+    Start Trimmed Percent: 520
+    End Trimmed: 530
+    End Trimmed Percent: 540
+    Middle Split: 550
+    Middle Split Percent: 560
   Filtlong:
-    Target bases: 500
+    Target bases: 600
+  nanoq:
+    Reads: 700
+    Read N50: 710
   BBDuk:
     Input reads: 800
     Total Removed bases percent: 810
@@ -203,6 +277,24 @@ table_columns_visible:
     percent_duplicates: False
     percent_gc: False
     percent_fails: False
+  Adapter Removal:
+    aligned_total: True
+    percent_aligned: True
+    percent_collapsed: True
+    percent_discarded: False
+  fastp:
+    pct_adapter: True
+    pct_surviving: True
+    pct_duplication: False
+    after_filtering_gc_content: False
+    after_filtering_q30_rate: False
+    after_filtering_q30_bases: False
+  nonpareil:
+    nonpareil_R: false
+    nonpareil_LR: false
+    nonpareil_kappa: true
+    nonpareil_C: true
+    nonpareil_diversity: true
   porechop:
     Input reads: False
     Start Trimmed:
@@ -211,20 +303,19 @@ table_columns_visible:
     End Trimmed Percent: True
     Middle Split: False
     Middle Split Percent: True
-  fastp:
-    pct_adapter: True
-    pct_surviving: True
-    pct_duplication: False
-    after_filtering_gc_content: False
-    after_filtering_q30_rate: False
-    after_filtering_q30_bases: False
+  porechop_abi:
+    Input reads: False
+    Start Trimmed:
+    Start Trimmed Percent: True
+    End Trimmed: False
+    End Trimmed Percent: True
+    Middle Split: False
+    Middle Split Percent: True
   Filtlong:
     Target bases: True
-  Adapter Removal:
-    aligned_total: True
-    percent_aligned: True
-    percent_collapsed: True
-    percent_discarded: False
+  nanoq:
+    ReadN50: True
+    Reads: True
   BBDuk:
     Input reads: False
     Total Removed bases Percent: False
@@ -276,6 +367,10 @@ extra_fn_clean_exts:
   - ".bbduk"
   - ".unmapped"
   - "_filtered"
+  - "porechop"
+  - "porechop_abi"
+  - "_processed"
+  - ".diamond"
   - type: remove
     pattern: "_falco"
 

assets/schema_database.json

@@ -39,6 +39,12 @@
                 "errorMessage": "Invalid database db_params entry. No quotes allowed.",
                 "meta": ["db_params"]
             },
+            "db_type": {
+                "type": "string",
+                "enum": ["short", "long", "short;long"],
+                "default": "short;long",
+                "meta": ["db_type"]
+            },
             "db_path": {
                 "type": "string",
                 "exists": true,