Release 0.7 #116

muffato · 2024-10-02T12:19:46Z

The sum of #113, #114, #117, #118, #120, #121, #125, #127, #124, #130, #139, #142, #143, #146, #156.

PR checklist

Use small 1.2G for base nextflow job

Use the org level variable TOWER_COMPUTE_ENV which point to small actually with 1.2G memory.

Use small tower compute env

… the beginning

This will save a lot of space

Support for grid view (and a few other things)

github-actions · 2024-10-02T12:21:27Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 1ac291b

+| ✅ 185 tests passed       |+
#| ❔  28 tests were ignored |#
!| ❗  20 tests had warnings |!

❗ Test warnings:

nextflow_config - Config manifest.version should end in dev: 0.7.0
pipeline_todos - TODO string in ro-crate-metadata.json: "description": "# sanger-tol/blobtoolkit\n\n\n \n\n\n\n\n\n\n\n\n## Introduction\n\nsanger-tol/blobtoolkit is a bioinformatics pipeline that ...\n\n TODO nf-core:\n Complete this sentence with a 2-3 sentence summary of what types of data the pipeline ingests, a brief overview of the\n major pipeline sections and the types of output it produces. You're giving an overview to someone new\n to nf-core here, in 15-20 seconds. For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#introduction\n\n\n Include a figure that guides the user through the major workflow steps. Many nf-core\n workflows use the "tube map" design for that. See https://nf-co.re/docs/contributing/design_guidelines#examples for examples. \n Fill in short bullet-pointed list of the default steps in the pipeline 1. Read QC (FastQC)2. Present QC for raw reads (MultiQC)\n\n## Usage\n\n> [!NOTE]\n> If you are new to Nextflow and nf-core, please refer to this page on how to set-up Nextflow. Make sure to test your setup with -profile test before running the workflow on actual data.\n\n Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.\n Explain what rows and columns represent. For instance (please edit as appropriate):\n\nFirst, prepare a samplesheet with your input data that looks as follows:\n\nsamplesheet.csv:\n\ncsv\nsample,fastq_1,fastq_2\nCONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz\n\n\nEach row represents a fastq file (single-end) or a pair of fastq files (paired end).\n\n\n\nNow, you can run the pipeline using:\n\n update the following command to include all required parameters for a minimal example \n\nbash\nnextflow run sanger-tol/blobtoolkit \\\n -profile <docker/singularity/.../institute> \\\n --input samplesheet.csv \\\n --outdir <OUTDIR>\n\n\n> [!WARNING]\n> Please provide pipeline parameters via the CLI or Nextflow -params-file option. Custom config files including those provided by the -c Nextflow option can be used to provide any configuration except for parameters; see docs.\n\n## Credits\n\nsanger-tol/blobtoolkit was originally written by priyanka-surana.\n\nWe thank the following people for their extensive assistance in the development of this pipeline:\n\n If applicable, make list of people who have also contributed \n\n## Contributions and Support\n\nIf you would like to contribute to this pipeline, please see the contributing guidelines.\n\n## Citations\n\n Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file. \n If you use sanger-tol/blobtoolkit for your analysis, please cite it using the following doi: 10.5281/zenodo.XXXXXX \n\n Add bibliography of tools and data used in your pipeline \n\nAn extensive list of references for the tools used by the pipeline can be found in the CITATIONS.md file.\n\nThis pipeline uses code and infrastructure developed and maintained by the nf-core community, reused here under the MIT license.\n\n> The nf-core framework for community-curated bioinformatics pipelines.\n>\n> Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.\n>\n> Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.\n",
local_component_structure - jsonify_taxdump.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
local_component_structure - create_bed.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
local_component_structure - nohit_list.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
local_component_structure - restructurebuscodir.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
local_component_structure - generate_config.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
local_component_structure - compressblobdir.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
local_component_structure - windowstats_input.nf in modules/local should be moved to a TOOL/SUBTOOL/main.nf structure
local_component_structure - run_blastx.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - view.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - collate_stats.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - run_blastn.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - minimap_alignment.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - prepare_genome.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - coverage_stats.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - finalise_blobdir.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - input_check.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - blobtools.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure
local_component_structure - busco_diamond_blastp.nf in subworkflows/local should be moved to a SUBWORKFLOW_NAME/main.nf structure

❔ Tests ignored:

files_exist - File is ignored: CODE_OF_CONDUCT.md
files_exist - File is ignored: assets/nf-core-blobtoolkit_logo_light.png
files_exist - File is ignored: docs/images/nf-core-blobtoolkit_logo_light.png
files_exist - File is ignored: docs/images/nf-core-blobtoolkit_logo_dark.png
files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: conf/igenomes.config
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting_comment.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/email_template.html
files_unchanged - File ignored due to lint config: assets/email_template.txt
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File ignored due to lint config: assets/nf-core-blobtoolkit_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-blobtoolkit_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-blobtoolkit_logo_dark.png
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/blobtoolkit/blobtoolkit/.github/workflows/awstest.yml
template_strings - template_strings
merge_markers - merge_markers

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes_ignored.config
files_exist - File found: modules.json
files_exist - File found: ro-crate-metadata.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-blobtoolkit_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowBlobtoolkit.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-schema plugin
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: validation.help.enabled
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable found: validation.help.beforeText
nextflow_config - Config variable found: validation.help.afterText
nextflow_config - Config variable found: validation.help.command
nextflow_config - Config variable found: validation.summary.beforeText
nextflow_config - Config variable found: validation.summary.afterText
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config variable (correctly) not found: params.max_cpus
nextflow_config - Config variable (correctly) not found: params.max_memory
nextflow_config - Config variable (correctly) not found: params.max_time
nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
nextflow_config - Config variable (correctly) not found: params.validationLenientMode
nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.image_format= png
nextflow_config - Config default value correct: params.outdir= results
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Nextflow minimum version badge matched config. Badge: 24.04.2, Config: 24.04.2
readme - README Zenodo placeholder was replaced with DOI.
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: sanger_test_full.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: template_version_comment.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: sanger_test.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - WINDOWSTATS_INPUT found in conf/base.config and Nextflow scripts.
base_config - BLOBTOOLKIT_WINDOWSTATS found in conf/base.config and Nextflow scripts.
base_config - FASTAWINDOWS found in conf/base.config and Nextflow scripts.
base_config - BUSCO_BUSCO found in conf/base.config and Nextflow scripts.
base_config - BLAST_BLASTN found in conf/base.config and Nextflow scripts.
modules_config - conf/modules.config found and not ignored.
modules_config - WINDOWMASKER_MKCOUNTS found in conf/modules.config and Nextflow scripts.
modules_config - WINDOWMASKER_USTAT found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_HIC found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_ILMN found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_CCS found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_CLR found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_ONT found in conf/modules.config and Nextflow scripts.
modules_config - MINIMAP2_ found in conf/modules.config and Nextflow scripts.
modules_config - SAMTOOLS_VIEW found in conf/modules.config and Nextflow scripts.
modules_config - SAMTOOLS_INDEX found in conf/modules.config and Nextflow scripts.
modules_config - PIGZ_COMPRESS found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTK_DEPTH found in conf/modules.config and Nextflow scripts.
modules_config - BUSCO_BUSCO found in conf/modules.config and Nextflow scripts.
modules_config - RESTRUCTUREBUSCODIR found in conf/modules.config and Nextflow scripts.
modules_config - DIAMOND_BLASTP found in conf/modules.config and Nextflow scripts.
modules_config - DIAMOND_BLASTX found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTK_DEPTH found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTOOLKIT_WINDOWSTATS found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTOOLKIT_CREATEBLOBDIR found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTOOLKIT_UPDATEBLOBDIR found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTOOLKIT_CHUNK found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTOOLKIT_UNCHUNK found in conf/modules.config and Nextflow scripts.
modules_config - NOHIT_LIST found in conf/modules.config and Nextflow scripts.
modules_config - BLAST_BLASTN found in conf/modules.config and Nextflow scripts.
modules_config - COMPRESSBLOBDIR found in conf/modules.config and Nextflow scripts.
modules_config - BLOBTK_IMAGES found in conf/modules.config and Nextflow scripts.
modules_config - JSONIFY_TAXDUMP found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.2.0

Run details

nf-core/tools version 3.2.0
Run at 2025-03-19 15:25:06

…rkflows can natively support multiple input files.

… be used over and over

Precomputed BUSCOs

Fixed the handling of the taxdump

Release changes

gq1 and others added 27 commits September 13, 2024 15:32

Update sanger_test_full.yml

2584645

Use small 1.2G for base nextflow job

Update sanger_test_full.yml

ea1b05a

Use the org level variable TOWER_COMPUTE_ENV which point to small actually with 1.2G memory.

Merge pull request #113 from sanger-tol/small_tower_compute_env

e0bf8ba

Use small tower compute env

Fixed the comment

d13d1ed

UPDATEBLOBDIR needs to pass the tsv files through certain arguments

2c793aa

Populate as much as possible of the metadata at the beginning

b55f76c

Also generate TSV files

0daf1e2

Example plugging the TSV files into blobtools commands

2cf12fd

Skip BLOBTOOLKIT_METADATA as we generate the yaml file correctly from…

b8b1fa4

… the beginning

Introduced a new samplesheet parameter to track single/paired readsets

41b101f

Populate as much as possible of the metadata at the beginning

80a100b

Version bumps

bea3a77

Updated the documentation

4edafb4

Version bump

904a512

Publish the windowmasker outputs too

51985ad

Convert input BAM/CRAM files to Fasta on the fly

f47dd16

This will save a lot of space

Bumped the version down

43254bc

Pin the version of editorconfig like in the readmapping pipeline

c89b1b9

Exclude sqlite databases from ECLint

152282b

add exclusions for nf-core lint

476b736

Update CITATIONS.md

303b221

prettier linting

27b0a51

The release is going to happen today

4eef0fa

Decreased the runtime requirements for illumina alignments

0d8a5a9

Wording of the changelog

1909646

Merge pull request #114 from sanger-tol/chrom_view

102dbf4

Support for grid view (and a few other things)

Documentation update

8a05dc2

muffato self-assigned this Oct 2, 2024

muffato requested a review from tkchafin October 14, 2024 16:20

muffato added 8 commits March 14, 2025 13:50

This is a path and must have the name method

8913dd1

The guideline is that all calls should have first() so that the subwo…

49f2156

…rkflows can natively support multiple input files.

Restored the multiqc output - but only for newly-computed BUSCOs

7bc8850

Aligned with the Python script: supports direct lineage names

18e4f95

bugfix: the genome must be made a value channel so that the value can…

30c3774

… be used over and over

Corrected the availability of short_summary.tsv

e9f5398

bugfix: the name of Busco has changed

87cab15

Expanded to cater for the new test profiles

b45b34a

muffato linked an issue Mar 14, 2025 that may be closed by this pull request

Add Label or distinctive name to different sequencing data #123

Closed

muffato added 9 commits March 14, 2025 16:16

Merge pull request #142 from sanger-tol/precomp_busco

5f11dda

Precomputed BUSCOs

Preparing for the relase

1b97dac

Set the ORCIDs

009f580

Added the logo back

3ba85db

Unused

d937fbb

Added missing file

d5fd283

[prettier]

5581e95

Tweaked editorconfig-checker for the cff file

ebe28d9

[prettier]

09c55f5

muffato marked this pull request as ready for review March 15, 2025 11:22

muffato and others added 8 commits March 17, 2025 09:19

Forgot to update this placeholder

3da1915

fixed the syntax to ensure the uniqueness of the first column

32702a7

Merge pull request #146 from sanger-tol/taxdump

2500f11

Fixed the handling of the taxdump

bugfix: the name of Busco has changed

e2ff7b0

Release date

2857e64

bugfix: some BUSCO results can be missing, e.g. archaea/bacteria

6ae12c7

v2 is now officially out

754db69

Updated the release date

ae3e10f

tkchafin approved these changes Mar 19, 2025

View reviewed changes

Merge pull request #156 from sanger-tol/release

1ac291b

Release changes

muffato merged commit 6745579 into main Mar 19, 2025
14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Release 0.7 #116

Release 0.7 #116

Uh oh!

muffato commented Oct 2, 2024 •

edited

Loading

Uh oh!