Add nf-test to CI by kedhammar · Pull Request #51 · nf-core/seqinspector

kedhammar · 2024-10-29T09:55:39Z

PR checklist

This comment contains a description of changes (with reason).
- Add nf-test to CI, based on the nf-test docs
- Closes Run nf-test in CI #22
~~[ ] If you've fixed a bug or added code that should be tested, add tests!~~
~~[ ] If you've added a new tool - have you followed the pipeline conventions in the contribution docs~~
~~[ ] If necessary, also make a PR on the nf-core/seqinspector branch on the nf-core/test-datasets repository.~~
Make sure your code lints (nf-core lint).
Ensure the test suite passes (nf-test test main.nf.test -profile test,docker).
~~[ ] Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).~~
~~[ ] Usage Documentation in docs/usage.md is updated.~~
~~[ ] Output Documentation in docs/output.md is updated.~~
CHANGELOG.md is updated.
~~[] README.md` is updated (including new tool citations and authors/contributors).~~

…rom nf-test docs

github-actions · 2024-10-29T09:57:17Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 69825df

+| ✅ 189 tests passed       |+
!| ❗  21 tests had warnings |!

Details

❗ Test warnings:

readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
pipeline_todos - TODO string in main.nf: Remove this line if you don't need a FASTA file
pipeline_todos - TODO string in nextflow.config: Specify your pipeline's command line flags
pipeline_todos - TODO string in nextflow.config: Optionally, you can add a pipeline-specific nf-core config at https://github.com/nf-core/configs
pipeline_todos - TODO string in README.md: TODO nf-core:
pipeline_todos - TODO string in README.md: Include a figure that guides the user through the major workflow steps. Many nf-core
pipeline_todos - TODO string in README.md: Fill in short bullet-pointed list of the default steps in the pipeline
pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
pipeline_todos - TODO string in test_full.config: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA)
pipeline_todos - TODO string in test_full.config: Give any required params for the test so that command line flags are not needed
pipeline_todos - TODO string in test.config: Specify the paths to your test data on nf-core/test-datasets
pipeline_todos - TODO string in test.config: Give any required params for the test so that command line flags are not needed
pipeline_todos - TODO string in base.config: Check the defaults for all processes
pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-seqinspector_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-seqinspector_logo_light.png
files_exist - File found: docs/images/nf-core-seqinspector_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: conf/igenomes_ignored.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-seqinspector_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowSeqinspector.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-schema plugin
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: validation.help.enabled
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable found: validation.help.beforeText
nextflow_config - Config variable found: validation.help.afterText
nextflow_config - Config variable found: validation.help.command
nextflow_config - Config variable found: validation.summary.beforeText
nextflow_config - Config variable found: validation.summary.afterText
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config variable (correctly) not found: params.max_cpus
nextflow_config - Config variable (correctly) not found: params.max_memory
nextflow_config - Config variable (correctly) not found: params.max_time
nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
nextflow_config - Config variable (correctly) not found: params.validationLenientMode
nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 1.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-seqinspector_logo_light.png matches the template
files_unchanged - docs/images/nf-core-seqinspector_logo_light.png matches the template
files_unchanged - docs/images/nf-core-seqinspector_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 24.04.2, Config: 24.04.2
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: nf-test.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: template_version_comment.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
modules_config - conf/modules.config found and not ignored.
modules_config - FASTQC found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC_GLOBAL found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC_PER_TAG found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.0.2

Run details

nf-core/tools version 3.0.2
Run at 2024-10-29 14:49:10

…imits

kedhammar · 2024-10-29T10:55:45Z

Since I've opted for conservative nf-test runs I'm running it manually to make sure we have full coverage.

I’m running into a single, very simple assertion error for the utils_nextflow_pipeline subworkflow.

  Test [dd2e962c] 'Should print version' Assertion failed: 

assert workflow.stdout.contains("nextflow_workflow v9.9.9")
       |        |      |
       workflow []     false

FAILED (8.614s)

  Assertion failed: 
  
  1 of 2 assertions failed
  
  Nextflow stdout:
  
  Nextflow stderr:

This is after running nf-core submodules update as well. I've asked about it in the nf-test slack channel now but it might not be our problem.

kedhammar · 2024-10-29T10:56:28Z

Ping @maxulysse

kedhammar · 2024-10-29T11:21:07Z

I’m running into a single, very simple assertion error for the utils_nextflow_pipeline subworkflow.

Got confirmation from @mirpedrol it's an issue on the nf-core/modules repo

Aratz · 2024-10-29T11:52:01Z

Awesome! I've tried to make some changes on my own repo to test it but I get this weird error in the ci 🤔 Is this something you've encountered yourself?
https://github.com/Aratz/seqinspector/actions/runs/11573234518/job/32214895326

Aratz · 2024-10-29T11:57:10Z

I think we might want to run all the tests all the time actually: if you modify some files, run the test and they fail, and then do some other modifications that don't trigger the tests again, it will not run anything and still show a green mark, which may be misleading, see https://github.com/Aratz/seqinspector/actions/runs/11573281505/job/32215045767, I just wanted to reduce the number of shards to 2 but it didn't run the tests because modifying the ci file does not trigger the tests...

maxulysse · 2024-10-29T11:58:15Z

.github/workflows/nf-test.yml

why not in the ci.yml file?

The ci.yml file runs the whole pipeline if I understand correctly.

I'd prefer to have them as separate GHA checks, if they do different things.

kedhammar · 2024-10-29T13:40:45Z

I’m running into a single, very simple assertion error for the utils_nextflow_pipeline subworkflow.

Got confirmation from @mirpedrol it's an issue on the nf-core/modules repo

Fixed here nf-core/modules#6887 and applied in 3443734.

All tests now pass on gitpod 🚀

…sts run on different nextflow versions

Aratz

Great!

Just one thing is that now it's running the tests for the modules we're using, even though these are supposed to be validated in https://github.com/nf-core/modules/. I think for now it's fine but it might get very slow as we add more and more modules. Maybe we can look into how to restrict nf-test. I tried to look it up now but didn't manage to go anywhere...

kedhammar added 2 commits October 29, 2024 10:51

Add CI config for parallellized and economical nf-test runs, ripped f…

0f7ff12

…rom nf-test docs

Add nf-test config to trigger full test on changes to critical files

389c50b

kedhammar self-assigned this Oct 29, 2024

prettier lint

38b0e45

kedhammar added 2 commits October 29, 2024 11:00

Bump changelog

b452763

Add modules testdata base path to test config and trim out resource l…

89ca390

…imits

kedhammar requested review from Aratz and maxulysse October 29, 2024 10:56

maxulysse reviewed Oct 29, 2024

View reviewed changes

nf-core subworkflows update to apply fix

3443734

kedhammar added 4 commits October 29, 2024 13:43

simplify CI, always run

ee42aa9

try specifying cwd as arg in nf-test command

4705376

harmonize syntax and correct comment

222a986

remove multiqc versions file from snapshot assertion, will fail if te…

69825df

…sts run on different nextflow versions

Aratz approved these changes Oct 29, 2024

View reviewed changes

kedhammar merged commit a2d589f into nf-core:dev Oct 29, 2024

kedhammar deleted the ci-nf-test branch October 29, 2024 15:49

kedhammar mentioned this pull request Oct 29, 2024

Speed up nf-test CI by excluding modules #61

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Add nf-test to CI#51

Add nf-test to CI#51
kedhammar merged 10 commits intonf-core:devfrom
kedhammar:ci-nf-test

kedhammar commented Oct 29, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Oct 29, 2024 •

edited

Loading

❗ Test warnings:

✅ Tests passed:

Run details

Uh oh!

kedhammar commented Oct 29, 2024

Uh oh!

kedhammar commented Oct 29, 2024

Uh oh!

kedhammar commented Oct 29, 2024 •

edited

Loading

Uh oh!

Aratz commented Oct 29, 2024

Uh oh!

Aratz commented Oct 29, 2024

Uh oh!

maxulysse Oct 29, 2024

Uh oh!

kedhammar Oct 29, 2024

Uh oh!

kedhammar commented Oct 29, 2024

Uh oh!

Aratz left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

kedhammar commented Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR checklist

Uh oh!

github-actions bot commented Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

✅ Tests passed:

Run details

Uh oh!

kedhammar commented Oct 29, 2024

Uh oh!

kedhammar commented Oct 29, 2024

Uh oh!

kedhammar commented Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aratz commented Oct 29, 2024

Uh oh!

Aratz commented Oct 29, 2024

Uh oh!

maxulysse Oct 29, 2024

Choose a reason for hiding this comment

Uh oh!

kedhammar Oct 29, 2024

Choose a reason for hiding this comment

Uh oh!

kedhammar commented Oct 29, 2024

Uh oh!

Aratz left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kedhammar commented Oct 29, 2024 •

edited

Loading

github-actions bot commented Oct 29, 2024 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

kedhammar commented Oct 29, 2024 •

edited

Loading