Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use setup_process_afterscript #314

Merged
merged 6 commits into from
Oct 30, 2024
Merged

use setup_process_afterscript #314

merged 6 commits into from
Oct 30, 2024

Conversation

sorelfitzgibbon
Copy link
Contributor

Description

Closes #301

Testing Results

  • NFtest
    • log: /hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/unreleased/sfitz-afterscript/log-nftest-20241015T222100Z.log
      cases: a_mini-all-tools-std-input

These extra files are created.

< process-log/intersect/compress_index_VCF_concat/check_compression_bgzip/log.command.begin
< process-log/intersect/compress_index_VCF_concat/compress_VCF_bgzip/log.command.begin
< process-log/intersect/compress_index_VCF_concat/index_VCF_tabix/log.command.begin
< process-log/intersect/compress_index_VCF_reordered/check_compression_bgzip/log.command.begin
< process-log/intersect/compress_index_VCF_reordered/index_VCF_tabix/log.command.begin
< process-log/muse/compress_index_VCF/check_compression_bgzip/log.command.begin
< process-log/muse/compress_index_VCF/index_VCF_tabix/log.command.begin
< process-log/mutect2/compress_index_VCF/check_compression_bgzip/log.command.begin
< process-log/mutect2/compress_index_VCF/index_VCF_tabix/log.command.begin
< process-log/somaticsniper/compress_index_VCF_fix/check_compression_bgzip/log.command.begin
< process-log/somaticsniper/compress_index_VCF_fix/index_VCF_tabix/log.command.begin
< process-log/somaticsniper/compress_index_VCF_hc/check_compression_bgzip/log.command.begin
< process-log/somaticsniper/compress_index_VCF_hc/compress_VCF_bgzip/log.command.begin
< process-log/somaticsniper/compress_index_VCF_hc/index_VCF_tabix/log.command.begin
< process-log/strelka2/compress_index_VCF/check_compression_bgzip/log.command.begin
< process-log/strelka2/compress_index_VCF/index_VCF_tabix/log.command.begin

They seem to be duplicates of the same files except including the additional (correct) tool-tool-version directory and the (correct) suffixes on the final directory. e.g.

> grep compress_index_VCF_hc after.txt 
process-log/SomaticSniper-1.0.5.0/somaticsniper/compress_index_VCF_hc/check_compression_bgzip-SNV/log.command.begin
process-log/SomaticSniper-1.0.5.0/somaticsniper/compress_index_VCF_hc/compress_VCF_bgzip-SNV/log.command.begin
process-log/SomaticSniper-1.0.5.0/somaticsniper/compress_index_VCF_hc/index_VCF_tabix-SNV/log.command.begin
process-log/somaticsniper/compress_index_VCF_hc/check_compression_bgzip/log.command.begin
process-log/somaticsniper/compress_index_VCF_hc/compress_VCF_bgzip/log.command.begin
process-log/somaticsniper/compress_index_VCF_hc/index_VCF_tabix/log.command.begin

I haven't been able to identify the reason and wonder if it's happening within the submodule; some kind of a default log being created as well as that specified by log_output_dir

Checklist

  • I have read the code review guidelines and the code review best practice on GitHub check-list.

  • I have reviewed the Nextflow pipeline standards.

  • The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].

  • I have set up or verified the branch protection rule following the github standards before opening this pull request.

  • I have added my name to the contributors listings in the manifest block in the nextflow.config as part of this pull request; I am listed already, or do not wish to be listed. (This acknowledgement is optional.)

  • I have added the changes included in this pull request to the CHANGELOG.md under the next release version or unreleased, and updated the date.

  • I have updated the version number in the metadata.yaml and manifest block of the nextflow.config file following semver, or the version number has already been updated. (Leave it unchecked if you are unsure about new version number and discuss it with the infrastructure team in this PR.)

  • I have tested the pipeline on at least one A-mini sample.

@nwiltsie
Copy link
Member

nwiltsie commented Oct 17, 2024

@sorelfitzgibbon You're running into the same issue as uclahs-cds/pipeline-Nextflow-module#42 - all of our modules have publishDir rules built-in, so now they are being duplicated.

  • log-call-sSNV-8.1.0-20241015T222108Z/process-log/mutect2/compress_index_VCF/check_compression_bgzip/log.command.log - Produced by the afterscript, should always be produced
  • log-call-sSNV-8.1.0-20241015T222108Z/process-log/Mutect2-4.5.0.0/mutect2/compress_index_VCF/check_compression_bgzip-snps/log.command.log - Produced by the publishDir rule in the submodule. Will not be produced if the command fails.

My fix from uclahs-cds/pipeline-Nextflow-module#42 was to add ext capture_logs: false to those processes (in the submodule, not in this repo). That ensures that only the publishDir rule will produce the logfiles, although that means that they will not be produced if the process fails.

So to fix this:

@nwiltsie
Copy link
Member

  • Revert all changes to addParams(options: [log_output_dir: ... lines in this PR.

Actually, I'm not sure about this one... those might be right. Maybe those don't need to be reverted.

@sorelfitzgibbon
Copy link
Contributor Author

Works now with pipeline-nextflow-module update, commit 9f07b9fee6bcd742ef8f461b62782522bb69f65e
NFtest log: /hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/unreleased/sfitz-afterscript/log-nftest-20241018T000427Z.log
Process logs match as confirmed with /hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/unreleased/tmp.sh

@sorelfitzgibbon
Copy link
Contributor Author

/fix-tests

Copy link

Bleep bloop, I am a robot.

You requested that I fix the tests, but I can only do so after posting a comment saying that I can do so.

Copy link

Bleep bloop, I am a robot.

Alas, some of the Nextflow configuration tests failed!

test/configtest-F16.json

@ ["process","afterScript"]
+ {"1":"","2":"","3":"","closure":""}
@ ["process","ext"]
+ {"capture_logs":true,"commonAfterScript":{"1":"","2":"","3":"","closure":""},"log_dir":{"1":"ext","2":"ext","3":"ext","closure":"ext"}}

test/configtest-F32.json

@ ["process","afterScript"]
+ {"1":"","2":"","3":"","closure":""}
@ ["process","ext"]
+ {"capture_logs":true,"commonAfterScript":{"1":"","2":"","3":"","closure":""},"log_dir":{"1":"ext","2":"ext","3":"ext","closure":"ext"}}

If the above changes are surprising, stop and determine what happened.

If the above changes are expected, there are two ways to fix this:

  1. Automatically: Post a comment starting with "/fix-tests" (without the quotes) and I will update the tests for you (you must review my work afterwards).
  2. Manually: Follow these steps on Confluence.

@nwiltsie
Copy link
Member

/fix-tests

Copy link

Bleep bloop, I am a robot.

I have updated all of the failing tests for you with 2d7af58. You must review my work before merging this pull request!

@wiz-inc-8da00b022c
Copy link

wiz-inc-8da00b022c bot commented Oct 28, 2024

Wiz Scan Summary

Scan Module Critical High Medium Low Info Total
IaC Misconfigurations 0 0 0 0 0 0
Vulnerabilities 0 0 0 0 0 0
Sensitive Data 0 0 0 0 1 1
Secrets 0 0 0 0 0 0
Total 0 0 0 0 1 1

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try using Wiz Code VS Code Extension.

@sorelfitzgibbon sorelfitzgibbon merged commit 78aa330 into main Oct 30, 2024
7 checks passed
@sorelfitzgibbon sorelfitzgibbon deleted the sfitz-afterscript branch October 30, 2024 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use methods.setup_process_afterscript() for process logs
3 participants