Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resource handler #73

Draft
wants to merge 17 commits into
base: main
Choose a base branch
from
Draft

Conversation

yashpatel6
Copy link
Collaborator

  • I have read the code review guidelines and the code review best practice on GitHub check-list.

  • I have reviewed the Nextflow pipeline standards.

  • The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].

  • I have set up or verified the branch protection rule following the github standards before opening this pull request.

  • I have added my name to the contributors listings in the
    metadata.yaml and the manifest block in the nextflow.config as part of this pull request, am listed
    already, or do not wish to be listed. (This acknowledgement is optional.)

  • I have added the changes included in this pull request to the CHANGELOG.md under the next release version or unreleased, and updated the date.

  • I have updated the version number in the metadata.yaml and manifest block of the nextflow.config file following semver, or the version number has already been updated. (Leave it unchecked if you are unsure about new version number and discuss it with the infrastructure team in this PR.)

  • I have tested the config being added or modified. Outline the tests below.

Closes #...

@yashpatel6
Copy link
Collaborator Author

Example pipeline: uclahs-cds/pipeline-call-gSNP#189

Steps to setup:

  1. Run the following to automatically generate a resources JSON from all node configs found in a given directory (i.e. full path to the pipeline's config directory):
# Run this from a directory that doesn't contain a nextflow.config file as that may cause issues since Nextflow will automatically try to load it from the working directory
nextflow run /hot/software/pipeline/pipeline-Nextflow-config/Nextflow/development/unreleased/yashpatel-generalized-resource-handler/converter_pipeline/main.nf --config_dir /path/to/config/dir

The pipeline will find and convert all node configs into a single resources.json file generated in the same directory as the node configs. Additionally, the JSON will copy the F72 allocations as the default allocations - the default allocations will require manual updates to make them generalized with proper limits.

  1. Delete the node-specific configs and update methods.config to remove calls to methods.set_resources_allocation(), methods.modify_base_allocations(), and retry.setup_retry() and replace with a single call to resource_handler.handle_resources("${projectDir}/config/resources.config"). See draft PR in call-gSNP linked above for an example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants