Skip to content

Nextflow eagerly terminates Azure Batch jobs during execution #5839

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
adamrtalbot opened this issue Mar 3, 2025 · 0 comments · May be fixed by #6159
Open

Nextflow eagerly terminates Azure Batch jobs during execution #5839

adamrtalbot opened this issue Mar 3, 2025 · 0 comments · May be fixed by #6159
Labels
executor/azure-batch

Comments

@adamrtalbot
Copy link
Collaborator

New feature

Based on comment: #3927 (comment)

Azure Batch jobs are terminated during pipeline closing down, see #3927.

During that PR, we discussed eagerly changing the setting of each job to onAllTasksComplete = 'terminate' but never looked further into it.

Use case

The main benefit of doing it this way is that if Nextflow fails to close properly (i.e. is killed), most active jobs will be marked as to be terminated and be removed from the active job quota.

Suggested implementation

We just need an "onProcessClose" event to notify when all tasks are created. I've already added it in a number of PRs, like this one: #3818

The suggestion from the first PR which could be implemented ☝
adamrtalbot added a commit that referenced this issue Jun 4, 2025
@adamrtalbot
feat: Azure Batch eagerly terminates jobs after all tasks have been s…
92c58a6 
…ubmitted

Azure Batch "job leak" is still an issue. This commit fixes #5839 which allows Nextflow to set jobs to auto terminate when all tasks have been submitted. This means that eventually jobs will move into terminated state even if something prevents nextflow reaching a graceful shutdown. Very early implementation and needs some refinement.

Signed-off-by: adamrtalbot <[email protected]>
@adamrtalbot adamrtalbot linked a pull request Jun 4, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
executor/azure-batch
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Azure Batch eagerly terminates jobs after all tasks have been submitted nextflow-io/nextflow
2 participants
@bentsherman @adamrtalbot