Skip to content

Conversation

chrisroberts
Copy link
Member

Description

Migrating batch allocations being marked as failed when killed was
introduced in dfa07e1 (#26025) to ensure that allocations were
properly rescheduled to another node when drained if no eligible
nodes were immediately available.

It appears that the change introduced in 333dd94 (#26292) resolved
the underlying issue affecting drained batch allocations by filtering
allocations with a desired state that is terminal when reconciling
the task group.

This reverts the changes to mark the migrating batch allocations
as failed. Draining batch nodes still behave as expected and the
status of the migrated allocations are now complete instead of
failed.

Testing & Reproduction steps

Links

Reverts changes introduced in #26025
Related changes that fixed the drain issue #26292

Contributor Checklist

  • Changelog Entry If this PR changes user-facing behavior, please generate and add a
    changelog entry using the make cl command.
  • Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
    ensure regressions will be caught.
  • Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
    and job configuration, please update the Nomad website documentation to reflect this. Refer to
    the website README for docs guidelines. Please also consider whether the
    change requires notes within the upgrade guide.

Reviewer Checklist

  • Backport Labels Please add the correct backport labels as described by the internal
    backporting document.
  • Commit Type Ensure the correct merge method is selected which should be "squash and merge"
    in the majority of situations. The main exceptions are long-lived feature branches or merges where
    history should be preserved.
  • Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
    within the public repository.
  • If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

Migrating batch allocations being marked as failed when killed was
introduced in dfa07e1 (#26025) to ensure that allocations were
properly rescheduled to another node when drained if no eligible
nodes were immediately available.

It appears that the change introduced in 333dd94 (#26292) resolved
the underlying issue affecting drained batch allocations by filtering
allocations with a desired state that is terminal when reconciling
the task group.

This reverts the changes to mark the migrating batch allocations
as failed. Draining batch nodes still behave as expected and the
status of the migrated allocations are now `complete` instead of
`failed`.
Copy link
Member

@tgross tgross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code change seems fine but after a little digging the mechanism is a little unclear to me.

I wanted to double-check the expected behavior, and the nomad node drain docs say:

Allocations for batch and sysbatch jobs will wait until they complete or the drain's deadline is reached, whichever comes first. These allocations will not be replaced.

So the original #26025 PR now seems like it's got an issue in terms of what we were trying to fix (I definitely missed this during the original review so that's on me). And the PR you mentioned in the description here #26292 was never backported because it wasn't considered a bug fix, but when I backported the patch in this PR to the release/1.8.x+ent branch for testing, everything still seemed to work as expected, so that couldn't be what's fixing the underlying issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants