Skip to content

Report SLURM runner status as launch-slurm-runner job outcome #540

Open
@yhtang

Description

@yhtang

We provision GitHub Actions runners for running JAX/Pallas/Levanter A100 unit tests using SLURM jobs that launch ephemeral runner instances inside the job. Currently, the launch-slurm-runner job will succeed as long as it is able to submit the job and wait for the job to finish/terminate. If the runner fails to register, e.g. due to network issue, then launch-slurm-runner would succeed, while the actual unit test job that needs to use the runner will wait forever. Even cancelling and then restarting the workflow cannot work around this, as the launcher job is marked as successfully and won't be rerun.

To fix that, we need the launch-slurm-runner job to query the status of the runner running inside the job. If the runner errors out without being able to handle a job, then the CI job should also fail.

Note the difference between

  1. launch-slurm-runner successfully starts an A100 runner, which picks up a job that run and failed, and
  2. launch-slurm-runner fails to start an A100 runner.

The latter is what we need to address in this issue.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions