Elastic Training Fast Resume

[lukebaumann]

Description

These changes are for the fast resume mode of elastic training where training is paused if a slice is lost and resumes when the slice has returned. It is intended to make workload startup time after hardware failure much quicker because there is no need to communicate to GCS to restore from a checkpoint. It accomplishes this by not restarting the workload and only restarting the affected slices.

Tests

It is not thoroughly tested at this state but it it has successfully fast resumed from a simulated slice failure.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed.