feat(app): allow robot restarts to track boot ID and timeout

[mcous]

Overview

This PR moves restart tracking logic from a smear across the robot-admin and buildroot modules into a single epic in the robot-admin module. The new epic:

• Keeps existing up/down tracking behavior (watch for robots to go down and come back up)
  • This logic has a known race condition that could result in a restart being "missed"
• Now also tracks boot ID as an additional "successful restart" avenue
  • Because we're comparing boot IDs, the race condition above does not apply
• Calls the restart "timed out" after 60 seconds, closing the spinner
  • This will resolve and "missed" restarts and drop the user into a state with a warning modal in case there is some actual problem which caused the robot to not come back

Closes #6585

Changelog

• feat(app): allow robot restarts to track boot ID and timeout
  • Includes general restarts, like clicking the "Restart" button (full page spinner)
  • Includes restarts initiated during a robot software update (progress animation in update modal)

Review requests

This is kind of a gnarly test matrix, but here's the behaviors we want to test. The "Restart" button on the robot page is the easiest way to trigger a restart. This works best with a non-Wi-Fi robot so you can pull the cable at any point to simulate failures.

• Successful restart: robot reboots, spinner pops up then goes away
• Unsuccessful restart: robot reboots, spinner pops up, robot never comes back, spinner goes away after timeout
  • Yanking the Ethernet cable after you click "Restart" can simulate this

We also need to test both of these for robot updates:

• Successful update: robot reboots, you end up on a "robot has updated" success modal
• Unsuccessful update: robot reboots, eventually it times out and gives you an error modal
  • To trigger: pull the Ethernet cable after you get to the "your robot is restarting" message

With those behaviors, we have two types of robots to test on:

• v4.2.x robots and higher - expose a bootId in their update server's health endpoint
  • Inspect Redux State > robotAdmin > ${robotName} > restart > bootId to make sure its being tracked
  • Also look at the payloads of robotAdmin:RESTART_STATUS_CHANGED actions, there should be a bootId in the action that changes the status to restart-pending
• Pre-4.2 robots do not have a bootId
  • We need to test that the restart spinner fallback logic still works

Finally:

• The restart timeout of 60 changed to 300 seconds (5 minutes) was an arbitrary choice! Is that too short? Too long?
  • Keep an eye on how long restarts take in your testing, especially during updates

If you need refresher on what the buildroot update epics are supposed to do, check out the app cookbook which has a mostly complete outline + flowcharts of the app-side logic of the update procedure.

Risk assessment

Medium. I have not tested this on a robot yet, and it required some changes to the robot update logic. After robot testing, this PR appears to be doing its job well when combined with #7608

Mitigating factors:

• Code is well covered by unit tests, though it's very possible I'm missing some edge cases
• Changes to the robot update logic should only affect code that happens after the update has been sent over, verified, and committed

[amitlissack]

Should be looked at by JS person.

[SyntaxColoring]

Calls the restart "timed out" after 60 seconds, closing the spinner

This seems pretty tight to me. Anecdotally, I think I've seen restarts take multiple minutes, depending on whether the motor controller board needs to update its firmware, and potentially even how old the SD card is. Admittedly, though, that's measuring from power-on to API server readiness—not update server readiness.

Should we instead set the timeout very very conservatively? Like 10 minutes?

Edit: And the current "Robot is restarting..." dialog copy says "this may take up to 3 minutes," which still seems kind of tight to me.

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (edge@c526b19). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             edge    #7589   +/-   ##
=======================================
  Coverage        ?   82.31%           
=======================================
  Files           ?      328           
  Lines           ?    22062           
  Branches        ?        0           
=======================================
  Hits            ?    18160           
  Misses          ?     3902           
  Partials        ?        0           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c526b19...8918e39. Read the comment docs.