armadactl watch --exit-if-inactive should only exit if the jobset is inactive now, not at the first point inactivity in its event stream

[sebwills]

Using armadactl watch --exit-if-inactive on a job set that currently has some running jobs, I expect it to display the history so far and then keep monitoring until the job set becomes active.

But if the job set has a period of inactivity in its past, this command terminates as soon as it it reaches that moment, even if there is activity after that point:

Full history:

➜   armadactl watch svc-autogca FCESUP-2410_3
Watching job set FCESUP-2410_3
Sep 13 08:36:43 | Queued:   0, Leased:   0, Pending:   0, Running:   0, Succeeded:   0, Failed:   0, Cancelled:   0 | JobSubmittedEvent, job id: 01k513gcqg2vm43h2pmm9at0bx
Sep 13 08:36:43 | Queued:   1, Leased:   0, Pending:   0, Running:   0, Succeeded:   0, Failed:   0, Cancelled:   0 | JobQueuedEvent, job id: 01k513gcqg2vm43h2pmm9at0bx
Sep 13 08:36:57 | Queued:   0, Leased:   1, Pending:   0, Running:   0, Succeeded:   0, Failed:   0, Cancelled:   0 | JobLeasedEvent, job id: 01k513gcqg2vm43h2pmm9at0bx
Sep 13 08:37:10 | Queued:   0, Leased:   0, Pending:   1, Running:   0, Succeeded:   0, Failed:   0, Cancelled:   0 | JobPendingEvent, job id: 01k513gcqg2vm43h2pmm9at0bx pod: 0
Sep 13 08:37:31 | Queued:   0, Leased:   0, Pending:   0, Running:   1, Succeeded:   0, Failed:   0, Cancelled:   0 | JobRunningEvent, job id: 01k513gcqg2vm43h2pmm9at0bx pod: 0
Sep 13 09:13:21 | Queued:   0, Leased:   0, Pending:   0, Running:   0, Succeeded:   1, Failed:   0, Cancelled:   0 | JobSucceededEvent, job id: 01k513gcqg2vm43h2pmm9at0bx pod: 0
Sep 13 09:42:28 | Queued:   0, Leased:   0, Pending:   0, Running:   0, Succeeded:   1, Failed:   0, Cancelled:   0 | JobSubmittedEvent, job id: 01k5178rzj2pc21r3ym0s5we42
Sep 13 09:42:28 | Queued:   1, Leased:   0, Pending:   0, Running:   0, Succeeded:   1, Failed:   0, Cancelled:   0 | JobQueuedEvent, job id: 01k5178rzj2pc21r3ym0s5we42
Sep 13 09:42:46 | Queued:   0, Leased:   1, Pending:   0, Running:   0, Succeeded:   1, Failed:   0, Cancelled:   0 | JobLeasedEvent, job id: 01k5178rzj2pc21r3ym0s5we42
Sep 13 09:42:55 | Queued:   0, Leased:   0, Pending:   1, Running:   0, Succeeded:   1, Failed:   0, Cancelled:   0 | JobPendingEvent, job id: 01k5178rzj2pc21r3ym0s5we42 pod: 0
Sep 13 09:43:16 | Queued:   0, Leased:   0, Pending:   0, Running:   1, Succeeded:   1, Failed:   0, Cancelled:   0 | JobRunningEvent, job id: 01k5178rzj2pc21r3ym0s5we42 pod: 0
[no more events - terminated with Ctrl-C]

Note that although the initially submitted job terminated, leaving the job set inactive, another job was later submitted into the same job set and is still running. But with --exit-if-inactive, the command terminates after printing the 09:13 event:

➜   armadactl watch svc-autogca FCESUP-2410_3 --exit-if-inactive
Watching job set FCESUP-2410_3
Sep 13 08:36:43 | Queued:   0, Leased:   0, Pending:   0, Running:   0, Succeeded:   0, Failed:   0, Cancelled:   0 | JobSubmittedEvent, job id: 01k513gcqg2vm43h2pmm9at0bx
Sep 13 08:36:43 | Queued:   1, Leased:   0, Pending:   0, Running:   0, Succeeded:   0, Failed:   0, Cancelled:   0 | JobQueuedEvent, job id: 01k513gcqg2vm43h2pmm9at0bx
Sep 13 08:36:57 | Queued:   0, Leased:   1, Pending:   0, Running:   0, Succeeded:   0, Failed:   0, Cancelled:   0 | JobLeasedEvent, job id: 01k513gcqg2vm43h2pmm9at0bx
Sep 13 08:37:10 | Queued:   0, Leased:   0, Pending:   1, Running:   0, Succeeded:   0, Failed:   0, Cancelled:   0 | JobPendingEvent, job id: 01k513gcqg2vm43h2pmm9at0bx pod: 0
Sep 13 08:37:31 | Queued:   0, Leased:   0, Pending:   0, Running:   1, Succeeded:   0, Failed:   0, Cancelled:   0 | JobRunningEvent, job id: 01k513gcqg2vm43h2pmm9at0bx pod: 0
Sep 13 09:13:21 | Queued:   0, Leased:   0, Pending:   0, Running:   0, Succeeded:   1, Failed:   0, Cancelled:   0 | JobSucceededEvent, job id: 01k513gcqg2vm43h2pmm9at0bx pod: 0
[terminates]

I would expect the command only to terminate when it reaches a state of inactivity and there are no more events in the stream (i.e. if the job set is currently inactive).