Job Runs Have Out Of Order Timestamps

[jparraga-stackav]

Describe the bug

We do reporting on jobs & job runs and sometimes the timestamps can appear out of order which corrupts some of the calculations we make. It appears that the scheduler may cancel or preempt a job run and mark the job run as finished but then later the underlying container can trigger a job running event. It seems to me that these events are processed almost unconditionally, which leads to out of order timestamps on the job runs.

Expected behavior

I would expect that the executor would not send job running events for jobs that have been cancelled or preempted. Also, I would expect the scheduler ingester to drop or warn about messages for invalid state transitions on job runs.