Random job selection for generated jobs not ideal #357
Description
Was debugging an issue this morning and noticed what I thought to be some gaps in generated job creation. Turns out I was looking at only completed jobs. Because there was a large backlog of generated jobs due to some slow jobs it took hours for a generated job to be picked up. In this scenario we're running consistency checker every 30 minutes. There were some jobs that got processed 5 hours after they were scheduled.
Recommendation: Attempt to pick generated job based on scheduled date. Pick oldest first. If cannot, then try a random non-generated job.