Dynamic priorities #469
Description
Right now we can assign priorities to each recipe so that they run in a more optimal ordering to complete the entire event sooner. Our servers are getting even busier these days so priorities often come up. One improvement that could be made would be to have a bit of dynamic priority assignment going on here and there. For instance, let's say we have PR launches will all of the systems are really busy. If that PR ends up with a failed box on it anywhere, it would be nice if the priority for all of the other (un-started) boxes would drop to the lowest priority. This would ensure that merges and other PRs that are succeeding would get through the system faster. Right now a PR that's failing, will continue to get services while things that are working wait around (or have the same priority). I'm sure there are even more things we can do with priorities, but this is a starting point.