Message Delivery Rate alert rule and overview panel should use deriv instead of rate

[jkuester]

The rate function is only for use with COUNTER metrics (values that only go up or are reset to 0, but never go down) and not GAUGE metrics (values that can go up or down). Basically the rate function has special logic to deal with the "reset to zero" case, but that logic behaves weirdly when the metric value starts going down.

The deriv function should be used for any GAUGE type metrics instead of rate. It is a similar calculate for the rate of change, but does not account for "reset to zero" (and it does not break if the metric value goes down).

The Message Delivery Rate alert rule (and panel) is an interesting case since technically the cht_messaging_outgoing_total metric value can never go down (and so is a COUNTER). However our calculations are based on values recorded for the failed and delivered labels. The values for these labels are based on data reported to the CHT by the external messaging platforms. While it seems like the number of messages in the failed status should never go down, based on prod data recorded on the Allies Watchdog instance, it apparently is possible. So, for these queries where we are filtering cht_messaging_outgoing_total by the status label, we should use deriv instead of rate to properly measure the change.