bug: Store message state in Redis to solve for some reliability state management issues (#1005)

Move the individual message state to Redis. This is to attempt to stop
some reliability counts from trending negative. This is after the code
paths were scrutinized to ensure that the struct cloning was minimized,
and some states (notably, ones where there was a hand-off between active
services) continued to trend negative.

Unfortunately, the currently available version of Redis is 7.2, so many
of the auto-expiring records (like
[HSETEX](https://redis.io/docs/latest/commands/hsetex/) are not present,
meaning that we'll have to use more keys, and use traditional `SET` with
an `EX`piration set. Fortunately, that should not be a problem, since
there appears to be plenty of available memory and CPU on the active
Redis servers.

The new approach _only_ updates counts if the prior state matches what
has been last recorded.

Closes: PUSH-546

Author: jrconlin

autoendpoint/src/extractors/notification.rs

@@ -234,6 +234,10 @@ impl Notification {
                 &self.reliable_state,
                 Some(self.timestamp + self.headers.ttl as u64),
             )
-            .await;
+            .await
+            .inspect_err(|e| {
+                warn!("🔍⚠️ Unable to record reliability state log: {:?}", e);
+            })
+            .unwrap_or(Some(state))
     }
 }

autopush-common/src/notification.rs

@@ -90,7 +90,11 @@ impl Notification {
                 &self.reliable_state,
                 Some(self.expiry()),
             )
-            .await;
+            .await
+            .inspect_err(|e| {
+                warn!("🔍⚠️ Unable to record reliability state log: {:?}", e);
+            })
+            .unwrap_or_default();
     }
 
     #[cfg(feature = "reliable_report")]