Frequent fatal rejected envelopes from backtracking while recovering large backlog

[leviramsey]

We just started a DynamoDB projection from a DynamoDB journal (using the 24.10.1 BOM). The journal has been being written to heavily for several days, so the projection has a backlog to start of a few million events.

We are observing consistent failures and restarts of the projection with

Failed to process envelopes with offsets from [TimestampOffset(2024-11-08T22:25:45.722977Z, 2024-11-13T17:55:53.021702Z, Map(EntityType|someId0 -> 9))] to [TimestampOffset(2024-11-08T22:25:04.180012Z, 2024-11-13T17:55:53.022282Z, Map(EntityType|someId1 -> 5))] after [4] attempts. Projection will stop as defined by recovery strategy. MDC:[akkaAddress=akka://[!FILTERED!]@10.236.134.254:2551, akkaUid=-7361407520859643022, sourceThread=actorSystemName-akka.actor.default-dispatcher-30, akkaSource=DynamoDBProjectionImpl(akka://actorSystemName), sourceActorSystem=actorSystemName, akkaTimestamp=17:56:05.236UTC] akka.projection.dynamodb.internal.DynamoDBOffsetStore$RejectedEnvelope: Rejected envelope from backtracking, persistenceId [EntityType|someId3], seqNr [24] due to unexpected sequence number.

These restarts are resulting in rerunning queries resulting in wasted RCU consumption.

[leviramsey]

Possibly related to #70 and #79