Dependence on persisting system clock timestamps is vulnerable to clock skew

[leviramsey]

The timestamp on persisted events is taken from the system clock. Since the eventsBySlices query uses timestamps for offsets, this can cause rejected sequence numbers, especially if backtracking cannot find the events (the backtracking window is exceeded by clock skew, or the projection is far behind and avoiding backtracking).

R2DBC defaults to avoiding this problem by deferring to the database's clock and ensuring monotonicity in the transaction; that approach is not applicable with dynamo. Perhaps it makes sense for the plugin to keep a cache, populated on replay and updated on persist, of the high watermark timestamps for entities (ensuring that the timestamp is monotonic increasing)?