Dropping pubsub event in the logs

[xeppaka]

Description is here: https://discuss.lightbend.com/t/akka-persistence-r2dbc-pubsub-event-dropped/10619

Versions used

akka-persistence-r2dbc 1.5.2

Akka version: 2.9.1

Expected Behavior

Event is processed by projection right after pubsub event is received.

Actual Behavior

Event is processed by projection with a delay.

Relevant logs

https://discuss.lightbend.com/t/akka-persistence-r2dbc-pubsub-event-dropped/10619

[patriknw]

I started to look at this, but it is not as easy as first thought. This was introduced in #345 and the reason is important for correctness.

I thought we could track that no events at all have arrived and therefore safely accept the incoming event because of the idle situation. However, then when second event arrives we are in the same situation until the backtracking event arrives. We don't know for how long we could just continue accepting pubsub events because then we could violate the reason for the "to far ahead" check.

This only happens if there are no events at all for a duration longer than the backtracking window, so it should be a rare situation. Increasing the backtracking window could be a way to mitigate it for a low throughput system. Pubsub should only be seen as an optimization, not a replacement for the refresh interval. There can always be situations where the latency goes up to the refresh interval.

With that conclusion I don't think we can do anything about this, unless someone has a brilliant suggestion?

[xeppaka]

Hmm, I thought that pubsub is used just to trigger the standard query to DB for a new events (I mean that no event is sent via pubsub). So maybe we can trigger this query in case "too far ahead" check is positive (the one that is done with refresh-interval period)?

[patriknw]

Hmm, I thought that pubsub is used just to trigger the standard query to DB for a new events

No, that's the Cassandra plugin. For r2dbc we send the events and they are processed without addition query if sequence numbers and such line up with expectations.

So maybe we can trigger this query in case "too far ahead" check is positive

That might work as a fallback. I'll take a look at if we can trigger more frequent query based on that.

[xeppaka]

That might work as a fallback. I'll take a look at if we can trigger more frequent query based on that.

That would be nice, thanks!