Async Projection Event Loss

[zhenyi2697]

Hi, we have encountered some strange behavior regarding events in async projection being lost. The behavior we noticed is:

• Event successfully created and updated in DB in mt_events table
• Our sync projection successfully executed for the event
• Async Projections logging shows that the batch successfully updated with message: Shard '{ProjectionShardIdentity}': Executed updates for {Range} and the correct event sequence
• However, we can't find any logging inside the async projection action method and nothing is executed inside the async projections for the particular event.
• We cannot find any additional information or error logs neither for the projection nor for the events.

Marten Version: 6.0.5
Async Daemon Type: HotCold
We have 2 instances running Marten as well.

Does anyone has any ideas on:

1. What could possibly cause the async event not being executed?
2. What additional logging can we implement to have better visibility on the async event lifecycle?

Thank you.

[Hawxy]

At minimum I'd suggesting updating to 6.4.2, ideally to v7. There's been a significantly amount of bug fixes around flakey daemon operations in the last year, plus some additional logging added for some weird edge-cases.