One stream, multiple SSPs, different versions & strong concurrency

[ross1296]

The base question here boils down to: If I have multiple, different SingleStreamProjections for the same stream that only care about certain events, should the Version in the JSON body of these SSPs be the same or different and should the mt_version column be the same or different?

Example
Using the IncidentService / HelpDesk example that is often used, let's pretend that you have the following events:

• IncidentLogged
• IncidentCategorised
• IncidentPrioritised
• IncidentClosed
• AgentAssignedToIncident
• AgentRespondedToIncident
• CustomerRespondedToIncident
• IncidentResolved
• ResolutionAcknowledgedByCustomer

And now let's say that you want to have multiple SSPs to serve your UI. In this scenario, let's say you have two pages on the UI: Details page & Notes page. The details page, as the name suggests, contains all the basic details of the incident. The notes page, also as the name suggests, contains all of the AgentRespondedToIncident and CustomerRespondedToIncident events

So we have an IncidentDetails projection and IncidentNotes projection. Both are registered as inline.

Our IncidentDetails projection has an Apply() method for all 9 of the events mentioned above. Our IncidentNotes projection has an Apply() method for only: IncidentLogged, AgentRespondedToIncident, CustomerRespondedToIncident

So now let's say we have this flow:

• LogIncident endpoint, which returns an IncidentLogged event
• Then we hit our CategoriseIncident endpoint, which returns an IncidentCategorised event
• Then we hit the CustomerRespondToIncident endpoint, which returns the CustomerRespondedToIncident event
• Then we hit the AgentRespondToIncident endpoint, which returns the AgentRespondedToIncident event

If we examine the mt_streams table, the stream's version is 4.

If we examine the mt_doc_incidentdetails table, both the version in the JSON body and the mt_version column have a value of 4

If we examine the mt_doc_incidentnotes table, both the version in the JSON body and the mt_version column have a value of 3.

Problem
If we have multiple SSPs for the same stream to feed pages of our UI (because we only care to get the info we need), but the versions differ due to the above example, then we have the wrong version value on the frontend when it comes to doing a POST/PUT/PATCH request to the backend, and that results in a concurrency exception.

So should SSPs for a stream have their version incremented regardless of if they have an Apply() method for all events?

Repro
I've cloned the IncidentService from the WolverineFX repo here: https://github.com/ross1296/IncidentServiceExample/

All I've done is:

• Create a new record called ShortIncidentDetails: https://github.com/ross1296/IncidentServiceExample/blob/690cf355d35b1ecddf37d0d4b8608b807d790fb4/IncidentService/Incident.cs#L78
• Create a SingleStreamProjection for ShortIncidentDetails which only does Apply() for a select few events: https://github.com/ross1296/IncidentServiceExample/blob/690cf355d35b1ecddf37d0d4b8608b807d790fb4/IncidentService/Incident.cs#L85
• Registered the SSP as inline: https://github.com/ross1296/IncidentServiceExample/blob/690cf355d35b1ecddf37d0d4b8608b807d790fb4/IncidentService/Program.cs#L28

I then make the following HTTP requests:

• POST to /incidents: https://github.com/ross1296/IncidentServiceExample/blob/690cf355d35b1ecddf37d0d4b8608b807d790fb4/IncidentService/IncidentService.http#L4
• POST to /incidents/{incidentId/categorise: https://github.com/ross1296/IncidentServiceExample/blob/690cf355d35b1ecddf37d0d4b8608b807d790fb4/IncidentService/IncidentService.http#L21

The result is:

• mt_streams version: 2
• mt_doc_incident version (data and version column): 2
• mt_doc_shortincidentdetails (data and version column): 1

Justification
I'm fully aware that I may be misunderstanding something here, or perhaps(?) this isn't idiomatic usage, but is it not reasonable to create multiple SSPs to feed UI pages in order to keep the request/response as small as possible, and also ensure strong consistency from UI > Backend?

I refer to this blog post: https://jeremydmiller.com/2023/12/05/building-a-critter-stack-application-dealing-with-concurrency/

Which states: "Marten will “automagically” set the value of a Version property of the aggregated document to the latest revision number of the event stream. This (hopefully) makes it relatively easy for systems built with Marten to transfer the current event stream revision number to user interfaces or other clients specifically to make optimistic concurrency protection easier."

But from the demonstration above, it does not appear that all SSPs have their Version property set to the latest revision number of the event stream. But instead, only by the events which the projection implements.

[jeremydmiller]

Hey man, this came up for the very, very first time last week and I admittedly told someone that this was more or less a completely bonkers use case.

I'm moving this to a discussion. I think this would be extremely hard to support. My suggestion would be to only use one single stream projection that catches every possible event type, then use a Live lifecycle for everything else and always use FetchLatest() or FetchForWriting when fetching the others.

[ross1296]

Hey @jeremydmiller, appreciate the reply and opening a discussion for this!

To touch more on your suggestion, am I right in assuming this workflow?

1. Create one single stream projection which, as you said, catches every possible event type
2. Register said SSP as inline (this likely changes use case to use case, or person to person. I typically care more about read times vs write times)
3. Have X number of query handlers / endpoints, such as GetIncidentDetails and GetIncidentNotes which both use the session.Query<T>.Select(...) syntax to select the property which we care about / project them into a DTO or anonymous object

This way we can provide the UI with the correct version number, and that same version number can then be POST/PUT to ensure strong concurrency F > B.

Apologies, as I didn't quite follow what you meant by: "then use a Live lifecycle for everything else and always use FetchLatest() or FetchForWriting when fetching the others" -- Would you be able to apply that to the example scenario mentioned in the original post?

Again, greatly appreciate opening this up for discussion! Keep up the great work.