Message versioning should be easier

[anonymous234]

I was pretty shocked to discover that Hyperion does not handle the case of adding a single extra, optional field to an existing object. This is a use case that is necessary for almost any distributed application.

I have seen the Article and the video. The solution recommended here is to manually write protobuf definitions, run the protobuf compiler to generate code to serialize each those types, then write a serializer that to call those serializers.

This is way more work than people expect to have to do! JSON serializers, while less efficient, can handle an extra field being added. And it's not an intrinsic property of "polymorphic serialization".

I noticed the https://github.com/akkadotnet/Hyperion repository says

Versioned, in this mode, Hyperion will emit both type names and field information in the output stream. This allows systems to have slightly different versions of the contract types where some fields may have been added or removed

This would really help. However after testing it by hand, using the WithVersionTolerance option it still throws an IndexOutOfRangeException if I add a field to an object (there's no documentation beyond that sentence so I don't know if that's even a bug).

Alternatively, just returning to the Newtonsoft.Json serializer could be a solution for people who don't need too much performance. Maybe that could be explained in the documentation?

On that note, is there a reason why we can't use System.Text.Json, which is the closest thing to a standard serializer .NET has? It's less efficient, and you'd still need to encode the type somewhere, but the advantage would be that all the limitations are widely understood and explained.

[Aaronontheweb]

I was pretty shocked to discover that Hyperion does not handle the case of adding a single extra, optional field to an existing object. This is a use case that is necessary for almost any distributed application.

We should probably just deprecate Hyperion at this point. It's too much of moving target trying to keep Polymorphic serialization congruent throughout all of the changes to the languages, runtimes, and end-user changes to message types to do this at runtime, really. Even Newtonsoft.Json and STJ can have trouble with this.

Message versioning is not a trivial problem to solve. This is why we recommend the Protobuf strategy today - that gives you the explicit control over how versioning is handled and helps ensure that you can follow best practices like extend-only design: https://aaronstannard.com/extend-only-design/

This is way more work than people expect to have to do!

1. People's expectations are wrong. This is a hard problem that has to be carefully considered.
2. I agree - it's more tedious than it should be.

What we are hoping to automate this with Akka.NET v1.6 using source generation https://petabridge.com/blog/akkadotnet-v1.6-roadmap/ - see the appropriate section there. But we haven't really started work in-earnest on that yet mostly due to other competing priorities.

[Aaronontheweb]

As for why Newtonsoft.Json and not STJ - it just comes down to us having to test and create a migration path for all of the ~10 years worth of data users have stored using Newtonsoft's serialization system and make sure that all works equivalently in STJ, which is a giant quality assurance surface area.

We can probably just introduce a new STJ-powered serializer and make Newtonsoft.Json available as a legacy package for folks who need that backwards compatibility, but it just hasn't been a high priority compared to trying to make schema-based serialization more convenient.