Fallback Id handler to handle migration from 8-byte ids to 4-byte schema ids #7106
+153
−5
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
During the testing of quarkusio/quarkus#51401, we realised that the migration to v3 involves handling the change of schema ids from using 8-bytes to 4-bytes.
While there are distinct IdHandler implementations available to the client application to use, there isn't any easy migration path to use with message processing applications producing/consuming on topics (eg. Kafka).
The problem of an active topic is explained here : quarkusio/quarkus#51401 (comment)
This change proposes an
OptimisticFallbackIdHandlerto write 4-byte ids and read both 4 and 8 byte ids to be used in a migration scenario.Because the payload format doesn't include any format versionning, this IdHandler makes the following assumption to distinguish between 4-bytes 8-bytes ids: The schema ids are greater than 0 and smaller than integer max value.
The change requires some non-breaking modifications to
AbstractDeserializer, and maybe to theIdHandlerinterface that can be discussed.Additionally, I couldn't figure out what to do for the Go SDK.