Kryo serializers for Entity and Event objects (and possibly TupleStore)

[cjnolet]

Working on #252 of Accumulo Recipes leads me to write serializers for Events and Entities and it got me thinking that it would be massivley useful if we had a mango-kryo module for this.

[cjnolet]

While we're at it, I'd also like to create a separate serializer for the metadata info. Assuming we use our type system to go to bytes, this shouldn't be too bad with the generic ByteArraySerializer

[eawagner]

My biggest concern with using Kryo is that in my experience if you use it you are tied to a specific version that your data was serialized with. Additionally, since Kryo doesn't exactly use interfaces, I question its source compatability between versions.

All of that was to get around to saying that I really really don't want mango to be the decider of the Kryo versions being used for other projects. This could potentially mean for instance that if we upgraged the kryo version in mango in a minor or bugfix release that things like accumulo-recipes would have to go through a potential major release as all the data would then potentially be unserializable.

Another option would be to make the kryo dependency optional/provided in maven so that it is guaranteed not to be brought in as a dependency with mango. Before we go down this route I would like us to investigate how binary compatible Kryo stays between versions. Atleast for the Serializer, Kryo, Input, and Output classes needed to write serializers. If their classes guarantee binary compatability between major versions then I would feel more comfortable with this option as we can compile mango against one version, but others can use there own version of the library.

[cjnolet]

As far as I know, the biggest Kryo incompatibility was between 1.04 and 2.x (when they basically redid the entire thing). Have you also noticed significant incompatibilities between 2.x versions?

Spark and Storm both use Kryo. I don't think it's something we can necessarily get away from and one of the utilities of this is that events and entities can natively get passed between nodes.

[eawagner]

Yeah I love Kryo and in places like Storm and Spark it makes a lot of sense, as the data is very short lived between serialization and deserialization, never persisted, and all serialization is compiled using the same version of Kryo that the framework uses.

The biggest problem that I have noticed between 2.x versions was when trying to persist Kryo serialized data. Even minor version changes in Kryo made the data unserializable.

What I don't know is how much of a stickler Kryo's API is for staying binary compatible. I do know that they don't use interfaces which means that any public method added to the class could potentially make the compiled mango binaries incompatable with a version of Kryo. By compiling and releasing a mango-kryo module we would be forcing users of it to only use Kryo versions that are binary compatible with the version mango uses. If the different Kryo versions are source compatable, atleast someone could rebuild mango with an updated Kryo version and use that, but that would be a hassle I would like to avoid if possible.

Just to avoid confusion between binary and source compatability here is what I am referring to https://blogs.oracle.com/darcy/entry/kinds_of_compatibility

[cjnolet]

Ah, I see what happened here. I don't intend on persisting the kryo. This is strictly for serializing the output of iterators so that data can be passed over the wire to the clients. A good example is in the AbstractEvaluatingIterator. This is being done currently, it' sjust not events/entities which are being serialized.

The problem is that Kryo (and other serialization mechanisms) don't work with Map<String,Object> very well at all. The generic solution is to still use the encoders underneath the Kryo serializer, which then puts the lightweight encoding back on us.

I'm more concerned @ this point with having the objects able to be passed across transports than I am with using kryo to persist. Tuple, TupleStore, Event, Entity, and IPv{4,6} are ones I can think of right away which would benefit.

I know what binary vs source compatibility is btw.

[cjnolet]

Wanted to get this going again-

I'm thinking of providing Kryo as a provided dependency (or providing an overridable version in the module). The API components that I'm using are very very basic so as long as it's >2.0, we should be good as long as it's compiled properly.

I want to pull over this class from Flowmix [1] since I now need it in Spark stuff as well as Flowmix.

[1] https://github.com/calrissian/flowmix/blob/master/src/main/java/org/calrissian/flowmix/api/kryo/EventSerializer.java