Compact type encoders

[cjnolet]

Thrift and Avro both supply really good standards for fast and lightweight compression of various different data formats for sending over the wire. For cases where we don't need lexicographically sortable types but instead just want something lightweight and compact to, for instance, place in the Value of an Accumulo key/value, I think it would be great it we could get a set of CompactTypeEncoders going.

The "Types" sunsection of the thrift paper would be a good place to start:

https://thrift.apache.org/static/files/thrift-20070401.pdf

[eawagner]

I see a few question we should answer before starting this.

1. For each type how should we encode for maximum compactness? I think this can depend entirely on the type of data. For instance large strings can be made more compact with a compression algorithm, where it can cause larger values for small strings.
2. For integer types should we look at varable length encodings ala Kryo or Thrift
3. Do we need to come up with a more compact alias representation for each of the types? For instance with the number of types that we have we can actually represent all of the types in 1 byte.

[eawagner]

After having implemented half a dozen set of encoders, I don't know how useful this will be. In each instance, I have had to tweak the encoding to the use case, while using the mango encoders where appropriate. Actually, the simple encoders in mango have been the ones that I have used the most for an actual compact string encoder.

For instance, for one of my lexicoders I have had to create an encoder which creates a more compact number encoding. The only downside is that it won't encode numbers > 10^128. For our usecases this is more than acceptable and nets us a use space savings on the encoding for most numbers.

For our over-the-wire encodings we also have a novel approach that helps number fit into the ascii text range. We simply encode all numbers into base 36 (MAX_RADIX). This usually halves the size of the data without requiring any special libraries for special handling.

For each of these we also used a 1 char alias with no special meaning. This allowed for a much smaller footprint over the wire and in storage.

While these encoding worked well for us, I doubt they would be very universal