Merge branch 'release/0.3.0'

Author: johnynek

.travis.yml

@@ -1,4 +1,4 @@
 language: scala
 scala:
   - 2.10.0
-  - 2.9.2
+  - 2.9.3

CHANGES.md

@@ -1,5 +1,28 @@
 # chill #
 
+### 0.3.0
+* Lots of refactoring around configuration (see chill.config in chill-java)
+* Issue #94 KryoPool for pooling Kryo instances and Output buffers
+* Issue #85 upgrade to bijection 0.5.2 and scala 2.9.3
+* Issue #78 Some, Left, Right serializers in chill(-scala)
+* Issue #70 create chill-java for Java only code
+* Issue #67 add chill-storm
+* Issue #65, #71 import cascading.kryo as chill-hadoop
+* Issue #64 fix KryoInjection to be from Any to Array[Byte]
+* Issues #61, #63 handle the case of small Map/Set in scala
+* Issue #57 use CanBuildFrom in Traversable serialization
+
+Contributors:
+
+94 commits
+* P. Oscar Boykin: 46 commits
+* Sam Ritchie: 38 commits
+* ryanlecompte: 5 commits
+* Ngoc Dao: 2 commits
+* Dao Ngoc: 1 commits
+* Michael Schmitz: 1 commits
+* Julien Le Dem: 1 commits
+
 ### 0.2.3
 * Update to Kryo 2.21
 * Update to Bijection 0.4.0

README.md

@@ -1,8 +1,73 @@
 ## Chill [![Build Status](https://secure.travis-ci.org/twitter/chill.png)](http://travis-ci.org/twitter/chill)
 
-Scala extensions for the [Kryo serialization library](http://code.google.com/p/kryo/).
+Extensions for the [Kryo serialization library](http://code.google.com/p/kryo/) including
+serializers and a set of classes to ease configuration of Kryo in systems like Hadoop, Storm, 
+Akka, etc.
 
-Chill provides a a number of Kryo serializers and an Option-like type called the MeatLocker. The MeatLocker allows you to box Kryo-serializable objects and deserialize them lazily on the first call to `get`:
+Chill has a set of subprojects: chill-java, chill-hadoop, chill-storm and chill-scala.  Other than
+chill-scala, all these projects are written in Java so they are easy to use on any JVM platform.
+
+## Chill-Java
+
+The chill-java package includes the `KryoInstantiator` class (factory for Kryo instances)
+and the `IKryoRegistrar` interface (adds Serializers to a given Kryo). These two are composable
+to build instantiators that create instances of Kryo that have the options and serializers you
+need. The benefit of this over a direct Kryo instance is that a Kryo instance is mutable and not
+serializable, which limits the safety and reusability of code that works directly with them.
+
+To deserialize or serialize easily, look at `KryoPool`:
+
+```java
+int POOL_SIZE = 10;
+KryoPool kryo = KryoPool.withByteArrayOutputStream(POOL_SIZE, new KryoInstantiator());
+byte[] ser = kryo.toBytesWithClass(myObj);
+Object deserObj = kryo.fromBytes(myObj);
+```
+
+The KryoPool is a thread-safe way to share Kryo instances and temporary output buffers.
+
+### Chill Config
+
+Hadoop, Storm, and Akka all use a configuration that is basically equivalent to a `Map[String,
+String]`. The `com.twitter.chill.config` package makes it easy to build up `KryoInstantiator`
+instances given a Config instance, which is an abstract class acting as a thin wrapper over
+whatever configuration data the system, such as Hadoop, Storm or Akka, might give.
+
+To configure a KryoInstantiator use `ConfiguredInstantiator` with either reflection,
+which takes a class name and instantiates that KryoInstantiator, or an instance of KryoInstantiator
+and serializes that instance to use later:
+```scala
+class TestInst extends KryoInstantiator { override def newKryo = sys.error("blow up") }
+
+// A new Config:
+val conf = new JavaMapConfig
+// Set-up class-based reflection of our instantiator:
+ConfiguredInstantiator.setReflect(conf, classOf[TestInst])
+val cci = new ConfiguredInstantiator(conf)
+cci.newKryo // uses TestInst
+//Or serialize a particular instance into the config to use later (or another node):
+
+ConfiguredInstantiator.setSerialized(conf, new TestInst)
+val cci2 = new ConfiguredInstantiator(conf)
+cci2.newKryo // uses the particular instance we passed above
+```
+
+## Chill in Scala
+
+Scala classes often have a number of properties that distinguish them from usual Java classes. Often
+scala classes are immutable, and thus have no zero argument constructor. Secondly, `object` in scala is
+a singleton that needs to be carefully serialized. Additionally, scala classes often have synthetic
+(compiler generated) fields that need to be serialized, and by default Kryo does not serialize
+those.
+
+In addition to a `ScalaKryoInstantiator` which generates Kryo instances with options suitable for
+scala, chill provides a number of Kryo serializers for standard scala classes (see below).
+
+### The MeatLocker
+
+Many existing systems use Java serialization. MeatLocker is an object that wraps a given instance
+using Kryo serialization internally, but the MeatLocker itself is Java serializable.
+The MeatLocker allows you to box Kryo-serializable objects and deserialize them lazily on the first call to `get`:
 
 ```scala
 import com.twitter.chill.MeatLocker
@@ -16,24 +81,17 @@ box.get == boxedItem.get // true!
 
 To retrieve the boxed item without caching the deserialized value, use `meatlockerInstance.copy`.
 
-To serialize to bytes and deserialize from bytes:
-
-```scala
-import com.twitter.chill.KryoInjection
-
-val bytes:  Array[Byte]    = KryoInjection(someItem)
-val option: Option[AnyRef] = KryoInjection.invert(bytes)  // None is returned on failure
-```
-
-### Handled classes
+### Serializers for Scala classes
 
-Chill provides support for singletons, scala Objects and the following types:
+These are found in the `chill-scala` directory in the chill jar (originally this project was
+only scala serializers).  Chill provides support for singletons, scala Objects and the following types:
 
 * Scala primitives
   * scala.Enumeration values
   * scala.Symbol
   * scala.reflect.Manifest
   * scala.reflect.ClassManifest
+  * scala.Function[0-22] closure cleaning (removing unused `$outer` references).
 * Collections and sequences
   * scala.collection.immutable.Map
   * scala.collection.immutable.List
@@ -42,9 +100,41 @@ Chill provides support for singletons, scala Objects and the following types:
   * scala.collection.mutable.{Map, Set, Buffer, WrappedArray}
   * all 22 scala tuples
 
+## Chill-bijection
+[Bijections and Injections](https://github.com/twitter/bijection)
+are useful when considering serialization. If you have an Injection from
+`T` to `Array[Byte]` you have a serialization.  Additionally, if you have a Bijection between `A`
+and `B`, and a serialization for `B`, then you have a serialization for `A`.  See
+`BijectionEnrichedKryo` for easy interop between bijection and chill.
+
+### KryoInjection: easy serialization to byte Arrays
+
+KryoInjection is an injection from `Any` to `Array[Byte]`. To serialize using it:
+
+```scala
+import com.twitter.chill.KryoInjection
+
+val bytes:  Array[Byte]    = KryoInjection(someItem)
+val tryDecode: scala.util.Try[Any] = KryoInjection.invert(bytes)
+```
+
+KryoInjection can be composed with Bijections and Injections from `com.twitter.bijection`.
+
 ## Maven
 
-Current version is `0.2.3`. groupid=`"com.twitter"` artifact=`"chill_2.9.2"` or artifact=`"chill_2.10"`.
+Chill modules are available on Maven Central. The current groupid and version for all modules is, respectively, `"com.twitter"` and  `0.3.0`.
+
+Current published artifacts are
+
+* `chill-java`
+* `chill-storm`
+* `chill-hadoop`
+* `chill_2.9.3`
+* `chill_2.10`
+* `chill-bijection_2.9.3`
+* `chill-bijection_2.10`
+
+The suffix denotes the scala version.
 
 ## Authors
 

chill-bijection/src/main/scala/com/twitter/chill/BijectionEnrichedKryo.scala

@@ -0,0 +1,69 @@
+/*
+Copyright 2012 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.twitter.chill
+
+import com.twitter.bijection.Injection
+import com.twitter.bijection.{ Bufferable, Bijection, ImplicitBijection, Injection }
+
+object BijectionEnrichedKryo {
+  implicit def enrich(k: Kryo): BijectionEnrichedKryo = new BijectionEnrichedKryo(k)
+
+  /** Use a bijection[A,B] then the KSerializer on B
+   */
+  def viaBijection[A,B](kser: KSerializer[B])(implicit bij: ImplicitBijection[A,B], cmf: ClassManifest[B]): KSerializer[A] =
+    new KSerializer[A] {
+      def write(k: Kryo, out: Output, obj: A) { kser.write(k, out, bij(obj)) }
+      def read(k: Kryo, in: Input, cls: Class[A]) =
+        bij.invert(kser.read(k, in, cmf.erasure.asInstanceOf[Class[B]]))
+    }
+
+  def viaBufferable[T](implicit b: Bufferable[T]): KSerializer[T] =
+    InjectiveSerializer.asKryo[T](Bufferable.injectionOf[T])
+}
+
+class BijectionEnrichedKryo(k: Kryo) {
+
+  def injectionForClass[T](implicit inj: Injection[T, Array[Byte]], cmf: ClassManifest[T]): Kryo = {
+    k.register(cmf.erasure, InjectiveSerializer.asKryo[T])
+    k
+  }
+
+  def injectionForSubclass[T](implicit inj: Injection[T, Array[Byte]], cmf: ClassManifest[T]): Kryo = {
+    k.addDefaultSerializer(cmf.erasure, InjectiveSerializer.asKryo[T])
+    k
+  }
+
+  def bufferableForClass[T](implicit b: Bufferable[T], cmf: ClassManifest[T]): Kryo = {
+    k.register(cmf.erasure, BijectionEnrichedKryo.viaBufferable[T])
+    k
+  }
+
+  /** B has to already be registered, then use the KSerializer[B] to create KSerialzer[A]
+   */
+  def forClassViaBijection[A,B](implicit bij: ImplicitBijection[A,B], acmf: ClassManifest[A], bcmf: ClassManifest[B]): Kryo = {
+    val kserb = k.getSerializer(bcmf.erasure).asInstanceOf[KSerializer[B]]
+    k.register(acmf.erasure, BijectionEnrichedKryo.viaBijection[A,B](kserb))
+    k
+  }
+
+  /** Helpful override to alleviate rewriting types. */
+  def forClassViaBijection[A,B](bij: Bijection[A,B])(implicit acmf: ClassManifest[A], bcmf: ClassManifest[B]): Kryo = {
+    implicit def implicitBij = bij
+    this.forClassViaBijection[A, B]
+  }
+
+}

chill-bijection/src/main/scala/com/twitter/chill/InjectionRegistrar.scala

@@ -0,0 +1,48 @@
+package com.twitter.chill
+
+import com.twitter.bijection.Injection
+
+// need the root, because java looks like chill.java to scalac. :(
+import _root_.java.io.Serializable
+
+/**
+ *  @author Sam Ritchie
+ *  @author Oscar Boykin
+ *
+ * Convenience class that holds both a Class[T] and an Injection from
+ * type T to Array[Byte].
+ */
+
+object InjectionRegistrar {
+  def apply[T](klass: Class[T], injection: Injection[T, Array[Byte]]): InjectionRegistrar[T] =
+    new InjectionRegistrar(klass, injection)
+}
+
+class InjectionRegistrar[T](val klass: Class[T], @transient b: Injection[T, Array[Byte]])
+  extends IKryoRegistrar with Serializable {
+  protected val bBox = MeatLocker(b)
+
+  implicit def injection = bBox.copy
+
+  def apply(k: Kryo) {
+    if(!k.alreadyRegistered(klass)) {
+      k.register(klass, InjectiveSerializer.asKryo[T])
+    }
+  }
+}
+
+object InjectionDefaultRegistrar {
+  def apply[T](klass: Class[T], injection: Injection[T, Array[Byte]]): InjectionDefaultRegistrar[T] =
+    new InjectionDefaultRegistrar(klass, injection)
+}
+
+class InjectionDefaultRegistrar[T](klass: Class[T], @transient b: Injection[T, Array[Byte]])
+  extends InjectionRegistrar(klass, b) {
+  override def apply(k: Kryo) {
+    if(!k.alreadyRegistered(klass)) {
+      k.addDefaultSerializer(klass, InjectiveSerializer.asKryo[T])
+      k.register(klass)
+    }
+  }
+}
+

chill-scala/src/main/scala/com/twitter/chill/InjectiveSerializer.scala chill-bijection/src/main/scala/com/twitter/chill/InjectiveSerializer.scala

@@ -1,7 +1,5 @@
 package com.twitter.chill
 
-import com.esotericsoftware.kryo.{ Kryo, Serializer => KSerializer }
-import com.esotericsoftware.kryo.io.{ Input, Output }
 import com.twitter.bijection.Injection
 
 /**
@@ -29,11 +27,6 @@ class InjectiveSerializer[T] private (injection: Injection[T, Array[Byte]]) exte
   def read(kser: Kryo, in: Input, cls: Class[T]): T = {
     val bytes = new Array[Byte](in.readInt(true))
     in.readBytes(bytes)
-    injection.invert(bytes) match {
-      case Some(t) => t
-      case None => throw new RuntimeException(
-        "Could not deserialize instance of " + cls.getName
-      )
-    }
+    injection.invert(bytes).get
   }
 }

chill-bijection/src/main/scala/com/twitter/chill/KryoInjection.scala

@@ -0,0 +1,65 @@
+/*
+Copyright 2012 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.twitter.chill
+
+import com.twitter.bijection.{ Bijection, Injection, Inversion }
+
+import _root_.java.io.Serializable
+
+/**
+ * A default KryoInjection that uses the ScalaKryoInstantiator with
+ * ByteArrayOutputStream as the backing buffer
+ */
+object KryoInjection extends Injection[Any, Array[Byte]] {
+  def apply(obj: Any): Array[Byte] = ScalaKryoInstantiator.defaultPool.toBytesWithClass(obj)
+  def invert(b: Array[Byte]) = Inversion.attempt(b) {
+    ScalaKryoInstantiator.defaultPool.fromBytes(_)
+  }
+
+  /**
+   * Create a new KryoInjection instance that serializes items using
+   * the supplied KryoPool instance. For this to be serializable, you
+   * need the call by name semantics
+   */
+  def instance(kryoPool: => KryoPool): Injection[Any, Array[Byte]] =
+    new KryoInjectionInstance(kryoPool)
+
+  /** Creates a small pool (size 2) and uses it as an Injection
+   *  Note the implicit in the package from () => Kryo to KryoInstatiator.
+   *  It is ESSENTIAL that this function is allocating new Kryos, or we will
+   *  not be thread-safe
+   */
+  def instance(ki: KryoInstantiator, poolSize: Int = 2): Injection[Any, Array[Byte]] =
+    new KryoInjectionInstance(KryoPool.withByteArrayOutputStream(poolSize, ki))
+}
+
+/**
+ * Use this if you want to control the KryoPool.
+ * This is thread-safe since the KryoPool is
+ */
+class KryoInjectionInstance(lazyKryoP: => KryoPool) extends Injection[Any, Array[Byte]] {
+  private val mutex = new AnyRef with Serializable // some serializable object
+  @transient private var kpool: KryoPool = null
+
+  private def kryoP: KryoPool = mutex.synchronized {
+    if(null == kpool) { kpool = lazyKryoP }
+    kpool
+  }
+
+  def apply(obj: Any): Array[Byte] = kryoP.toBytesWithClass(obj)
+  def invert(b: Array[Byte]) = Inversion.attempt(b) { kryoP.fromBytes(_) }
+}