The options parameter, which appears in a few places, is used to pass several
+ * pieces of information to the encoder. In the "higher level" methods such as encodeBytes( bytes,
+ * options ) the options parameter can be used to indicate such things as first gzipping the bytes
+ * before encoding them, not inserting linefeeds, and encoding using the URL-safe and Ordered
+ * dialects.
+ *
+ *
Note, according to RFC3548, Section 2.1,
+ * implementations should not add line feeds unless explicitly told to do so. I've got Base64 set to
+ * this behavior now, although earlier versions broke lines by default.
+ *
+ *
The constants defined in Base64 can be OR-ed together to combine options, so you might make a
+ * call like this:
+ * String encoded = Base64.encodeBytes( mybytes, Base64.GZIP | Base64.DO_BREAK_LINES );
+ *
+ *
to compress the data before encoding it and then making the output have newline characters.
+ *
+ *
v2.3.7 - Fixed subtle bug when base 64 input stream contained the value 01111111, which is
+ * an invalid base 64 character but should not throw an ArrayIndexOutOfBoundsException either.
+ * Led to discovery of mishandling (or potential for better handling) of other bad input
+ * characters. You should now get an IOException if you try decoding something that has bad
+ * characters in it.
+ *
v2.3.6 - Fixed bug when breaking lines and the final byte of the encoded string ended in
+ * the last column; the buffer was not properly shrunk and contained an extra (null) byte that
+ * made it into the string.
+ *
v2.3.5 - Fixed bug in {@link #encodeFromFile} where estimated buffer size was wrong for
+ * files of size 31, 34, and 37 bytes.
+ *
v2.3.4 - Fixed bug when working with gzipped streams whereby flushing the
+ * Base64.OutputStream closed the Base64 encoding (by padding with equals signs) too soon.
+ * Also added an option to suppress the automatic decoding of gzipped streams. Also added
+ * experimental support for specifying a class loader when using the {@link
+ * #decodeToObject(java.lang.String, int, java.lang.ClassLoader)} method.
+ *
v2.3.3 - Changed default char encoding to US-ASCII which reduces the internal Java
+ * footprint with its CharEncoders and so forth. Fixed some javadocs that were inconsistent.
+ * Removed imports and specified things like java.io.IOException explicitly inline.
+ *
v2.3.2 - Reduced memory footprint! Finally refined the "guessing" of how big the final
+ * encoded data will be so that the code doesn't have to create two output arrays: an
+ * oversized initial one and then a final, exact-sized one. Big win when using the {@link
+ * #encodeBytesToBytes(byte[])} family of methods (and not using the gzip options which uses a
+ * different mechanism with streams and stuff).
+ *
v2.3.1 - Added {@link #encodeBytesToBytes(byte[], int, int, int)} and some similar helper
+ * methods to be more efficient with memory by not returning a String but just a byte array.
+ *
v2.3 - This is not a drop-in replacement! This is two years of comments
+ * and bug fixes queued up and finally executed. Thanks to everyone who sent me stuff, and I'm
+ * sorry I wasn't able to distribute your fixes to everyone else. Much bad coding was cleaned
+ * up including throwing exceptions where necessary instead of returning null values or
+ * something similar. Here are some changes that may affect you:
+ *
+ *
Does not break lines, by default. This is to keep in compliance with RFC3548.
+ *
Throws exceptions instead of returning null values. Because some operations
+ * (especially those that may permit the GZIP option) use IO streams, there is a
+ * possiblity of an java.io.IOException being thrown. After some discussion and thought,
+ * I've changed the behavior of the methods to throw java.io.IOExceptions rather than
+ * return null if ever there's an error. I think this is more appropriate, though it
+ * will require some changes to your code. Sorry, it should have been done this way to
+ * begin with.
+ *
Removed all references to System.out, System.err, and the like. Shame on me.
+ * All I can say is sorry they were ever there.
+ *
Throws NullPointerExceptions and IllegalArgumentExceptions as needed such as
+ * when passed arrays are null or offsets are invalid.
+ *
Cleaned up as much javadoc as I could to avoid any javadoc warnings. This was
+ * especially annoying before for people who were thorough in their own projects and
+ * then had gobs of javadoc warnings on this file.
+ *
+ *
v2.2.1 - Fixed bug using URL_SAFE and ORDERED encodings. Fixed bug when using very small
+ * files (~< 40 bytes).
+ *
v2.2 - Added some helper methods for encoding/decoding directly from one file to the next.
+ * Also added a main() method to support command line encoding/decoding from one file to the
+ * next. Also added these Base64 dialects:
+ *
+ *
The default is RFC3548 format.
+ *
Calling Base64.setFormat(Base64.BASE64_FORMAT.URLSAFE_FORMAT) generates URL and file
+ * name friendly format as described in Section 4 of RFC3548.
+ * http://www.faqs.org/rfcs/rfc3548.html
+ *
Calling Base64.setFormat(Base64.BASE64_FORMAT.ORDERED_FORMAT) generates URL and file
+ * name friendly format that preserves lexical ordering as described in
+ * http://www.faqs.org/qa/rfcc-1940.html
+ *
+ * Special thanks to Jim Kellerman at http://www.powerset.com/ for contributing the new
+ * Base64 dialects.
+ *
v2.1 - Cleaned up javadoc comments and unused variables and methods. Added some convenience
+ * methods for reading and writing to and from files.
+ *
v2.0.2 - Now specifies UTF-8 encoding in places where the code fails on systems with other
+ * encodings (like EBCDIC).
+ *
v2.0.1 - Fixed an error when decoding a single byte, that is, when the encoded data was a
+ * single byte.
+ *
v2.0 - I got rid of methods that used booleans to set options. Now everything is more
+ * consolidated and cleaner. The code now detects when data that's being decoded is
+ * gzip-compressed and will decompress it automatically. Generally things are cleaner. You'll
+ * probably have to change some method calls that you were making to support the new options
+ * format (ints that you "OR" together).
+ *
v1.5.1 - Fixed bug when decompressing and decoding to a byte[] using
+ * decode( String s, boolean gzipCompressed ). Added the ability to "suspend" encoding
+ * in the Output Stream so you can turn on and off the encoding if you need to embed base64
+ * data in an otherwise "normal" stream (like an XML file).
+ *
v1.5 - Output stream pases on flush() command but doesn't do anything itself. This helps
+ * when using GZIP streams. Added the ability to GZip-compress objects before encoding them.
+ *
v1.4 - Added helper methods to read/write files.
+ *
v1.3.6 - Fixed OutputStream.flush() so that 'position' is reset.
+ *
v1.3.5 - Added flag to turn on and off line breaks. Fixed bug in input stream where last
+ * buffer being read, if not completely full, was not returned.
+ *
v1.3.4 - Fixed when "improperly padded stream" error was thrown at the wrong time.
+ *
v1.3.3 - Fixed I/O streams which were totally messed up.
+ *
+ *
+ *
I am placing this code in the Public Domain. Do with it as you will. This software comes with
+ * no guarantees or warranties but with plenty of well-wishing instead! Please visit http://iharder.net/base64 periodically to check for updates
+ * or to contribute improvements.
+ *
+ * @author Robert Harder
+ * @author rob@iharder.net
+ * @version 2.3.7
+ */
+public class Base64 {
+
+ /* ******** P U B L I C F I E L D S ******** */
+
+ /** No options specified. Value is zero. */
+ public static final int NO_OPTIONS = 0;
+
+ /** Specify encoding in first bit. Value is one. */
+ public static final int ENCODE = 1;
+
+ /** Specify decoding in first bit. Value is zero. */
+ public static final int DECODE = 0;
+
+ /** Specify that data should be gzip-compressed in second bit. Value is two. */
+ public static final int GZIP = 2;
+
+ /** Specify that gzipped data should not be automatically gunzipped. */
+ public static final int DONT_GUNZIP = 4;
+
+ /** Do break lines when encoding. Value is 8. */
+ public static final int DO_BREAK_LINES = 8;
+
+ /**
+ * Encode using Base64-like encoding that is URL- and Filename-safe as described in Section 4 of
+ * RFC3548: http://www.faqs.org/rfcs/rfc3548.html. It is
+ * important to note that data encoded this way is not officially valid Base64, or at the
+ * very least should not be called Base64 without also specifying that is was encoded using the
+ * URL- and Filename-safe dialect.
+ */
+ public static final int URL_SAFE = 16;
+
+ /**
+ * Encode using the special "ordered" dialect of Base64 described here: http://www.faqs.org/qa/rfcc-1940.html.
+ */
+ public static final int ORDERED = 32;
+
+ /* ******** P R I V A T E F I E L D S ******** */
+
+ /** Maximum line length (76) of Base64 output. */
+ private static final int MAX_LINE_LENGTH = 76;
+
+ /** The equals sign (=) as a byte. */
+ private static final byte EQUALS_SIGN = (byte) '=';
+
+ /** The new line character (\n) as a byte. */
+ private static final byte NEW_LINE = (byte) '\n';
+
+ /** Preferred encoding. */
+ private static final String PREFERRED_ENCODING = "US-ASCII";
+
+ private static final byte WHITE_SPACE_ENC = -5; // Indicates white space in encoding
+ private static final byte EQUALS_SIGN_ENC = -1; // Indicates equals sign in encoding
+
+ /* ******** S T A N D A R D B A S E 6 4 A L P H A B E T ******** */
+
+ /** The 64 valid Base64 values. */
+ /* Host platform me be something funny like EBCDIC, so we hardcode these values. */
+ private static final byte[] _STANDARD_ALPHABET = {
+ (byte) 'A', (byte) 'B', (byte) 'C', (byte) 'D', (byte) 'E', (byte) 'F', (byte) 'G', (byte) 'H',
+ (byte) 'I', (byte) 'J', (byte) 'K', (byte) 'L', (byte) 'M', (byte) 'N', (byte) 'O', (byte) 'P',
+ (byte) 'Q', (byte) 'R', (byte) 'S', (byte) 'T', (byte) 'U', (byte) 'V', (byte) 'W', (byte) 'X',
+ (byte) 'Y', (byte) 'Z', (byte) 'a', (byte) 'b', (byte) 'c', (byte) 'd', (byte) 'e', (byte) 'f',
+ (byte) 'g', (byte) 'h', (byte) 'i', (byte) 'j', (byte) 'k', (byte) 'l', (byte) 'm', (byte) 'n',
+ (byte) 'o', (byte) 'p', (byte) 'q', (byte) 'r', (byte) 's', (byte) 't', (byte) 'u', (byte) 'v',
+ (byte) 'w', (byte) 'x', (byte) 'y', (byte) 'z', (byte) '0', (byte) '1', (byte) '2', (byte) '3',
+ (byte) '4', (byte) '5', (byte) '6', (byte) '7', (byte) '8', (byte) '9', (byte) '+', (byte) '/'
+ };
+
+ /**
+ * Translates a Base64 value to either its 6-bit reconstruction value or a negative number
+ * indicating some other meaning.
+ */
+ private static final byte[] _STANDARD_DECODABET = {
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 0 - 8
+ -5,
+ -5, // Whitespace: Tab and Linefeed
+ -9,
+ -9, // Decimal 11 - 12
+ -5, // Whitespace: Carriage Return
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 14 - 26
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 27 - 31
+ -5, // Whitespace: Space
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 33 - 42
+ 62, // Plus sign at decimal 43
+ -9,
+ -9,
+ -9, // Decimal 44 - 46
+ 63, // Slash at decimal 47
+ 52,
+ 53,
+ 54,
+ 55,
+ 56,
+ 57,
+ 58,
+ 59,
+ 60,
+ 61, // Numbers zero through nine
+ -9,
+ -9,
+ -9, // Decimal 58 - 60
+ -1, // Equals sign at decimal 61
+ -9,
+ -9,
+ -9, // Decimal 62 - 64
+ 0,
+ 1,
+ 2,
+ 3,
+ 4,
+ 5,
+ 6,
+ 7,
+ 8,
+ 9,
+ 10,
+ 11,
+ 12,
+ 13, // Letters 'A' through 'N'
+ 14,
+ 15,
+ 16,
+ 17,
+ 18,
+ 19,
+ 20,
+ 21,
+ 22,
+ 23,
+ 24,
+ 25, // Letters 'O' through 'Z'
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 91 - 96
+ 26,
+ 27,
+ 28,
+ 29,
+ 30,
+ 31,
+ 32,
+ 33,
+ 34,
+ 35,
+ 36,
+ 37,
+ 38, // Letters 'a' through 'm'
+ 39,
+ 40,
+ 41,
+ 42,
+ 43,
+ 44,
+ 45,
+ 46,
+ 47,
+ 48,
+ 49,
+ 50,
+ 51, // Letters 'n' through 'z'
+ -9,
+ -9,
+ -9,
+ -9,
+ -9 // Decimal 123 - 127
+ ,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 128 - 139
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 140 - 152
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 153 - 165
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 166 - 178
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 179 - 191
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 192 - 204
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 205 - 217
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 218 - 230
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 231 - 243
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9 // Decimal 244 - 255
+ };
+
+ /* ******** U R L S A F E B A S E 6 4 A L P H A B E T ******** */
+
+ /**
+ * Used in the URL- and Filename-safe dialect described in Section 4 of RFC3548: http://www.faqs.org/rfcs/rfc3548.html. Notice
+ * that the last two bytes become "hyphen" and "underscore" instead of "plus" and "slash."
+ */
+ private static final byte[] _URL_SAFE_ALPHABET = {
+ (byte) 'A', (byte) 'B', (byte) 'C', (byte) 'D', (byte) 'E', (byte) 'F', (byte) 'G', (byte) 'H',
+ (byte) 'I', (byte) 'J', (byte) 'K', (byte) 'L', (byte) 'M', (byte) 'N', (byte) 'O', (byte) 'P',
+ (byte) 'Q', (byte) 'R', (byte) 'S', (byte) 'T', (byte) 'U', (byte) 'V', (byte) 'W', (byte) 'X',
+ (byte) 'Y', (byte) 'Z', (byte) 'a', (byte) 'b', (byte) 'c', (byte) 'd', (byte) 'e', (byte) 'f',
+ (byte) 'g', (byte) 'h', (byte) 'i', (byte) 'j', (byte) 'k', (byte) 'l', (byte) 'm', (byte) 'n',
+ (byte) 'o', (byte) 'p', (byte) 'q', (byte) 'r', (byte) 's', (byte) 't', (byte) 'u', (byte) 'v',
+ (byte) 'w', (byte) 'x', (byte) 'y', (byte) 'z', (byte) '0', (byte) '1', (byte) '2', (byte) '3',
+ (byte) '4', (byte) '5', (byte) '6', (byte) '7', (byte) '8', (byte) '9', (byte) '-', (byte) '_'
+ };
+
+ /** Used in decoding URL- and Filename-safe dialects of Base64. */
+ private static final byte[] _URL_SAFE_DECODABET = {
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 0 - 8
+ -5,
+ -5, // Whitespace: Tab and Linefeed
+ -9,
+ -9, // Decimal 11 - 12
+ -5, // Whitespace: Carriage Return
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 14 - 26
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 27 - 31
+ -5, // Whitespace: Space
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 33 - 42
+ -9, // Plus sign at decimal 43
+ -9, // Decimal 44
+ 62, // Minus sign at decimal 45
+ -9, // Decimal 46
+ -9, // Slash at decimal 47
+ 52,
+ 53,
+ 54,
+ 55,
+ 56,
+ 57,
+ 58,
+ 59,
+ 60,
+ 61, // Numbers zero through nine
+ -9,
+ -9,
+ -9, // Decimal 58 - 60
+ -1, // Equals sign at decimal 61
+ -9,
+ -9,
+ -9, // Decimal 62 - 64
+ 0,
+ 1,
+ 2,
+ 3,
+ 4,
+ 5,
+ 6,
+ 7,
+ 8,
+ 9,
+ 10,
+ 11,
+ 12,
+ 13, // Letters 'A' through 'N'
+ 14,
+ 15,
+ 16,
+ 17,
+ 18,
+ 19,
+ 20,
+ 21,
+ 22,
+ 23,
+ 24,
+ 25, // Letters 'O' through 'Z'
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 91 - 94
+ 63, // Underscore at decimal 95
+ -9, // Decimal 96
+ 26,
+ 27,
+ 28,
+ 29,
+ 30,
+ 31,
+ 32,
+ 33,
+ 34,
+ 35,
+ 36,
+ 37,
+ 38, // Letters 'a' through 'm'
+ 39,
+ 40,
+ 41,
+ 42,
+ 43,
+ 44,
+ 45,
+ 46,
+ 47,
+ 48,
+ 49,
+ 50,
+ 51, // Letters 'n' through 'z'
+ -9,
+ -9,
+ -9,
+ -9,
+ -9 // Decimal 123 - 127
+ ,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 128 - 139
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 140 - 152
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 153 - 165
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 166 - 178
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 179 - 191
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 192 - 204
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 205 - 217
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 218 - 230
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 231 - 243
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9 // Decimal 244 - 255
+ };
+
+ /* ******** O R D E R E D B A S E 6 4 A L P H A B E T ******** */
+
+ /**
+ * I don't get the point of this technique, but someone requested it, and it is described here: http://www.faqs.org/qa/rfcc-1940.html.
+ */
+ private static final byte[] _ORDERED_ALPHABET = {
+ (byte) '-', (byte) '0', (byte) '1', (byte) '2', (byte) '3', (byte) '4', (byte) '5', (byte) '6',
+ (byte) '7', (byte) '8', (byte) '9', (byte) 'A', (byte) 'B', (byte) 'C', (byte) 'D', (byte) 'E',
+ (byte) 'F', (byte) 'G', (byte) 'H', (byte) 'I', (byte) 'J', (byte) 'K', (byte) 'L', (byte) 'M',
+ (byte) 'N', (byte) 'O', (byte) 'P', (byte) 'Q', (byte) 'R', (byte) 'S', (byte) 'T', (byte) 'U',
+ (byte) 'V', (byte) 'W', (byte) 'X', (byte) 'Y', (byte) 'Z', (byte) '_', (byte) 'a', (byte) 'b',
+ (byte) 'c', (byte) 'd', (byte) 'e', (byte) 'f', (byte) 'g', (byte) 'h', (byte) 'i', (byte) 'j',
+ (byte) 'k', (byte) 'l', (byte) 'm', (byte) 'n', (byte) 'o', (byte) 'p', (byte) 'q', (byte) 'r',
+ (byte) 's', (byte) 't', (byte) 'u', (byte) 'v', (byte) 'w', (byte) 'x', (byte) 'y', (byte) 'z'
+ };
+
+ /** Used in decoding the "ordered" dialect of Base64. */
+ private static final byte[] _ORDERED_DECODABET = {
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 0 - 8
+ -5,
+ -5, // Whitespace: Tab and Linefeed
+ -9,
+ -9, // Decimal 11 - 12
+ -5, // Whitespace: Carriage Return
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 14 - 26
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 27 - 31
+ -5, // Whitespace: Space
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 33 - 42
+ -9, // Plus sign at decimal 43
+ -9, // Decimal 44
+ 0, // Minus sign at decimal 45
+ -9, // Decimal 46
+ -9, // Slash at decimal 47
+ 1,
+ 2,
+ 3,
+ 4,
+ 5,
+ 6,
+ 7,
+ 8,
+ 9,
+ 10, // Numbers zero through nine
+ -9,
+ -9,
+ -9, // Decimal 58 - 60
+ -1, // Equals sign at decimal 61
+ -9,
+ -9,
+ -9, // Decimal 62 - 64
+ 11,
+ 12,
+ 13,
+ 14,
+ 15,
+ 16,
+ 17,
+ 18,
+ 19,
+ 20,
+ 21,
+ 22,
+ 23, // Letters 'A' through 'M'
+ 24,
+ 25,
+ 26,
+ 27,
+ 28,
+ 29,
+ 30,
+ 31,
+ 32,
+ 33,
+ 34,
+ 35,
+ 36, // Letters 'N' through 'Z'
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 91 - 94
+ 37, // Underscore at decimal 95
+ -9, // Decimal 96
+ 38,
+ 39,
+ 40,
+ 41,
+ 42,
+ 43,
+ 44,
+ 45,
+ 46,
+ 47,
+ 48,
+ 49,
+ 50, // Letters 'a' through 'm'
+ 51,
+ 52,
+ 53,
+ 54,
+ 55,
+ 56,
+ 57,
+ 58,
+ 59,
+ 60,
+ 61,
+ 62,
+ 63, // Letters 'n' through 'z'
+ -9,
+ -9,
+ -9,
+ -9,
+ -9 // Decimal 123 - 127
+ ,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 128 - 139
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 140 - 152
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 153 - 165
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 166 - 178
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 179 - 191
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 192 - 204
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 205 - 217
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 218 - 230
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9, // Decimal 231 - 243
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9,
+ -9 // Decimal 244 - 255
+ };
+
+ /* ******** D E T E R M I N E W H I C H A L H A B E T ******** */
+
+ /**
+ * Returns one of the _SOMETHING_ALPHABET byte arrays depending on the options specified. It's
+ * possible, though silly, to specify ORDERED and URLSAFE in which case one of them will be
+ * picked, though there is no guarantee as to which one will be picked.
+ */
+ private static final byte[] getAlphabet(int options) {
+ if ((options & URL_SAFE) == URL_SAFE) {
+ return _URL_SAFE_ALPHABET;
+ } else if ((options & ORDERED) == ORDERED) {
+ return _ORDERED_ALPHABET;
+ } else {
+ return _STANDARD_ALPHABET;
+ }
+ } // end getAlphabet
+
+ /**
+ * Returns one of the _SOMETHING_DECODABET byte arrays depending on the options specified. It's
+ * possible, though silly, to specify ORDERED and URL_SAFE in which case one of them will be
+ * picked, though there is no guarantee as to which one will be picked.
+ */
+ private static final byte[] getDecodabet(int options) {
+ if ((options & URL_SAFE) == URL_SAFE) {
+ return _URL_SAFE_DECODABET;
+ } else if ((options & ORDERED) == ORDERED) {
+ return _ORDERED_DECODABET;
+ } else {
+ return _STANDARD_DECODABET;
+ }
+ } // end getAlphabet
+
+ /** Defeats instantiation. */
+ private Base64() {}
+
+ /* ******** E N C O D I N G M E T H O D S ******** */
+
+ /**
+ * Encodes up to the first three bytes of array threeBytes and returns a four-byte
+ * array in Base64 notation. The actual number of significant bytes in your array is given by
+ * numSigBytes. The array threeBytes needs only be as big as
+ * numSigBytes. Code can reuse a byte array by passing a four-byte array as
+ * b4.
+ *
+ * @param b4 A reusable byte array to reduce array instantiation
+ * @param threeBytes the array to convert
+ * @param numSigBytes the number of significant bytes in your array
+ * @return four byte array in Base64 notation.
+ * @since 1.5.1
+ */
+ private static byte[] encode3to4(byte[] b4, byte[] threeBytes, int numSigBytes, int options) {
+ encode3to4(threeBytes, 0, numSigBytes, b4, 0, options);
+ return b4;
+ } // end encode3to4
+
+ /**
+ * Encodes up to three bytes of the array source and writes the resulting four Base64
+ * bytes to destination. The source and destination arrays can be manipulated anywhere
+ * along their length by specifying srcOffset and destOffset. This method
+ * does not check to make sure your arrays are large enough to accommodate srcOffset +
+ * 3 for the source array or destOffset + 4 for the destination
+ * array. The actual number of significant bytes in your array is given by numSigBytes.
+ *
+ *
This is the lowest level of the encoding methods with all possible parameters.
+ *
+ * @param source the array to convert
+ * @param srcOffset the index where conversion begins
+ * @param numSigBytes the number of significant bytes in your array
+ * @param destination the array to hold the conversion
+ * @param destOffset the index where output will be put
+ * @return the destination array
+ * @since 1.3
+ */
+ private static byte[] encode3to4(
+ byte[] source,
+ int srcOffset,
+ int numSigBytes,
+ byte[] destination,
+ int destOffset,
+ int options) {
+
+ byte[] ALPHABET = getAlphabet(options);
+
+ // 1 2 3
+ // 01234567890123456789012345678901 Bit position
+ // --------000000001111111122222222 Array position from threeBytes
+ // --------| || || || | Six bit groups to index ALPHABET
+ // >>18 >>12 >> 6 >> 0 Right shift necessary
+ // 0x3f 0x3f 0x3f Additional AND
+
+ // Create buffer with zero-padding if there are only one or two
+ // significant bytes passed in the array.
+ // We have to shift left 24 in order to flush out the 1's that appear
+ // when Java treats a value as negative that is cast from a byte to an int.
+ int inBuff =
+ (numSigBytes > 0 ? ((source[srcOffset] << 24) >>> 8) : 0)
+ | (numSigBytes > 1 ? ((source[srcOffset + 1] << 24) >>> 16) : 0)
+ | (numSigBytes > 2 ? ((source[srcOffset + 2] << 24) >>> 24) : 0);
+
+ switch (numSigBytes) {
+ case 3:
+ destination[destOffset] = ALPHABET[(inBuff >>> 18)];
+ destination[destOffset + 1] = ALPHABET[(inBuff >>> 12) & 0x3f];
+ destination[destOffset + 2] = ALPHABET[(inBuff >>> 6) & 0x3f];
+ destination[destOffset + 3] = ALPHABET[(inBuff) & 0x3f];
+ return destination;
+
+ case 2:
+ destination[destOffset] = ALPHABET[(inBuff >>> 18)];
+ destination[destOffset + 1] = ALPHABET[(inBuff >>> 12) & 0x3f];
+ destination[destOffset + 2] = ALPHABET[(inBuff >>> 6) & 0x3f];
+ destination[destOffset + 3] = EQUALS_SIGN;
+ return destination;
+
+ case 1:
+ destination[destOffset] = ALPHABET[(inBuff >>> 18)];
+ destination[destOffset + 1] = ALPHABET[(inBuff >>> 12) & 0x3f];
+ destination[destOffset + 2] = EQUALS_SIGN;
+ destination[destOffset + 3] = EQUALS_SIGN;
+ return destination;
+
+ default:
+ return destination;
+ } // end switch
+ } // end encode3to4
+
+ /**
+ * Performs Base64 encoding on the raw ByteBuffer, writing it to the encoded
+ * ByteBuffer. This is an experimental feature. Currently it does not pass along any
+ * options (such as {@link #DO_BREAK_LINES} or {@link #GZIP}.
+ *
+ * @param raw input buffer
+ * @param encoded output buffer
+ * @since 2.3
+ */
+ public static void encode(java.nio.ByteBuffer raw, java.nio.ByteBuffer encoded) {
+ byte[] raw3 = new byte[3];
+ byte[] enc4 = new byte[4];
+
+ while (raw.hasRemaining()) {
+ int rem = Math.min(3, raw.remaining());
+ raw.get(raw3, 0, rem);
+ Base64.encode3to4(enc4, raw3, rem, Base64.NO_OPTIONS);
+ encoded.put(enc4);
+ } // end input remaining
+ }
+
+ /**
+ * Performs Base64 encoding on the raw ByteBuffer, writing it to the encoded
+ * CharBuffer. This is an experimental feature. Currently it does not pass along any
+ * options (such as {@link #DO_BREAK_LINES} or {@link #GZIP}.
+ *
+ * @param raw input buffer
+ * @param encoded output buffer
+ * @since 2.3
+ */
+ public static void encode(java.nio.ByteBuffer raw, java.nio.CharBuffer encoded) {
+ byte[] raw3 = new byte[3];
+ byte[] enc4 = new byte[4];
+
+ while (raw.hasRemaining()) {
+ int rem = Math.min(3, raw.remaining());
+ raw.get(raw3, 0, rem);
+ Base64.encode3to4(enc4, raw3, rem, Base64.NO_OPTIONS);
+ for (int i = 0; i < 4; i++) {
+ encoded.put((char) (enc4[i] & 0xFF));
+ }
+ } // end input remaining
+ }
+
+ /**
+ * Serializes an object and returns the Base64-encoded version of that serialized object.
+ *
+ *
As of v 2.3, if the object cannot be serialized or there is another error, the method will
+ * throw an java.io.IOException. This is new to v2.3! In earlier versions, it just returned
+ * a null value, but in retrospect that's a pretty poor way to handle it. The object is not
+ * GZip-compressed before being encoded.
+ *
+ * @param serializableObject The object to encode
+ * @return The Base64-encoded object
+ * @throws java.io.IOException if there is an error
+ * @throws NullPointerException if serializedObject is null
+ * @since 1.4
+ */
+ public static String encodeObject(java.io.Serializable serializableObject)
+ throws java.io.IOException {
+ return encodeObject(serializableObject, NO_OPTIONS);
+ } // end encodeObject
+
+ /**
+ * Serializes an object and returns the Base64-encoded version of that serialized object.
+ *
+ *
As of v 2.3, if the object cannot be serialized or there is another error, the method will
+ * throw an java.io.IOException. This is new to v2.3! In earlier versions, it just returned
+ * a null value, but in retrospect that's a pretty poor way to handle it. The object is not
+ * GZip-compressed before being encoded.
+ *
+ *
Example options:
+ *
+ *
+ * GZIP: gzip-compresses object before encoding it.
+ * DO_BREAK_LINES: break lines at 76 characters
+ *
+ *
+ *
Example: encodeObject( myObj, Base64.GZIP ) or
+ *
+ *
Example: encodeObject( myObj, Base64.GZIP | Base64.DO_BREAK_LINES )
+ *
+ * @param serializableObject The object to encode
+ * @param options Specified options
+ * @return The Base64-encoded object
+ * @see Base64#GZIP
+ * @see Base64#DO_BREAK_LINES
+ * @throws java.io.IOException if there is an error
+ * @since 2.0
+ */
+ public static String encodeObject(java.io.Serializable serializableObject, int options)
+ throws java.io.IOException {
+
+ if (serializableObject == null) {
+ throw new NullPointerException("Cannot serialize a null object.");
+ } // end if: null
+
+ // Streams
+ java.io.ByteArrayOutputStream baos = null;
+ java.io.OutputStream b64os = null;
+ java.util.zip.GZIPOutputStream gzos = null;
+ java.io.ObjectOutputStream oos = null;
+
+ try {
+ // ObjectOutputStream -> (GZIP) -> Base64 -> ByteArrayOutputStream
+ baos = new java.io.ByteArrayOutputStream();
+ b64os = new Base64.OutputStream(baos, ENCODE | options);
+ if ((options & GZIP) != 0) {
+ // Gzip
+ gzos = new java.util.zip.GZIPOutputStream(b64os);
+ oos = new java.io.ObjectOutputStream(gzos);
+ } else {
+ // Not gzipped
+ oos = new java.io.ObjectOutputStream(b64os);
+ }
+ oos.writeObject(serializableObject);
+ } // end try
+ catch (java.io.IOException e) {
+ // Catch it and then throw it immediately so that
+ // the finally{} block is called for cleanup.
+ throw e;
+ } // end catch
+ finally {
+ try {
+ oos.close();
+ } catch (Exception e) {
+ }
+ try {
+ gzos.close();
+ } catch (Exception e) {
+ }
+ try {
+ b64os.close();
+ } catch (Exception e) {
+ }
+ try {
+ baos.close();
+ } catch (Exception e) {
+ }
+ } // end finally
+
+ // Return value according to relevant encoding.
+ try {
+ return new String(baos.toByteArray(), PREFERRED_ENCODING);
+ } // end try
+ catch (java.io.UnsupportedEncodingException uue) {
+ // Fall back to some Java default
+ return new String(baos.toByteArray());
+ } // end catch
+ } // end encode
+
+ /**
+ * Encodes a byte array into Base64 notation. Does not GZip-compress data.
+ *
+ * @param source The data to convert
+ * @return The data in Base64-encoded form
+ * @throws NullPointerException if source array is null
+ * @since 1.4
+ */
+ public static String encodeBytes(byte[] source) {
+ // Since we're not going to have the GZIP encoding turned on,
+ // we're not going to have an java.io.IOException thrown, so
+ // we should not force the user to have to catch it.
+ String encoded = null;
+ try {
+ encoded = encodeBytes(source, 0, source.length, NO_OPTIONS);
+ } catch (java.io.IOException ex) {
+ assert false : ex.getMessage();
+ } // end catch
+ assert encoded != null;
+ return encoded;
+ } // end encodeBytes
+
+ /**
+ * Encodes a byte array into Base64 notation.
+ *
+ *
Example options:
+ *
+ *
+ * GZIP: gzip-compresses object before encoding it.
+ * DO_BREAK_LINES: break lines at 76 characters
+ * Note: Technically, this makes your encoding non-compliant.
+ *
+ *
+ *
Example: encodeBytes( myData, Base64.GZIP ) or
+ *
+ *
As of v 2.3, if there is an error with the GZIP stream, the method will throw an
+ * java.io.IOException. This is new to v2.3! In earlier versions, it just returned a null
+ * value, but in retrospect that's a pretty poor way to handle it.
+ *
+ * @param source The data to convert
+ * @param options Specified options
+ * @return The Base64-encoded data as a String
+ * @see Base64#GZIP
+ * @see Base64#DO_BREAK_LINES
+ * @throws java.io.IOException if there is an error
+ * @throws NullPointerException if source array is null
+ * @since 2.0
+ */
+ public static String encodeBytes(byte[] source, int options) throws java.io.IOException {
+ return encodeBytes(source, 0, source.length, options);
+ } // end encodeBytes
+
+ /**
+ * Encodes a byte array into Base64 notation. Does not GZip-compress data.
+ *
+ *
As of v 2.3, if there is an error, the method will throw an java.io.IOException. This is
+ * new to v2.3! In earlier versions, it just returned a null value, but in retrospect that's a
+ * pretty poor way to handle it.
+ *
+ * @param source The data to convert
+ * @param off Offset in array where conversion should begin
+ * @param len Length of data to convert
+ * @return The Base64-encoded data as a String
+ * @throws NullPointerException if source array is null
+ * @throws IllegalArgumentException if source array, offset, or length are invalid
+ * @since 1.4
+ */
+ public static String encodeBytes(byte[] source, int off, int len) {
+ // Since we're not going to have the GZIP encoding turned on,
+ // we're not going to have an java.io.IOException thrown, so
+ // we should not force the user to have to catch it.
+ String encoded = null;
+ try {
+ encoded = encodeBytes(source, off, len, NO_OPTIONS);
+ } catch (java.io.IOException ex) {
+ assert false : ex.getMessage();
+ } // end catch
+ assert encoded != null;
+ return encoded;
+ } // end encodeBytes
+
+ /**
+ * Encodes a byte array into Base64 notation.
+ *
+ *
Example options:
+ *
+ *
+ * GZIP: gzip-compresses object before encoding it.
+ * DO_BREAK_LINES: break lines at 76 characters
+ * Note: Technically, this makes your encoding non-compliant.
+ *
+ *
+ *
Example: encodeBytes( myData, Base64.GZIP ) or
+ *
+ *
As of v 2.3, if there is an error with the GZIP stream, the method will throw an
+ * java.io.IOException. This is new to v2.3! In earlier versions, it just returned a null
+ * value, but in retrospect that's a pretty poor way to handle it.
+ *
+ * @param source The data to convert
+ * @param off Offset in array where conversion should begin
+ * @param len Length of data to convert
+ * @param options Specified options
+ * @return The Base64-encoded data as a String
+ * @see Base64#GZIP
+ * @see Base64#DO_BREAK_LINES
+ * @throws java.io.IOException if there is an error
+ * @throws NullPointerException if source array is null
+ * @throws IllegalArgumentException if source array, offset, or length are invalid
+ * @since 2.0
+ */
+ public static String encodeBytes(byte[] source, int off, int len, int options)
+ throws java.io.IOException {
+ byte[] encoded = encodeBytesToBytes(source, off, len, options);
+
+ // Return value according to relevant encoding.
+ try {
+ return new String(encoded, PREFERRED_ENCODING);
+ } // end try
+ catch (java.io.UnsupportedEncodingException uue) {
+ return new String(encoded);
+ } // end catch
+ } // end encodeBytes
+
+ /**
+ * Similar to {@link #encodeBytes(byte[])} but returns a byte array instead of instantiating a
+ * String. This is more efficient if you're working with I/O streams and have large data sets to
+ * encode.
+ *
+ * @param source The data to convert
+ * @return The Base64-encoded data as a byte[] (of ASCII characters)
+ * @throws NullPointerException if source array is null
+ * @since 2.3.1
+ */
+ public static byte[] encodeBytesToBytes(byte[] source) {
+ byte[] encoded = null;
+ try {
+ encoded = encodeBytesToBytes(source, 0, source.length, Base64.NO_OPTIONS);
+ } catch (java.io.IOException ex) {
+ assert false
+ : "IOExceptions only come from GZipping, which is turned off: " + ex.getMessage();
+ }
+ return encoded;
+ }
+
+ /**
+ * Similar to {@link #encodeBytes(byte[], int, int, int)} but returns a byte array instead of
+ * instantiating a String. This is more efficient if you're working with I/O streams and have
+ * large data sets to encode.
+ *
+ * @param source The data to convert
+ * @param off Offset in array where conversion should begin
+ * @param len Length of data to convert
+ * @param options Specified options
+ * @return The Base64-encoded data as a String
+ * @see Base64#GZIP
+ * @see Base64#DO_BREAK_LINES
+ * @throws java.io.IOException if there is an error
+ * @throws NullPointerException if source array is null
+ * @throws IllegalArgumentException if source array, offset, or length are invalid
+ * @since 2.3.1
+ */
+ public static byte[] encodeBytesToBytes(byte[] source, int off, int len, int options)
+ throws java.io.IOException {
+
+ if (source == null) {
+ throw new NullPointerException("Cannot serialize a null array.");
+ } // end if: null
+
+ if (off < 0) {
+ throw new IllegalArgumentException("Cannot have negative offset: " + off);
+ } // end if: off < 0
+
+ if (len < 0) {
+ throw new IllegalArgumentException("Cannot have length offset: " + len);
+ } // end if: len < 0
+
+ if (off + len > source.length) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Cannot have offset of %d and length of %d with array of length %d",
+ off, len, source.length));
+ } // end if: off < 0
+
+ // Compress?
+ if ((options & GZIP) != 0) {
+ java.io.ByteArrayOutputStream baos = null;
+ java.util.zip.GZIPOutputStream gzos = null;
+ Base64.OutputStream b64os = null;
+
+ try {
+ // GZip -> Base64 -> ByteArray
+ baos = new java.io.ByteArrayOutputStream();
+ b64os = new Base64.OutputStream(baos, ENCODE | options);
+ gzos = new java.util.zip.GZIPOutputStream(b64os);
+
+ gzos.write(source, off, len);
+ gzos.close();
+ } // end try
+ catch (java.io.IOException e) {
+ // Catch it and then throw it immediately so that
+ // the finally{} block is called for cleanup.
+ throw e;
+ } // end catch
+ finally {
+ try {
+ gzos.close();
+ } catch (Exception e) {
+ }
+ try {
+ b64os.close();
+ } catch (Exception e) {
+ }
+ try {
+ baos.close();
+ } catch (Exception e) {
+ }
+ } // end finally
+
+ return baos.toByteArray();
+ } // end if: compress
+
+ // Else, don't compress. Better not to use streams at all then.
+ else {
+ boolean breakLines = (options & DO_BREAK_LINES) != 0;
+
+ // int len43 = len * 4 / 3;
+ // byte[] outBuff = new byte[ ( len43 ) // Main 4:3
+ // + ( (len % 3) > 0 ? 4 : 0 ) // Account for padding
+ // + (breakLines ? ( len43 / MAX_LINE_LENGTH ) : 0) ]; // New lines
+ // Try to determine more precisely how big the array needs to be.
+ // If we get it right, we don't have to do an array copy, and
+ // we save a bunch of memory.
+ int encLen = (len / 3) * 4 + (len % 3 > 0 ? 4 : 0); // Bytes needed for actual encoding
+ if (breakLines) {
+ encLen += encLen / MAX_LINE_LENGTH; // Plus extra newline characters
+ }
+ byte[] outBuff = new byte[encLen];
+
+ int d = 0;
+ int e = 0;
+ int len2 = len - 2;
+ int lineLength = 0;
+ for (; d < len2; d += 3, e += 4) {
+ encode3to4(source, d + off, 3, outBuff, e, options);
+
+ lineLength += 4;
+ if (breakLines && lineLength >= MAX_LINE_LENGTH) {
+ outBuff[e + 4] = NEW_LINE;
+ e++;
+ lineLength = 0;
+ } // end if: end of line
+ } // en dfor: each piece of array
+
+ if (d < len) {
+ encode3to4(source, d + off, len - d, outBuff, e, options);
+ e += 4;
+ } // end if: some padding needed
+
+ // Only resize array if we didn't guess it right.
+ if (e <= outBuff.length - 1) {
+ // If breaking lines and the last byte falls right at
+ // the line length (76 bytes per line), there will be
+ // one extra byte, and the array will need to be resized.
+ // Not too bad of an estimate on array size, I'd say.
+ byte[] finalOut = new byte[e];
+ System.arraycopy(outBuff, 0, finalOut, 0, e);
+ // System.err.println("Having to resize array from " + outBuff.length + " to " + e );
+ return finalOut;
+ } else {
+ // System.err.println("No need to resize array.");
+ return outBuff;
+ }
+ } // end else: don't compress
+ } // end encodeBytesToBytes
+
+ /* ******** D E C O D I N G M E T H O D S ******** */
+
+ /**
+ * Decodes four bytes from array source and writes the resulting bytes (up to three of
+ * them) to destination. The source and destination arrays can be manipulated anywhere
+ * along their length by specifying srcOffset and destOffset. This method
+ * does not check to make sure your arrays are large enough to accomodate srcOffset + 4
+ * for the source array or destOffset + 3 for the destination
+ * array. This method returns the actual number of bytes that were converted from the Base64
+ * encoding.
+ *
+ *
This is the lowest level of the decoding methods with all possible parameters.
+ *
+ * @param source the array to convert
+ * @param srcOffset the index where conversion begins
+ * @param destination the array to hold the conversion
+ * @param destOffset the index where output will be put
+ * @param options alphabet type is pulled from this (standard, url-safe, ordered)
+ * @return the number of decoded bytes converted
+ * @throws NullPointerException if source or destination arrays are null
+ * @throws IllegalArgumentException if srcOffset or destOffset are invalid or there is not enough
+ * room in the array.
+ * @since 1.3
+ */
+ private static int decode4to3(
+ byte[] source, int srcOffset, byte[] destination, int destOffset, int options) {
+
+ // Lots of error checking and exception throwing
+ if (source == null) {
+ throw new NullPointerException("Source array was null.");
+ } // end if
+ if (destination == null) {
+ throw new NullPointerException("Destination array was null.");
+ } // end if
+ if (srcOffset < 0 || srcOffset + 3 >= source.length) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Source array with length %d cannot have offset of %d and still process four bytes.",
+ source.length, srcOffset));
+ } // end if
+ if (destOffset < 0 || destOffset + 2 >= destination.length) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Destination array with length %d cannot have offset of %d and still store three bytes.",
+ destination.length, destOffset));
+ } // end if
+
+ byte[] DECODABET = getDecodabet(options);
+
+ // Example: Dk==
+ if (source[srcOffset + 2] == EQUALS_SIGN) {
+ // Two ways to do the same thing. Don't know which way I like best.
+ // int outBuff = ( ( DECODABET[ source[ srcOffset ] ] << 24 ) >>> 6 )
+ // | ( ( DECODABET[ source[ srcOffset + 1] ] << 24 ) >>> 12 );
+ int outBuff =
+ ((DECODABET[source[srcOffset]] & 0xFF) << 18)
+ | ((DECODABET[source[srcOffset + 1]] & 0xFF) << 12);
+
+ destination[destOffset] = (byte) (outBuff >>> 16);
+ return 1;
+ }
+
+ // Example: DkL=
+ else if (source[srcOffset + 3] == EQUALS_SIGN) {
+ // Two ways to do the same thing. Don't know which way I like best.
+ // int outBuff = ( ( DECODABET[ source[ srcOffset ] ] << 24 ) >>> 6 )
+ // | ( ( DECODABET[ source[ srcOffset + 1 ] ] << 24 ) >>> 12 )
+ // | ( ( DECODABET[ source[ srcOffset + 2 ] ] << 24 ) >>> 18 );
+ int outBuff =
+ ((DECODABET[source[srcOffset]] & 0xFF) << 18)
+ | ((DECODABET[source[srcOffset + 1]] & 0xFF) << 12)
+ | ((DECODABET[source[srcOffset + 2]] & 0xFF) << 6);
+
+ destination[destOffset] = (byte) (outBuff >>> 16);
+ destination[destOffset + 1] = (byte) (outBuff >>> 8);
+ return 2;
+ }
+
+ // Example: DkLE
+ else {
+ // Two ways to do the same thing. Don't know which way I like best.
+ // int outBuff = ( ( DECODABET[ source[ srcOffset ] ] << 24 ) >>> 6 )
+ // | ( ( DECODABET[ source[ srcOffset + 1 ] ] << 24 ) >>> 12 )
+ // | ( ( DECODABET[ source[ srcOffset + 2 ] ] << 24 ) >>> 18 )
+ // | ( ( DECODABET[ source[ srcOffset + 3 ] ] << 24 ) >>> 24 );
+ int outBuff =
+ ((DECODABET[source[srcOffset]] & 0xFF) << 18)
+ | ((DECODABET[source[srcOffset + 1]] & 0xFF) << 12)
+ | ((DECODABET[source[srcOffset + 2]] & 0xFF) << 6)
+ | ((DECODABET[source[srcOffset + 3]] & 0xFF));
+
+ destination[destOffset] = (byte) (outBuff >> 16);
+ destination[destOffset + 1] = (byte) (outBuff >> 8);
+ destination[destOffset + 2] = (byte) (outBuff);
+
+ return 3;
+ }
+ } // end decodeToBytes
+
+ /**
+ * Low-level access to decoding ASCII characters in the form of a byte array. Ignores
+ * GUNZIP option, if it's set. This is not generally a recommended method, although it is
+ * used internally as part of the decoding process. Special case: if len = 0, an empty array is
+ * returned. Still, if you need more speed and reduced memory footprint (and aren't gzipping),
+ * consider this method.
+ *
+ * @param source The Base64 encoded data
+ * @return decoded data
+ * @since 2.3.1
+ */
+ public static byte[] decode(byte[] source) throws java.io.IOException {
+ byte[] decoded = null;
+ // try {
+ decoded = decode(source, 0, source.length, Base64.NO_OPTIONS);
+ // } catch( java.io.IOException ex ) {
+ // assert false : "IOExceptions only come from GZipping, which is turned off: " +
+ // ex.getMessage();
+ // }
+ return decoded;
+ }
+
+ /**
+ * Low-level access to decoding ASCII characters in the form of a byte array. Ignores
+ * GUNZIP option, if it's set. This is not generally a recommended method, although it is
+ * used internally as part of the decoding process. Special case: if len = 0, an empty array is
+ * returned. Still, if you need more speed and reduced memory footprint (and aren't gzipping),
+ * consider this method.
+ *
+ * @param source The Base64 encoded data
+ * @param off The offset of where to begin decoding
+ * @param len The length of characters to decode
+ * @param options Can specify options such as alphabet type to use
+ * @return decoded data
+ * @throws java.io.IOException If bogus characters exist in source data
+ * @since 1.3
+ */
+ public static byte[] decode(byte[] source, int off, int len, int options)
+ throws java.io.IOException {
+
+ // Lots of error checking and exception throwing
+ if (source == null) {
+ throw new NullPointerException("Cannot decode null source array.");
+ } // end if
+ if (off < 0 || off + len > source.length) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Source array with length %d cannot have offset of %d and process %d bytes.",
+ source.length, off, len));
+ } // end if
+
+ if (len == 0) {
+ return new byte[0];
+ } else if (len < 4) {
+ throw new IllegalArgumentException(
+ "Base64-encoded string must have at least four characters, but length specified was "
+ + len);
+ } // end if
+
+ byte[] DECODABET = getDecodabet(options);
+
+ int len34 = len * 3 / 4; // Estimate on array size
+ byte[] outBuff = new byte[len34]; // Upper limit on size of output
+ int outBuffPosn = 0; // Keep track of where we're writing
+
+ byte[] b4 = new byte[4]; // Four byte buffer from source, eliminating white space
+ int b4Posn = 0; // Keep track of four byte input buffer
+ int i = 0; // Source array counter
+ byte sbiDecode = 0; // Special value from DECODABET
+
+ for (i = off; i < off + len; i++) { // Loop through source
+
+ sbiDecode = DECODABET[source[i] & 0xFF];
+
+ // White space, Equals sign, or legit Base64 character
+ // Note the values such as -5 and -9 in the
+ // DECODABETs at the top of the file.
+ if (sbiDecode >= WHITE_SPACE_ENC) {
+ if (sbiDecode >= EQUALS_SIGN_ENC) {
+ b4[b4Posn++] = source[i]; // Save non-whitespace
+ if (b4Posn > 3) { // Time to decode?
+ outBuffPosn += decode4to3(b4, 0, outBuff, outBuffPosn, options);
+ b4Posn = 0;
+
+ // If that was the equals sign, break out of 'for' loop
+ if (source[i] == EQUALS_SIGN) {
+ break;
+ } // end if: equals sign
+ } // end if: quartet built
+ } // end if: equals sign or better
+ } // end if: white space, equals sign or better
+ else {
+ // There's a bad input character in the Base64 stream.
+ throw new java.io.IOException(
+ String.format(
+ "Bad Base64 input character decimal %d in array position %d",
+ ((int) source[i]) & 0xFF, i));
+ } // end else:
+ } // each input character
+
+ byte[] out = new byte[outBuffPosn];
+ System.arraycopy(outBuff, 0, out, 0, outBuffPosn);
+ return out;
+ } // end decode
+
+ /**
+ * Decodes data from Base64 notation, automatically detecting gzip-compressed data and
+ * decompressing it.
+ *
+ * @param s the string to decode
+ * @return the decoded data
+ * @throws java.io.IOException If there is a problem
+ * @since 1.4
+ */
+ public static byte[] decode(String s) throws java.io.IOException {
+ return decode(s, NO_OPTIONS);
+ }
+
+ /**
+ * Decodes data from Base64 notation, automatically detecting gzip-compressed data and
+ * decompressing it.
+ *
+ * @param s the string to decode
+ * @param options encode options such as URL_SAFE
+ * @return the decoded data
+ * @throws java.io.IOException if there is an error
+ * @throws NullPointerException if s is null
+ * @since 1.4
+ */
+ public static byte[] decode(String s, int options) throws java.io.IOException {
+
+ if (s == null) {
+ throw new NullPointerException("Input string was null.");
+ } // end if
+
+ byte[] bytes;
+ try {
+ bytes = s.getBytes(PREFERRED_ENCODING);
+ } // end try
+ catch (java.io.UnsupportedEncodingException uee) {
+ bytes = s.getBytes();
+ } // end catch
+ //
+
+ // Decode
+ bytes = decode(bytes, 0, bytes.length, options);
+
+ // Check to see if it's gzip-compressed
+ // GZIP Magic Two-Byte Number: 0x8b1f (35615)
+ boolean dontGunzip = (options & DONT_GUNZIP) != 0;
+ if ((bytes != null) && (bytes.length >= 4) && (!dontGunzip)) {
+
+ int head = ((int) bytes[0] & 0xff) | ((bytes[1] << 8) & 0xff00);
+ if (java.util.zip.GZIPInputStream.GZIP_MAGIC == head) {
+ java.io.ByteArrayInputStream bais = null;
+ java.util.zip.GZIPInputStream gzis = null;
+ java.io.ByteArrayOutputStream baos = null;
+ byte[] buffer = new byte[2048];
+ int length = 0;
+
+ try {
+ baos = new java.io.ByteArrayOutputStream();
+ bais = new java.io.ByteArrayInputStream(bytes);
+ gzis = new java.util.zip.GZIPInputStream(bais);
+
+ while ((length = gzis.read(buffer)) >= 0) {
+ baos.write(buffer, 0, length);
+ } // end while: reading input
+
+ // No error? Get new bytes.
+ bytes = baos.toByteArray();
+
+ } // end try
+ catch (java.io.IOException e) {
+ e.printStackTrace();
+ // Just return originally-decoded bytes
+ } // end catch
+ finally {
+ try {
+ baos.close();
+ } catch (Exception e) {
+ }
+ try {
+ gzis.close();
+ } catch (Exception e) {
+ }
+ try {
+ bais.close();
+ } catch (Exception e) {
+ }
+ } // end finally
+ } // end if: gzipped
+ } // end if: bytes.length >= 2
+
+ return bytes;
+ } // end decode
+
+ /**
+ * Attempts to decode Base64 data and deserialize a Java Object within. Returns null
+ * if there was an error.
+ *
+ * @param encodedObject The Base64 data to decode
+ * @return The decoded and deserialized object
+ * @throws NullPointerException if encodedObject is null
+ * @throws java.io.IOException if there is a general error
+ * @throws ClassNotFoundException if the decoded object is of a class that cannot be found by the
+ * JVM
+ * @since 1.5
+ */
+ public static Object decodeToObject(String encodedObject)
+ throws java.io.IOException, java.lang.ClassNotFoundException {
+ return decodeToObject(encodedObject, NO_OPTIONS, null);
+ }
+
+ /**
+ * Attempts to decode Base64 data and deserialize a Java Object within. Returns null
+ * if there was an error. If loader is not null, it will be the class loader used
+ * when deserializing.
+ *
+ * @param encodedObject The Base64 data to decode
+ * @param options Various parameters related to decoding
+ * @param loader Optional class loader to use in deserializing classes.
+ * @return The decoded and deserialized object
+ * @throws NullPointerException if encodedObject is null
+ * @throws java.io.IOException if there is a general error
+ * @throws ClassNotFoundException if the decoded object is of a class that cannot be found by the
+ * JVM
+ * @since 2.3.4
+ */
+ public static Object decodeToObject(String encodedObject, int options, final ClassLoader loader)
+ throws java.io.IOException, java.lang.ClassNotFoundException {
+
+ // Decode and gunzip if necessary
+ byte[] objBytes = decode(encodedObject, options);
+
+ java.io.ByteArrayInputStream bais = null;
+ java.io.ObjectInputStream ois = null;
+ Object obj = null;
+
+ try {
+ bais = new java.io.ByteArrayInputStream(objBytes);
+
+ // If no custom class loader is provided, use Java's builtin OIS.
+ if (loader == null) {
+ ois = new java.io.ObjectInputStream(bais);
+ } // end if: no loader provided
+
+ // Else make a customized object input stream that uses
+ // the provided class loader.
+ else {
+ ois =
+ new java.io.ObjectInputStream(bais) {
+ @Override
+ public Class resolveClass(java.io.ObjectStreamClass streamClass)
+ throws java.io.IOException, ClassNotFoundException {
+ Class c = Class.forName(streamClass.getName(), false, loader);
+ if (c == null) {
+ return super.resolveClass(streamClass);
+ } else {
+ return c; // Class loader knows of this class.
+ } // end else: not null
+ } // end resolveClass
+ }; // end ois
+ } // end else: no custom class loader
+
+ obj = ois.readObject();
+ } // end try
+ catch (java.io.IOException e) {
+ throw e; // Catch and throw in order to execute finally{}
+ } // end catch
+ catch (java.lang.ClassNotFoundException e) {
+ throw e; // Catch and throw in order to execute finally{}
+ } // end catch
+ finally {
+ try {
+ bais.close();
+ } catch (Exception e) {
+ }
+ try {
+ ois.close();
+ } catch (Exception e) {
+ }
+ } // end finally
+
+ return obj;
+ } // end decodeObject
+
+ /**
+ * Convenience method for encoding data to a file.
+ *
+ *
As of v 2.3, if there is a error, the method will throw an java.io.IOException. This is
+ * new to v2.3! In earlier versions, it just returned false, but in retrospect that's a pretty
+ * poor way to handle it.
+ *
+ * @param dataToEncode byte array of data to encode in base64 form
+ * @param filename Filename for saving encoded data
+ * @throws java.io.IOException if there is an error
+ * @throws NullPointerException if dataToEncode is null
+ * @since 2.1
+ */
+ public static void encodeToFile(byte[] dataToEncode, String filename) throws java.io.IOException {
+
+ if (dataToEncode == null) {
+ throw new NullPointerException("Data to encode was null.");
+ } // end iff
+
+ Base64.OutputStream bos = null;
+ try {
+ bos = new Base64.OutputStream(new java.io.FileOutputStream(filename), Base64.ENCODE);
+ bos.write(dataToEncode);
+ } // end try
+ catch (java.io.IOException e) {
+ throw e; // Catch and throw to execute finally{} block
+ } // end catch: java.io.IOException
+ finally {
+ try {
+ bos.close();
+ } catch (Exception e) {
+ }
+ } // end finally
+ } // end encodeToFile
+
+ /**
+ * Convenience method for decoding data to a file.
+ *
+ *
As of v 2.3, if there is a error, the method will throw an java.io.IOException. This is
+ * new to v2.3! In earlier versions, it just returned false, but in retrospect that's a pretty
+ * poor way to handle it.
+ *
+ * @param dataToDecode Base64-encoded data as a string
+ * @param filename Filename for saving decoded data
+ * @throws java.io.IOException if there is an error
+ * @since 2.1
+ */
+ public static void decodeToFile(String dataToDecode, String filename) throws java.io.IOException {
+
+ Base64.OutputStream bos = null;
+ try {
+ bos = new Base64.OutputStream(new java.io.FileOutputStream(filename), Base64.DECODE);
+ bos.write(dataToDecode.getBytes(PREFERRED_ENCODING));
+ } // end try
+ catch (java.io.IOException e) {
+ throw e; // Catch and throw to execute finally{} block
+ } // end catch: java.io.IOException
+ finally {
+ try {
+ bos.close();
+ } catch (Exception e) {
+ }
+ } // end finally
+ } // end decodeToFile
+
+ /**
+ * Convenience method for reading a base64-encoded file and decoding it.
+ *
+ *
As of v 2.3, if there is a error, the method will throw an java.io.IOException. This is
+ * new to v2.3! In earlier versions, it just returned false, but in retrospect that's a pretty
+ * poor way to handle it.
+ *
+ * @param filename Filename for reading encoded data
+ * @return decoded byte array
+ * @throws java.io.IOException if there is an error
+ * @since 2.1
+ */
+ public static byte[] decodeFromFile(String filename) throws java.io.IOException {
+
+ byte[] decodedData = null;
+ Base64.InputStream bis = null;
+ try {
+ // Set up some useful variables
+ java.io.File file = new java.io.File(filename);
+ byte[] buffer = null;
+ int length = 0;
+ int numBytes = 0;
+
+ // Check for size of file
+ // See: https://stackoverflow.com/questions/3038392/do-java-arrays-have-a-maximum-size
+ if (file.length() > Integer.MAX_VALUE - 8) {
+ throw new java.io.IOException(
+ "File is too big for this convenience method (" + file.length() + " bytes).");
+ } // end if: file too big for int index
+ buffer = new byte[(int) file.length()];
+
+ // Open a stream
+ bis =
+ new Base64.InputStream(
+ new java.io.BufferedInputStream(new java.io.FileInputStream(file)), Base64.DECODE);
+
+ // Read until done
+ while ((numBytes = bis.read(buffer, length, 4096)) >= 0) {
+ length += numBytes;
+ } // end while
+
+ // Save in a variable to return
+ decodedData = new byte[length];
+ System.arraycopy(buffer, 0, decodedData, 0, length);
+
+ } // end try
+ catch (java.io.IOException e) {
+ throw e; // Catch and release to execute finally{}
+ } // end catch: java.io.IOException
+ finally {
+ try {
+ bis.close();
+ } catch (Exception e) {
+ }
+ } // end finally
+
+ return decodedData;
+ } // end decodeFromFile
+
+ /**
+ * Convenience method for reading a binary file and base64-encoding it.
+ *
+ *
As of v 2.3, if there is a error, the method will throw an java.io.IOException. This is
+ * new to v2.3! In earlier versions, it just returned false, but in retrospect that's a pretty
+ * poor way to handle it.
+ *
+ * @param filename Filename for reading binary data
+ * @return base64-encoded string
+ * @throws java.io.IOException if there is an error
+ * @since 2.1
+ */
+ public static String encodeFromFile(String filename) throws java.io.IOException {
+
+ String encodedData = null;
+ Base64.InputStream bis = null;
+ try {
+ // Set up some useful variables
+ java.io.File file = new java.io.File(filename);
+ byte[] buffer =
+ new byte
+ [Math.max(
+ (int) (file.length() * 1.4 + 1),
+ 40)]; // Need max() for math on small files (v2.2.1); Need +1 for a few corner
+ // cases (v2.3.5)
+ int length = 0;
+ int numBytes = 0;
+
+ // Open a stream
+ bis =
+ new Base64.InputStream(
+ new java.io.BufferedInputStream(new java.io.FileInputStream(file)), Base64.ENCODE);
+
+ // Read until done
+ while ((numBytes = bis.read(buffer, length, 4096)) >= 0) {
+ length += numBytes;
+ } // end while
+
+ // Save in a variable to return
+ encodedData = new String(buffer, 0, length, Base64.PREFERRED_ENCODING);
+
+ } // end try
+ catch (java.io.IOException e) {
+ throw e; // Catch and release to execute finally{}
+ } // end catch: java.io.IOException
+ finally {
+ try {
+ bis.close();
+ } catch (Exception e) {
+ }
+ } // end finally
+
+ return encodedData;
+ } // end encodeFromFile
+
+ /**
+ * Reads infile and encodes it to outfile.
+ *
+ * @param infile Input file
+ * @param outfile Output file
+ * @throws java.io.IOException if there is an error
+ * @since 2.2
+ */
+ public static void encodeFileToFile(String infile, String outfile) throws java.io.IOException {
+
+ String encoded = Base64.encodeFromFile(infile);
+ java.io.OutputStream out = null;
+ try {
+ out = new java.io.BufferedOutputStream(new java.io.FileOutputStream(outfile));
+ out.write(encoded.getBytes("US-ASCII")); // Strict, 7-bit output.
+ } // end try
+ catch (java.io.IOException e) {
+ throw e; // Catch and release to execute finally{}
+ } // end catch
+ finally {
+ try {
+ out.close();
+ } catch (Exception ex) {
+ }
+ } // end finally
+ } // end encodeFileToFile
+
+ /**
+ * Reads infile and decodes it to outfile.
+ *
+ * @param infile Input file
+ * @param outfile Output file
+ * @throws java.io.IOException if there is an error
+ * @since 2.2
+ */
+ public static void decodeFileToFile(String infile, String outfile) throws java.io.IOException {
+
+ byte[] decoded = Base64.decodeFromFile(infile);
+ java.io.OutputStream out = null;
+ try {
+ out = new java.io.BufferedOutputStream(new java.io.FileOutputStream(outfile));
+ out.write(decoded);
+ } // end try
+ catch (java.io.IOException e) {
+ throw e; // Catch and release to execute finally{}
+ } // end catch
+ finally {
+ try {
+ out.close();
+ } catch (Exception ex) {
+ }
+ } // end finally
+ } // end decodeFileToFile
+
+ /* ******** I N N E R C L A S S I N P U T S T R E A M ******** */
+
+ /**
+ * A {@link Base64.InputStream} will read data from another java.io.InputStream,
+ * given in the constructor, and encode/decode to/from Base64 notation on the fly.
+ *
+ * @see Base64
+ * @since 1.3
+ */
+ public static class InputStream extends java.io.FilterInputStream {
+
+ private boolean encode; // Encoding or decoding
+ private int position; // Current position in the buffer
+ private byte[] buffer; // Small buffer holding converted data
+ private int bufferLength; // Length of buffer (3 or 4)
+ private int numSigBytes; // Number of meaningful bytes in the buffer
+ private int lineLength;
+ private boolean breakLines; // Break lines at less than 80 characters
+ private int options; // Record options used to create the stream.
+ private byte[] decodabet; // Local copies to avoid extra method calls
+
+ /**
+ * Constructs a {@link Base64.InputStream} in DECODE mode.
+ *
+ * @param in the java.io.InputStream from which to read data.
+ * @since 1.3
+ */
+ public InputStream(java.io.InputStream in) {
+ this(in, DECODE);
+ } // end constructor
+
+ /**
+ * Constructs a {@link Base64.InputStream} in either ENCODE or DECODE mode.
+ *
+ *
Valid options:
+ *
+ *
+ * ENCODE or DECODE: Encode or Decode as data is read.
+ * DO_BREAK_LINES: break lines at 76 characters
+ * (only meaningful when encoding)
+ *
+ *
+ *
Example: new Base64.InputStream( in, Base64.DECODE )
+ *
+ * @param in the java.io.InputStream from which to read data.
+ * @param options Specified options
+ * @see Base64#ENCODE
+ * @see Base64#DECODE
+ * @see Base64#DO_BREAK_LINES
+ * @since 2.0
+ */
+ public InputStream(java.io.InputStream in, int options) {
+
+ super(in);
+ this.options = options; // Record for later
+ this.breakLines = (options & DO_BREAK_LINES) > 0;
+ this.encode = (options & ENCODE) > 0;
+ this.bufferLength = encode ? 4 : 3;
+ this.buffer = new byte[bufferLength];
+ this.position = -1;
+ this.lineLength = 0;
+ this.decodabet = getDecodabet(options);
+ } // end constructor
+
+ /**
+ * Reads enough of the input stream to convert to/from Base64 and returns the next byte.
+ *
+ * @return next byte
+ * @since 1.3
+ */
+ @Override
+ public int read() throws java.io.IOException {
+
+ // Do we need to get data?
+ if (position < 0) {
+ if (encode) {
+ byte[] b3 = new byte[3];
+ int numBinaryBytes = 0;
+ for (int i = 0; i < 3; i++) {
+ int b = in.read();
+
+ // If end of stream, b is -1.
+ if (b >= 0) {
+ b3[i] = (byte) b;
+ numBinaryBytes++;
+ } else {
+ break; // out of for loop
+ } // end else: end of stream
+ } // end for: each needed input byte
+
+ if (numBinaryBytes > 0) {
+ encode3to4(b3, 0, numBinaryBytes, buffer, 0, options);
+ position = 0;
+ numSigBytes = 4;
+ } // end if: got data
+ else {
+ return -1; // Must be end of stream
+ } // end else
+ } // end if: encoding
+
+ // Else decoding
+ else {
+ byte[] b4 = new byte[4];
+ int i = 0;
+ for (i = 0; i < 4; i++) {
+ // Read four "meaningful" bytes:
+ int b = 0;
+ do {
+ b = in.read();
+ } while (b >= 0 && decodabet[b & 0x7f] <= WHITE_SPACE_ENC);
+
+ if (b < 0) {
+ break; // Reads a -1 if end of stream
+ } // end if: end of stream
+
+ b4[i] = (byte) b;
+ } // end for: each needed input byte
+
+ if (i == 4) {
+ numSigBytes = decode4to3(b4, 0, buffer, 0, options);
+ position = 0;
+ } // end if: got four characters
+ else if (i == 0) {
+ return -1;
+ } // end else if: also padded correctly
+ else {
+ // Must have broken out from above.
+ throw new java.io.IOException("Improperly padded Base64 input.");
+ } // end
+ } // end else: decode
+ } // end else: get data
+
+ // Got data?
+ if (position >= 0) {
+ // End of relevant data?
+ if (
+ /*!encode &&*/ position >= numSigBytes) {
+ return -1;
+ } // end if: got data
+
+ if (encode && breakLines && lineLength >= MAX_LINE_LENGTH) {
+ lineLength = 0;
+ return '\n';
+ } // end if
+ else {
+ lineLength++; // This isn't important when decoding
+ // but throwing an extra "if" seems
+ // just as wasteful.
+
+ int b = buffer[position++];
+
+ if (position >= bufferLength) {
+ position = -1;
+ } // end if: end
+
+ return b & 0xFF; // This is how you "cast" a byte that's
+ // intended to be unsigned.
+ } // end else
+ } // end if: position >= 0
+
+ // Else error
+ else {
+ throw new java.io.IOException("Error in Base64 code reading stream.");
+ } // end else
+ } // end read
+
+ /**
+ * Calls {@link #read()} repeatedly until the end of stream is reached or len bytes
+ * are read. Returns number of bytes read into array or -1 if end of stream is encountered.
+ *
+ * @param dest array to hold values
+ * @param off offset for array
+ * @param len max number of bytes to read into array
+ * @return bytes read into array or -1 if end of stream is encountered.
+ * @since 1.3
+ */
+ @Override
+ public int read(byte[] dest, int off, int len) throws java.io.IOException {
+ int i;
+ int b;
+ for (i = 0; i < len; i++) {
+ b = read();
+
+ if (b >= 0) {
+ dest[off + i] = (byte) b;
+ } else if (i == 0) {
+ return -1;
+ } else {
+ break; // Out of 'for' loop
+ } // Out of 'for' loop
+ } // end for: each byte read
+ return i;
+ } // end read
+ } // end inner class InputStream
+
+ /* ******** I N N E R C L A S S O U T P U T S T R E A M ******** */
+
+ /**
+ * A {@link Base64.OutputStream} will write data to another java.io.OutputStream,
+ * given in the constructor, and encode/decode to/from Base64 notation on the fly.
+ *
+ * @see Base64
+ * @since 1.3
+ */
+ public static class OutputStream extends java.io.FilterOutputStream {
+
+ private boolean encode;
+ private int position;
+ private byte[] buffer;
+ private int bufferLength;
+ private int lineLength;
+ private boolean breakLines;
+ private byte[] b4; // Scratch used in a few places
+ private boolean suspendEncoding;
+ private int options; // Record for later
+ private byte[] decodabet; // Local copies to avoid extra method calls
+
+ /**
+ * Constructs a {@link Base64.OutputStream} in ENCODE mode.
+ *
+ * @param out the java.io.OutputStream to which data will be written.
+ * @since 1.3
+ */
+ public OutputStream(java.io.OutputStream out) {
+ this(out, ENCODE);
+ } // end constructor
+
+ /**
+ * Constructs a {@link Base64.OutputStream} in either ENCODE or DECODE mode.
+ *
+ *
Valid options:
+ *
+ *
+ * ENCODE or DECODE: Encode or Decode as data is read.
+ * DO_BREAK_LINES: don't break lines at 76 characters
+ * (only meaningful when encoding)
+ *
+ *
+ *
Example: new Base64.OutputStream( out, Base64.ENCODE )
+ *
+ * @param out the java.io.OutputStream to which data will be written.
+ * @param options Specified options.
+ * @see Base64#ENCODE
+ * @see Base64#DECODE
+ * @see Base64#DO_BREAK_LINES
+ * @since 1.3
+ */
+ public OutputStream(java.io.OutputStream out, int options) {
+ super(out);
+ this.breakLines = (options & DO_BREAK_LINES) != 0;
+ this.encode = (options & ENCODE) != 0;
+ this.bufferLength = encode ? 3 : 4;
+ this.buffer = new byte[bufferLength];
+ this.position = 0;
+ this.lineLength = 0;
+ this.suspendEncoding = false;
+ this.b4 = new byte[4];
+ this.options = options;
+ this.decodabet = getDecodabet(options);
+ } // end constructor
+
+ /**
+ * Writes the byte to the output stream after converting to/from Base64 notation. When encoding,
+ * bytes are buffered three at a time before the output stream actually gets a write() call.
+ * When decoding, bytes are buffered four at a time.
+ *
+ * @param theByte the byte to write
+ * @since 1.3
+ */
+ @Override
+ public void write(int theByte) throws java.io.IOException {
+ // Encoding suspended?
+ if (suspendEncoding) {
+ this.out.write(theByte);
+ return;
+ } // end if: supsended
+
+ // Encode?
+ if (encode) {
+ buffer[position++] = (byte) theByte;
+ if (position >= bufferLength) { // Enough to encode.
+
+ this.out.write(encode3to4(b4, buffer, bufferLength, options));
+
+ lineLength += 4;
+ if (breakLines && lineLength >= MAX_LINE_LENGTH) {
+ this.out.write(NEW_LINE);
+ lineLength = 0;
+ } // end if: end of line
+
+ position = 0;
+ } // end if: enough to output
+ } // end if: encoding
+
+ // Else, Decoding
+ else {
+ // Meaningful Base64 character?
+ if (decodabet[theByte & 0x7f] > WHITE_SPACE_ENC) {
+ buffer[position++] = (byte) theByte;
+ if (position >= bufferLength) { // Enough to output.
+
+ int len = Base64.decode4to3(buffer, 0, b4, 0, options);
+ out.write(b4, 0, len);
+ position = 0;
+ } // end if: enough to output
+ } // end if: meaningful base64 character
+ else if (decodabet[theByte & 0x7f] != WHITE_SPACE_ENC) {
+ throw new java.io.IOException("Invalid character in Base64 data.");
+ } // end else: not white space either
+ } // end else: decoding
+ } // end write
+
+ /**
+ * Calls {@link #write(int)} repeatedly until len bytes are written.
+ *
+ * @param theBytes array from which to read bytes
+ * @param off offset for array
+ * @param len max number of bytes to read into array
+ * @since 1.3
+ */
+ @Override
+ public void write(byte[] theBytes, int off, int len) throws java.io.IOException {
+ // Encoding suspended?
+ if (suspendEncoding) {
+ this.out.write(theBytes, off, len);
+ return;
+ } // end if: supsended
+
+ for (int i = 0; i < len; i++) {
+ write(theBytes[off + i]);
+ } // end for: each byte written
+ } // end write
+
+ /**
+ * Method added by PHIL. [Thanks, PHIL. -Rob] This pads the buffer without closing the stream.
+ *
+ * @throws java.io.IOException if there's an error.
+ */
+ public void flushBase64() throws java.io.IOException {
+ if (position > 0) {
+ if (encode) {
+ out.write(encode3to4(b4, buffer, position, options));
+ position = 0;
+ } // end if: encoding
+ else {
+ throw new java.io.IOException("Base64 input not properly padded.");
+ } // end else: decoding
+ } // end if: buffer partially full
+ } // end flush
+
+ /**
+ * Flushes and closes (I think, in the superclass) the stream.
+ *
+ * @since 1.3
+ */
+ @Override
+ public void close() throws java.io.IOException {
+ // 1. Ensure that pending characters are written
+ flushBase64();
+
+ // 2. Actually close the stream
+ // Base class both flushes and closes.
+ super.close();
+
+ buffer = null;
+ out = null;
+ } // end close
+
+ /**
+ * Suspends encoding of the stream. May be helpful if you need to embed a piece of
+ * base64-encoded data in a stream.
+ *
+ * @throws java.io.IOException if there's an error flushing
+ * @since 1.5.1
+ */
+ public void suspendEncoding() throws java.io.IOException {
+ flushBase64();
+ this.suspendEncoding = true;
+ } // end suspendEncoding
+
+ /**
+ * Resumes encoding of the stream. May be helpful if you need to embed a piece of base64-encoded
+ * data in a stream.
+ *
+ * @since 1.5.1
+ */
+ public void resumeEncoding() {
+ this.suspendEncoding = false;
+ } // end resumeEncoding
+ } // end inner class OutputStream
+} // end class Base64
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/ClassRegistrar.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/ClassRegistrar.java
new file mode 100644
index 0000000000..b69829f66e
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/ClassRegistrar.java
@@ -0,0 +1,53 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill;
+
+import com.esotericsoftware.kryo.Kryo;
+
+/** Register this class to be used with the default serializer for this class */
+public class ClassRegistrar implements IKryoRegistrar {
+ final Class klass;
+
+ public ClassRegistrar(Class cls) {
+ klass = cls;
+ }
+
+ public Class getRegisteredClass() {
+ return klass;
+ }
+
+ @Override
+ public void apply(Kryo k) {
+ k.register(klass);
+ }
+
+ @Override
+ public int hashCode() {
+ return klass.hashCode();
+ }
+
+ @Override
+ public boolean equals(Object that) {
+ if (null == that) {
+ return false;
+ } else if (that instanceof ClassRegistrar) {
+ return klass.equals(((ClassRegistrar) that).klass);
+ } else {
+ return false;
+ }
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/IKryoRegistrar.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/IKryoRegistrar.java
new file mode 100644
index 0000000000..6df97d42fa
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/IKryoRegistrar.java
@@ -0,0 +1,29 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill;
+
+import com.esotericsoftware.kryo.Kryo;
+import java.io.Serializable;
+
+/**
+ * A Registrar adds registrations to a given Kryo instance. Examples would be a registrar that
+ * registers serializers for all objects in a given package. comes from Storm, which took it from
+ * cascading.kryo
+ */
+public interface IKryoRegistrar extends Serializable {
+ void apply(Kryo k);
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/KryoInstantiator.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/KryoInstantiator.java
new file mode 100644
index 0000000000..34c1ddece2
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/KryoInstantiator.java
@@ -0,0 +1,112 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill;
+
+import com.esotericsoftware.kryo.Kryo;
+import java.io.Serializable;
+import org.objenesis.strategy.InstantiatorStrategy;
+
+/**
+ * Class to create a new Kryo instance. Used in initial configuration or pooling of Kryo objects.
+ * These objects are immutable (and hopefully Kryo serializable)
+ */
+public class KryoInstantiator implements Serializable {
+ public Kryo newKryo() {
+ return new Kryo();
+ }
+
+ /** Use this to set a specific classloader */
+ public KryoInstantiator setClassLoader(final ClassLoader cl) {
+ return new KryoInstantiator() {
+ public Kryo newKryo() {
+ Kryo k = KryoInstantiator.this.newKryo();
+ k.setClassLoader(cl);
+ return k;
+ }
+ };
+ }
+ /** If true, Kryo will error if it sees a class that has not been registered */
+ public KryoInstantiator setInstantiatorStrategy(final InstantiatorStrategy inst) {
+ return new KryoInstantiator() {
+ public Kryo newKryo() {
+ Kryo k = KryoInstantiator.this.newKryo();
+ k.setInstantiatorStrategy(inst);
+ return k;
+ }
+ };
+ }
+
+ /**
+ * If true, Kryo keeps a map of all the objects it has seen. this can use a ton of memory on
+ * hadoop, but save serialization costs in some cases
+ */
+ public KryoInstantiator setReferences(final boolean ref) {
+ return new KryoInstantiator() {
+ public Kryo newKryo() {
+ Kryo k = KryoInstantiator.this.newKryo();
+ /**
+ * Kryo 2.17, used in storm, has this method returning void, 2.21 has it returning boolean.
+ * Try not to call the method if you don't need to.
+ */
+ if (k.getReferences() != ref) {
+ k.setReferences(ref);
+ }
+ return k;
+ }
+ };
+ }
+
+ /** If true, Kryo will error if it sees a class that has not been registered */
+ public KryoInstantiator setRegistrationRequired(final boolean req) {
+ return new KryoInstantiator() {
+ public Kryo newKryo() {
+ Kryo k = KryoInstantiator.this.newKryo();
+ /**
+ * Try to avoid calling this method if you don't need to. We've been burned by binary
+ * compatibility with Kryo
+ */
+ if (k.isRegistrationRequired() != req) {
+ k.setRegistrationRequired(req);
+ }
+ return k;
+ }
+ };
+ }
+ /**
+ * Use Thread.currentThread().getContextClassLoader() as the ClassLoader where ther newKryo is
+ * called
+ */
+ public KryoInstantiator setThreadContextClassLoader() {
+ return new KryoInstantiator() {
+ public Kryo newKryo() {
+ Kryo k = KryoInstantiator.this.newKryo();
+ k.setClassLoader(Thread.currentThread().getContextClassLoader());
+ return k;
+ }
+ };
+ }
+
+ public KryoInstantiator withRegistrar(final IKryoRegistrar r) {
+ return new KryoInstantiator() {
+ public Kryo newKryo() {
+ Kryo k = KryoInstantiator.this.newKryo();
+ r.apply(k);
+ return k;
+ }
+ };
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/KryoPool.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/KryoPool.java
new file mode 100644
index 0000000000..02c6c9cbad
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/KryoPool.java
@@ -0,0 +1,135 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill;
+
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+
+/** Simple ResourcePool to save on Kryo instances, which are expensive to allocate */
+public abstract class KryoPool extends ResourcePool {
+
+ protected KryoPool(int poolSize) {
+ super(poolSize);
+ }
+
+ @Override
+ public void release(SerDeState st) {
+ st.clear();
+ super.release(st);
+ }
+ /** Output is created with new Output(outBufferMin, outBufferMax); */
+ public static KryoPool withBuffer(
+ int poolSize, final KryoInstantiator ki, final int outBufferMin, final int outBufferMax) {
+ return new KryoPool(poolSize) {
+ protected SerDeState newInstance() {
+ return new SerDeState(ki.newKryo(), new Input(), new Output(outBufferMin, outBufferMax));
+ }
+ };
+ }
+
+ /**
+ * Output is created with new Output(new ByteArrayOutputStream()) This will automatically resize
+ * internally
+ */
+ public static KryoPool withByteArrayOutputStream(int poolSize, final KryoInstantiator ki) {
+ return new KryoPool(poolSize) {
+ protected SerDeState newInstance() {
+ return new SerDeState(ki.newKryo(), new Input(), new Output(new ByteArrayOutputStream())) {
+ /*
+ * We have to take extra care of the ByteArrayOutputStream
+ */
+ @Override
+ public void clear() {
+ super.clear();
+ ByteArrayOutputStream byteStream = (ByteArrayOutputStream) output.getOutputStream();
+ byteStream.reset();
+ }
+
+ @Override
+ public byte[] outputToBytes() {
+ output.flush();
+ ByteArrayOutputStream byteStream = (ByteArrayOutputStream) output.getOutputStream();
+ return byteStream.toByteArray();
+ }
+
+ @Override
+ public void writeOutputTo(OutputStream os) throws IOException {
+ output.flush();
+ ByteArrayOutputStream byteStream = (ByteArrayOutputStream) output.getOutputStream();
+ byteStream.writeTo(os);
+ }
+ };
+ }
+ };
+ }
+
+ public T deepCopy(T obj) {
+ return (T) fromBytes(toBytesWithoutClass(obj), obj.getClass());
+ }
+
+ public Object fromBytes(byte[] ary) {
+ SerDeState serde = borrow();
+ try {
+ serde.setInput(ary);
+ return serde.readClassAndObject();
+ } finally {
+ release(serde);
+ }
+ }
+
+ public T fromBytes(byte[] ary, Class cls) {
+ SerDeState serde = borrow();
+ try {
+ serde.setInput(ary);
+ return serde.readObject(cls);
+ } finally {
+ release(serde);
+ }
+ }
+
+ public byte[] toBytesWithClass(Object obj) {
+ SerDeState serde = borrow();
+ try {
+ serde.writeClassAndObject(obj);
+ return serde.outputToBytes();
+ } finally {
+ release(serde);
+ }
+ }
+
+ public byte[] toBytesWithoutClass(Object obj) {
+ SerDeState serde = borrow();
+ try {
+ serde.writeObject(obj);
+ return serde.outputToBytes();
+ } finally {
+ release(serde);
+ }
+ }
+
+ public boolean hasRegistration(Class obj) {
+ SerDeState serde = borrow();
+ try {
+ return serde.hasRegistration(obj);
+ } finally {
+ release(serde);
+ }
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/ReflectingDefaultRegistrar.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/ReflectingDefaultRegistrar.java
new file mode 100644
index 0000000000..0f2d581276
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/ReflectingDefaultRegistrar.java
@@ -0,0 +1,62 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.Serializer;
+
+/** Set the default serializers for subclasses of the given class */
+public class ReflectingDefaultRegistrar implements IKryoRegistrar {
+ final Class klass;
+ // Some serializers handle any class (FieldsSerializer, for instance)
+ final Class> serializerKlass;
+
+ public ReflectingDefaultRegistrar(Class cls, Class> ser) {
+ klass = cls;
+ serializerKlass = ser;
+ }
+
+ public Class getRegisteredClass() {
+ return klass;
+ }
+
+ public Class> getSerializerClass() {
+ return serializerKlass;
+ }
+
+ @Override
+ public void apply(Kryo k) {
+ k.addDefaultSerializer(klass, serializerKlass);
+ }
+
+ @Override
+ public int hashCode() {
+ return klass.hashCode() ^ serializerKlass.hashCode();
+ }
+
+ @Override
+ public boolean equals(Object that) {
+ if (null == that) {
+ return false;
+ } else if (that instanceof ReflectingDefaultRegistrar) {
+ return klass.equals(((ReflectingDefaultRegistrar) that).klass)
+ && serializerKlass.equals(((ReflectingDefaultRegistrar) that).serializerKlass);
+ } else {
+ return false;
+ }
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/ReflectingRegistrar.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/ReflectingRegistrar.java
new file mode 100644
index 0000000000..480773c2e9
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/ReflectingRegistrar.java
@@ -0,0 +1,75 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.Serializer;
+import com.esotericsoftware.kryo.util.Util;
+
+/**
+ * Use reflection to instantiate a serializer. Used when serializer classes are written to config
+ * files
+ */
+public class ReflectingRegistrar implements IKryoRegistrar {
+ final Class klass;
+ // Some serializers handle any class (FieldsSerializer, for instance)
+ final Class> serializerKlass;
+
+ public Class getRegisteredClass() {
+ return klass;
+ }
+
+ public Class> getSerializerClass() {
+ return serializerKlass;
+ }
+
+ public ReflectingRegistrar(Class cls, Class> ser) {
+ klass = cls;
+ serializerKlass = ser;
+ }
+
+ @Override
+ public void apply(Kryo k) {
+ try {
+ k.register(klass, serializerKlass.newInstance());
+ } catch (Exception ex) {
+ throw new IllegalArgumentException(
+ "Unable to create serializer \""
+ + serializerKlass.getName()
+ + "\" for class: "
+ + Util.className(klass),
+ ex);
+ }
+ }
+
+ @Override
+ public int hashCode() {
+ return klass.hashCode() ^ serializerKlass.hashCode();
+ }
+
+ @Override
+ public boolean equals(Object that) {
+ if (null == that) {
+ return false;
+ } else if (that instanceof ReflectingRegistrar) {
+ return klass.equals(((ReflectingRegistrar) that).klass)
+ && serializerKlass.equals(((ReflectingRegistrar) that).serializerKlass);
+ } else {
+ return false;
+ }
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/ResourcePool.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/ResourcePool.java
new file mode 100644
index 0000000000..18c0c2e039
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/ResourcePool.java
@@ -0,0 +1,51 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill;
+
+import java.util.concurrent.ArrayBlockingQueue;
+
+/** Simple ResourcePool to save on Kryo instances, which are expensive to allocate */
+public abstract class ResourcePool {
+ private ArrayBlockingQueue pool;
+
+ protected abstract T newInstance();
+
+ public ResourcePool(int size) {
+ pool = new ArrayBlockingQueue(size);
+ }
+
+ public T borrow() {
+ try {
+ T res = pool.poll();
+ if (null == res) {
+ return newInstance();
+ } else {
+ return res;
+ }
+ } catch (Exception x) {
+ throw new RuntimeException(x);
+ }
+ }
+
+ public void release(T item) {
+ try {
+ pool.offer(item);
+ } catch (Exception x) {
+ throw new RuntimeException(x);
+ }
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/SerDeState.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/SerDeState.java
new file mode 100644
index 0000000000..3b7bdc9e4b
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/SerDeState.java
@@ -0,0 +1,97 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+
+/**
+ * This holds a Kryo Instance, Input and Output so that these objects can be pooled and no
+ * reallocated on each serialization.
+ */
+public class SerDeState {
+ protected final Kryo kryo;
+ protected final Input input;
+ protected final Output output;
+ // To reset the Input
+ static final byte[] EMPTY_BUFFER = new byte[0];
+
+ protected SerDeState(Kryo k, Input in, Output out) {
+ kryo = k;
+ input = in;
+ output = out;
+ }
+
+ /** Call this when to reset the state to the initial state */
+ public void clear() {
+ input.setBuffer(EMPTY_BUFFER);
+ output.clear();
+ }
+
+ public void setInput(byte[] in) {
+ input.setBuffer(in);
+ }
+
+ public void setInput(byte[] in, int offset, int count) {
+ input.setBuffer(in, offset, count);
+ }
+
+ public void setInput(InputStream in) {
+ input.setInputStream(in);
+ }
+
+ public int numOfWrittenBytes() {
+ return (int) output.total();
+ }
+
+ public int numOfReadBytes() {
+ return (int) input.total();
+ }
+
+ // Common operations:
+ public T readObject(Class cls) {
+ return kryo.readObject(input, cls);
+ }
+
+ public Object readClassAndObject() {
+ return kryo.readClassAndObject(input);
+ }
+
+ public void writeObject(Object o) {
+ kryo.writeObject(output, o);
+ }
+
+ public void writeClassAndObject(Object o) {
+ kryo.writeClassAndObject(output, o);
+ }
+
+ public byte[] outputToBytes() {
+ return output.toBytes();
+ }
+ // There for ByteArrayOutputStream cases this can be optimized
+ public void writeOutputTo(OutputStream os) throws IOException {
+ os.write(output.getBuffer(), 0, output.position());
+ }
+
+ public boolean hasRegistration(Class obj) {
+ return kryo.getRegistration(obj) != null;
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/SingleDefaultRegistrar.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/SingleDefaultRegistrar.java
new file mode 100644
index 0000000000..01d2badf24
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/SingleDefaultRegistrar.java
@@ -0,0 +1,36 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.Serializer;
+
+/** Set the default serializers for subclasses of the given class */
+public class SingleDefaultRegistrar implements IKryoRegistrar {
+ final Class klass;
+ final Serializer serializer;
+
+ public SingleDefaultRegistrar(Class cls, Serializer ser) {
+ klass = cls;
+ serializer = ser;
+ }
+
+ @Override
+ public void apply(Kryo k) {
+ k.addDefaultSerializer(klass, serializer);
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/SingleRegistrar.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/SingleRegistrar.java
new file mode 100644
index 0000000000..c2c18fddaa
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/SingleRegistrar.java
@@ -0,0 +1,35 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.Serializer;
+
+public class SingleRegistrar implements IKryoRegistrar {
+ final Class klass;
+ final Serializer serializer;
+
+ public SingleRegistrar(Class cls, Serializer ser) {
+ klass = cls;
+ serializer = ser;
+ }
+
+ @Override
+ public void apply(Kryo k) {
+ k.register(klass, serializer);
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/Config.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/Config.java
new file mode 100644
index 0000000000..6cbc58d9a3
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/Config.java
@@ -0,0 +1,64 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill.config;
+
+/** Particular systems subclass this to reuse existing configured Instantiators */
+public abstract class Config {
+ /** Return null if this key is undefined */
+ public abstract String get(String key);
+
+ public abstract void set(String key, String value);
+
+ public String getOrElse(String key, String def) {
+ String val = get(key);
+ if (null == val) {
+ return def;
+ } else {
+ return val;
+ }
+ }
+
+ public boolean contains(String key) {
+ return get(key) != null;
+ }
+
+ public Boolean getBoolean(String key) {
+ String bval = get(key);
+ if (null == bval) {
+ return null;
+ } else {
+ return Boolean.valueOf(bval);
+ }
+ }
+
+ public boolean getBoolean(String key, boolean defval) {
+ String bval = get(key);
+ if (null == bval) {
+ return defval;
+ } else {
+ return Boolean.valueOf(bval).booleanValue();
+ }
+ }
+
+ public void setBoolean(String key, Boolean v) {
+ if (null == v) {
+ set(key, null);
+ } else {
+ set(key, v.toString());
+ }
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/ConfigurationException.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/ConfigurationException.java
new file mode 100644
index 0000000000..1180256c31
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/ConfigurationException.java
@@ -0,0 +1,31 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill.config;
+
+public class ConfigurationException extends Exception {
+ public ConfigurationException(String message) {
+ super(message);
+ }
+
+ public ConfigurationException(Exception ex) {
+ super(ex);
+ }
+
+ public ConfigurationException(String message, Exception ex) {
+ super(message, ex);
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/ConfiguredInstantiator.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/ConfiguredInstantiator.java
new file mode 100644
index 0000000000..d99db57641
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/ConfiguredInstantiator.java
@@ -0,0 +1,198 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill.config;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import com.spotify.scio.vendor.chill.Base64;
+import com.spotify.scio.vendor.chill.KryoInstantiator;
+import java.lang.reflect.InvocationTargetException;
+
+/**
+ * This is the standard Config based KryoInstantiator. It delegates to another KryoInstantiator that
+ * is described a Config object. This is either done via reflection or reflection AND serialization.
+ *
+ *
If the KEY is not set, the delegate is the default: new KryoInstantiator()
+ *
+ *
In the case of reflection, the class name of the delegate instantiator is given.
+ *
+ *
In the case of serialization, we first reflect to create the KryoInstatiator we use to get the
+ * Kryo we need to deserialize.
+ */
+public class ConfiguredInstantiator extends KryoInstantiator {
+ protected final KryoInstantiator delegate;
+
+ /**
+ * Key we use to configure this class. Format: {@literal(:)} if there is no serialized instantiator, we use the reflected
+ * instance as the delegate
+ */
+ public static final String KEY = "com.spotify.scio.vendor.chill.config.configuredinstantiator";
+
+ public ConfiguredInstantiator(Config conf) throws ConfigurationException {
+ String key = conf.get(KEY);
+ if (null == key) {
+ delegate = new KryoInstantiator();
+ } else {
+ String[] parts = fastSplitKey(key);
+ if (parts == null) {
+ throw new ConfigurationException("Invalid Config Key: " + conf.get(KEY));
+ }
+ KryoInstantiator reflected = null;
+ try {
+ reflected =
+ reflect(
+ (Class)
+ Class.forName(parts[0], true, Thread.currentThread().getContextClassLoader()),
+ conf);
+ } catch (ClassNotFoundException x) {
+ throw new ConfigurationException("Could not find class for: " + parts[0], x);
+ }
+
+ if (parts.length == 2) {
+ delegate = fastDeserialize(reflected.newKryo(), parts[1]);
+ if (null == delegate) {
+ throw new ConfigurationException("Null delegate from: " + parts[1]);
+ }
+ } else {
+ delegate = reflected;
+ }
+ }
+ }
+
+ /** Calls through to the delegate */
+ public Kryo newKryo() {
+ return delegate.newKryo();
+ }
+
+ /** Return the delegated KryoInstantiator */
+ public KryoInstantiator getDelegate() {
+ return delegate;
+ }
+
+ /**
+ * In this mode, we are just refecting to another delegated class. This is preferred if you don't
+ * have any configuration to do at runtime (i.e. you can make a named class that has all the logic
+ * for your KryoInstantiator).
+ */
+ public static void setReflect(Config conf, Class instClass) {
+ conf.set(KEY, instClass.getName());
+ }
+
+ /**
+ * This instantiates a KryoInstantiator by: 1) checking if it has a constructor that takes Config
+ * 2) checking for a no-arg constructor
+ */
+ static KryoInstantiator reflect(Class instClass, Config optConf)
+ throws ConfigurationException {
+ try {
+ try {
+ return instClass.getConstructor(Config.class).newInstance(optConf);
+ } catch (NoSuchMethodException ex3) {
+ return instClass.newInstance();
+ }
+ } catch (InstantiationException x) {
+ throw new ConfigurationException(x);
+ } catch (IllegalAccessException x) {
+ throw new ConfigurationException(x);
+ } catch (InvocationTargetException x) {
+ throw new ConfigurationException(x);
+ }
+ }
+
+ /**
+ * Use the default KryoInstantiator to serialize the KryoInstantiator ki same as:
+ * setSerialized(conf, KryoInstantiator.class, ki)
+ */
+ public static void setSerialized(Config conf, KryoInstantiator ki) throws ConfigurationException {
+ setSerialized(conf, KryoInstantiator.class, ki);
+ }
+
+ /**
+ * If this reflector needs config to be set, that should be done PRIOR to making this call. This
+ * mode serializes an instance (ki) to be used as the delegate. Only use this mode if reflection
+ * alone will not work.
+ */
+ public static void setSerialized(
+ Config conf, Class reflector, KryoInstantiator ki)
+ throws ConfigurationException {
+ KryoInstantiator refki = reflect(reflector, conf);
+ String kistr = serialize(refki.newKryo(), ki);
+ // Verify, that deserialization works:
+ KryoInstantiator deser =
+ deserialize(refki.newKryo(), kistr); // ignore the result, just see if it throws
+ deser.newKryo(); // just see if we can still create it
+ conf.set(KEY, reflector.getName() + ":" + kistr);
+ }
+
+ protected static KryoInstantiator deserialize(Kryo k, String base64Value)
+ throws ConfigurationException {
+ try {
+ return (KryoInstantiator) k.readClassAndObject(new Input(Base64.decode(base64Value)));
+ } catch (java.io.IOException iox) {
+ throw new ConfigurationException("could not deserialize: " + base64Value, iox);
+ }
+ }
+
+ protected static String serialize(Kryo k, KryoInstantiator ki) {
+ Output out = new Output(1 << 10, 1 << 19); // 1 MB in config is too much
+ k.writeClassAndObject(out, ki);
+ return Base64.encodeBytes(out.toBytes());
+ }
+
+ /**
+ * Simple class to hold the cached copy of the latest kryo instantiator. As well as its
+ * corresponding base64 encoded data.
+ */
+ private static class CachedKryoInstantiator {
+ public final KryoInstantiator kryoInstantiator;
+ public final String base64Value;
+
+ public CachedKryoInstantiator(KryoInstantiator ki, String bv) {
+ kryoInstantiator = ki;
+ base64Value = bv;
+ }
+ }
+
+ private static CachedKryoInstantiator cachedKryoInstantiator = null;
+
+ private static synchronized KryoInstantiator fastDeserialize(Kryo k, String base64Value)
+ throws ConfigurationException {
+ if (cachedKryoInstantiator == null || !cachedKryoInstantiator.base64Value.equals(base64Value)) {
+ cachedKryoInstantiator = new CachedKryoInstantiator(deserialize(k, base64Value), base64Value);
+ }
+ return cachedKryoInstantiator.kryoInstantiator;
+ }
+
+ /**
+ * Java's string split is very expensive due to regexes. Implement our own simple version instead.
+ */
+ public static String[] fastSplitKey(String key) {
+ int i = key.indexOf(':');
+ if (-1 == i) {
+ return new String[] {key};
+ } else {
+ int j = key.indexOf(':', i + 1);
+ if (-1 != j) {
+ return null;
+ } else {
+ return new String[] {key.substring(0, i), key.substring(i + 1)};
+ }
+ }
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/JavaMapConfig.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/JavaMapConfig.java
new file mode 100644
index 0000000000..1c88f4b33d
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/JavaMapConfig.java
@@ -0,0 +1,49 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill.config;
+
+import java.util.Map;
+
+/** This takes a raw Map and calls toString on the objects before returning them as values */
+public class JavaMapConfig extends Config {
+
+ final Map conf;
+
+ public JavaMapConfig(Map conf) {
+ this.conf = conf;
+ }
+
+ public JavaMapConfig() {
+ this(new java.util.HashMap());
+ }
+ /** Return null if this key is undefined */
+ /** Return null if this key is undefined */
+ @Override
+ public String get(String key) {
+ Object value = conf.get(key);
+ if (null != value) {
+ return value.toString();
+ } else {
+ return null;
+ }
+ }
+
+ @Override
+ public void set(String key, String value) {
+ conf.put(key, value);
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/ReflectingInstantiator.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/ReflectingInstantiator.java
new file mode 100644
index 0000000000..16b5cc461c
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/config/ReflectingInstantiator.java
@@ -0,0 +1,302 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill.config;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.Serializer;
+import com.spotify.scio.vendor.chill.ClassRegistrar;
+import com.spotify.scio.vendor.chill.IKryoRegistrar;
+import com.spotify.scio.vendor.chill.KryoInstantiator;
+import com.spotify.scio.vendor.chill.ReflectingDefaultRegistrar;
+import com.spotify.scio.vendor.chill.ReflectingRegistrar;
+import java.util.ArrayList;
+import java.util.List;
+import org.objenesis.strategy.InstantiatorStrategy;
+import org.objenesis.strategy.StdInstantiatorStrategy;
+
+public class ReflectingInstantiator extends KryoInstantiator {
+
+ final boolean regRequired;
+ final boolean skipMissing;
+ final Class kryoClass;
+ final Class instStratClass;
+ final List registrations;
+ final List defaultRegistrations;
+
+ public ReflectingInstantiator(Config conf) throws ConfigurationException {
+ regRequired = conf.getBoolean(REGISTRATION_REQUIRED, REGISTRATION_REQUIRED_DEFAULT);
+ skipMissing = conf.getBoolean(SKIP_MISSING, SKIP_MISSING_DEFAULT);
+
+ try {
+ kryoClass =
+ (Class)
+ Class.forName(
+ conf.getOrElse(KRYO_CLASS, KRYO_CLASS_DEFAULT),
+ true,
+ Thread.currentThread().getContextClassLoader());
+ instStratClass =
+ (Class)
+ Class.forName(
+ conf.getOrElse(INSTANTIATOR_STRATEGY_CLASS, INSTANTIATOR_STRATEGY_CLASS_DEFAULT),
+ true,
+ Thread.currentThread().getContextClassLoader());
+
+ registrations = (List) buildRegistrars(conf.get(REGISTRATIONS), false);
+ defaultRegistrations =
+ (List) buildRegistrars(conf.get(DEFAULT_REGISTRATIONS), true);
+ // Make sure we can make a newKryo, this throws a runtime exception if not.
+ newKryoWithEx();
+ } catch (ClassNotFoundException x) {
+ throw new ConfigurationException(x);
+ } catch (InstantiationException x) {
+ throw new ConfigurationException(x);
+ } catch (IllegalAccessException x) {
+ throw new ConfigurationException(x);
+ }
+ }
+
+ /** Create an instance using the defaults for non-listed params */
+ public ReflectingInstantiator(
+ Iterable classRegistrations,
+ Iterable registrations,
+ Iterable defaults) {
+ this(
+ Kryo.class,
+ StdInstantiatorStrategy.class,
+ classRegistrations,
+ registrations,
+ defaults,
+ REGISTRATION_REQUIRED_DEFAULT,
+ SKIP_MISSING_DEFAULT);
+ }
+
+ public ReflectingInstantiator(
+ Class kryoClass,
+ Class stratClass,
+ Iterable classRegistrations,
+ Iterable registrations,
+ Iterable defaults,
+ boolean regRequired,
+ boolean skipMissing) {
+
+ this.kryoClass = kryoClass;
+ instStratClass = stratClass;
+ this.regRequired = regRequired;
+ this.skipMissing = skipMissing;
+
+ this.registrations = new ArrayList();
+ for (IKryoRegistrar cr : classRegistrations) {
+ this.registrations.add(cr);
+ }
+ for (IKryoRegistrar rr : registrations) {
+ this.registrations.add(rr);
+ }
+
+ defaultRegistrations = new ArrayList();
+ for (ReflectingDefaultRegistrar rdr : defaults) {
+ defaultRegistrations.add(rdr);
+ }
+ }
+
+ public void set(Config conf) throws ConfigurationException {
+ conf.setBoolean(REGISTRATION_REQUIRED, regRequired);
+ conf.setBoolean(SKIP_MISSING, skipMissing);
+
+ conf.set(KRYO_CLASS, kryoClass.getName());
+ conf.set(INSTANTIATOR_STRATEGY_CLASS, instStratClass.getName());
+
+ conf.set(REGISTRATIONS, registrarsToString(registrations));
+ conf.set(DEFAULT_REGISTRATIONS, registrarsToString(defaultRegistrations));
+ }
+
+ // This one adds expeption annotations that the interface does not have
+ protected Kryo newKryoWithEx() throws InstantiationException, IllegalAccessException {
+ Kryo k = kryoClass.newInstance();
+ k.setInstantiatorStrategy(instStratClass.newInstance());
+ k.setRegistrationRequired(regRequired);
+ for (IKryoRegistrar kr : registrations) {
+ kr.apply(k);
+ }
+ for (IKryoRegistrar dkr : defaultRegistrations) {
+ dkr.apply(k);
+ }
+ return k;
+ }
+
+ @Override
+ public Kryo newKryo() {
+ try {
+ return newKryoWithEx();
+ } catch (InstantiationException x) {
+ throw new RuntimeException(x);
+ } catch (IllegalAccessException x) {
+ throw new RuntimeException(x);
+ }
+ }
+
+ /** All keys are prefixed with this string */
+ public static final String prefix = "com.spotify.scio.vendor.chill.config.reflectinginstantiator";
+
+ /**
+ * Name of the subclass of kryo to instantiate to start with. If this is empty, we use Kryo.class
+ */
+ public static final String KRYO_CLASS = prefix + ".kryoclass";
+
+ public static final String KRYO_CLASS_DEFAULT = Kryo.class.getName();
+ /**
+ * Name of the InstatiatorStrategy to use. If this is empty, we use
+ * org.objenesis.strategy.StdInstantiatorStrategy
+ */
+ public static final String INSTANTIATOR_STRATEGY_CLASS = prefix + ".instantiatorstrategyclass";
+
+ public static final String INSTANTIATOR_STRATEGY_CLASS_DEFAULT =
+ StdInstantiatorStrategy.class.getName();
+ /**
+ * KRYO_REGISTRATIONS holds a colon-separated list of classes to register with Kryo. For example,
+ * the following value:
+ *
+ *
will direct KryoFactory to register someClass and thirdClass with custom serializers and
+ * otherClass with Kryo's FieldsSerializer.
+ */
+ public static final String REGISTRATIONS = prefix + ".registrations";
+
+ /**
+ * DEFAULT_REGISTRATIONS holds a colon-separated list of classes or interfaces to register with
+ * Kryo. Default Registrations are searched after basic registrations, and have the ability to
+ * capture objects that are assignable from the hierarchy's superclass. For example, the following
+ * value:
+ *
+ *
will configure to serializeobjects that extend from someClass with someSerializer, and
+ * objects that extend someInterface with otherSerializer.
+ */
+ public static final String DEFAULT_REGISTRATIONS = prefix + ".defaultregistrations";
+
+ /**
+ * If SKIP_MISSING is set to false, Kryo will throw an error when Cascading tries to register a
+ * class or serialization that doesn't exist.
+ */
+ public static final String SKIP_MISSING = prefix + ".skipmissing";
+
+ public static final boolean SKIP_MISSING_DEFAULT = false;
+
+ /**
+ * If REGISTRATION_REQUIRED is set to false, Kryo will try to serialize all java objects, not just
+ * those with custom serializations registered.
+ */
+ public static final String REGISTRATION_REQUIRED = prefix + ".registrationrequired";
+
+ public static final boolean REGISTRATION_REQUIRED_DEFAULT = false;
+
+ protected List buildRegistrars(String base, boolean isAddDefault)
+ throws ConfigurationException {
+ List builder = new ArrayList();
+
+ if (base == null) return builder;
+
+ for (String s : base.split(":")) {
+ String[] pair = s.split(",");
+ try {
+ switch (pair.length) {
+ case 1:
+ if (isAddDefault) {
+ throw new ConfigurationException(
+ "default serializers require class and serializer: " + base);
+ }
+ builder.add(
+ new ClassRegistrar(
+ Class.forName(pair[0], true, Thread.currentThread().getContextClassLoader())));
+ break;
+ case 2:
+ @SuppressWarnings("unchecked")
+ Class kls =
+ Class.forName(pair[0], true, Thread.currentThread().getContextClassLoader());
+ Class serializerClass =
+ (Class)
+ Class.forName(pair[1], true, Thread.currentThread().getContextClassLoader());
+ if (isAddDefault) {
+ builder.add(new ReflectingDefaultRegistrar(kls, serializerClass));
+ } else {
+ builder.add(new ReflectingRegistrar(kls, serializerClass));
+ }
+ break;
+ default:
+ throw new ConfigurationException(base + " is not well-formed.");
+ }
+ } catch (ClassNotFoundException e) {
+ if (skipMissing) {
+ System.err.println(
+ "Could not find serialization or class for " + pair[1] + ". Skipping registration.");
+ } else {
+ throw new ConfigurationException(e);
+ }
+ }
+ }
+ return builder;
+ }
+
+ protected String registrarsToString(Iterable registrars)
+ throws ConfigurationException {
+ StringBuilder builder = new StringBuilder();
+ boolean isFirst = true;
+ for (IKryoRegistrar reg : registrars) {
+ if (!isFirst) builder.append(":");
+ isFirst = false;
+ String part = null;
+ if (reg instanceof ClassRegistrar) {
+ ClassRegistrar r = (ClassRegistrar) reg;
+ part = r.getRegisteredClass().getName();
+ } else if (reg instanceof ReflectingRegistrar) {
+ ReflectingRegistrar r = (ReflectingRegistrar) reg;
+ part = r.getRegisteredClass().getName() + "," + r.getSerializerClass().getName();
+ } else if (reg instanceof ReflectingDefaultRegistrar) {
+ ReflectingDefaultRegistrar r = (ReflectingDefaultRegistrar) reg;
+ part = r.getRegisteredClass().getName() + "," + r.getSerializerClass().getName();
+ } else {
+ throw new ConfigurationException(
+ "Unknown type of reflecting registrar: " + reg.getClass().getName());
+ }
+ builder.append(part);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return kryoClass.hashCode() ^ registrations.hashCode() ^ defaultRegistrations.hashCode();
+ }
+
+ @Override
+ public boolean equals(Object that) {
+ if (null == that) {
+ return false;
+ } else if (that instanceof ReflectingInstantiator) {
+ ReflectingInstantiator thatri = (ReflectingInstantiator) that;
+ return (regRequired == thatri.regRequired)
+ && (skipMissing == thatri.skipMissing)
+ && kryoClass.equals(thatri.kryoClass)
+ && instStratClass.equals(thatri.instStratClass)
+ && registrations.equals(thatri.registrations)
+ && defaultRegistrations.equals(thatri.defaultRegistrations);
+ } else {
+ return false;
+ }
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/ArraysAsListSerializer.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/ArraysAsListSerializer.java
new file mode 100644
index 0000000000..cf3b79508a
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/ArraysAsListSerializer.java
@@ -0,0 +1,113 @@
+/*
+ * Copyright 2010 Martin Grotzke
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package com.spotify.scio.vendor.chill.java;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.Serializer;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import com.spotify.scio.vendor.chill.IKryoRegistrar;
+import com.spotify.scio.vendor.chill.SingleRegistrar;
+import java.lang.reflect.Array;
+import java.lang.reflect.Field;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A kryo {@link Serializer} for lists created via {@link Arrays#asList(Object...)}.
+ *
+ *
Note: This serializer does not support cyclic references, so if one of the objects gets set
+ * the list as attribute this might cause an error during deserialization.
+ *
+ * @author Martin Grotzke
+ */
+public class ArraysAsListSerializer extends Serializer> {
+
+ private static final Map, Class> primitives =
+ new HashMap, Class>(8, 1.0F);
+
+ static {
+ primitives.put(byte.class, Byte.class);
+ primitives.put(short.class, Short.class);
+ primitives.put(int.class, Integer.class);
+ primitives.put(long.class, Long.class);
+ primitives.put(char.class, Character.class);
+ primitives.put(float.class, Float.class);
+ primitives.put(double.class, Double.class);
+ primitives.put(boolean.class, Boolean.class);
+ }
+
+ public static IKryoRegistrar registrar() {
+ return new SingleRegistrar(Arrays.asList("").getClass(), new ArraysAsListSerializer());
+ }
+
+ private Field _arrayField;
+
+ public ArraysAsListSerializer() {
+ try {
+ _arrayField = Class.forName("java.util.Arrays$ArrayList").getDeclaredField("a");
+ _arrayField.setAccessible(true);
+ } catch (final Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Override
+ public List read(final Kryo kryo, final Input input, final Class> type) {
+ final int length = input.readInt(true);
+ Class componentType = kryo.readClass(input).getType();
+ try {
+ final Object items = Array.newInstance(getBoxedClass(componentType), length);
+ for (int i = 0; i < length; i++) {
+ Array.set(items, i, kryo.readClassAndObject(input));
+ }
+ return Arrays.asList((Object[]) items);
+ } catch (final Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Override
+ public void write(final Kryo kryo, final Output output, final List obj) {
+ try {
+ final Object[] array = (Object[]) _arrayField.get(obj);
+ output.writeInt(array.length, true);
+ final Class componentType = array.getClass().getComponentType();
+ kryo.writeClass(output, componentType);
+ for (final Object item : array) {
+ kryo.writeClassAndObject(output, item);
+ }
+ } catch (final RuntimeException e) {
+ // Don't eat and wrap RuntimeExceptions because the ObjectBuffer.write...
+ // handles SerializationException specifically (resizing the buffer)...
+ throw e;
+ } catch (final Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private static Class getBoxedClass(final Class c) {
+ if (c.isPrimitive()) {
+ Class x;
+ return (x = primitives.get(c)) != null ? x : c;
+ }
+ return c;
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/BitSetSerializer.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/BitSetSerializer.java
new file mode 100644
index 0000000000..0e3b13e7a9
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/BitSetSerializer.java
@@ -0,0 +1,117 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill.java;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoException;
+import com.esotericsoftware.kryo.Serializer;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import com.spotify.scio.vendor.chill.IKryoRegistrar;
+import com.spotify.scio.vendor.chill.SingleRegistrar;
+import java.io.Serializable;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.BitSet;
+
+public class BitSetSerializer extends Serializer implements Serializable {
+
+ public static IKryoRegistrar registrar() {
+ return new SingleRegistrar(BitSet.class, new BitSetSerializer());
+ }
+
+ private static final Field wordsField;
+ private static final Constructor bitSetConstructor;
+ private static final Method recalculateWordsInUseMethod;
+
+ static {
+ try {
+ wordsField = BitSet.class.getDeclaredField("words");
+ wordsField.setAccessible(true);
+ } catch (NoSuchFieldException e) {
+ throw new KryoException("Error while getting field 'words' of bitSet", e);
+ }
+ try {
+ bitSetConstructor = BitSet.class.getDeclaredConstructor(long[].class);
+ bitSetConstructor.setAccessible(true);
+ } catch (NoSuchMethodException e) {
+ throw new KryoException("Unable to get BitSet(long[]) constructor", e);
+ }
+ try {
+ recalculateWordsInUseMethod = BitSet.class.getDeclaredMethod("recalculateWordsInUse");
+ recalculateWordsInUseMethod.setAccessible(true);
+ } catch (NoSuchMethodException e) {
+ throw new KryoException("Unable to get BitSet.recalculateWordsInUse() method", e);
+ }
+ }
+
+ @Override
+ public void write(Kryo kryo, Output output, BitSet bitSet) {
+ long words[] = null;
+ // its sufficent to get only the 'words' field because
+ // we can recompute the wordsInUse after deserialization
+ try {
+ words = (long[]) wordsField.get(bitSet);
+ } catch (IllegalAccessException e) {
+ throw new KryoException("Error while accessing field 'words' of bitSet", e);
+ }
+ output.writeInt(words.length, true);
+
+ for (int i = 0; i < words.length; i++) {
+ output.writeLong(words[i]);
+ }
+ }
+
+ @Override
+ public BitSet read(Kryo kryo, Input input, Class bitSetClass) {
+ int len = input.readInt(true);
+ long[] target = new long[len];
+
+ for (int i = 0; i < len; i++) {
+ target[i] = input.readLong();
+ }
+
+ BitSet ret = null;
+
+ try {
+ ret = bitSetConstructor.newInstance(target);
+ } catch (InstantiationException e) {
+ throw new KryoException("Exception thrown while creating new instance BitSetConstructor", e);
+ } catch (IllegalAccessException e) {
+ throw new KryoException(
+ "Exception thrown while creating new instance of BitSetConstructor", e);
+ } catch (InvocationTargetException e) {
+ throw new KryoException(
+ "Exception thrown while creating new instance of BitSetConstructor", e);
+ } catch (IllegalArgumentException e) {
+ throw new KryoException(
+ "Exception thrown while creating new instance of BitSetConstructor", e);
+ }
+ try {
+ recalculateWordsInUseMethod.invoke(ret);
+ } catch (InvocationTargetException e) {
+ throw new KryoException("Exception thrown while invoking recalculateWordsInUseMethod", e);
+ } catch (IllegalAccessException e) {
+ throw new KryoException("Exception thrown while invoking recalculateWordsInUseMethod", e);
+ } catch (IllegalArgumentException e) {
+ throw new KryoException("Exception thrown while invoking recalculateWordsInUseMethod", e);
+ }
+ return ret;
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/InetSocketAddressSerializer.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/InetSocketAddressSerializer.java
new file mode 100644
index 0000000000..405d79cf37
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/InetSocketAddressSerializer.java
@@ -0,0 +1,45 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill.java;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.Serializer;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import com.spotify.scio.vendor.chill.IKryoRegistrar;
+import com.spotify.scio.vendor.chill.SingleRegistrar;
+import java.net.InetSocketAddress;
+
+public class InetSocketAddressSerializer extends Serializer {
+
+ public static IKryoRegistrar registrar() {
+ return new SingleRegistrar(InetSocketAddress.class, new InetSocketAddressSerializer());
+ }
+
+ @Override
+ public void write(Kryo kryo, Output output, InetSocketAddress obj) {
+ output.writeString(obj.getHostName());
+ output.writeInt(obj.getPort(), true);
+ }
+
+ @Override
+ public InetSocketAddress read(Kryo kryo, Input input, Class klass) {
+ String host = input.readString();
+ int port = input.readInt(true);
+ return new InetSocketAddress(host, port);
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/IterableRegistrar.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/IterableRegistrar.java
new file mode 100644
index 0000000000..e1fee2682a
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/IterableRegistrar.java
@@ -0,0 +1,54 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill.java;
+
+import com.esotericsoftware.kryo.DefaultSerializer;
+import com.esotericsoftware.kryo.Kryo;
+import com.spotify.scio.vendor.chill.IKryoRegistrar;
+import java.util.Arrays;
+
+/**
+ * A Registrar adds registrations to a given Kryo instance. Examples would be a registrar that
+ * registers serializers for all objects in a given package. comes from Storm, which took it from
+ * cascading.kryo
+ */
+@DefaultSerializer(IterableRegistrarSerializer.class)
+public class IterableRegistrar implements IKryoRegistrar {
+ private final Iterable registrarList;
+
+ public IterableRegistrar(IKryoRegistrar... registrars) {
+ this(Arrays.asList(registrars));
+ }
+
+ public IterableRegistrar(Iterable registrars) {
+ registrarList = registrars;
+ for (IKryoRegistrar kr : registrarList) {
+ if (null == kr) throw new IllegalArgumentException("null Registrars not allowed");
+ }
+ }
+
+ @Override
+ public void apply(Kryo k) {
+ for (IKryoRegistrar kr : registrarList) {
+ kr.apply(k);
+ }
+ }
+
+ public Iterable getRegistrars() {
+ return registrarList;
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/IterableRegistrarSerializer.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/IterableRegistrarSerializer.java
new file mode 100644
index 0000000000..eae1d990f8
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/IterableRegistrarSerializer.java
@@ -0,0 +1,44 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill.java;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.Serializer;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import com.spotify.scio.vendor.chill.IKryoRegistrar;
+import java.util.ArrayList;
+
+public class IterableRegistrarSerializer extends Serializer {
+ public void write(Kryo kryo, Output output, IterableRegistrar obj) {
+ for (IKryoRegistrar kr : obj.getRegistrars()) {
+ kryo.writeClassAndObject(output, kr);
+ output.flush();
+ }
+ kryo.writeClassAndObject(output, null);
+ }
+
+ public IterableRegistrar read(Kryo kryo, Input input, Class type) {
+ ArrayList krs = new ArrayList();
+ IKryoRegistrar thisKr = (IKryoRegistrar) kryo.readClassAndObject(input);
+ while (thisKr != null) {
+ krs.add(thisKr);
+ thisKr = (IKryoRegistrar) kryo.readClassAndObject(input);
+ }
+ return new IterableRegistrar(krs);
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/Java8ClosureRegistrar.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/Java8ClosureRegistrar.java
new file mode 100644
index 0000000000..27e020134a
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/Java8ClosureRegistrar.java
@@ -0,0 +1,36 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill.java;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.serializers.ClosureSerializer;
+import com.spotify.scio.vendor.chill.IKryoRegistrar;
+
+/** Enables Java 8 lambda serialization if running on Java 8; no-op otherwise. */
+public class Java8ClosureRegistrar implements IKryoRegistrar {
+
+ @Override
+ public void apply(Kryo k) {
+ try {
+ Class.forName("java.lang.invoke.SerializedLambda");
+ } catch (ClassNotFoundException e) {
+ // Not running on Java 8.
+ return;
+ }
+ k.register(ClosureSerializer.Closure.class, new ClosureSerializer());
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/LocaleSerializer.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/LocaleSerializer.java
new file mode 100644
index 0000000000..c5cab889f0
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/LocaleSerializer.java
@@ -0,0 +1,32 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill.java;
+
+import com.esotericsoftware.kryo.serializers.JavaSerializer;
+import com.spotify.scio.vendor.chill.IKryoRegistrar;
+import com.spotify.scio.vendor.chill.SingleRegistrar;
+import java.util.Locale;
+
+/**
+ * The java serializer uses an cache of allocated instances so it is probably a bit hard to beat, so
+ * why bother
+ */
+public class LocaleSerializer extends JavaSerializer {
+ public static IKryoRegistrar registrar() {
+ return new SingleRegistrar(Locale.class, new LocaleSerializer());
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/PackageRegistrar.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/PackageRegistrar.java
new file mode 100644
index 0000000000..633111b5cf
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/PackageRegistrar.java
@@ -0,0 +1,45 @@
+/*
+Copyright 2013 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill.java;
+
+import com.spotify.scio.vendor.chill.IKryoRegistrar;
+
+/** Creates a registrar for all the serializers in the chill.java package */
+public class PackageRegistrar {
+
+ public static IKryoRegistrar all() {
+ return new IterableRegistrar(
+ ArraysAsListSerializer.registrar(),
+ BitSetSerializer.registrar(),
+ PriorityQueueSerializer.registrar(),
+ RegexSerializer.registrar(),
+ SqlDateSerializer.registrar(),
+ SqlTimeSerializer.registrar(),
+ TimestampSerializer.registrar(),
+ URISerializer.registrar(),
+ InetSocketAddressSerializer.registrar(),
+ UUIDSerializer.registrar(),
+ LocaleSerializer.registrar(),
+ SimpleDateFormatSerializer.registrar(),
+ UnmodifiableCollectionSerializer.registrar(),
+ UnmodifiableListSerializer.registrar(),
+ UnmodifiableMapSerializer.registrar(),
+ UnmodifiableSetSerializer.registrar(),
+ UnmodifiableSortedMapSerializer.registrar(),
+ UnmodifiableSortedSetSerializer.registrar());
+ }
+}
diff --git a/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/PriorityQueueSerializer.java b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/PriorityQueueSerializer.java
new file mode 100644
index 0000000000..178ec76b31
--- /dev/null
+++ b/scio-core/src/main/java/com/spotify/scio/vendor/chill/java/PriorityQueueSerializer.java
@@ -0,0 +1,79 @@
+/*
+Copyright 2012 Twitter, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package com.spotify.scio.vendor.chill.java;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.Serializer;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import com.spotify.scio.vendor.chill.IKryoRegistrar;
+import com.spotify.scio.vendor.chill.SingleRegistrar;
+import java.lang.reflect.Field;
+import java.util.Comparator;
+import java.util.PriorityQueue;
+
+class PriorityQueueSerializer extends Serializer> {
+ private Field compField;
+
+ public static IKryoRegistrar registrar() {
+ return new SingleRegistrar(PriorityQueue.class, new PriorityQueueSerializer());
+ }
+
+ public PriorityQueueSerializer() {
+ try {
+ compField = PriorityQueue.class.getDeclaredField("comparator");
+ compField.setAccessible(true);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public Comparator getComparator(PriorityQueue q) {
+ try {
+ return (Comparator) compField.get(q);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public void write(Kryo k, Output o, PriorityQueue q) {
+ k.writeClassAndObject(o, getComparator(q));
+ o.writeInt(q.size(), true);
+ for (Object a : q) {
+ k.writeClassAndObject(o, a);
+ o.flush();
+ }
+ }
+
+ public PriorityQueue read(Kryo k, Input i, Class> c) {
+ Comparator