Efficient binary serialization for Clojure/Java

Question

I'm looking for a way to efficiently serialize Clojure objects into a binary format - i.e. not just doing the classic print and read text serialization.

i.e. I want to do something like:

(def orig-data {:name "Data Object" 
                :data (get-big-java-array) 
                :other (get-clojure-data-stuff)})

(def binary (serialize orig-data))

;; here "binary" is a raw binary form, e.g. a Java byte array
;; so it can be persisted in key/value store or sent over network etc.

;; now check it works!

(def new-data (deserialize binary))

(= new-data orig-data)
=> true

The motivation is that I have some large data structures that contain a significant amount of binary data (in Java arrays), and I want to avoid the overhead of converting these all to text and back again. In addition, I'm trying to keep the format compact in order to minimise network bandwidth usage.

Specific features I'd like to have:

Lightweight, pure-Java implementation
Support all of Clojure's standard data structures as well as all Java primitives, arrays etc.
No need for extra build steps / configuration files - I'd rather it just worked "out of the box"
Good performance both in terms of processing time required
Compactness in terms of binary encoded representation

What's the best / standard approach to doing this in Clojure?

Depending on what you are encoding and how you are encoding it, text can be faster and more compact than some binary formats. Often the overhead in serialization is the use of reflection rather than the conversion so the need to support arbitrary data structures is most likely to be your real issue. This compares canned vs Java Serialization, vanillajava.blogspot.com/2011/08/… — Peter Lawrey
– Peter Lawrey, Commented Oct 9, 2011 at 7:33
Does it have to be able to handle references to runtime objects such as atoms and refs? — bmillare
– bmillare, Commented Oct 9, 2011 at 14:24
I've had great success using Kryo with Cascalog, ElephantDB and Storm. carbonite, by revelytix, has serializers for most of the Clojure data structures out of the box. — Sam Ritchie
– Sam Ritchie, Commented Feb 21, 2012 at 1:12

j-g-faustus · Accepted Answer · 2011-10-10 01:37:40Z

12

I may be missing something here, but what's wrong with the standard Java serialization? Too slow, too big, something else?

A Clojure wrapper for plain Java serialization could be something like this:

(defn serializable? [v]
  (instance? java.io.Serializable v))

(defn serialize 
  "Serializes value, returns a byte array"
  [v]
  (let [buff (java.io.ByteArrayOutputStream. 1024)]
    (with-open [dos (java.io.ObjectOutputStream. buff)]
      (.writeObject dos v))
    (.toByteArray buff)))

(defn deserialize 
  "Accepts a byte array, returns deserialized value"
  [bytes]
  (with-open [dis (java.io.ObjectInputStream.
                   (java.io.ByteArrayInputStream. bytes))]
    (.readObject dis)))

 user> (= (range 10) (deserialize (serialize (range 10))))
 true

There are values that cannot be serialized, e.g. Java streams and Clojure atom/agent/future, but it should work for most plain values, including Java primitives and arrays and Clojure functions, collections and records.

Whether you actually save anything depends. In my limited testing on smallish data sets serializing to text and binary seems to be about the same time and space.

But for the special case where the bulk of the data is arrays of Java primitives, Java serialization can be orders of magnitude faster and save a significant chunk of space. (Quick test on a laptop, 100k random bytes: serialize 0.9 ms, 100kB; text 490 ms, 700kB.)

Note that the (= new-data orig-data) test doesn't work for arrays (it delegates to Java's equals, which for arrays just tests whether it's the same object), so you may want/need to write your own equality function to test the serialization.

user> (def a (range 10))
user> (= a (range 10))
true
user> (= (into-array a) (into-array a))
false
user> (.equals (into-array a) (into-array a))
false
user> (java.util.Arrays/equals (into-array a) (into-array a))
true

edited Oct 10, 2011 at 1:37

answered Oct 9, 2011 at 23:49

j-g-faustus

9,0574 gold badges35 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

mikera Over a year ago

interesting - I'd had bad experiences with Java serialisation before (too slow, bloated message sizes) but it sounds from your tests like it might actually work quite well for large arrays.

j-g-faustus Over a year ago

@mikera I think the main problem with Java serialization is unexpected dependencies, so you may end up serializing half the JVM. But if you stick to simple values, it's pretty good. This blog found Java serialization to be (slightly) faster and smaller than Google protobufs for a simple POJO.

mpenet · Accepted Answer · 2013-06-16 17:16:41Z

5

Nippy is one of the best choices imho: https://github.com/ptaoussanis/nippy

answered Jun 16, 2013 at 17:16

mpenet

3673 silver badges6 bronze badges

Comments

Nano Taboada · Accepted Answer · 2011-10-09 06:23:49Z

4

Have you considered Google's protobuf? You might want to check the GitHub repository with the interface for Clojure.

edited Oct 9, 2011 at 6:23

answered Oct 9, 2011 at 6:18

Nano Taboada

4,18212 gold badges64 silver badges90 bronze badges

3 Comments

mikera Over a year ago

interesting.... good to see there is a Clojure wrapper for this! However unless I'm mistaken this doesn't allow serialisation of arbitrary Clojure objects, i.e. you have to pre-specific your structure in a .proto file?

Nano Taboada Over a year ago

I'm glad you found it helpful! Unfortunately I'm not following what do you mean with arbitraty but either way, as far as I'm concerned, the .proto file is just where you define your data structures (think: schema), everything else is done for you by the API.

mikera Over a year ago

What I mean is: I want to be able to serialize and deserialize Clojure structures without having to pre-define the data structures in advance. Seems like this ought to be possible since all Clojure data structures are just maps, sets, lists etc. with a few basic Java objects thrown in. Since many data structures are dynamically created at runtime in Clojure it's pretty much impossible to specify them all in advance.......

amalloy · Accepted Answer · 2011-10-09 17:41:13Z

3

If you don't have a schema ahead of time, serializing to text is probably your best bet. To serialize arbitrary data in general, you need to do a lot of work to preserve the object graph, and do reflection to see how to serialize everything...at least Clojure's printer can do a static, no-reflection lookup of the print-method for each item.

Conversely, if you really want an optimized wire format, you need to define a schema. I've used thrift from java, and protobuf from clojure: neither is loads of fun, but it's not hideously onerous if you plan in advance.

answered Oct 9, 2011 at 17:41

amalloy

93k8 gold badges148 silver badges215 bronze badges

Collectives™ on Stack Overflow

Efficient binary serialization for Clojure/Java

4 Answers 4

2 Comments

Comments

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

2 Comments

Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related