[ANN] Nippy, a fast Clojure serializer and drop-in reader replacement

[Peter Taoussanis]

Hey everyone! First of two library releases today...

Github: https://github.com/ptaoussanis/nippy
Clojars: https://clojars.org/com.taoensso/nippy

Features (taken from readme):
* Simple, high-performance all-Clojure de/serializer.
* Comprehensive, extensible support for all major data types.
* Reader-fallback for difficult/future types.
* Full test coverage for every supported type.
* Snappy integrated de/compression for efficient storage and network transfer.

Cheers!

- Peter Taoussanis (@ptaoussanis)

[Brent Millare]

Nice work, is there support for tagged literals and clojurescript?

[Peter Taoussanis]

Hi Brent,

Tagged literals are supported: Nippy falls back to the reader whenever it encounters something it doesn't know how to serialize:

(def my-uuid (java.util.UUID/randomUUID))

=> #uuid "c463d8d3-49f4-4e40-9937-8a9699b1af1d"

(thaw-from-bytes (freeze-to-bytes my-uuid))

=> #uuid "c463d8d3-49f4-4e40-9937-8a9699b1af1d"

There isn't an example in the reference data set since I'm targeting 1.3+, but I'll make a note about this!

No Clojurescript support unfortunately since the lib relies on JVM features. It's an interesting question... I haven't personally seen a need (yet?) for fast binary serialization script-side since the reader performance (usually?) isn't as much of an issue there.

What I'd normally do for example is serialize server-side to and from the db, then communicate with the CLJS client through the reader. This could still be a bottleneck under certain circumstances though, so I'll try think about it...

If any CLJS gurus have comments, I'd be very happy to hear them! 

In real world use, are people hitting reader performance limits client-side?

[Sun Ning]

Really nice work, I have been looking for such library for my RPC
framework for a long time.
Can't wait to test it out.

By the way, do you have a performance comparison between Nippy and
carbonite(the one wraps kryo) ?

I think it's a pretty good idea to exchange data in clojure literal,
and also a faster reader/writer is high appropriate.
Thank you for this library.

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient
> with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

[Peter Taoussanis]

Hi Sun,

Can't wait to test it out.

Great- thank you! Let me know how it goes and if you run into any problems.

 

By the way, do you have a performance comparison between Nippy and
carbonite(the one wraps kryo) ?

Not yet, but plan to. I should probably also compare to JSON and a few others. I would expect Carbonite to be faster since Kryo is faster than standard Java serialization (which is basically what Nippy uses).

The goal for Nippy isn't outright speed but a particular balance of speed, simplicity, and usability that I was looking for for some of my own applications. Actually, come to think of it, I will add links to some alternatives like Kryo.

[Frank Siebenlist]

Just trying to understand the issues/solutions for the (de-)serializing of clojure data structures…

With the tag literal support of 1.4+, is that kind of the idiomatic way of the future to structure protocols for serialized clojure data/code?

When you say that Nippy is much faster than the reader, are there ways to improve on or optimize the reader?
(guess the speed improvement is only achieved for true data (?))

Thanks, Frank.

[Peter Taoussanis]

Hey Frank,

Thanks for getting in touch!

With the tag literal support of 1.4+, is that kind of the idiomatic way of the future to structure protocols for serialized clojure data/code?

Sorry, could you clarify what you're asking here - I'm not sure if I follow what you mean by "structure protocols"? I'm not familiar with the implementation details of the tagged literal support btw, so someone else should probably chime in if that's the domain.

When you say that Nippy is much faster than the reader, are there ways to improve on or optimize the reader?
(guess the speed improvement is only achieved for true data (?))

Well again not knowing much about the particular implementation, my general inclination would be to say that the reader's performance doesn't seem especially bad given all that it's doing and its flexibility. Optimizing it for speed at this point is, I'd guess, a low priority. Though I'm curious to see what impact if any Datomic will have on Clojure core, since I'm assuming they're likely to start butting into some of these performance edge-cases eventually.

Naturally, these things are a trade-off. The Java serialization libs are a good example of this: each brings its own particular opinion about the correct simplicity/flexibility/speed balance. Given that context, it probably makes sense to let the reader focus on simplicity+flexibility, and to outsource to a lib when you (really+rarely) need the extra speed and where you have a knob to choose how much flexibility you're prepared to give up.

Not sure what you mean by "true data" btw, so it's possible I've completely missed what you're getting at ^^

[Alex Miller]

If you are serializing data, then using the reader implies serializing to a string (by printing your data) and reading it from a string.  It should be obvious that marshalling data to/from strings is neither fast nor small compared to other binary alternatives.  Using the reader is nice for certain applications (tagged literals help) and using serializers is necessary for other applications.

Nippy looks pretty clean and I like that it doesn't have the baggage of carbonite's dependency on kryo. [I wrote carbonite.]

Looking at the code, I would guess that it suffers quite a bit on performance and serialization size based on my experiences with Carbonite.  Some key ideas for improvement:

1) buffer reuse - reusing cached ThreadLocal byte[] for converting from data to bytes is a nice trick and made possible in most serialization libs but I don't that's possible in the current code. 

2) transients while deserializing collections (in coll-thaw!) - I found this to be the only way to get anywhere close to Java serialization while benchmarking.

3) more efficient primitive writing - this is the main reason I wrapped Kryo - there are lots of well-known tricks for writing chars, ints, etc.  Thrift, Protobuff, Kryo, and others have made these pretty standard and they make a huge difference vs the basic DataInputStream serialization.  

I did a lot of benchmarking vs standard Java serialization (which is supported on all Clojure data types already).  Despite its well-known issues, Java serialization is so deeply hacked into the language (some people say Java's greatest design mistake) that it is actually really fast.  For small pieces of data, it is incredibly bloated, but for larger graphs, that actually amortizes out a bit if you have repeated object references.  

I found the use of transients in deserializing collections to be the only way I could get anywhere close to Java serialization performance - they are critical.  The resulting data is definitely smaller, especially on smaller amounts of data, and for our uses that's actually really important so it was still a win overall.  

The other major issue that drove the creation of carbonite was that Java serialization of LazySeqs could easily blow the stack by pushing every cons serialization onto the stack.  This is a trivial issue to solve in any Clojure-aware serializer.  For that reason alone, we needed a better solution.

If you want to steal ideas from Carbonite, please feel free. :)  https://github.com/revelytix/carbonite

Alex

[Peter Taoussanis]

Hi Alex,

Thanks for your input (and all your work on Carbonite)- much appreciated!

1) buffer reuse - reusing cached ThreadLocal byte[] for converting from data to bytes is a nice trick and made possible in most serialization libs but I don't that's possible in the current code.

I experimented with a number of different buffering approaches, including a reusable ByteBuffer. Didn't see as big of a performance boost as I was expecting, so elected to keep it simple for the moment. It's very possible I missed something or tested incorrectly: I'll certainly check out your approach!

 

2) transients while deserializing collections (in coll-thaw!) - I found this to be the only way to get anywhere close to Java serialization while benchmarking.

I'm relying on `into` when possible, which is using transients under-the-hood. The notable exception is for maps - but that was because the only approaches I could think of that'd allow transients actually turned out to be slower in practice due to other overhead. A transient-based zipmap was the best perf I could get, but it was only ~10% faster and it incurred some other effects I didn't like.

3) more efficient primitive writing - this is the main reason I wrapped Kryo - there are lots of well-known tricks for writing chars, ints, etc.  Thrift, Protobuff, Kryo, and others have made these pretty standard and they make a huge difference vs the basic DataInputStream serialization.

This I'm completely unfamiliar with and it sounds interesting: I'll definitely check it out as well, thank you!

 

I found the use of transients in deserializing collections to be the only way I could get anywhere close to Java serialization performance - they are critical.

To be honest, performance for Nippy was actually a relatively low priority for me since in its current/obvious form, it's already fast enough that it ceases to be the bottleneck in any practical contexts in which I would be using it (mostly DB-related). That's to say, I haven't spent an awful lot of effort tweaking for performance and there's probably lots of low-hanging fruit.

I'll start by checking out what you've suggested, and I'm very open to input if you (or anyone else) thinks of other ideas!

Cheers!

[Frank Siebenlist]

On Jul 10, 2012, at 1:51 AM, Peter Taoussanis wrote:

> With the tag literal support of 1.4+, is that kind of the idiomatic way of the future to structure protocols for serialized clojure data/code?
>
> Sorry, could you clarify what you're asking here - I'm not sure if I follow what you mean by "structure protocols"? I'm not familiar with the implementation details of the tagged literal support btw, so someone else should probably chime in if that's the domain.

Sorry for being a little terse with my initial Q…

The idea is that you could use tagged literals for the serialized data format (not "protocol" as I mentioned).

In your example:

> (def my-uuid (java.util.UUID/randomUUID))
> => #uuid "c463d8d3-49f4-4e40-9937-8a9699b1af1d"
>
> (thaw-from-bytes (freeze-to-bytes my-uuid))
> => #uuid "c463d8d3-49f4-4e40-9937-8a9699b1af1d"

You can see that the tag #uuid is taken by the reader as kind of a constructor that takes the string as an argument and returns/substitutes it by the uuid object/instance.

You could extend it such that your custom contructor function could take a stringified, compressed binary representation of the object.

Alex mentions that "marshalling data to/from strings is neither fast nor small compared to other binary alternatives", which is true in general, but because those tag-contructors are yours to write, you could optimize the data format and encodings based on the datatype, and possibly overcome some of those speed/size limitations.

The advantage is that it's all clojure and doesn't rely on java serialization, while the disadvantage is that it will probably always be slower/bigger than pure binary formats... but maybe not that much.

(disclaimer: I have no experience writing those kind of encoders/decoders - just interested in how they work/could-work with pros&cons)


-FrankS.