LZ4 compression

[Robert DiFalco]

I'm wondering if anyone knows of a library that has a KryoSerializer for LZ4 compression. Oh, and if you know, have you found it much faster than the JSK Deflater/Inflater for Kryo under Java 1.8? Thanks.

[mongonix]

On Tuesday, December 16, 2014 2:18:44 PM UTC-8, Robert DiFalco wrote:

I'm wondering if anyone knows of a library that has a KryoSerializer for LZ4 compression. Oh, and if you know, have you found it much faster than the JSK Deflater/Inflater for Kryo under Java 1.8? Thanks.

Please have a look at this: https://github.com/romix/akka-kryo-serialization

It supports LZ4 for Kryo.  The library is for Scala, but you may extract the LZ4-relevant code from it.

-Leo

[Robert DiFalco]

So I tried this out and interestingly there is probably some minimum size where it does not make sense. I have a lot of small objects I serialize over a channel and for many of these the LZ4 compressed size is a couple of bytes or more large than the original. However, when it gets to 1K or more then it starts having an effect. So I'm going to use a byte to indicate if the byte array is compressed and only compress it if the raw serialized bytes are greater than 1024.

Thoughts?

--
You received this message because you are subscribed to the "kryo-users" group.
http://groups.google.com/group/kryo-users
---
You received this message because you are subscribed to a topic in the Google Groups "kryo-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kryo-users/0vrfvQxQPAA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kryo-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[mongonix]

On Tuesday, December 16, 2014 4:57:38 PM UTC-8, Robert DiFalco wrote:

So I tried this out and interestingly there is probably some minimum size where it does not make sense. I have a lot of small objects I serialize over a channel and for many of these the LZ4 compressed size is a couple of bytes or more large than the original. However, when it gets to 1K or more then it starts having an effect. So I'm going to use a byte to indicate if the byte array is compressed and only compress it if the raw serialized bytes are greater than 1024.

Thoughts?

It could be that LZ4 splits input in chunks of 1025 bytes and then compresses or something like this. May be it is configurable. You may want to ask Luben, who contributed LZ4 support to https://github.com/romix/akka-kryo-serialization. He has more experience with LZ4.

[Tumi]

Hi all,

 

I use https://github.com/ning/compress (which also supports Unsafe mode and vanilla/safe mode), an is really fast and realiable in its unsafe implementation.

 

We use its LZFOutputStream and LZFInputStream decorators, and pass these streams to Kryo’s Output and Input instances (invoking Input.setInputStream() and Output.setOutputStream() ).

If you want full control, you can reuse a BufferRecycler instance instead of relying in its default ThreadLocal behavior (of course you have now the responsibility of thread-safety over its buffers).

 

LZF if the first half part of a GZip process, so as claimed in the referred link: “LZF algorithm itself is optimized for speed, with somewhat more modest compression: compared to Deflate (algorithm gzip uses) LZF can be 5-6 times as fast to compress, and twice as fast to decompress”.

 

Of course this means better performance in terms of CPU process, but worst performance in terms of IO (because of the result size), so depending on entropy of your data and depending on how fast is your filesystem backend or network, you should empirically choose one option or another (or no compression at all!).

 

In my case, by default I use LZF because I think it is a good midpoing between both extremes.

 

And interestingly not only offers me (in my case, in my environments) better performance than Deflate/GZ, but also better than (or draw) not using compression because of some filesystem IO-waits on remote shared directories….

 

My 2 cents.

 

Cheers,

-Tumi

You received this message because you are subscribed to the Google Groups "kryo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kryo-users+...@googlegroups.com.