Updgrading LZF codec, +50-100% speedup...

[Tatu Saloranta]

As some of you know, I have worked on improving the stand-alone
pure-Java LZF codec (https://github.com/ning/compress), as well as
worked on measurements as part of jvm-compressor benchmark project
(https://github.com/ning/jvm-compressor-benchmark).
Progress with performance improvements has been significant both for
compression and uncompression speeds, so I am wondering if it would
make sense to upgrade LZF codec that Voldemort has (written by Ismael
and I)? Since format has not changed this would be purely performance
improvement without backwards compatibility problems.
External codec could be shaded in to minimize versioning issues; or
added as a dependency if that is preferred.

What do you think?

-+ Tatu +-

ps. I might be able to take current embedded V LZF codec and get
actual measurements using jvm-compress-benchmark as well, to get exact
numbers

[Roshan Sumbaly]

Hi Tatu,

That would be awesome! I already have a version in my local branch but haven't had a chance to merge it to master ( also might be a couple of months old - https://github.com/rsumbaly/voldemort/commit/2e44db2c3c160b0e49f49ce058bc11a372737be2 ). It would be great to have a cleaner updated version in case any further changes have been made to the 'compress' library. 

Best,

Roshan

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To post to this group, send email to project-...@googlegroups.com.
To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.

[ijuma]

Good to see support for Snappy in that branch. That would close http://code.google.com/p/project-voldemort/issues/detail?id=356 . Having said that, it's a bad idea to use a stream during compression/decompression as it will affect performance negatively. ning-compress has methods to compress/decompress without using streams if I remember correctly.

Best,

Ismael

[Tatu Saloranta]

Correct. I believe Snappy codecs also have block-based alternatives as well.

-+ Tatu +-

[Tatu Saloranta]

On Thu, Nov 24, 2011 at 2:36 PM, Roshan Sumbaly <rsum...@gmail.com> wrote:
> Hi Tatu,
> That would be awesome! I already have a version in my local branch but
> haven't had a chance to merge it to master ( also might be a couple of
> months old
> - https://github.com/rsumbaly/voldemort/commit/2e44db2c3c160b0e49f49ce058bc11a372737be2
> ). It would be great to have a cleaner updated version in case any further
> changes have been made to the 'compress' library.

Yes, some performance optimizations have been added since original LZF
codec was added (and from the first ning compress versions).

For what it is worth, throuhgput increase seems to be around +10% for
compression (modest but measurable), and +75% for decompression. I
will still need to see how much performance depends on usage of
Sun.misc.unsafe (LZF codec comes with two variants; this, and
"vanilla" which only uses basic array access), which matters quite a
bit for decompression speed.

-+ Tatu +-

[Alex Feinberg]

+ 1, Would welcome and merge a patch contribution this.

- af

[ijuma]

On Thursday, 1 December 2011 18:46:57 UTC, Alex Feinberg wrote:

+ 1, Would welcome and merge a patch contribution this.

There you go. :)

https://github.com/ijuma/voldemort/commits/snappy-lzf

Best,

Ismael

[Alex Feinberg]

Go ahead and push to master, or if you prefer, I can do this for you.

- af

> --
> You received this message because you are subscribed to the Google Groups
> "project-voldemort" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/project-voldemort/-/EXxFJW4Vk_kJ.

[ijuma]

On Monday, 5 December 2011 20:09:33 UTC, Alex Feinberg wrote:

Go ahead and push to master, or if you prefer, I can do this for you.

Excellent, pushed.

Best,

Ismael