"Snappy" algorithm in deflate
235 views
Skip to first unread message
Klaus Post
Sep 18, 2015, 10:46:21 AM9/18/15
to golang-nuts
Hi!
I was looking at the snappy implementation in Go, and really liked the simplicity and performance of it. Just as an experiment I wanted to see how the same code would perform if it was outputting deflate-compatible data. The implementation was really easy, and to my surprise it was both very fast and very close to "BestSpeed" compression setting in deflate.
In my previous experiments with deflate the most common case for "level 1" was that it provided no significant speedup, only lower compression compared to level 2 and sometimes even 3.
However, the modified Snappy algorithm provides a very good sweet spot. Usually about 75% faster and with only little compression loss. Therefore I decided to replace level 1 with this mode entirely.
Input is split into blocks between 32 and 64kb, and they are encoded independently (no back-references across blocks) for the best speed. Contrary to Snappy the output is entropy-encoded, so you will almost always see better compression than Snappy. But Snappy is still about twice as fast as "snappy" in deflate mode.
If you want to try/use it, you can download/update the compression package from here:
I was looking at the snappy implementation in Go, and really liked the simplicity and performance of it. Just as an experiment I wanted to see how the same code would perform if it was outputting deflate-compatible data. The implementation was really easy, and to my surprise it was both very fast and very close to "BestSpeed" compression setting in deflate.
In my previous experiments with deflate the most common case for "level 1" was that it provided no significant speedup, only lower compression compared to level 2 and sometimes even 3.
However, the modified Snappy algorithm provides a very good sweet spot. Usually about 75% faster and with only little compression loss. Therefore I decided to replace level 1 with this mode entirely.
Input is split into blocks between 32 and 64kb, and they are encoded independently (no back-references across blocks) for the best speed. Contrary to Snappy the output is entropy-encoded, so you will almost always see better compression than Snappy. But Snappy is still about twice as fast as "snappy" in deflate mode.
If you want to try/use it, you can download/update the compression package from here:
* go get -u github.com/klauspost/compress
You will find fully complatible flate/gzip/zip/zlib packages. So to try it, you simply replace the import path of the standard libraries.
Have a nice weekend!
/Klaus
Mirolim Mirzakhmedov
Sep 19, 2015, 9:57:45 AM9/19/15
to golang-nuts
Thank you, I will definitely try it.
Reply all
Reply to author
Forward
0 new messages