[ANN] pgzip: Parallel gzip

[Klaus Post]

Hi!

For a recent project, I needed to gzip a big amount of data. Since the gzip in the standard library only operates in a single goroutine I saw a possibility to make it faster.

This is a modification of the standard library that encodes independent blocks, and where the reader decodes ahead in a separate goroutine.

Major features:

* Fully multithreaded gzip compression.

* Asynchronous gzip decompression.

* Fully gzip compatible, both read and write.

* Size overhead < 1%

* Fully compatible drop-in "replacement" for compress/gzip.

See more at https://github.com/klauspost/pgzip  (documentation, benchmarks, etc)

TL;DR: Use this if you are (de)compressing more than 1MB gzip data at the time.

Feedback, comments and experiences are very welcome.

/Klaus

[Dan Kortschak]

It looks like you ignore decompression checksum errors since you drop error returns from z.decompressor.Close().

[Dan Kortschak]

Also, I suspect that there is an error at line 297 of gunzip.go.

[Klaus Post]

Hi!

Thanks for looking through it - it is always nice to have other eyes on code.

>It looks like you ignore decompression checksum errors since you drop error returns from z.decompressor.Close().

Actually the checksum is calculated outside the inflater (gunzip:368), but of course the error shouldn't be ignored.

 >Also, I suspect that there is an error at line 297 of gunzip.go.

No - we must be sure that the crc digester has finished before we can return, so we do a lock before we return.


Thanks for reviewing it! If you find more, feel free to add it. I will add the missing error check as an issue, until I get back to work tomorrow!

/Klaus

[Dan Kortschak]

On 11/03/2015, at 7:19 AM, "Klaus Post" <klau...@gmail.com> wrote:

> Actually the checksum is calculated outside the inflater (gunzip:368), but of course the error shouldn't be ignored.

Yes, sorry. I misremembered what errors flate.decompressor returned.

> No - we must be sure that the crc digester has finished before we can return, so we do a lock before we return.

Ahh, you are using that as a semaphore.

[Klaus Post]

Hi!

I have updated the code to forward any error from the inflater to the Close() method to mimic the behavior of the standard library.

Thanks again for reporting!

/Klaus

[Camilo Aguilar]

Why not make those enhancements part of the standard Go library? Being Go Go, I would assume the standard library would offert as many parallel algorithms as possible...

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Benjamin Measures]

> Being Go Go, I would assume the standard library would offert as many parallel algorithms as possible...

http://blog.golang.org/concurrency-is-not-parallelism
If there's one thing most people know about Go, is that it is designed for concurrency. [...]
But when people hear the word concurrency they often think of parallelism, a related but quite distinct concept.

[Klaus Post]

On Thursday, March 12, 2015 at 7:55:30 PM UTC+1, Camilo Aguilar wrote:

Why not make those enhancements part of the standard Go library? Being Go Go, I would assume the standard library would offert as many parallel algorithms as possible...

The thing is that this "version" of the library only gives improvements on bigger data sizes, something like at least 1MB at the time. In a setting like a webserver, serving average of 10KB per request, it would only be unneeded overhead, since it spawns goroutines. An even if goroutines are cheap, if you have a webserver with 10k+ requests/second, you want to minimize the overhead as much as possible.

Therefore I recommend to use it in some cases - and just benchmark it - it is as simple as changing an import line.

/Klaus