Paralell zipping

[Slawomir Pryczek]

Hi Guys, im looking for a way to make paralell compression of large amounts of data, eg. there's 30GB text file i need to put inside .ZIP

Golang built-in zip class doesn't seem to support it... any suggestiong?

[Mauro Risonho de Paula Assumpção]

Hi.

Link: http://stackoverflow.com/questions/23005917/parallel-zip-compression-in-go

@firebitsbr

2015-08-06 11:46 GMT-03:00 Slawomir Pryczek <slawe...@gmail.com>:

Hi Guys, im looking for a way to make paralell compression of large amounts of data, eg. there's 30GB text file i need to put inside .ZIP

Golang built-in zip class doesn't seem to support it... any suggestiong?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cesar Alvarado Peragallo]

You can try http://blog.klauspost.com/optimized-gzipzip-packages-30-50-faster/

[Klaus Post]

On Thursday, 6 August 2015 16:52:57 UTC+2, Slawomir Pryczek wrote:

Hi Guys, im looking for a way to make paralell compression of large amounts of data, eg. there's 30GB text file i need to put inside .ZIP

Golang built-in zip class doesn't seem to support it... any suggestiong?

As Cesar very kindly pointed out, I have been working on improving deflate performance you can access a modified zip package that uses the new deflate and crc32 function. That should give you about 50% extra performance on text files.

There are a few shortcomings of the standard library zip package, I would like to address at some point. Most prominently compression is locked at level 5. It isn't a very bad default, through for your content I would probably go for level 3, which on high compression content can compress 300MB/sec (single core).

If you use my modified library it is easy to change. In $GOPATH$/github.com/klauspost/compress/zip/register.go change line 32 to the level you would like to use.

I have only done a multicore version of the gzip package (called pgzip). If gzip is an option you could look into that.

I cannot make any promises, but I am very likely to add a compression level selector to the zip package, and I may do a multicore version of zip as well.

/Klaus