What's the correct way to decompress zlib data?

[Leo Jay]

Hi All,

I encountered a strange slowness of compress/zlib

Here is the source code to reproduce this:
http://pastebin.com/UUbVP2i6

And I wrote a python version for comparison
http://pastebin.com/2110EjcY


The result is very strange to me:
$ time go run zlibtest.go
sum = 161220000

real    0m4.195s
user    0m4.150s
sys    0m0.010s


$ time python zlibtest.py
161220000

real    0m0.743s
user    0m0.730s
sys    0m0.010s


What's wrong with my code?
Thank you.

--
Best Regards,
Leo Jay

[Nigel Tao]

On 7 May 2012 21:53, Leo Jay <python...@gmail.com> wrote:
> The result is very strange to me:
> $ time go run zlibtest.go

This measures the time to compile, link and run that program. Another
data point would be to measure just the portion spent running:
$ go build zlibtest.go && time ./zlibtest

Also, are you on the release (also known as go1.0.1) or on tip? ZLIB
wraps flate, and I submitted an optimization [0] to flate just a week
or so ago, on tip, that speeds up some decompression microbenchmarks
by up to 2x, depending on the workload.

Speaking of workloads, can you describe what sort of data your
zlibtest.dat file is, i.e. how it was generated?

[0] https://groups.google.com/group/golang-dev/browse_thread/thread/8f9eb9fda3412ba4/3ba7b24a2127951a

[Leo Jay]

On Mon, May 7, 2012 at 8:17 PM, Nigel Tao <nige...@golang.org> wrote:
>
> On 7 May 2012 21:53, Leo Jay <python...@gmail.com> wrote:
> > The result is very strange to me:
> > $ time go run zlibtest.go
>
> This measures the time to compile, link and run that program. Another
> data point would be to measure just the portion spent running:
> $ go build zlibtest.go && time ./zlibtest
>

I tried, but the result is almost the same. And I don't think go needs
a lot of time to compile such a simple program.

> Also, are you on the release (also known as go1.0.1) or on tip? ZLIB
> wraps flate, and I submitted an optimization [0] to flate just a week
> or so ago, on tip, that speeds up some decompression microbenchmarks
> by up to 2x, depending on the workload.
>

I tried 1.0, 1.0.1 and the tip version, the result is almost the same
for all of them.
And I also tried to set the for loop max value to 100000 (10 times to
the previous version),
The go version needs about 40 seconds, and python needs about 7 seconds.

> Speaking of workloads, can you describe what sort of data your
> zlibtest.dat file is, i.e. how it was generated?
>
> [0] https://groups.google.com/group/golang-dev/browse_thread/thread/8f9eb9fda3412ba4/3ba7b24a2127951a

I used some normal text files.
You may try to download this script and compress it with zlib, and use
this as the zlibtest.dat:
http://python-distribute.org/distribute_setup.py