Modifying zlib window

[Rob Napier]

Following up on https://groups.google.com/forum/#!searchin/golang-nuts/zlib/golang-nuts/s7xpW4pBTPk/QT5mUkbrl1QJ, I'm also using this spdy library and would like to shrink the zlib window size. The headers we're sending are tiny and will benefit little from compression, so the 32k buffer per connection is overly expensive when dealing with 30,000 connections.

Looking at the zlib code, the window size appears to be hardcoded as "default" (2^15).

From src/compress/zlib/writer.go:

1. // ZLIB has a two-byte header (as documented in RFC 1950).
2. // The first four bits is the CINFO (compression info), which is 7 for the default deflate window size.
3. // The next four bits is the CM (compression method), which is 8 for deflate.
4. z.scratch[0] = 0x78

It also appears to be hardcoded in compress/flate. Making this configurable seems possible, but not trivial. The actual change inside of deflate.go would be pretty easy, but the API doesn't provide any hooks for more configuration. It looks like it would need a NewWriterDictWindow() function (and maybe a NewWriterWindow() function). At the gzip level, it would require NewWriterLevelDictWindow() (which is quite a mouthful, but consistent).

Is there a better way to configure this window?

-Rob

[Brad Fitzpatrick]

I think I'd prefer if NewWriter returned a *Writer with a few public fields you could tweak before your first call to Write, at which time the Writer is initialized and you can't tweak them anymore. We use that pattern sometimes, and I wish we would've used it here. But I think we still could retroactively, keeping the old constructors.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rob Napier]

Totally agree. I'm embarrassed to say that I looked at "Writer" and my brain substituted "io.Writer". Yes, the sensible way to manage it would be via flate.Writer and zlib.Writer.

I assume these should accessed through functions like SetWindow() rather than directly modifying public fields? That way Write() could enforce locking them, and zlib could pass along updates to the underlying flate.Writer.

Do you have a particular data structure that would be a good model?

-Rob

[Brad Fitzpatrick]

Go in general doesn't force you to play by the rules. Just document the rules in English and it's their fault if there's a race. A public field on the Writer struct seems fine to me.

[Rob Napier]

Got it. I'll put together a CL for review. Thanks.

-Rob

--

Rob Napier
Cocoaphony blog -- http://robnapier.net/blog

iOS Programming Pushing the Limits -- http://robnapier.net/book

[Rob Napier]

(Copying Ivan Krasin because I believe this is his code originally.)

This has turned out to be a bit harder than I hoped. I've been running into trouble with deflate when the window size is smaller than the maximum match length (258).

If flate.logWindowSize == 8 (which is the minimum allowed by spec), then the window will be 256 bytes and flate.deflate() will crash when the window is adjusted because the index will be negative:

panic: runtime error: index out of range

goroutine 1 [running]:

ztest/compress/flate.(*compressor).deflate(0x2081c8900)

/Users/rnapier/go/src/ztest/compress/flate/deflate.go:235 +0xff9

I've posted example code to demonstrate: https://github.com/rnapier/ztest

I'm still digging into how to address this, consulting the equivalent zlib code. Simply reducing maxMatchLength to 128 seems to fix it, but zlib doesn't require this hack, so I'm thinking that's the wrong direction.

Insights welcome.

Thanks,

-Rob

[Ivan Krasin]

Hi Rob,

while I don't remember all the details, I would not be surprised if the code had some assumptions about the window size to be large enough.

I have time to look at that, if the Go team agrees it's something we want to have in the standard library. I would also recommend to file an issue for tracking, but my recommendation is completely optional.

krasin

[Rob Napier]

Looks like one was already open: https://github.com/golang/go/issues/3155

I've added some comments.

-Rob

[A Keeton]

Looks like this issue has been dead for quite some time. I'd like to resurrect it :) Any chance configurable window sizes for flate might be on the horizon?

[thebroke...@gmail.com]

Feel free to upvote the issue on github, which will help guage interest in the issue. There is no doubt that is something recurrent that people have requested throughout the years. There's a number of things I would like to see done in compression packages and I'm not sure where this fits into all of that work.

Adding this functionality for the decompressor is fairly trivial. On the other hand, doing so for the compressor will require some work, but the process for doing so will probably be a good opportunity to clean-up the logic and unify some constants. There may be a slight performance hit since the compiler can no longer assume that the window is always 32KiB and will have to do bounds checking in places that we previously could have elided the check. That remains to be investigated.