Pure python implementation of zlib deflate with dictionary support?

[Berend-Jan Wever]

Hey all,

I'm working on a SPDY server in Python (2.x) for use in fuzzing. I want the code to be platform independent and 100% Python if possible. I noticed that SPDY uses zlib deflate with a dictionary to compress the headers. However, Python's zlib implementation does not support dictionaries. All the example code I've found online is platform dependent and for Windows IA-32 (they all seem to load a separate zlib dll and call its code to do the compression).

In order to address this issue, and to have control over compression during fuzzing, I'd like to use a pure Python implementation of a zlib deflate compressor/decompressor. Can anybody help me find that or suggest other solutions that I could consider?

Cheers,

BJ

Berend-Jan Wever (SkyL...@google.com) | Security Software Engineer
Google Netherlands B.V. | Reg: Claude Debussylaan 34, 15th floor 1082 MD Amsterdam
34198589 | NETHERLANDS | VAT / Tax ID:- 812788515 B01

[Adam Langley]

On Thu, Apr 26, 2012 at 11:06 AM, Berend-Jan Wever
<skyl...@chromium.org> wrote:
> In order to address this issue, and to have control over compression during
> fuzzing, I'd like to use a pure Python implementation of a zlib deflate
> compressor/decompressor. Can anybody help me find that or suggest other
> solutions that I could consider?

I'm not aware of any pure Python implementations of zlib I'm afraid.
You could certainly wrap libzlib in a way that exposed the ability to
set a dictionary in order to get going, but the clearest explanation
of the zlib format is probably the Go source if you wanted to write
your own: http://tip.golang.org/src/pkg/compress/flate/


Cheers

AGL

[Berend-Jan Wever]

Thanks! If all else fails, at least I have something to port :)

Berend-Jan Wever (SkyL...@google.com) | Security Software Engineer
Google Netherlands B.V. | Reg: Claude Debussylaan 34, 15th floor 1082 MD Amsterdam
34198589 | NETHERLANDS | VAT / Tax ID:- 812788515 B01

[Stephen Lamm]

Take a look at mnot's nbhttp:

https://github.com/mnot/nbhttp/tree/spdy/src

https://github.com/mnot/nbhttp/blob/spdy/src/c_zlib.py

That was used by Web Page Replay (WPR) to test spdy/2 performance. It would be fantastic to have a spdy/3 Python implementation.

-slamm

[Berend-Jan Wever]

I'm not sure what I am supposed to look at? That code is one of the samples I looked at before: it loads a zlib dll and is therefore platform dependent and not a Python implementation of deflate. Am I missing something?

Berend-Jan Wever (SkyL...@google.com) | Security Software Engineer
Google Netherlands B.V. | Reg: Claude Debussylaan 34, 15th floor 1082 MD Amsterdam
34198589 | NETHERLANDS | VAT / Tax ID:- 812788515 B01

[Stephen Lamm]

Sorry, my mind was stuck on the idea of not having to compile anything extra. Lately, I have tested on Linux and Mac where the library is available.

-slamm

[Mark Nottingham]

FWIW, my intent is to do it in Thor:
https://github.com/mnot/thor/tree/spdy
I'd like to get Thor using bytearrays first, though.

Now, if I could just find some spare time lying around...

Cheers,

--
Mark Nottingham http://www.mnot.net/

[Berend-Jan Wever]

Ah, I was not aware that ctypes.cdll.LoadLibrary was platform independent. I assumed it would be Windows specific and require a dll that matches the cpu architecture of your Python binary (x86 or x64). From what you are saying, it seems that first assumption is wrong? On Windows you still need to make sure you get a x86 or x64 dll, depending on your Python binary. But one could just try to load both and use whichever succeeds.

It seems my platform dependency problem doesn't actually exist, as long as I get dll/shared libraries for all platforms and cpu architectures I want to target.

It would still be nice to have a pure Python implementation to be able to play with compression in my fuzzer.

Berend-Jan Wever (SkyL...@google.com) | Security Software Engineer
Google Netherlands B.V. | Reg: Claude Debussylaan 34, 15th floor 1082 MD Amsterdam
34198589 | NETHERLANDS | VAT / Tax ID:- 812788515 B01

[Colin Marc]

Hi BJ,

I wrote a pure python spdy implementation here: https://github.com/colinmarc/python-spdy. It includes a Cython-wrapped zlib implementation with the spdy custom dictionary, and you're welcome to steal all or some of it. 

[theyok]

Python 3.3 now has all the necessary zlib methods:

http://docs.python.org/dev/library/zlib.html 

I've used it and it works great, a lot simpler than wrapping zlib with CTypes or Cython.