gzip HTTP results problem

[Bill Loren]

Hey ppl,

I've encountered a problem trying to decode gzip data
returned from an HTTP server I communicate with (I use urllib2).
I've tried to use both the gzip and zlib come-along python libraries but
alas.

Have anyone of you ppl succeeded in talking gzip with an HTTP server ?

thanks in advance,
~B
(btw I do not wish to change my Encoding-Type to deflate...)

[Fredrik Lundh]

Bill Loren wrote:

> I've encountered a problem trying to decode gzip data
> returned from an HTTP server I communicate with (I use urllib2).
> I've tried to use both the gzip and zlib come-along python libraries but
> alas.
>
> Have anyone of you ppl succeeded in talking gzip with an HTTP
> server ?

this might help:

http://effbot.org/zone/consumer-gzip.htm

(that piece of code is used in production code, so it should
work...)

</F>

[Richie Hindle]

[Fredrik]

> this might help:
>
> http://effbot.org/zone/consumer-gzip.htm
>
> (that piece of code is used in production code, so it should
> work...)

This looks wrong to me:

except IndexError:
self.data = data

Shouldn't it be:

except IndexError:
self.__data = data

?

--
Richie Hindle
ric...@entrian.com

[John J. Lee]

"Fredrik Lundh" <fre...@pythonware.com> writes:

That would go nicely with my unreleased latest version of
ClientCookie, which is plug-compatible with urllib2. but makes it
easier to add new functionality like this (I submitted a patch to
Python library based on this a while back, and am hoping for
comments).

The idea is that you pass processor objects to build_opener just as if
they were handler objects. Processors pre-process requests and
post-process responses. This stops you having to subclass things like
AbstractHTTPHandler. Ask if you want a copy.

(code below is completely untested, unworking, purely illustrative!)

import ClientCookie
from GzipConsumer import GzipConsumer
from cStringIO import StringIO

class stupid_gzip_consumer:
def __init__(self): self.data = []
def feed(self, data): self.data.append(data)

class stupid_gzip_wrapper:
def __init__(self, response):
self._response = response

c = stupid_gzip_consumer()
gzc = GzipConsumer(c)
gzc.feed(response.read())
self.__data = StringIO("".join(c.data))

def __getattr__(self, key):
# delegate unknown methods/attributes
if key in ("read", "readline", "readlines"):
return getattr(self.__data, key)
else:
return getattr(self._response, key)

class HTTPGzipProcessor(ClientCookie.BaseProcessor):
def http_response(self, request, response):
# post-process response
enc = response.hdrs.get["content-encoding"]
if ("gzip" in enc) or ("compress" in enc):
return stupid_gzip_wrapper(response)
else:
return response

https_response = http_response

opener = ClientCookie.build_opener(HTTPGzipProcessor)
ClientCookie.install_opener(opener)

response = urlopen("http://www.example.com/")
print response.read()
response.close()

John