Problems using zlib - any help?
Don Bruder
Perhaps someone here can tell me what I'm doing wrong?
I've got a project that's intended to look like a web browser (as far as
a web server is concerned - it's not going to be a full-blown browser,
though - it's simply going to speak to a web server as if it were
FireFox doing the talking) What makes it "tricky" is that the web server
sends back (whether I send the "accept-encoding" tag for gzip or not) a
block of gzipped data. I need to work with this data. In a "real"
FireFox, the data gets decompressed and used properly, so the data must
be good.
What I'm trying to do leads, naturally enough, to using zlib to unpack
the data I get back.
Except that I'm having no luck at all...
The headers for the server response follow conventions by telling me
that there's gzipped data using the "Content-Encoding: gzip" tag. I can
parse out the zipped data to a buffer without problems. From another
entry in the headers, I know the compressed length of the zipped data
(which matches up correctly with my "hand-calculated" length based on
seeking out the "01/8b" magic bytes that indicate the start of the
zipped data, and finding the end of the packet)
I set up a z_stream struct - Fill in the next_in/avail_in members with
the address of the buffer holding the data, and its length, stuff the
next_out/avail_out members with the address and length of a malloc()ed
buffer that should be *WAY* beyond overkill based on the typical amount
of compressed data I get handed, set the zalloc, zfree, and opaque
members to Z_NULL, then call inflateInit() with the address of my
z_stream struct. This returns Z_OK.
So far, so good...
I then try the incantation "inflate(&My_z_stream, Z_FINISH);", and get
back "Z_DATA_ERROR", and get no uncompressed data.
The data I pass in is a malloc()ed block stuffed with the data the
server handed back, with the "01" byte of the "01/8b" signature pair at
location 0 of the buffer, followed by the rest of the data-block that
the server supplies me with.
I've tried stripping off the "01/8b" bytes, putting the "CM" byte
discussed in the RFCs (value "0x80") at location 0 of the buffer. No
joy.
I've tried using inflateSync - no joy - inflateSync() returns Z_OK, but
generates no output. Following up the inflateSync call with a call to
inflate() gives me back a "-3" value (defined as Z_DATA_ERROR in zlib.h,
which explains it as meaning the data is corrupted, or the checksum test
failed), and no useful data output.
FireFox, on the other hand, apparently has no problem with the data - no
sign of any trouble of any kind.
Everything I'm doing *LOOKS* right, based on what I can see, but I can't
get any useful data out of it.
Can anybody give me an idea of what I'm doing wrong? Or better yet, a
pointer to some source code that works on "in memory" compressed data?
All of the zlib examples I've found so far deal strictly with a "read a
chunk from a file, process it, read another chunk, process,
lather-rinse-repeat until out of data in the file" model, but none
address the "all in memory" method I'm trying to use (but which zlib.h
says is perfectly fine, so long as all the buffers are big enough to
hold the data - whether compressed or not - that'll be used/result)
Should I be stripping off the ID bytes (01/8b) and the CM/FLG/etc bytes
to leave only the compressed data? Reading the zlib.h file, this doesn't
seem to be the right way to do things, since there are multiple mentions
of these bits of data being used by the process.
Any help out there?
--
Email shown is deceased. If you would like to contact me by email, please
post something that makes it obvious in this or another group you see me
posting in with a "how to contact you" address, and I'll get back to you.
Don Bruder
Don Bruder <dak...@sonic.net> wrote:
> Perhaps someone here can tell me what I'm doing wrong?
>
<snip for space>
> Everything I'm doing *LOOKS* right, based on what I can see, but I can't
> get any useful data out of it.
After a suggestion from a poster in another group, found the difficulty,
and now all is well. Turns out I needed to be using
"inflateInit2(&MyZStream, 47)" to signal that I wanted gzip
decompression, rather than "inflateInit(&MyZStream)", which says I want
to decompress a "naked" (no headers of any kind) deflate stream.
Mark Adler
> Turns out I needed to be using
> "inflateInit2(&MyZStream, 47)" to signal that I wanted gzip
> decompression, rather than "inflateInit(&MyZStream)", which says I want
> to decompress a "naked" (no headers of any kind) deflate stream.
Glad you solved your problem. But just in case someone comes across
this thread using a search, inflateInit() says you want to decompress a
zlib stream, which has a zlib header and trailer. inflateInit2()
allows you to ask for gzip (31 for the second parameter) decoding, gzip
/ zlib auto-detection (47), or raw or "naked" (-15).
Mark
Don Bruder
Mark Adler <mad...@alumni.caltech.edu> wrote:
Hi Mark - nice to see an author chime in.
Suggestion:
Maybe I'm a bit thick, but I had *ABSOLUTE FITS* trying to troubleshoot
this - Suggest a bit clearer/more explicit wording in the zlib.h
header/docs. (Although aside from zlib.h itself, "other docs", including
example code of any kind, for this package seems like something that's
pretty hard to come by...)
The other poster I mentioned pointed out that inflateInit2() might be
needed. From there, it was a bit of a slog to get things to work -
My choice of "47" for param 2 was the result of trial-and-error that
started at "if 15 = default window size..." and "+32 = auto-detection?
Hmmm...", which turned into "if I feed it 47, it should go". And it did.
But now, my bald spot is significantly larger :)
Aside from that bit of difficulty, it's a really handy package - If I
hadn't "hit the wall" of thinking I should be using inflateInit(), then
spent several days beating my head against it trying to figure out why I
got an endless string of Z_DATA_ERROR results despite other code being
able to handle the data I was getting back without a problem, the whole
thing would likely have been up and running inside of an hour - probably
less than that, even.
Thanks again, both for the reply, and the package!
Mark Adler
I will look into improving the documentation.
This problem is caused by an all-too-common implicit assumption that is
unfortunately false and hard to dispell, which is that zlib == gzip. I
try to dispell that right at the beginning of zlib.h, but I think that
that part is often not read at all, with the reader heading straight
for the function descriptions. That preface says:
The compressed data format used by default by the in-memory
functions is the zlib format, which is a zlib wrapper documented in RFC
1950, wrapped around a deflate stream, which is itself documented in
RFC 1951.
The library also supports reading and writing files in gzip (.gz)
format with an interface similar to that of stdio using the functions
that start with "gz". The gzip format is different from the zlib
format. gzip is a gzip wrapper, documented in RFC 1952, wrapped around
a deflate stream.
This library can optionally read and write gzip streams in memory as well.
The zlib format was designed to be compact and fast for use in
memory and on communications channels. The gzip format was designed
for single-file compression on file systems, has a larger header than
zlib to maintain directory information, and uses a different, slower
check method than zlib.
On 2009-06-19 22:09:53 -0700, Don Bruder <dak...@sonic.net> said:
> (Although aside from zlib.h itself, "other docs", including example
> code of any kind, for this package seems like something that's pretty
> hard to come by...)
There is a link on zlib.net for a detailed "zlib Usage Example", which goes to:
There are many examples of the use of zlib in the examples directory of
the zlib distribution, including a copy of the web page above and the
zpipe.c example described in that web page. There are more examples in
the contrib directory. Also there is the example.c file used for
testing a build.
> The other poster I mentioned pointed out that inflateInit2() might be needed.
From the FAQ at http://zlib.net/zlib_faq.html#faq19 :
19. Ok, so why are there two different formats?
The gzip format was designed to retain the directory information about
a single file, such as the name and last modification date. The zlib
format on the other hand was designed for in-memory and communication
channel applications, and has a much more compact header and trailer
and uses a faster integrity check than gzip.
20. Well that's nice, but how do I make a gzip file in memory?
You can request that deflate write the gzip format instead of the zlib
format using deflateInit2(). You can also request that inflate decode
the gzip format using inflateInit2(). Read zlib.h for more details.
> then spent several days beating my head against it trying to figure out
> why I got an endless string of Z_DATA_ERROR results
From zlib.h in the description of the meaning of the inflate() return values:
inflate() returns ... Z_DATA_ERROR if the input data was corrupted
(input stream not conforming to the zlib format or incorrect check
value) ...
Mark