Zlib question : using Z_FULL_FLUSH to build random access index into compressed data
Qubeley
compressed data. I split input data into fixed size chunks and
compress each chunk separately using the same zstream with the flush
parameter set to Z_FULL_FLUSH mode. I expect to get byte aligned
compressed blocks so I can record the offset of blocks and directly
jump desired block during decompression, w/o touching previous
blocks.
My problem is this indexing sometimes work, sometimes not. For
example, I could observe a bit pattern in compressed data sometimes
that the presence of "0000 FFFF" trailer near end of the block which
was said to be a trait (is it?) for a compressed block. If this bit
pattern exhibits, then my index will work. Otherwise, the index will
not working : though I can decompress some bits starting from the
index but I can not get the complete original block (with size equal
to the fixed chunk size).
I am hoping if someone here would shed some lights, thanks
Mark Adler
> My problem is this indexing sometimes work, sometimes not. For
> example, I could observe a bit pattern in compressed data sometimes
> that the presence of "0000 FFFF" trailer near end of the block which
> was said to be a trait (is it?) for a compressed block. If this bit
> pattern exhibits, then my index will work. Otherwise, the index will
> not working : though I can decompress some bits starting from the
> index but I can not get the complete original block (with size equal
> to the fixed chunk size).
It sounds like you are not getting all of the output data from deflate
after issuing the Z_FULL_FLUSH. This would be because you are not
providing enough output space with the first deflate() call. That's
fine, but then you need to keep calling deflate() with no more input,
Z_FULL_FLUSH, and making available more output space until deflate()
returns with avail_out != 0.
If you use deflate() correctly, the data that comes back from a
Z_FULL_FLUSH will always end with 00 00 ff ff.
Mark
Qubeley
My compressed data always missed the last several pending bytes in
zlib buffer (0000FFFF + one or more previous 00 for padding previous
block) because of not checking the avail_out at the final step of
compression. After adding the checking it works well now and I can
constantly observe the trailing byte pattern at the end of each block.