Frame Decompression Feasibility on Chunk Decompression

Dhruv Upadhyay

unread,

Oct 20, 2022, 6:33:11 PM10/20/22

to LZ4c

Hello,

I am working on an application where my file would ideally be compressed as one unit, and then sent, chunk by chunk to a downstream target where it would be decompressed as they arrive. Is it possible to have this done via frame decompression? If so, how would I know the size of the next header? Is there a way to determine the size of the next header without decompression?

Best,

Dhruv

Yann Collet

unread,

Oct 20, 2022, 9:38:29 PM10/20/22

to LZ4c

This is exactly the way streaming works.

This scenario is supported by the `lz4` command line utility,

and the `lz4frame` API.

If you want to create and decode the frames yourself,

you can have a look at the frame specification:

https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md

Dhruv Upadhyay

unread,

Oct 21, 2022, 1:01:22 PM10/21/22

to LZ4c

Thank you! I went thorugh the lz4Frame API, but one question I had was regarding the size of the next block. Is it only possible to determine the size of the compressed frame chunks by decompressing it? Or would they be equal in size?

Dhruv Upadhyay

unread,

Oct 21, 2022, 1:20:08 PM10/21/22

to LZ4c

*next compressed block

Yann Collet

unread,

Oct 21, 2022, 2:42:53 PM10/21/22

to LZ4c

You are looking at the decompressed size of the next compressed block.

Presuming it's a compressed block (otherwise it's obvious),

there are only 2 guarantees :

- The decompressed size is larger than the compressed size

- The decompressed size is bounded by the maximum block size (determined at the beginning of the frame)

That's all that can be told beforehand, without decompressing the block.

Dhruv Upadhyay

unread,

Oct 21, 2022, 3:50:26 PM10/21/22

to LZ4c

But then how would one determine how much of the next compressed chunk to request before decompressing? If there is no way to determine the size of the next compressed chunk?

Yann Collet

unread,

Oct 21, 2022, 4:23:30 PM10/21/22

to LZ4c

This is a different topic.

The decompressor needs to know the size of the next _compressed_ block.

And this information is provided up-front, in the block header.

Dhruv Upadhyay

unread,

Oct 21, 2022, 4:26:40 PM10/21/22

to LZ4c

Ohh so the block header of one chunk has the size of the next compressed block in its header? Is there a way to read that without decompressing the block?

Dhruv Upadhyay

unread,

Oct 21, 2022, 4:27:05 PM10/21/22

to LZ4c

And if so, how would one determine the size of the first compressed chunk?

Reply all

Reply to author

Forward