Cannot decompress files compressed with lz4 tools (CLI and Windows)

[Felix Bytow]

Hi guys,

I'm the maintainer of Compress::LZ4Frame (perl binding for LZ4) and need your help :-)

As far as I see, the lz4 tools generate frames, where the content size header is 0 (unknown).

The current stable version of Compress::LZ4Frame requires a known content size,

which works for data that was compressed with it, but may be bad it the header is 0 :/

So in this branch: https://github.com/autinitysystems/Compress-LZ4Frame/tree/decompress_zero_size

I implemented the decompression when the content size is unknown.

Now for testing I generated a text file and compressed it using the LZ4 program yielding the t/lorem.txt and t/lorem.txt.lz4

I wrote a little test loading the two files, decompressing the compressed one and then comparing the original to the decompressed version.

Funnily now the code errors before I even get to check if the content size header is 0.

LZ4F_getFrameInfo returns ERROR_frameType_incomplete (master branch) and ERROR_frameType_unknown (dev branch)

which is rather confusing.

Using the LZ4 tools I am able to decompress the .lz4 file.

Currently I am stuck, so if someone could take a look at the code that would be great.

The relevant parts are CompressLZ4Frame.xs (C) and t/01-compression.t (Perl, just in case I have a bug in the testing code... you never know O.o)

As for the XS file the interesting part is the decompress_single_frame function directly at the beginning of the file.

It is called repeatedly for concatenated LZ4 frames and so far works for normal data.

Only problem is stuff compressed with the tools :/

In line 30 the frame info is read which returns the errors above (currently ERROR_frameType_unknown, because I replaced the LZ4 sources with current versions from the dev branch).

Thanks in advance

Felix

[Cyan]

Hi Felix !

Is your problem basically the same as :

https://github.com/Cyan4973/lz4/issues/157 ?

[Felix Bytow]

I guess so.

Also the linked issue was supposed to be fixed by the pull request I commented on.

That's why I get ERROR_frameType_unknown now instead of ERROR_frameType_incomplete.

Also it is not exactly the same problem as I did not compress empty data.

I just used the lz4 windows tool. (I also tested it with the lz4 command line tool which had the same result).

[Takayuki Matsuoka]

Hi Felix,

> I just used the lz4 windows tool.

I think we should make sure what is "lz4 windows tool" before investigation.
Do you mean "lz4 windows tool" as Yann's legacy LZ4 for Windows?
I mean did you download and use LZ4 from http://fastcompression.blogspot.com/p/lz4.html ?

[Cyan]

If I do understand your issue correctly Felix,

your program requires knowing the size of content.

This information is optional, and by default, not provided (that's what `0` means).

It's possible to add it, using the lz4 cli, with the command `--content-size`.

Note that this command is not available on the windows cli available at http://fastcompression.blogspot.fr/p/lz4.html , which is much older and now considered obsolete.

Content size or not, it's unclear why it would generate an ERROR_frameType_unknown error message.

It's supposed to mean the header has not been properly identified.

Looking at compressed files with an Hex editor, I see no good reason for that.

[Felix Bytow]

Exactly that one. But AFAIK it also just uses the cli tool.

[Felix Bytow]

My code works correctly if the content size header is set.

I do know that it is optional and can be enforced by specifying "--content-size".

The branch I referred to in my initial post was meant to fix this by also supporting content size headers of 0.

Yet, as said, this does not work, because LZ4F_getFrameInfo already returns ERROR_frameType_unknown.

[Felix Bytow]

Wow thank you, you gave me the right hints and now I was able to fix it.

You may read the added comments and the last commit message if you are interested in the details.

Am Donnerstag, 11. August 2016 17:47:44 UTC+2 schrieb Cyan: