lzo vs lz4

[Henrik Nilsson]

I have recently backported lz4 and latest changes to lzo from linux-next to a linux kernel I am using. I was under the impression that if I compressed with lz4c -c1 I would get not only higher compression ratio than lzo, but also higher decompression speed. That is how I should read the chart I suppose?

LZ4 HC (r101)   2.720     25    2080

vs

LZO 2.06        2.106    414     600

What happens is that the lzo compressed kernel is about 4.8M, and the lz4 compressed kernel is about 5.1M. It was compressed using lz4c -c1, that is equivalent with hc I hope. What could I be doing wrong?

Can you clarify if the lzo results in your chart are done with or without unaligned access for lzo? I guess they are done with unaligned access for lz4.

Cheers,

HN

[Yann Collet]

Hi Henrik

There 2 parts in your question :

1) LZO has several versions. They are not comparable, nor always compatible either.

The version used in the benchmark is 1x_1, which is considered the fastest.

(I'll add it to the chart, so that there is no confusion).

Other version compress more, such as 1x_999.

The Linux compressed kernel you refer to was compressed using a high compression derivative of LZO.

2) The benchmark was achieved on a PC (CPU spec provided).

The situation is not at all comparable with an ARM environment,

especially when it comes to unaligned memory accesses, since modern x86/x64 CPU are very good at that.

Regards

[Henrik Nilsson]

How are they not comparable? Do you mean in the same way that lz4 and lz4hc is not comparable? Which only means that they differ a lot, not that they are not actually comparable.

Which versions that are actually not compatible I don't know. What I do know is that the linux command line utility lzo is able to compress something that the linux kernel then can decompress. That indicates compatibility to me. I am sure there are lzo compressors that compress in a manner that the linux kernel cannot decompress, I have not tested those, nor do I think they are relevant.

Regarding the benchmark, you say it was done on pc, and you also say that unaligned access is most efficient on pc, but the tests chart does not indicate which de/compressors were run using it or not.

Also, unaligned access seemingly does make quite a dramatic difference on arm too, a comment from the git repo at http://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=cffb78b0e0b3a30b059b27a1d97500cf6464efa9

"1. ARMv7, 1.5GHz based board

Kernel: linux 3.4 Uncompressed Kernel Size: 14MB Compressed Size Decompression Speed LZO 6.7MB 20.1MB/s, 25.2MB/s(UA) LZ4 7.3MB 29.1MB/s, 45.6MB/s(UA) 2. ARMv7, 1.7GHz based board Kernel: linux 3.7 Uncompressed Kernel Size: 14MB Compressed Size Decompression Speed LZO 6.0MB 34.1MB/s, 52.2MB/s(UA) LZ4 6.5MB 86.7MB/s - UA: Unaligned memory Access support - Latest patch set for LZO applied"

I agree performance is not the same using the above kind of arm architecture or a modern pc, but the relative performance increases you see from unaligned access is comparable I'd say.

Anyway, my main point is that just looking at the chart like I did, and then comparing results from my kernel compressed with lzo and lz4hc, it is inconsistent. The reason for that is that the chart uses another lzo compressor than the command line utility on most linuxes. That was what was unclear for me, until now. Please correct me if I'm still wrong! =)

As a side note, the actual command used to compress the lz4 kernel is lz4c -l -c1. I have not found any documentation for the -l switch just yet. (My guess is legacy format)

[Yann Collet]

>  the actual command used to compress the lz4 kernel is lz4c -l -c1. I have not found any documentation for the -l switch just yet. (My guess is legacy format)

Indeed, -l stands for legacy format.

There is no documentation on it to discourage its usage; it's becoming increasingly important to spread a common file format across architectures and languages.