LZ4 on audio data

[Basj]

Hello Yann and LZ4-users,

First of all congratulations for this wonderful tool that I just discovered a few days ago.

I'm using LZ4 on raw audio data (similar to .wav files), mainly for 2 reasons :
* speed (many times 100 MB /s for encoding)
* simplicity : in Python, I can do it in one line :   lz4.dumps(my_numpy_array)     and it's done !

Of course, I know that LZ4 is not mainly designed for audio data, so the compressed data is approx. 80% size of the original data.

Is it possible, with a few tricks, to have a higher compression ratio with LZ4 on .WAV audio data, or would this involve a totally new algorithm, and it would be out of topic of LZ4 (which is not designed for audio) ?

Remark 1 : I have looked at FLAC, but unfortunately, there is no simple tool like   flac.dumps(my_numpy_array)   in Python...

Remark 2 : this question may concern any kind of sampled data measured with a microphone or any other sensor, i.e. arrays containing the result of the sampling of a continuous function... So it may concern a wider range of data than just audio.

Thanks in advance,
Basj

[Yann Collet]

Hi Basj

Compressing files such as a .wav requires a different algorithm,

typically samples are delta-encoded, or sometimes order-1 delta encoded,

and the result of this transformation is provided to a regular compressor,

such as LZ4, or more likely, to an Huffman (or FSE) compressor.

This is the kind of thing that FLAC is doing.

I'm afraid there is no simple solution available around, it would require to rewrite a new algorithm.

Regards

Yann

[Basj]

Hi Yann,

Thanks for your answer.
I tried to do :
delta_array[k] = input[k] - input[k-1]
and to LZ4-compress the delta_array,  but the compression ratio wasn't better (and that's strange, it was even worse! maybe have I done something wrong?)

Have you ever tested to compress some .wav with LZ4 + some tricks ?

It could be really interesting because FLAC is much slower compared to LZ4... In fact, I'm searching for something in the middle of FLAC and LZ4 :
* faster than FLAC (it's ok if it's a little bit slower than LZ4)
* better ratio than LZ4 for audio (it's ok if it's not as good as FLAC for audio)

Any idea ?

Basj

[Yann Collet]

> I tried to do :
> delta_array[k] = input[k] - input[k-1]
> and to LZ4-compress the delta_array,  but the compression ratio wasn't better

Your file is likely an 16-bits audio wav, so delta_array[k] = input[k] - input[k-1]
should work on 16-bits values too.

It would also be better to know if it is mono or stereo.

Basically, that's an issue with format specific compressors : you need to properly assess the data format you are compressing, since it is a fundamental assumption of the following algorithm.

Blosc might help here, simply because it reorders high-bytes together. If the periodicity of Blosc and wav files matches, it will improve compression.

Low-bytes are probably equivalent to "noise", and therefore won't compress well. Better pack them together, so they don't hamper compression of high-bytes.

[Basj]

Hello,

Thanks for your answer.

It would also be better to know if it is mono or stereo.

It did benchmarking on mono files (but it may be stereo in the future).
 

Blosc might help here, simply because it reorders high-bytes together. If the periodicity of Blosc and wav files matches, it will improve compression.

Low-bytes are probably equivalent to "noise", and therefore won't compress well. Better pack them together, so they don't hamper compression of high-bytes

That's true ! If I replace

w = np.fromfile('myfile.bin', dtype=np.int16)         
z = lz4.compress(w)

by

w = np.fromfile('myfile.bin', dtype=np.int8).reshape((-1,2)).ravel(order='F')   # this groups all high-bytes together
z = lz4.compress(w),

I now get the same compression with "standard lz4" (lz4.compress)  and "blosc-lz4" (blosc.pack_array(w, cname='lz4'))

(something like a +10 / +15% compression gain).

Thank you very much !

Best regards.