Is it possible to do compression C++ and decompression in pure JS?

[Atanai Wuttisetpaiboon]

I have compressed data in C++ by using "LZ4_compress_default" but failed to decompress data in pure JS by calling "var decompressedData = lz4.decode(decodedData);"

I am getting this error in pure JS - Uncaught Error: Invalid magic number: 7B42B3C3 @0

[Takayuki Matsuoka]

Hi Atanai,

> Is it possible to do compression C++ and decompression in pure JS?

Yes, it is.  But you should use same format in both (C/C++ and JS) side.  LZ4 has two formats:

LZ4 Frame Format
https://github.com/Cyan4973/lz4/blob/master/lz4_Frame_format.md

LZ4 Block Format
https://github.com/Cyan4973/lz4/blob/master/lz4_Block_format.md

If you are new to LZ4, I recommend you to use "LZ4 Frame Format".

> I have compressed data in C++ by using "LZ4_compress_default" but failed to decompress data in pure JS by calling "var decompressedData = lz4.decode(decodedData);"
> I am getting this error in pure JS - Uncaught Error: Invalid magic number: 7B42B3C3 @0

It seems you are using "Block Format" in C++ and "Frame Format" in JS.  It's wrong usage.  You should use same format.
For example, it would be resolved by using "LZ4F_compressFrame()" (lz4frame.h, Frame Format) instead of "LZ4_compress_default" (lz4.h, Block Format).

Hope this helps.

Regards,

[Atanai Wuttisetpaiboon]

Thanks Takayuki. I have changed JS to use "block format" and it is working now :D

so which format is better? as I see that the performance of decompression in JS by using "block format" is not good as C++.

[Takayuki Matsuoka]

> so which format is better?

I'm not sure about your situation, but you can do your own benchmark with node-lz4's benchmark and example codes:
https://github.com/pierrec/node-lz4/blob/master/benchmark/bench.js
https://github.com/pierrec/node-lz4/tree/master/examples

> as I see that the performance of decompression in JS by using "block format" is not good as C++.

If you're using node-lz4, it sounds strange.  Technically, it's native binding.  So basically its performance is almost same as native lz4 library.
Here is my rough benchmark and result: https://gist.github.com/t-mat/18c3423eec5cf27b559e782196f9f39c

[Takayuki Matsuoka]

> If you're using node-lz4, it sounds strange.

Oops, I think I completely misunderstood your situation.  Do you use node-lz4/build/lz4.js for decoding?

[PierreC]

Hello,

I am the author of the JS version of LZ4.

Indeed, as you have found out, the pure JS version (not the node one which uses bindings to the C API), is slower than the C/C++ implementations, partly due to the way JS handles unsigned integers (see the cuint library to emulate C behaviour).

There is not much you can do (at least that I could) to improve further although it is not that bad?

Any input to improve it is welcome though!

Pierre

[Takayuki Matsuoka]

Hi,

Just an FYI, it seems lz4.js ('lz4-asm' at npm) https://github.com/ukyo/lz4.js performs better in my browser (FireFox and Chrome) for now.
You can try its example code : https://github.com/ukyo/lz4.js/blob/master/example/bench.html

I'm not sure lz4.js always performs better in all browsers, but it's interesting.

[PierreC]

Ah nice one, using emscripten :). I will check the benchmarks when I get around to it.

Thanks!