How to enable LZ4c multi-threading Support?

[Memory Box]

Hi,

I'm testing LZ4c compress, I found description about multi-threading at

http://fastcompression.blogspot.com/p/lz4.html

What's new :
- Supports the LZ4 Streaming format (version 1.4)
- Switch to gzip-compatible command line arguments

Compression level comparison :
In-memory benchmark (-b command), using Silesia Corpus
Benchmark platform : Core i5-3340M (2.7GHz), Window Seven 64-bits 

Compression / Thread	Ratio	Speed	Decoding
-1 / 4 threads	2.195	880 MB/s	3.80 GB/s
-1 / 1 thread	2.195	333 MB/s	1.76 GB/s
-9 / 4 threads	2.716	58.0 MB/s	4.38 GB/s
-9 / 1 thread	2.716	23.1 MB/s	2.13 GB/s

But when i run 

lz4  input.file output.lz4

or

lz4 -9 input.file output.lz4

it seems only use one core,  How to enable LZ4c multi-threading Support?

Could that have been the problem?

Env:

CentOS release 6.5 (Final)

Linux rdtest181 2.6.32-431.23.3.el6.x86_64 #1 SMP Thu Jul 31 17:20:51 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz 24 Core

gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)

source:lz4-r121

Thank you

[Yann Collet]

There is no multi-threading support within lz4c currently.

The windows binary you mention uses a custom, windows-specific, piece of code to provide multi-threading.

This code is closed source and not portable.

Multi-threading is currently a low-priority item for reference lz4c,

because portable multi-threading is quite difficult to achieve using standard C99, 

and single-threaded file compression seems fast enough to keep up with most I/O (excluding lz4hc compression obviously).

So, from a global execution time perspective, there is little gain to look for.

Multi-threading is more worthwhile at application level.

But then, at this level, programmers use the block-compression API directly, not an lz4c binary.

However, if you are nonetheless interested in having multi-threaded lz4 compression/decompression,

you can use today Takayuki Matsuoka C++11 version, at https://github.com/t-mat/lz4mt.

Regards

[Francesc Alted]

Another possibility is using LZ4 through Blosc as its multithreading capabilities allow all the supported compressors (including lz4 and lz4hc) to leverage them for free.  Multithreading in Blosc supports all major platforms (Linux, Mac and Win).  See some examples for linking with Blosc that I just added:

https://github.com/Blosc/c-blosc/tree/master/examples

and for using it with lz4/lz4hc, see specially:

https://github.com/Blosc/c-blosc/blob/master/examples/many_compressors.c

Francesc

El 21/08/14 a les 13:14, Yann Collet ha escrit:

--
Vous recevez ce message, car vous êtes abonné au groupe Google Groupes "LZ4c".
Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le concernant, envoyez un e-mail à l'adresse lz4c+uns...@googlegroups.com.
Pour obtenir davantage d'options, consultez la page https://groups.google.com/d/optout.

-- 
Francesc Alted

[Yann Collet]

So it's possible to use Blosc without the shuffling ?

Quite interesting, I wasn't aware of this

Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le concernant, envoyez un e-mail à l'adresse lz4c+unsubscribe@googlegroups.com.

Pour obtenir davantage d'options, consultez la page https://groups.google.com/d/optout.

-- 
Francesc Alted

[Francesc Alted]

Sure.  In:

https://github.com/Blosc/c-blosc/blob/master/examples/many_compressors.c#L84

just change the second parameter to 0 and shuffle is out.  Full info for blosc_compress() call:

https://github.com/Blosc/c-blosc/blob/master/blosc/blosc.h#L115

Francesc

El 21/08/14 a les 14:24, Yann Collet ha escrit:

Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le concernant, envoyez un e-mail à l'adresse lz4c+uns...@googlegroups.com.

Pour obtenir davantage d'options, consultez la page https://groups.google.com/d/optout.

-- 
Francesc Alted

[Memory Box]

Thanks Francesc Alted,

It seems that blosc is what i need, I'll try it.

 

在 2014年8月21日星期四UTC+8下午7时14分22秒，Yann Collet写道：

[Memory Box]

Thanks for your reply, Yann

在 2014年8月21日星期四UTC+8下午7时14分22秒，Yann Collet写道：

[Roger Pack]

But just turning off shuffle is not enough to create a "normal LZ4 decodable" stream, right, you still need to de-multi-thread it on the receiving end [using blosc]?

Thanks!

-roger-

Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le concernant, envoyez un e-mail à l'adresse lz4c+uns...@googlegroups.com.
Pour obtenir davantage d'options, consultez la page https://groups.google.com/d/optout.

-- 
Francesc Alted

--
Vous recevez ce message, car vous êtes abonné au groupe Google Groupes "LZ4c".
Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le concernant, envoyez un e-mail à l'adresse lz4c+uns...@googlegroups.com.
Pour obtenir davantage d'options, consultez la page https://groups.google.com/d/optout.

-- 
Francesc Alted

[Francesc Alted]

Yes, I think you are right.  But nevertheless, you can still use Blosc for benchmarking LZ4 in multithreading mode.

Francesc