LZO compression for zfs-fuse-0.7.0

[Marcin Szychowski]

Hello there,

I have just updated Eric's lzo patch found here: http://goo.gl/fEeMJ

How to use it:
1. Download & untar zfs-fuse-0.7.0
2. Download & unpack attached patch
3. Apply it with command: patch -p0 < zfs-fuse-0.7.0-lzo.patch
4. Recompile & install zfs-fuse as described in INSTALL file
5. Set compression=lzo on selected filesystems.

Beware:
1. This is not an official patch - it is not known if LZO compression will ever be incorporated into original ZFS.
2. Forget about deduplication - it's a simple update (as for now).


--
Greetings,
Marcin Szychowski
--
http://backblaze.pl/
Backup before you wish you had

[Emmanuel Anne]

Thanks, seems quite interesting, could be incorporated to the official repository here.
I'll test it first !

2011/3/29 Marcin Szychowski <szy...@gmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

--
zfs-fuse git repository : http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

[Nizamov Shawkat]

What about compression library Google opensourced a few days ago -
snappy? Looks like it should be very efficient in filesystems. Just
curios, obviously, I can't help with it due to lack of experience.

wbr,
Shavkat

[Emmanuel Anne]

Wow this patch was almost completely lost for 4 years !!!
Tsss... !!!

Not totally sure about the compatibility with different zfs pools versions since he says it changes something at offset 30 ? Didn't read the code yet.
Anyway it's applied, I'll test it.

2011/3/29 Nizamov Shawkat <nizamov...@gmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Emmanuel Anne]

After reading the code : no compatibility problem, the 30 story is just because he was too paranoid and decideded that it would be compression method number 30, adding something like 24 empty declarations between the last compression method and this one.

Which means that in case some new compression methods are officially added, they can add something like 24 new ones before colliding with this one, that's all.

Except that the patch seems ok, it should probably have been added earlier.
I'll use it for a while just to be sure.

2011/3/29 Emmanuel Anne <emmanu...@gmail.com>

[Marcin Szychowski]

On Mar 29, 3:51 pm, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:
> After reading the code : no compatibility problem, the 30 story is just
> because he was too paranoid and decideded that it would be compression
> method number 30, adding something like 24 empty declarations between the
> last compression method and this one.
>
> Which means that in case some new compression methods are officially added,
> they can add something like 24 new ones before colliding with this one,
> that's all.
>
> Except that the patch seems ok, it should probably have been added earlier.
> I'll use it for a while just to be sure.
>

> 2011/3/29 Emmanuel Anne <emmanuel.a...@gmail.com>
>
>

Deduplication doesn't work with lzo-compressed data. It actually works
with uncompressible data on filesystems using compress=lzo. I am not a
C hacker so I can't work it out in predictible timeframe ;-)

Perhaps we need to initialize some buffers with zeroes or something?

>
>
>
>
>
>
>
> > Wow this patch was almost completely lost for 4 years !!!
> > Tsss... !!!
>
> > Not totally sure about the compatibility with different zfs pools versions
> > since he says it changes something at offset 30 ? Didn't read the code yet.
> > Anyway it's applied, I'll test it.
>

> > 2011/3/29 Nizamov Shawkat <nizamov.shaw...@gmail.com>

>
> >> What about compression library Google opensourced a few days ago -
> >> snappy? Looks like it should be very efficient in filesystems. Just
> >> curios, obviously, I can't help with it due to lack of experience.
>
> >> wbr,
> >> Shavkat
>
> >> --
> >> To post to this group, send email to zfs-...@googlegroups.com

> >> To visit our Web site, click onhttp://zfs-fuse.net/

[Emmanuel Anne]

Yes after checking there is something weird. I won't commit the patch for now, there is something not normal somewhere.

2011/3/29 Marcin Szychowski <szy...@gmail.com>

To visit our Web site, click on http://zfs-fuse.net/

[Emmanuel Anne]

Ok. Finally it's not a bug, it's a "feature".

Apparently lzo tries to build a dictionary while compressing. Which means that the 1st time you pass a block to compress, it returns something it really tried to compress (normal).
Then if you pass the same block, it will return something telling "already seen this, in this block".
Well too bad, the result is that the return value can't be deduped at all.
More than that zfs doesn't seem to cut its blocks to be compressed the same way when passing them to lzo, and lzo seems to always start its compressed streams with the same kind of bytes.
The result : it works very badly with dedup.

Even without dedup, you can wonder if it's really useful with zfs-fuse : lzo is intended to be very fast, even faster than gzip for decompressing. Its main goal is speed, not compression ratio. Here in zfs-fuse, compression time is not really an issue, I guess we could spend a little more time compressing without noticing any difference at the user level.

So is it really useful ?
If adding it, we could add support for the real lzo library, which adds at least another compression method, but I am not convinced it's very useful finally.

But the good news is that adding a new compression algorithm to zfs is VERY easy finally !

2011/3/29 Emmanuel Anne <emmanu...@gmail.com>

[Marcin Szychowski]

On Mar 29, 11:57 pm, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:
> Ok. Finally it's not a bug, it's a "feature".
>
> Apparently lzo tries to build a dictionary while compressing. Which means
> that the 1st time you pass a block to compress, it returns something it
> really tried to compress (normal).
> Then if you pass the same block, it will return something telling "already
> seen this, in this block".
> Well too bad, the result is that the return value can't be deduped at all.
> More than that zfs doesn't seem to cut its blocks to be compressed the same
> way when passing them to lzo, and lzo seems to always start its compressed
> streams with the same kind of bytes.
> The result : it works very badly with dedup.

Hmmm, that's strange. As you can find from this page: http://www.lessfs.com/
LZO & data deduplication can work together pretty well.

>
> Even without dedup, you can wonder if it's really useful with zfs-fuse : lzo
> is intended to be very fast, even faster than gzip for decompressing. Its
> main goal is speed, not compression ratio. Here in zfs-fuse, compression
> time is not really an issue, I guess we could spend a little more time
> compressing without noticing any difference at the user level.
>
> So is it really useful?
> If adding it, we could add support for the real lzo library, which adds at
> least another compression method, but I am not convinced it's very useful
> finally.

Even if deduplication won't work with LZO, it could show some benefits
on systems with lots of read()'s of highly compressible, unique data.

>
> But the good news is that adding a new compression algorithm to zfs is VERY
> easy finally !
>

As I wrote it is not my work, I only updated it to 0.7.0. If anyone
finds it useful - that's the power of open source! I believe the
deduplication issue can be eventually resolved by someone - maybe Mr.
Oberhumer could take care of his child? ;-)

Thanks for your message, Good night!

> 2011/3/29 Emmanuel Anne <emmanuel.a...@gmail.com>

[Emmanuel Anne]

Well if you are curious, I pushed this to a new branch on my repository : compress.
I also added on the same model bz2 compression.
This one works really well with dedup by the way ! ;-)
It uses the libbz2 for linking obviously, and uses the same compression level as the default level used by bzip2 : 9.

I think it could be merged in master. This is something which can be added very easily in zfs-fuse (contrary to zfs in kernel - wait, there are lots of compression algorithms in the kernel these days too, so maybe it's easy even in kernel !), and lzo is ok if you don't use dedup with it.

I think your article meant that lzo tries to use dedup itself to a certain level, but deduping an lzo stream doesn't work so well (without reading it, I might take a look later).

Thanks again for the link anyway, it would have been a pity to have missed this !

2011/3/30 Marcin Szychowski <szy...@gmail.com>

To visit our Web site, click on http://zfs-fuse.net/

[Emmanuel Anne]

By the way I almost didn't change the lzo patch finally. I just added something to avoid to call lzo_init all the time, even if it's a macro it shouldn't be called everytime we want to compress/decompress something.
Also notice that we are loosing some speed because we are obliged to call umem_alloc/umem_free everytime we want to compress something since it requires a separate buffer to be able to tell if the compressed buffer will fit or not in the available destionation space. Well I guess umem is very fast so it's not a big issue, but it's still more complex that what can be done with gzip or bz2.

2011/3/30 Emmanuel Anne <emmanu...@gmail.com>

[Daniel Smedegaard Buus]

How about LZMA? Is that easily added to ZFS as well? For those of us
looking for really high ratios at the cost of a severe performance
impact? :)

[sgheeren]

hmmm I always remembered bz2 is the 'best' in that direction?

[Fajar A. Nugraha]

http://stephane.lesimple.fr/wiki/blog/lzop_vs_compress_vs_gzip_vs_bzip2_vs_lzma_vs_lzma2-xz_benchmark_reloaded

xz (a.k.a lzma2) is much faster during decompression than bz2, with reasonable memory requirements for decompression if you use level 3-6 (6 is default). 

The downside is compression time and memory requirement is much higher. Good if your load is mostly read to highly-compressible data.

-- 

Fajar

[Emmanuel Anne]

Yes lzma might be interesting, that's what 7zip uses...

I ran a few tests with all the compression methods we have so far. The idea is to read linux-2.6.1.tar.bz2 from cache and uncompressing it to a zfs image, compressed with one of the method. The idea is that ascii gets very good compression ratios usually, and the file is big enough to have something to test.

Results : (time, the compression ratio obtained from zfs get compressratio)
lzjb (on) : 39,694s 1.97
gzip : 36,095 3.48x
lzo : 37,991 2.23x
bz2 : 45,502 3.59x

I wonder if someone still uses compress=on ? You should really switch to compress=gzip !
Except that bz2 is noticeably slower finally (but not too much), but it still has the best compression ratio. I guess lzma should be better, and maybe not slower...
I'll try that tonight or tomorrow...

2011/3/30 Fajar A. Nugraha <li...@fajar.net>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Emmanuel Anne]

dedup is fixed for lzo !
I switched to liblzo2 instead of using the incorporated minilzo, and it just did the trick, now dedup works correctly. (there was no reason to keep minilzo, it added some useless code, and liblzo has another compression method).

It also adds yet another compression method, lzo9, which is better but slower to compress of course. The times/ratios are a little different :
lzo1 2.43x 38.180
lzo9 3.02x 44.473

The time difference for lzo1 is too small to really matter, it's measured using the command time and so the variation is too small to matter here (it can be produced by something else in the system).
But it's funny to see the compression is now really different with a better compression ratio, and it works with dedup.

For info, with dedup enabled, I get :
lzo1 2.40x (?) 42.559
and then when creating a directory to uncompress the same tar inside I get a much slower time, but it might be because it's on a 200 Mb image. But this time zpool get dedupratio correctly returns 2.0, finally !

So it's really a good thing you found this patch after all ! Thanks again ! :)
(everything is pushed to the compress branch).

2011/3/30 Emmanuel Anne <emmanu...@gmail.com>

[Marcin Szychowski]

I didn't know you were THAT fast ;-)

This is something really great for me, I'll give it a deep testing.

I can't wait to test it tomorrow, BUT I looked into the source and
have small, tiny questions:

[...] from src/lib/libzfscommon/include/sys/zio.h:
ZIO_COMPRESS_ZLE,
ZIO_COMPRESS_LZO=30,
ZIO_COMPRESS_BZIP2=31,
ZIO_COMPRESS_LZO9,
ZIO_COMPRESS_FUNCTIONS
1. What is the value of ZIO_COMPRESS_FUNCTIONS?
2. Will it vary depending on number of declared compression methods?
If so, is it ok?
3. Before people create / populate filesystems: can we keep some order
here, probably reserving space for other compression methods and
possibly sub-methods a.k.a compression levels? LZO offers five
compression levels for instance, to quote lzop manual:

-3 the default level offers pretty fast compression. -2, -3, -4, -5
and -6 are currently all equivalent - this may change in a future
release. (these produce files labeled LZO1X-1 by command `lzop -l')

-1, --fast
can be even a little bit faster in some cases - but most times
you won't notice the difference (a.k.a. LZO1X-1(15))

-7, -8, -9, --best
these compression levels are mainly intended for generating pre-
compressed data - especially -9 can be somewhat slow (a.k.a.
LZO1X-999/7, LZO1X-999/8, LZO1X-999/9, respectively)

Moreover, bzip2 and LZMA/LZMA2 also have several (nine? more?)
compression presets. Especially LZMA family have wide range of
parameters that influence both resource (memory, cpu) requirements and
compression ratio. Simple `xz -9vv' command shows following parameters
for LZMA2:

lzma2: dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0

Not diving too deeply it is pretty obvious that people might want
experiment with all these switches in the future. Introducing a new
(family of) algorithm(s) is the best time to layout the future shape
of the code, just like Eric and Ricardo did four years ago. Why not
reserve values, say, 30-34(39?) for LZO variants, 40-49(59?) for BZIP2
and so on? I am not trying to convince you to implement all mentioned
algorithms in all variants till tomorrow, I am just begging for a
little better house(code?)keeping ;-)

Once again - thank you for great work you've done.

--
Greetings,
Marcin Szychowski.

On Mar 30, 8:27 pm, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:
> dedup is fixed for lzo !
> I switched to liblzo2 instead of using the incorporated minilzo, and it just
> did the trick, now dedup works correctly. (there was no reason to keep
> minilzo, it added some useless code, and liblzo has another compression
> method).
>
> It also adds yet another compression method, lzo9, which is better but
> slower to compress of course. The times/ratios are a little different :
> lzo1 2.43x 38.180
> lzo9 3.02x 44.473
>
> The time difference for lzo1 is too small to really matter, it's measured
> using the command time and so the variation is too small to matter here (it
> can be produced by something else in the system).
> But it's funny to see the compression is now really different with a better
> compression ratio, and it works with dedup.
>
> For info, with dedup enabled, I get :
> lzo1 2.40x (?) 42.559
> and then when creating a directory to uncompress the same tar inside I get a
> much slower time, but it might be because it's on a 200 Mb image. But this
> time zpool get dedupratio correctly returns 2.0, finally !
>
> So it's really a good thing you found this patch after all ! Thanks again !
> :)
> (everything is pushed to the compress branch).
>

> 2011/3/30 Emmanuel Anne <emmanuel.a...@gmail.com>

>
>
>
>
>
>
>
>
>
> > Yes lzma might be interesting, that's what 7zip uses...
>
> > I ran a few tests with all the compression methods we have so far. The idea
> > is to read linux-2.6.1.tar.bz2 from cache and uncompressing it to a zfs
> > image, compressed with one of the method. The idea is that ascii gets very
> > good compression ratios usually, and the file is big enough to have
> > something to test.
>
> > Results : (time, the compression ratio obtained from zfs get compressratio)
> > lzjb (on) : 39,694s 1.97
> > gzip : 36,095 3.48x
> > lzo : 37,991 2.23x
> > bz2 : 45,502 3.59x
>
> > I wonder if someone still uses compress=on ? You should really switch to
> > compress=gzip !
> > Except that bz2 is noticeably slower finally (but not too much), but it
> > still has the best compression ratio. I guess lzma should be better, and
> > maybe not slower...
> > I'll try that tonight or tomorrow...
>

> > 2011/3/30 Fajar A. Nugraha <l...@fajar.net>

>
> > On Wed, Mar 30, 2011 at 6:36 PM, sgheeren <sghee...@hotmail.com> wrote:
>
> >>> On 03/30/2011 01:03 PM, Daniel Smedegaard Buus wrote:
>
> >>>> How about LZMA? Is that easily added to ZFS as well? For those of us
> >>>> looking for really high ratios at the cost of a severe performance
> >>>> impact? :)
>
> >>>>  hmmm I always remembered bz2 is the 'best' in that direction?
>

> >>http://stephane.lesimple.fr/wiki/blog/lzop_vs_compress_vs_gzip_vs_bzi...

>
> >> xz (a.k.a lzma2) is much faster during decompression than bz2, with
> >> reasonable memory requirements for decompression if you use level 3-6 (6 is
> >> default).
>
> >> The downside is compression time and memory requirement is much higher.
> >> Good if your load is mostly read to highly-compressible data.
>
> >> --
> >> Fajar
>
> >> --
> >> To post to this group, send email to zfs-...@googlegroups.com

> >> To visit our Web site, click onhttp://zfs-fuse.net/

[Emmanuel Anne]

2011/3/31 Marcin Szychowski <szy...@gmail.com>

[...] from src/lib/libzfscommon/include/sys/zio.h:
       ZIO_COMPRESS_ZLE,
       ZIO_COMPRESS_LZO=30,
       ZIO_COMPRESS_BZIP2=31,
       ZIO_COMPRESS_LZO9,
       ZIO_COMPRESS_FUNCTIONS
1. What is the value of ZIO_COMPRESS_FUNCTIONS?

It's an enum, so  ZIO_COMPRESS_FUNCTIONS=ZIO_COMPRESS_LZO9+1=33 here.

2. Will it vary depending on number of declared compression methods?
If so, is it ok?

Yes and yes.
 

To visit our Web site, click on http://zfs-fuse.net/

[Emmanuel Anne]

Sorry for double post, I accidentally pressed "send" (morning error !).
Anyway :

2011/3/31 Marcin Szychowski <szy...@gmail.com>

3. Before people create / populate filesystems: can we keep some order
here, probably reserving space for other compression methods and
possibly sub-methods a.k.a compression levels? LZO offers five
compression levels for instance, to quote lzop manual:

 -3  the default level offers pretty fast compression.  -2, -3, -4, -5
and -6 are currently all equivalent - this may change in a future
release. (these produce files labeled LZO1X-1 by command `lzop -l')

 -1, --fast
    can be even a little bit faster in some cases - but most times
you won't notice the difference (a.k.a. LZO1X-1(15))

 -7, -8, -9, --best
    these compression levels are mainly intended for generating pre-
compressed data - especially -9 can be somewhat slow (a.k.a.
LZO1X-999/7, LZO1X-999/8, LZO1X-999/9, respectively)

Yeah well if you look into lzo api doc, you'll see there are tons of compression methods, and a lot of them with compression levels ! All this is quite messy, I don't want to add all possible lzo combination, it wouldn't make much sense here.
I added the original compression method in your patch, lzo, which is the basic compression algorithm  and which doesn't accept any compression level (it's called lzo1x_1, and the doc says it's the one which should be used in most cases, also the fastest one).
The 2nd, lzo9, stands for lzo1x_999, but this particular function I used (lzo1x_999_compress) doesn't take any compression level neither. Now I saw there are versions of this with a compression level, but it's not very well documented, so I prefered to stay away from it.
Now if you do some testing with the lzo utilities and find that some particular compression method/level can be useful, you can always post about it, for now it can be added easily.

Moreover, bzip2 and LZMA/LZMA2 also have several (nine? more?)
compression presets. Especially LZMA family have wide range of
parameters that influence both resource (memory, cpu) requirements and
compression ratio. Simple `xz -9vv' command shows following parameters
for LZMA2:

lzma2: dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0

Not diving too deeply it is pretty obvious that people might want
experiment with all these switches in the future. Introducing a new
(family of) algorithm(s) is the best time to layout the future shape
of the code, just like Eric and Ricardo did four years ago. Why not
reserve values, say, 30-34(39?) for LZO variants, 40-49(59?) for BZIP2
and so on? I am not trying to convince you to implement all mentioned
algorithms in all variants till tomorrow, I am just begging for a
little better house(code?)keeping ;-)

Not a big deal anyway, even if you set the value 1987 for a lzo variant, it would still work without breaking anything, and the user doesn't see this value anyway.
Where this value is important is only when compared to the values used by the official zfs implementation from solaris, that's why we add all these methods starting at 30, to leave some free space for any (unlikely) new official method.
The thing to remember is that all these methods will make filesystems unreadable with other zfs implementations...

For all the parameters : well sorry you'll have to do without most of them, you can't pass everything here. Actually even the compression level can't be passed directly, that's why they created the names gzip-1 to gzip-9 for all the compression levels for gzip.
I might do the same for bzip2, but I am not sure it will be useful : if you want speed, use gzip. If you want compression level, then the default is 9 for bzip2, so it's the one you really want to use.
Except that I saw you can pass an environment variable to lzma to limit the amount of memory it can use. It might be useful, and it would work here too anyway, but for now it's not added yet.

Once again - thank you for great work you've done.

It was fun so far ! :)

To visit our Web site, click on http://zfs-fuse.net/

[Stefan G. Weichinger]

Am 30.03.2011 20:27, schrieb Emmanuel Anne:

> So it's really a good thing you found this patch after all ! Thanks
> again ! :)
> (everything is pushed to the compress branch).

Created a test-ebuild here on gentoo, pulling your compress branch.

Unfortunately it does not build yet.

I get:

gcc -o zfs-fuse/zfs_operations.o -c -pipe -Wall -std=c99 -Wno-switch
-Wno-unused -Wno-missing-braces -Wno-parentheses -Wno-uninitialized
-fno-strict-aliasing -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_REENTRANT
-DTEXT_DOMAIN=\"zfs-fuse\" -s -O2 -DNDEBUG -D_KERNEL -DLINUX_AIO
-Ilib/libavl/include -Ilib/libnvpair/include -Ilib/libumem/include
-Ilib/libzfscommon/include -Ilib/libsolkerncompat/include
zfs-fuse/zfs_operations.c
zfs-fuse/zfs_operations.c: In function 'zfsfuse_listxattr':
zfs-fuse/zfs_operations.c:249:5: error: jump into scope of identifier
with variably modified type
zfs-fuse/zfs_operations.c:314:1: note: label 'out' defined here
zfs-fuse/zfs_operations.c:262:4: note: 'entry' declared here
zfs-fuse/zfs_operations.c:249:5: error: jump into scope of identifier
with variably modified type
zfs-fuse/zfs_operations.c:314:1: note: label 'out' defined here
zfs-fuse/zfs_operations.c:259:2: note: '({anonymous})' declared here
zfs-fuse/zfs_operations.c:253:2: error: jump into scope of identifier
with variably modified type
zfs-fuse/zfs_operations.c:314:1: note: label 'out' defined here
zfs-fuse/zfs_operations.c:262:4: note: 'entry' declared here
zfs-fuse/zfs_operations.c:253:2: error: jump into scope of identifier
with variably modified type
zfs-fuse/zfs_operations.c:314:1: note: label 'out' defined here
zfs-fuse/zfs_operations.c:259:2: note: '({anonymous})' declared here
scons: *** [zfs-fuse/zfs_operations.o] Error 1
scons: building terminated because of errors.

Any idea? Thanks, Stefan

[Daniel Smedegaard Buus]

On Mar 30, 4:25 pm, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:
> Except that bz2 is noticeably slower finally (but not too much), but it
> still has the best compression ratio. I guess lzma should be better, and
> maybe not slower...

Perhaps LZMA only starts shining with larger datasets that it can use
large dictionary and word sizes on, with added massive solid
archiving. Stuff that makes no sense when encoding blocks of what, 4k?

Also, I don't think the memory tweaks make much difference so long as
you're compressing this small amounts of data. The one setting in 7-
zip's LZMA algo that makes an impact on memory usage is the dictionary
size, which goes from 64k (38 M used to compress, 3 to decompress) to
1G (10789 M used to compress, 1026 to decompress). Regardless of which
one you choose, the dictionary would never grow larger than the actual
data being compressed.

Probably LZMA is not an obvious candidate for single block
compression.

Kudos on the work, by the way! Very nice :)

[Stefan G. Weichinger]

Am 31.03.2011 12:15, schrieb Stefan G. Weichinger:

> Created a test-ebuild here on gentoo, pulling your compress branch.
>
> Unfortunately it does not build yet.

Update: Done.

corrected my ebuild and modified the patch from

http://bugs.gentoo.org/show_bug.cgi?id=332395

to apply to the compress-branch from Emmanuel.

Also added a dependency to the ebuild to pull in the lzo-libraries.

It compiles now and works so far on my testbox.

I think I should add the files to bugs.gentoo.org?
gentoo-users, let me know if you want to test.

Stefan

[Emmanuel Anne]

Finally added the lzma compression for testing.

The results are insanely slow :
still the same test where I had with bzip2 :
3.58x in about 45s
with lzma it becomes
3.67x in 2:03 !!!

To reply to a previous mail : it can still be efficient because zfs blocks can be up to 128k, not just 4k.
I used here the same code as in their example code in doc/examples/xz_pipe_compress.c, which means level = 6, COMPRESSION_EXTREME = true.
Maybe I should try without this compression_extreme thing to see if it makes a difference ?

Anyway it's pushed, still in the compress branch, if someone else wants to test...

2011/3/31 Stefan G. Weichinger <gm...@oops.co.at>

--
zfs-fuse git repository : http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

[Stefan G. Weichinger]

Am 31.03.2011 20:27, schrieb Emmanuel Anne:
> Finally added the lzma compression for testing.
> The results are insanely slow :
> still the same test where I had with bzip2 :
> 3.58x in about 45s
> with lzma it becomes
> 3.67x in 2:03 !!!
>
> To reply to a previous mail : it can still be efficient because zfs
> blocks can be up to 128k, not just 4k.
> I used here the same code as in their example code in
> doc/examples/xz_pipe_compress.c, which means level = 6,
> COMPRESSION_EXTREME = true.
> Maybe I should try without this compression_extreme thing to see if it
> makes a difference ?

Who knows? ;-)

> Anyway it's pushed, still in the compress branch, if someone else wants
> to test...

Just "emerged" it here, fine.

Do you have a specific test-script or so? Just to make things comparable ...

Stefan

[Fajar A. Nugraha]

On Fri, Apr 1, 2011 at 1:28 AM, Emmanuel Anne <emmanu...@gmail.com> wrote:
> Finally added the lzma compression for testing.
> The results are insanely slow :
> still the same test where I had with bzip2 :
> 3.58x in about 45s
> with lzma it becomes
> 3.67x in 2:03 !!!
>
> To reply to a previous mail : it can still be efficient because zfs blocks
> can be up to 128k, not just 4k.
> I used here the same code as in their example code in
> doc/examples/xz_pipe_compress.c, which means level = 6, COMPRESSION_EXTREME
> = true.
> Maybe I should try without this compression_extreme thing to see if it makes
> a difference ?

... as well as changing the level :D

On the benchmark link I sent earlier, xz-2 is somewhere between
bzip2-2 and bzip2-3 in terms of compression ratio and time, xz-3 is
better than bzip2-9, while anything beyond that is just memory hog.

I'm guessing the optimal level would be somewhere between xz-3 and 4.

--
Fajar

[Emmanuel Anne]

Ok, good advice, so I just added lzma-1 to lzma-9 for testing, and set compress=lzma becomes equivalent to lzma-3.
It produces for my test a ratio of 3.51 in 54.790s, which is closer to bzip2 -9 but bz2 is still better (faster and better ratio).
Now maybe lzma-3 produces better results on some other kind of data, I know gzip is optimized to compress ascii very efficiently (and it shows !), so it needs testing...

2011/4/1 Fajar A. Nugraha <li...@fajar.net>

Fajar

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Aneurin Price]

On Wed, Mar 30, 2011 at 15:25, Emmanuel Anne <emmanu...@gmail.com> wrote:
> Yes lzma might be interesting, that's what 7zip uses...
>
> I ran a few tests with all the compression methods we have so far. The idea
> is to read linux-2.6.1.tar.bz2 from cache and uncompressing it to a zfs
> image, compressed with one of the method. The idea is that ascii gets very
> good compression ratios usually, and the file is big enough to have
> something to test.
>
> Results : (time, the compression ratio obtained from zfs get compressratio)
> lzjb (on) : 39,694s 1.97
> gzip : 36,095 3.48x
> lzo : 37,991 2.23x
> bz2 : 45,502 3.59x
>
> I wonder if someone still uses compress=on ? You should really switch to
> compress=gzip !

I'm using compression=lzjb on all of my datasets that I access
frequently or don't expect to be particularly compressible. When
making this decision I based it on benchmarking reads and writes of
uncompressible files, hoping that would represent the worst case.
Reading times were the same for all the compression methods I tried;
writing times as follows:

no compression: 1m18s
lzjb: 1m18s
gzip: 1m35s
gzip-1: 1m30s
gzip-9: 1m35s

So lzjb seems to come with no overhead for me in the worst case, and I
decided I might as well turn it on everywhere. Note that the file in
this test is 1.09GB, which comes to a write rate of just 14MB/s.
Anyone on a system that's faster to start with may well start noticing
a slowdown even with lzjb.

Nye

[Seth Heeren]

Completely agreed.

Also, see this absolutely awesome comment by Jeff Bonwick:

http://don.blogs.smugmug.com/2008/10/13/zfs-mysqlinnodb-compression-update/#comment-3814

On 1 apr, 12:02, Aneurin Price <aneurin.pr...@gmail.com> wrote:

[Emmanuel Anne]

So it appears lzjb can be useful on highly compressible data after all ! ;)
I am quite surprised gzip had such a bad performance, too bad the test can't be reproduced with zfs-fuse...
For me it was the opposite, lzjb was even slower than gzip, but it was with a debug build, maybe the numbers change with an optimized binary.

Interesting anyway !

2011/4/1 Seth Heeren <sghe...@hotmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Seth Heeren]

Well, the benchmark I linked to was

- InnoDb on OpenSolaris (specific load, different OS)

It is not really general purpose, and probably doesn't compare. So
what we really need, is general-purpose metric on the compression

On 1 apr, 15:51, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:
> So it appears lzjb can be useful on highly compressible data after all ! ;)
> I am quite surprised gzip had such a bad performance, too bad the test can't
> be reproduced with zfs-fuse...
> For me it was the opposite, lzjb was even slower than gzip, but it was with
> a debug build, maybe the numbers change with an optimized binary.
>
> Interesting anyway !
>

> 2011/4/1 Seth Heeren <sghee...@hotmail.com>

>
>
>
>
>
> > Completely agreed.
>
> > Also, see this absolutely awesome comment by Jeff Bonwick:
>

> >http://don.blogs.smugmug.com/2008/10/13/zfs-mysqlinnodb-compression-u...

> > To visit our Web site, click onhttp://zfs-fuse.net/

[Emmanuel Anne]

Well I took inspiration from what they did and redid a few tests. I am sure a lot of people will prefer this version. The differences :
 - optimized build this time, it changes only lzjb I think even gzip uses an external libz, but not sure about that.
 - instead of using tar xvfj, I just used :
time bunzip2 --stdout /usr/src/linux-2.6.1.tar.bz2 > linux-2.6.1.tar
The thing is that zfs-fuse is extremely unefficient at unpacking lots of files (tar xvfj) because it means creating lots of small files, and changing attributes for all of them, and it calls opendir/readdir/closdir after each file. It's a nightmare, but for now I have no idea about how to speed this up. So using bunzip2 --stdout instead is much more efficient, here it has something big to compress, perfect situation.

So the results are :
type            size        time
uncompressed    186501120   17,479
lzjb (on)       88276k      14,768
lzo             64793k      13,541
gzip            42696k      15,352
lzma            39383k      19,933
bzip2           37992k      23,397

The uncompressed size is in bytes, sorry I should have put kbytes here. It's sorted by size.
So : lzo is the fastest here, even faster than uncompressed because it creates less output.
In an ideal situation like this lzma becomes more interesting, it's almost as good as bzip2, but noticeably faster !
lzjb is indeed slightly faster than gzip, but slower than lzo !
The compressed size is just taken from the output of ls -sk.
 

2011/4/1 Seth Heeren <sghe...@hotmail.com>

To visit our Web site, click on http://zfs-fuse.net/

[Emmanuel Anne]

Ah and forgot lzo9 :

type            size        time
uncompressed    186501120   17,479
lzjb (on)       88276k      14,768
lzo             64793k      13,541

lzo9            48213k      25,551

gzip            42696k      15,352
lzma            39383k      19,933
bzip2           37992k      23,397

Conclusion : don't use lzo9, it's less efficient than gzip, and slower than bzip2 !

2011/4/1 Emmanuel Anne <emmanu...@gmail.com>

[Marcin Szychowski]

On Apr 1, 6:44 pm, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:
> Ah and forgot lzo9 :

> lzo             64793k      13,541
> lzo9            48213k      25,551
>

> Conclusion : don't use lzo9, it's less efficient than gzip, and slower than
> bzip2 !
>

> 2011/4/1 Emmanuel Anne <emmanuel.a...@gmail.com>
>
>

Keep in mind that LZO's *decompression* "is very fast for all levels,
and decompression speed is not affected by the compression level".

That I confirmed in few minutes by decompressing from memory tar
archive of my /usr/src directory (source files + compiled objects).

method compressed uncompr. ratio uncompressed_name
LZO1X-1 1441222612 3930705920 36.7% usr-src.tar
LZO1X-999/9 1153994668 3930705920 29.4% usr-src.tar

method speed/compressed speed/original
LZO1X-1: 102.656 MB/s 279.98 MB/s
LZO1X-999/9: 91.704 MB/s 312.36 MB/s

I picked highest result from eight subsequent tests. As you can see,
while test did not confirm manual's claim, decompression speed per
original megabyte is even higher by 11.5%, while data takes up 20%
less space on disk.

So with lzo9 you might gain from few to several dozens percent of
performance improvement with rarely updated and frequently read, such
as ... your root or /usr partition! It is not even hard to imagine
following sequence:

zfs set compression=lzo9 mypool/src
cd /mypool/src
tar xf linux-kernel-2.6.xxx.tar.bz2
zfs set compression=lzo mypool/src
cd linux-kernel-2.6.xxx/
make ....


Anyone wants to perform such benchmark? For 5-15 times? :-)


Cheers,
Marcin Szychowski.

> > 2011/4/1 Seth Heeren <sghee...@hotmail.com>

[Emmanuel Anne]

The problem is how do you measure decompression time ?
The problem with zfs is that if some data is uncompressible, then it's just stored uncompressed -> reading it is like reading the original, do difference.

And I think uncompressing is also very fast for gzip and bzip2...
I guess it can be measured with something like that
bzcat linux-2.6.1.tar.bz2 > zfs/tar (measures compression time)
export
import (to clear the cache)
md5sum zfs/tar -> measures decompression time.
Most of the methods should be very close to what you get with uncompressed data, though.
Oh well, it's worth trying to test it anyway...

2011/4/3 Marcin Szychowski <szy...@gmail.com>

--
my zfs-fuse git repository : http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

[sgheeren]

On 04/03/2011 12:22 AM, Emmanuel Anne wrote:
> bzcat linux-2.6.1.tar.bz2 > zfs/tar (measures compression time)

Wouldn't that measure the compression time _WHILE_ decompressing in bzcat?

I'd do

bzcat linux-2.6.1.tar.bz2 > /tmp/cachefile
md5sum /tmp/cachefile

cat /tmp/cachefile > zfs/tar # (measures compression time)

[Emmanuel Anne]

Yes but since there is the same bz2 decompression for all the methods, it becomes a constant time and can be ignored. What matters here is the differences between the final results, not the values themselves.
But yes you can work from already uncompressed data if you prefer, it's just that it takes more space, especially if you want to read it from cache.

2011/4/3 sgheeren <sghe...@hotmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[sgheeren]

On 04/03/2011 12:29 AM, Emmanuel Anne wrote:
> Yes but since there is the same bz2 decompression for all the methods,
> it becomes a constant time and can be ignored. What matters here is
> the differences between the final results, not the values themselves.
> But yes you can work from already uncompressed data if you prefer,
> it's just that it takes more space, especially if you want to read it
> from cache.

Point taken

[Emmanuel Anne]

Ok, I get these times :
type            size        time    uncompress time
uncompressed    182283      18.287  0.927
lzjb (on)       88276k      14,768  1.130
lzo             64793k      13,541  1.068
lzo9            48213k      25,522  0.990
gzip            42696k      15,352  1.110
lzma            39383k      21.769  2.394
bzip2           37628k      23,397  3.278

Comments : the results are not very precise, times obtained this way (through time command) vary sometimes, so maybe it would be better to get a bigger file to test this...
Anyway it seems to show indeed that lzo is very efficient on this side too.
gzip is almost as good.
lzma and bzip2 might be too much, except if you are ready to sacrifice some time then.

2011/4/3 Marcin Szychowski <szy...@gmail.com>

To visit our Web site, click on http://zfs-fuse.net/

--
my zfs-fuse git repository : http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

[Marcin Szychowski]

Not quite.

BZIP2[-9] is about ten times slower to decompress than LZO9.
21s / 2s = 10.5
(21s + 21s) / (21s + 2s) = 42s / 23s = 1.83

Adding a constant to both sides of equation is irrelevant for adding
and subtracting only.

Now how I can tell bzip2 is that slower?

I took fresh kernel source tarball from kernel.org
(linux-2.6.38.2.tar.bz2) and recompressed it using few different
programs:
gzip (default a.k.a -6), gzip -1, gzip -9, xz -1[,2,3], lzop
(default), lzop -1 and -9, yielding:

74789685 linux-2.6.38.2.tar.bz2 (original unmodified source
tarball)
75013320 linux-2.6.38.2.tar.xz3 (xz -3)
76896764 linux-2.6.38.2.tar.xz2
80830872 linux-2.6.38.2.tar.xz1
94153668 linux-2.6.38.2.tar.gz9 (gzip -9)
94974362 linux-2.6.38.2.tar.gz (gzip): default
108411308 linux-2.6.38.2.tar.lzo9 (lzop -9)
117808111 linux-2.6.38.2.tar.gz1
148424286 linux-2.6.38.2.tar.lzo1 (lzop -1)
149227104 linux-2.6.38.2.tar.lzo (lzop): default

Then I put everything above on /dev/shm (1.5 GiB), turned off swap,
synced few times ;-) and after setting cpufreq governor to powersave
(800 MHz), I have run following script:

echo lzo; time cat linux-2.6.38.2.tar.lzo | lzop -dc > /dev/null
echo lzo1; time cat linux-2.6.38.2.tar.lzo1 | lzop -dc > /dev/null
echo lzo9; time cat linux-2.6.38.2.tar.lzo9 | lzop -dc > /dev/null
echo gz; time cat linux-2.6.38.2.tar.gz | gzip -dc > /dev/null
echo gz1; time cat linux-2.6.38.2.tar.gz1 | gzip -dc > /dev/null
echo gz9; time cat linux-2.6.38.2.tar.gz9 | gzip -dc > /dev/null
echo xz1; time cat linux-2.6.38.2.tar.xz1 | xz -dc > /dev/null
echo xz2; time cat linux-2.6.38.2.tar.xz2 | xz -dc > /dev/null
echo xz3; time cat linux-2.6.38.2.tar.xz3 | xz -dc > /dev/null
echo bz2; time cat linux-2.6.38.2.tar.bz2 | bzip2 -dc > /dev/null

which gave:
lzo9: 0m4.490s
lzo1: 0m5.197s
lzo: 0m5.249s
gz9: 0m10.607s
gz: 0m10.643s
gz1: 0m12.028s
xz3: 0m22.323s
xz2: 0m23.302s
xz1: 0m24.573s
bz2: 0m47.193s

That shows the power of LZO: two times faster than gzip, five times
faster than LZMA2 (a.k.a xz) and 10.5 times faster than BZIP2[-9] when
decompressing.

Remember that LZ77-based algorithms (all of the above except for bzip)
are general-purpose compression algorithms, while BWT-based algorithms
(such as BZIP2) are best suited for compressing textual data (such as
source code). Decompression times with LZ77-based algorithms are
roughly proportional to compressed data size only, while with BZIP2
higher compression methods request more decompression cycles,
effectively slowing down decompression, too.


Thank you for your attention
Goodnight! ;-)

Marcin Szychowski.

[Christ Schlacta]

I know I'm a little late to this thread, But I'm going to chime in with
input regarding reserving decompression bits..
I think we should only have 2 per compression type. a "gzip best" and a
"gzip fast", and so on for each type, and just pick good optimizations
for each one through thorough testing and experimentation.

[Marcin Szychowski]

On Apr 3, 2:36 am, Christ Schlacta <aarc...@gmail.com> wrote:
> I know I'm a little late to this thread, But I'm going to chime in with
> input regarding reserving decompression bits..
> I think we should only have 2 per compression type.  a "gzip best" and a
> "gzip fast", and so on for each type, and just pick good optimizations
> for each one through thorough testing and experimentation.
>

lzjb, zle and gzip-1 to gzip-9 are official methods provided by Sun.
All other and this entire thread is a free-form variation on zfs.

> > Marcin Szychowski.

[Marcin Szychowski]

On Apr 3, 1:12 am, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:
> Ok, I get these times :
> type            size        time    uncompress time
> uncompressed    182283      18.287  0.927
> lzjb (on)       88276k      14,768  1.130
> lzo             64793k      13,541  1.068
> lzo9            48213k      25,522  0.990
> gzip            42696k      15,352  1.110
> lzma            39383k      21.769  2.394
> bzip2           37628k      23,397  3.278
>
> Comments : the results are not very precise, times obtained this way
> (through time command) vary sometimes, so maybe it would be better to get a
> bigger file to test this...
> Anyway it seems to show indeed that lzo is very efficient on this side too.
> gzip is almost as good.
> lzma and bzip2 might be too much, except if you are ready to sacrifice some
> time then.
>

By 'LZMA' you mean original LZMA algorithm or it's improved version
LZMA2 (also referred to as XZ)? ;-)

[Emmanuel Anne]

Lots of things here.
Nice benchmarks, but out of context here. What we are testing here is not the latest compression utilities at their best, but what the compression libs can produce inside zfs-fuse, which is quite different.
A lot different actually.
As I said, my benchmark is an ideal situation, in real condition you have about no chance to reproduce it (who stores an uncompressed tar directly on disk nowdays ?). It's even rare to be able to compress well disk images, but the worst is for small files, in zfs you can have something like : compress this 1024 bytes buffer in a 512 bytes buffer, if you can't do it, then bye bye (and here most compression methods fail most of the time).
So if you want the "real world" situation, instead of just using bzcat to extract the tar, use tar xvfj, and then check the compression ratio you get from that (zfs get compressratio). Here lzo becomes much less interesting (from memory it gets a ratio around 2, when gzip is at 3.4 if I remember correctly, this makes a big difference here).
Actually it makes all the difference, because the basic idea for trying to add bz2 and lzma was that even if they are slow, they have to cope with zfs-fuse which is already slow, so it's possible to tolerate the overhead (and it would probably be much harder to tolerate with the speed of something like ext4 !).
Same thing for "lzo in n times better in decompression" : sorry here too, but all the decompression methods have to cope with what zfs sends them, and it's never a big archive to decompress, most of the time it's just small blocks (128k at most when in luck). So here too, it's better to compare from the absolute value you get without compression. You can't multiply here, too many things come in the way for that, you might be able to eliminate bzip2, buf if you also eliminate zfs-fuse then you are totally out of the subject !

Anyway you made your point : lzo is more interesting than what I first thought, but it's not the super graal neither.

And for lzma, since it's liblzma2 it's supposed to be the latest available version.

2011/4/3 Marcin Szychowski <szy...@gmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Emmanuel Anne]

The only place where I added 10 sub methods is for lzma. Testing would be long because lzma-3 (the default for lzma) produces slilghtly larger files for ascii than bz2, but at a better speed, but the results might be different for some binary data.
The default should be lzma-3 or 4, and then probably still keep lzma-1 and lzma-9.
It's easier to keep the 10 variants in this case, unless someone wants to try to test everything here.

2011/4/3 Christ Schlacta <aar...@gmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/