ZFS-FUSE so slow, help

[michel marcon (aka cmic)]

Hi

Context: Debian Lenny Linux 2.6.26-2-486 on CPU 2.8GHZ,
RAM 1GB, 3 HD, IDE 40 GB each (hda, hdb, hdd); zfs-fuse v0.6
compiled from GIT site sources.

I am testing zfs-fuse on a 100MB partition /dev/hdb1 and I am
disapointed
with the poor performance. Maybe I missed something ?

Copy a 59 MB directory file from /home/cmic to /tmp => about 1.5s
Copy a 59 MB directory file from /home/cmic to /tank/my => about 14s

I tried the find command with approx the same probleme.
Why is it so slow ?

TYA
--
michel marcon (aka cmic) Sysadmin
email: cmic |at| live.fr

--------------details here -------------------------
zorro:/home/cmic# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
tank 97M 58,7M 38,3M 60% ONLINE -

#ps ax | grep zfs
5114 ? Ssl 1:19 /usr/local/sbin/zfs-fuse -a 1000 -e 1000

zorro:/home/cmic$ time cp -rp test /tank/my

real 0m14.122s
user 0m0.028s
sys 0m0.248s
zorro:/home/cmic$ time cp -rp test /tmp

real 0m1.484s
user 0m0.016s
sys 0m0.148s

[devsk]

Try reversing the operations. In second copy, you are referring to
cache while in first, you are reading from source filesystem, which
may be slow to read a full directory. Depends on what's in the test
directory.

-devsk

[sgheeren]

Tuning, my friend.

Also write speed will be lower than on Solaris. It seems you missed the fuse-attr-timeout/fuse-entry-timeout optimizations (find being slow...)

Now for the tuning:

(a) use debug=0 build (0.6.9 packages do this)
(b) use proper options. (the default /etc/zfs/zfsrc should be fine; Here is my list:

fuse-attr-timeout = 1
fuse-entry-timeout = 1
fuse-mount-options = default_permissions,noatime
max-arc-size = 100
no-kstat-mount
stack-size = 2
vdev-cache-size = 10

 (c) use proper pool options. You say nothing about your pool. It seems  small (100Mb) so I doubt whether it uses more than one vdev (disk).

• If you use more vdevs be sure that all slices are on _diferent_ physical disks (important)
• Since memory is low (1GB) do not try to enable DEDUP. Dedup has a large performance trade-off to be made so is only interesting in specific scenarios where you expect huge dedup wins. Otherwise, just use compression[1]
• If dedup has _ever_ been enabled on a pool, the memory footprint will be incurred. So use 'zpool history' to find out if you did. Start over without dedup if you need to

Check for settings on performance-sensitive properties :

    zfs get -s local all | egrep 'dedup|compression|atime|copies|casesensitivity|normalization'

Recommended:
dedup=off
compression=off
atime=off
copies=1
inherit casesensitivity (default: sensitive) # use zfs inherit to 'unset'
normalization # should only be used when casesensitivity is modified

[1] as always, see the Dedup FAQ ( http://hub.opensolaris.org/bin/view/Community+Group+zfs/dedup) and favour snapshots/clones over dedup

[Stefan G. Weichinger]

Am 25.06.2010 08:47, schrieb sgheeren:

> (b) use proper options. (the default /etc/zfs/zfsrc should be fine; Here
> is my list:
>
> fuse-attr-timeout = 1
> fuse-entry-timeout = 1
> fuse-mount-options = default_permissions,noatime

[...]

hint here: note the non-existing whitespace between the two options.
Add a whitespace and you get errors thrown ... just "tested" here ;-)

Stefan

[sgheeren]

Huh? This is the default contrib zfsrc. It is well tested by many. Could
it be that you actually meant the leading whitespace (which *might* be a
problem, in which case you could report a bug for that).

> Stefan
>
>

[Aneurin Price]

I think Stefan is pointing out that it's important to use
'default_permissions,noatime' (or whatever fuse options are wanted)
exactly as specified, and that adding any space after the comma will
cause it to fail (somewhat surprisingly).

And BTW, does anyone know if FUSE supports the 'relatime' mount
option, and which version is needed if so?

[Stefan G. Weichinger]

Am 26.06.2010 12:17, schrieb Aneurin Price:

> I think Stefan is pointing out that it's important to use
> 'default_permissions,noatime' (or whatever fuse options are wanted)
> exactly as specified, and that adding any space after the comma will
> cause it to fail (somewhat surprisingly).

Use

fuse-mount-options = default_permissions, noatime

and you get some errors, misleading (not really pointing at the reason).

For me zfs-fuse does not start with this setting, I get the
usage-instructions displayed then ...

With

fuse-mount-options = default_permissions,noatime

(no space after the ",")

it starts fine.

BUT:

If I use

fuse-mount-options =default_permissions,noatime

(no leading space, none after ",")

it doesn't start up as well.

And with

fuse-mount-options =default_permissions, noatime

it starts.

Another small bug to file?

Stefan

[sgheeren]

Ok, this then might warrant a manpage for zfsrc :)

It is actually quite obvious. zfsrc is actually just a bunch of
commandline options in long-hand spelling but without the --

So specifying

fuse-mount-options = default_permissions, noatime

Actually results in a command line of roughly

zfs-fuse ... --fuse-mount-options=default_permissions, noatime

Everyone with shell experience will know that unless specificly quoted
that will be parsed as

zfs-fuse ... "--fuse-mount-options=default_permissions," "noatime"

instead of

zfs-fuse ... "--fuse-mount-options=default_permissions, noatime"

Now the thing to document is really, that zfsrc is not a shell script
and cannot employ shell quoting or any other shell construct.
I'll think I'll even throw in a bugfix quoting the entries from zfsrc
line by line, because it is not very probable that someone is relying on
something like the following to work (although he could, technically):

fuse-mount-options = default_permissions,noatime --no-daemon -s 8

Hmmm... how broken and confusing that would be :)

[sgheeren]

Now that I finally got some idea how the parsing was trying to work, I decided to revamp it, because, well, it needed it

It aims to be fully backwards compatible (modulo the bugs being reported). Together with prior patches[1], this makes the whole zfsrc/commandline experience a bit more user-friendly.

See http://zfs-fuse.net/issues/67 for details

+1 for reporting the small issues too!

[1]
* 47f7f84 Issue #67 Improve parsing of /etc/zfs/zfsrc
* 8e5e013 more: option disable-page-cache deprecated
|\ 
* | 5363a60 Update zfs-fuse manpage to reflect obsoletion of disable-page-cache
* | 425a6bf deprecated DAEMON_OPTS in init scripts
* | 16e046c option disable-page-cache deprecated
* | a8eb43b fixed zfsrc parse error when no line-end on last line

[Nikola M]

michel marcon (aka cmic) wrote:

> I am testing zfs-fuse on a 100MB partition /dev/hdb1 and I am
>

I was reading somewhere today that testing image should be at least
128MB or biger.

[sgheeren]

Well, the technical lower limit is that each vdev has to be 64MB
Easily verified by trying to create with smaller:

bash$ zpool create -O mountpoint=/tmp/toosmall1 toosmall1
/tmp/toosmall1_blk/za1 /tmp/toosmall1_blk/zb1
cannot create 'toosmall1': one or more devices is less than the minimum
size (64M)

[michel marcon (aka cmic)]

Hi

On 25 juin, 08:47, sgheeren <sghee...@hotmail.com> wrote:
> Tuning, my friend.
>
> Also write speed will be lower than on Solaris. It seems you missed the
> fuse-attr-timeout/fuse-entry-timeout optimizations (find being slow...)

Ha. Now i'm on zfs-fuse-0.6.9. (same platform, Debian, 3 little HD,
etc.). But
the raw performance is quite acceptable. I mean time to copy to /tmp
or copy
to /tank are the same. Next, I'm going to follow you hints/recipes
about opimizing.

I understand that 1- i have to have a big vdev (100 mB seems to short;
I'm
going to verify this) and 2- i have not enough RAM (1Gig) to try
dedup: a
first run show there is no difference with or whitout it.

Thanks you all.
--
michel marcon (aka cmic) SysAdmin
cmic |at| live.fr

>
> Now for the tuning:
>
> (a) use debug=0 build (0.6.9 packages do this)
> (b) use proper options. (the default /etc/zfs/zfsrc should be fine; Here
> is my list:
>
>         fuse-attr-timeout = 1
>         fuse-entry-timeout = 1
>         fuse-mount-options = default_permissions,noatime
>         max-arc-size = 100
>         no-kstat-mount
>         stack-size = 2
>         vdev-cache-size = 10
>
>  (c) use proper pool options. You say nothing about your pool. It seems
> small (100Mb) so I doubt whether it uses more than one vdev (disk).
>

>     * If you use more vdevs be sure that all slices are on _diferent_
>       physical disks (important)
>     * Since memory is low (1GB) do not try to enable DEDUP. Dedup has a

>       large performance trade-off to be made so is only interesting in
>       specific scenarios where you expect huge dedup wins. Otherwise,
>       just use compression[1]

>     * If dedup has _ever_ been enabled on a pool, the memory footprint

>       will be incurred. So use 'zpool history' to find out if you did.
>       Start over without dedup if you need to
>
> Check for settings on performance-sensitive properties :
>
>     zfs get -s local all | egrep
> 'dedup|compression|atime|copies|casesensitivity|normalization'
>
> Recommended:
> dedup=off
> compression=off
> atime=off
> copies=1
> inherit casesensitivity (default: sensitive) # use zfs inherit to 'unset'
> normalization # should only be used when casesensitivity is modified
>

> [1] as always, see the Dedup FAQ (http://hub.opensolaris.org/bin/view/Community+Group+zfs/dedup) and

[sgheeren]

On 06/27/2010 06:53 PM, michel marcon (aka cmic) wrote:
> Hi
>

> On 25 juin, 08:47, sgheeren <sghee...@hotmail.com> wrote:
>
>> Tuning, my friend.
>>
>> Also write speed will be lower than on Solaris. It seems you missed the
>> fuse-attr-timeout/fuse-entry-timeout optimizations (find being slow...)
>>
> Ha. Now i'm on zfs-fuse-0.6.9. (same platform, Debian, 3 little HD,
> etc.). But
> the raw performance is quite acceptable. I mean time to copy to /tmp
> or copy
> to /tank are the same. Next, I'm going to follow you hints/recipes
> about opimizing.
>
> I understand that 1- i have to have a big vdev (100 mB seems to short;
> I'm
> going to verify this) and 2- i have not enough RAM (1Gig) to try
> dedup: a
> first run show there is no difference with or whitout it.
>

Well the amount of RAM consumed by DEDUP tables is propertional to the
capacity of your pool. Note however, that once a pool init-ed it's DDT
(dedup tables) there is no way to revert that and the footprint will
always be there. This will also be true once you grow your pool (by
extending vdevs or by adding legs to your pool layout)

[Stefan G. Weichinger]

Am 27.06.2010 00:52, schrieb sgheeren:
> Now that I finally got some idea how the parsing was trying to work, I
> decided to revamp it, because, well, it needed it
>
> It aims to be fully backwards compatible (modulo the bugs being
> reported). Together with prior patches[1], this makes the whole
> zfsrc/commandline experience a bit more user-friendly.
>
> See http://zfs-fuse.net/issues/67 for details
>
> +1 for reporting the small issues too!

;-) thanks.

Looking forward to the changes ....

Stefan

[sgheeren]

As stated they're in testing. They won't be in maint as I don't think it is a minor bugfix (being a rewrite). So pick it up from

http://zfs-fuse.net/releases/0.7.0

or the related testing branch

http://gitweb.zfs-fuse.net/?p=official;a=shortlog;h=refs/heads/testing

or by doing 'git checkout origin/testing' or the equivalent on your working copy

[Stefan G. Weichinger]

Am 27.06.2010 23:22, schrieb sgheeren:

> As stated they're in testing. They won't be in maint as I don't think it
> is a minor bugfix (being a rewrite). So pick it up from
>
> http://zfs-fuse.net/releases/0.7.0
>
> or the related testing branch
>
> http://gitweb.zfs-fuse.net/?p=official;a=shortlog;h=refs/heads/testing
>
> or by doing 'git checkout origin/testing' or the equivalent on your
> working copy

OK, have my own ebuild for "testing" here ... just compiled and merged
it (gentoo-terms ...).

A quick test shows:

fuse-mount-options = default_permissions, noatime

does not work for me (the "2 spaces" form).

fuse-mount-options = default_permissions,noatime
WORKS

fuse-mount-options =default_permissions,noatime
DOES NOT WORK

fuse-mount-options =default_permissions, noatime
WORKS

--

I double checked if I really have the correct code ... seems OK to me.

?

S

[sgheeren]

On 06/28/2010 10:29 AM, Stefan G. Weichinger wrote

> OK, have my own ebuild for "testing" here ... just compiled and merged
> it (gentoo-terms ...).
>
> A quick test shows:
>
> fuse-mount-options = default_permissions, noatime
>
> does not work for me (the "2 spaces" form).
>

Could you tell me exactly what "does not work for me" means? See below:
> fuse-mount-options = default_permissions,noatime
> WORKS
>
Breakpoint 1, read_cfg () at zfs-fuse/main.c:397
397 parse_args(argc,argv);
3: argc = 3
2: argv[1] = 0x81ff290 "--fuse-mount-options"
1: argv[2] = 0xbffff361 "default_permissions,noatime"

> fuse-mount-options =default_permissions,noatime
> DOES NOT WORK
>

Breakpoint 1, read_cfg () at zfs-fuse/main.c:397
397 parse_args(argc,argv);
3: argc = 3
2: argv[1] = 0x81ff290 "--fuse-mount-options"
1: argv[2] = 0xbffff360 "default_permissions,noatime"
(gdb)

> fuse-mount-options =default_permissions, noatime
> WORKS
>
Breakpoint 1, read_cfg () at zfs-fuse/main.c:397
397 parse_args(argc,argv);
3: argc = 3
2: argv[1] = 0x81ff290 "--fuse-mount-options"
1: argv[2] = 0xbffff360 "default_permissions, noatime"
(gdb) c

> --
>
> I double checked if I really have the correct code ... seems OK to me.
>

git log -1
> ?
>
> S
>
>

[Seth Heeren]

git log -1

47f7f849680be6bcda9f03ae5c5b7bc023414be9

Also, a better place to supply the feedback would be http://zfs-fuse.net/issues/67

[Stefan G. Weichinger]

Am 28.06.2010 11:25, schrieb sgheeren:

>> I double checked if I really have the correct code ... seems OK to me.
>>
> git log -1

Something wrong with my checkout, sorry.
I get:

commit 1529820b05a00af40f874c6d22f0565f9fca2802
Author: Seth Heeren <sghe...@hotmail.com>
Date: Fri Jun 4 16:41:54 2010 +0200

issue #53 - O_CLOEXEC undefined on RHEL5.5

Will fix that after lunch ...
Stefan

[Stefan G. Weichinger]

Am 28.06.2010 11:59, schrieb Stefan G. Weichinger:

> Will fix that after lunch ...

Yep, got "testing" now.
Somehow I had the wrong code pulled in.

1) fuse-mount-options =default_permissions,noatime
restart works.
says: cannot mount '/tank': directory is not empty
zfs list OK, stuff mounted

2) fuse-mount-options = default_permissions,noatime
same as 1)

3) fuse-mount-options = default_permissions, noatime

* Mounting ZFS filesystems ...
cannot mount '/tank': directory is not empty
cannot mount 'tank/libvirt': Input/output error.
Make sure the FUSE module is loaded.
cannot mount 'tank/src': Input/output error.
Make sure the FUSE module is loaded.

zfs list OK, stuff mounted

4) fuse-mount-options =default_permissions, noatime
same as 3)

At least it restarts and mounts ... ;-)

If important:

Kernel 2.6.34
fuse 2.8.1

Stefan

[sgheeren]

On 06/28/2010 01:02 PM, Stefan G. Weichinger wrote:
> Kernel 2.6.34
> fuse 2.8.1
>
>
Only when you are reporting problems, friend :)

> directory is not empty
This should be pretty selfexplanatory

You might simply ensure nothing is mounted under /tank (stop zfs-fuse,
for starters) and do something smartish like

find . -type d -print0 | sort -rz | xargs -0 rmdir -v

(note how I try to be needlessly smart in order to remove nested subdirs
in reverse order _and_ allow for funky filenames).

[Stefan G. Weichinger]

Am 28.06.2010 13:17, schrieb sgheeren:
> On 06/28/2010 01:02 PM, Stefan G. Weichinger wrote:
>> Kernel 2.6.34
>> fuse 2.8.1
>>
>>
> Only when you are reporting problems, friend :)

ok then ;)

>> directory is not empty
> This should be pretty selfexplanatory

yep

> You might simply ensure nothing is mounted under /tank (stop zfs-fuse,
> for starters) and do something smartish like
>
> find . -type d -print0 | sort -rz | xargs -0 rmdir -v
>
> (note how I try to be needlessly smart in order to remove nested subdirs
> in reverse order _and_ allow for funky filenames).

sure. but I didn't create anything in /tank manually, so shouldn't
zfs-fuse take care of this itself?

No problem for me, maybe cosmetics ?

Stefan

[sgheeren]

Well, _normally_ yes. But once you do manual mounting (ever) you might run into this, exactly like you would with other fs-es. This happens on Solaris too.

Consider:

Layout:

pool
pool/fs1
pool/fs2

Let's take the following scenarios

(0) zfs-fuse stop; rm -rf /pool; zfs-fuse start;

# auto-mount all fs-es creating the proper mountpoints:
(1) zpool import pool

# unmount all fs-ses, removing the same mountpoints
(2) zfs umount -a

# mount only fs1, creating /pool and /pool/fs1
(3) zfs mount pool/fs1

# mount only fs2 in addition, creating /pool/fs2
(4) zfs mount pool/fs2

# umount either fs2, removing /pool/fs2
(5) zfs umount pool/fs2

# umount either fs1, removing /pool/fs1
(6) zfs umount pool/fs1

# observe how you are 'stuck' with the empty mountpoint /pool, while nothing is actually mounted. ZFS cannot clean that up because it doesn't know whether it is possible to do so on umounting pool/fs[12]

Now this is the scenario of empty mountpoints sticking around. Imagine what would have happened if the layout was acually less simple:

pool
pool/nest/fs1
pool/nest/fs2

Then umountint pool/test/fs[12] will leave you with /pool/test. This makes 'zfs mount pool' complain about the non-empty mountpoint /pool (since it still contains /pool/test). Hence your predicament.

Either this has happened, or your daemon was killed inapropriately (so the wasn't a way to remove the mountpoints) [1]

[1] or, of course, electrons run wild inside your systems. Ceterum censeo Carthaginem delendam esse.

[Stefan G. Weichinger]

Thanks for the explanation. Got it now ;-)
Stefan