xattr support broken, mangles permission bits on cp -a (http://zfs-fuse.net/issues/136)

[sgheeren]

A new issue has been submitted to the Issue tracker tracker by Andras Korn and awaits confirmation.


Issue Details:

Hi,

I'm using the prepackaged Ubuntu package, version 0.7.0-1ubuntu1~2.gbp43db46.

I enabled xattr support because I want to store posix ACLs on zfs (I realise they're not effective yet; I just wanted to be able to set them).

I noticed that when I copied or moved files, they would become mode 0600.

This appears to be caused by broken/half-working posix ACL support. Excerpt from strace cp -a file1 file2:

stat("file2", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
lstat("file1", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
stat("file2", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
open("file1", O_RDONLY|O_NOFOLLOW) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
open("file2", O_WRONLY|O_TRUNC) = 4
fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
read(3, "", 32768) = 0
utimensat(4, NULL, {{1315852659, 0}, {1315852659, 0}}, 0) = 0
flistxattr(3, (nil), 0) = 0
flistxattr(3, 0x7fffca5c2b80, 0) = 0
fgetxattr(3, "system.posix_acl_access", 0x7fffca5c2a60, 132) = -1 ENODATA (No data available)
fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
fsetxattr(4, "system.posix_acl_access", "\x02\x00\x00\x00\x01\x00\x06\x00\xff\xff\xff\xff\x04\x00\x04\x00\xff\xff\xff\xff \x00\x04\x00\xff\xff\xff\xff", 28, 0) = 0
close(4) = 0
close(3) = 0

The fsetxattr() call seems to set file2 mode 0600 irrespective of the actual mode bits inside.

Steps to reproduce:

enable xattr support
touch file1
chmod 644 file1
cp -a file1 file2
ls -la file1 file2
-rw-r--r-- 0 root root 2 Sep 12 20:41 file1
-rw------- 0 root root 2 Sep 12 20:41 file2

[Seth Heeren]

somehow clobbered the url: http://zfs-fuse.net/issues/136

[Emmanuel Anne]

It's cp's fault.

Since the setxattr call succeeds in the end, it doesn't call chmod.

If you do the strace again of the same command with xattrs disabled, you'll notice there is an chmod call just after the setxattr call fails.

That also explains why I never noticed this, the posix test suite uses more low level commands, so it doesn't have this problem.

2011/9/13 Seth Heeren <sghe...@hotmail.com>

somehow clobbered the url: http://zfs-fuse.net/issues/136

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

--
my zfs-fuse git repository : http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

[András]

On Sep 13, 3:11 pm, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:

> It's cp's fault.
> Since the setxattr call succeeds in the end, it doesn't call chmod.

Yes, but the setxattr call is supposed to set the file modes too. You
can see that the permission bits are passed in; it's just that the ACL
is not an extended ACL, so it only contains u::, g:: and o:: bits.

cp behaves the same way with other filesystems (e.g. xfs) and the
modes are set correctly there.

The bug seems to be that instead of applying the non-extended ACL as
simple file modes, zfs-fuse ignores the actual group and other bits
being passed in (takes them to be zero) and only applies the user
bits.

Andras

[Emmanuel Anne]

Ok, well maybe the right formulation is not "cp's fault", maybe it's just calling something like setacl, which is translated to setxattr by some high level layer (fuse ?), and you get this result.

But anyway to be sure I checked this with my posix_acl branch (which is still not finished because it's still just the basic support, no extended acl verfication at all, but it maps correctly between acls and standard chmod masks), and with this branch it works.

cp -a file1 file2 produces the right file mask for file2 with xattrs enabled.

If you wish I can push this branch to my repository, but since it's not finished you won't see that in any stable version before long.

It should be stable though.

2011/9/13 András <korn....@gmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[KORN Andras]

On Tue, Sep 13, 2011 at 05:58:53PM +0200, Emmanuel Anne wrote:

Hi,

> But anyway to be sure I checked this with my posix_acl branch (which is still
> not finished because it's still just the basic support, no extended acl
> verfication at all, but it maps correctly between acls and standard chmod
> masks), and with this branch it works.
> cp -a file1 file2 produces the right file mask for file2 with xattrs enabled.
>
> If you wish I can push this branch to my repository, but since it's not
> finished you won't see that in any stable version before long.
> It should be stable though.

Then yes please, I'd like to have it.

Thanks.

--
Andras Korn <korn at elan.rulez.org>
A bug is a feature that didn't make it into the manual.

[Emmanuel Anne]

Ok, pushed it.

I merged master in it on jully 31st, so it's pretty current.

And I deleted the old useless compress branch while I was at it.

2011/9/13 KORN Andras <korn....@gmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[KORN Andras]

On Tue, Sep 13, 2011 at 06:19:08PM +0200, Emmanuel Anne wrote:

> Ok, pushed it.
> I merged master in it on jully 31st, so it's pretty current.
> And I deleted the old useless compress branch while I was at it.

Either I'm looking at the wrong tree or you didn't remove all compression
related bits; what I downloaded from
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=snapshot;h=refs/heads/posix_acl;sf=tgz
still contains references to lzma, lzo and bz.

Am I looking at the wrong tree?

Andras

--
Andras Korn <korn at elan.rulez.org>

A picture is worth a thousand words, but it uses ten thousand times the memory.

[KORN Andras]

On Tue, Sep 13, 2011 at 10:00:19PM +0200, KORN Andras wrote:

> > Ok, pushed it.
> > I merged master in it on jully 31st, so it's pretty current.
> > And I deleted the old useless compress branch while I was at it.
>
> Either I'm looking at the wrong tree or you didn't remove all compression
> related bits; what I downloaded from
> http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=snapshot;h=refs/heads/posix_acl;sf=tgz
> still contains references to lzma, lzo and bz.
>
> Am I looking at the wrong tree?

It also doesn't compile due to incompatible definitions of acl_type_t in
/usr/include/sys/acl.h and src/lib/libsolkerncompat/include/sys/acl_impl.h;
I've looked at them but I can't fix this on my own.

--
Andras Korn <korn at elan.rulez.org>

A seven day honeymoon makes one weak.

[Emmanuel Anne]

For the compliaton errors, I had some with gcc-4.6, but they are gone with gcc-4.5.

For now I stay with gcc-4.5, following my last problems with 4.6.

For the compression, it was merged in master long ago, I kept the compress branch much longer than needed. So yeah it requires quite a few dev packages now, but nothing complex.

2011/9/13 KORN Andras <korn....@gmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Tue, Sep 13, 2011 at 10:51:54PM +0200, Emmanuel Anne wrote:

> For the compliaton errors, I had some with gcc-4.6, but they are gone with gcc-
> 4.5.

I must admit I'm somewhat amazed, but it did compile with gcc-4.5.

However, I quickly found a reliable way of causing it to segfault: ls -la in
a directory that contained a broken symlink. I'm not yet sure whether this
"works" for any directory or only the two particular ones I tried (I'm also
not sure the problem was the broken symlink; it's the only special feature
of the two directories that comes to mind, though, and the target of the
symlink didn't show up in the ls -l output).

zfs-fuse[11392]: segfault at 121 ip 00000000004f1091 sp 00002b6203ca9ce0 error 4 in zfs-fuse[400000+189000]

I'll play with it some more in a virtual environment, maybe try to obtain a
more useful backtrace than this one:

Core was generated by `/sbin/zfs-fuse --pidfile /var/run/zfs-fuse.pid'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000000004f1091 in vn_rele ()
(gdb) bt
#0 0x00000000004f1091 in vn_rele ()
#1 0x0000000000445285 in ?? ()
#2 0x00000000004421c6 in ?? ()
#3 0x00002b6203f5eb40 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00002b62056bf36d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

But that will probably take days. :(

--
Andras Korn <korn at elan.rulez.org>

Time is nature's way of keeping everything from happening at once.

[Emmanuel Anne]

Did I mention this one is really considered unstable this time ?

The reason is the only thing I ever ran on it was the posix conformance tests, that's all (contrary to the master branch which I use daily).

Anyway this one is very easy to fix, it's just the old bug about xattrs on symlinks which was fixed on master a short time ago. So I just remerged master in this branch and pushed the result, it fixes this problem (verified). You can pull the update on your side.

It's probably a good thing that you test this anyway, maybe it will be enough motivation to finish the work on these acls.

2011/9/14 Andras Korn <korn....@gmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Wed, Sep 14, 2011 at 08:59:35AM +0200, Emmanuel Anne wrote:

> Did I mention this one is really considered unstable this time ?

No :) In fact your words were "it should be stable" :)

> Anyway this one is very easy to fix, it's just the old bug about xattrs on
> symlinks which was fixed on master a short time ago. So I just remerged master
> in this branch and pushed the result, it fixes this problem (verified). You can
> pull the update on your side.

Thanks, will try it later (today, I hope).

> It's probably a good thing that you test this anyway, maybe it will be enough
> motivation to finish the work on these acls.

That would be awesome!

--
Andras Korn <korn at elan.rulez.org>

Hard work has a future payoff. Laziness pays off now.

[sgheeren]

On 09/14/2011 09:54 AM, Andras Korn wrote:
>> Did I mention this one is really considered unstable this time ?
> No :) In fact your words were "it should be stable" :)
>

The joy of quoting selectively. In all fairness it said (emphasis mine)

> but since **it's not finished** you **won't see that in any stable
version** before long.

And then the famous last words:

> It should be stable though.

In context I'd consider it 'unstable' but 'probably not lethal' (to
humans or data)...

[Andras Korn]

On Wed, Sep 14, 2011 at 01:24:49PM +0200, sgheeren wrote:

> On 09/14/2011 09:54 AM, Andras Korn wrote:
> >> Did I mention this one is really considered unstable this time ?
> > No :) In fact your words were "it should be stable" :)
> >
> The joy of quoting selectively. In all fairness it said (emphasis mine)

(That's why I put the smilies there)

--
Andras Korn <korn at elan.rulez.org>

According to statistics the main cause of divorce is marriage.

[Emmanuel Anne]

Yeah it's my eternal optimistic point of view, but its clear that it had to be considered unstable.

But since the posix tests are fairly complete, it should really be almost stable, but just not tested enough.

2011/9/14 Andras Korn <korn....@gmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Wed, Sep 14, 2011 at 02:55:34PM +0200, Emmanuel Anne wrote:

> Yeah it's my eternal optimistic point of view, but its clear that it had to be
> considered unstable.
> But since the posix tests are fairly complete, it should really be almost
> stable, but just not tested enough.

Just mounting and remounting a few dozen zfs filesystems, or mounting and
umounting some other fs under a subdirectory in zfs, often causes a segfault
for me; I haven't yet been able to obtain a backtrace but will do so if the
opportunity arises.

--
Andras Korn <korn at elan.rulez.org>

Do infants enjoy infancy as much as adults enjoy adultery?

[Andras Korn]

On Fri, Sep 16, 2011 at 01:00:08AM +0200, Andras Korn wrote:

> > Yeah it's my eternal optimistic point of view, but its clear that it had to be
> > considered unstable.
> > But since the posix tests are fairly complete, it should really be almost
> > stable, but just not tested enough.
>
> Just mounting and remounting a few dozen zfs filesystems, or mounting and
> umounting some other fs under a subdirectory in zfs, often causes a segfault
> for me; I haven't yet been able to obtain a backtrace but will do so if the
> opportunity arises.

Is there an easy way to make it dump core in /tmp instead of /? My rootfs
doesn't have enough space for the core file.

bind mounting a zfs and then umounting the bind mount seems to be a pretty
reliable way of triggering a segfault (in Emmanuel's posixacl tree).

--
Andras Korn <korn at elan.rulez.org>

Excuse me for butting in, but I'm interrupt-driven.

[Andras Korn]

On Fri, Sep 16, 2011 at 01:13:29AM +0200, Andras Korn wrote:

> bind mounting a zfs and then umounting the bind mount seems to be a pretty
> reliable way of triggering a segfault (in Emmanuel's posixacl tree).

During one of the crashes one of my filesystems suffered some accident that
now causes an apparent hang while mounting it (minutes pass and nothing
happens when I try to mount it; zpool iostat shows no activity). Any way to
recover from that or find out what's happening?

I've gone back to Seth's .deb for now, but that version is also unable to
mount this particular fs.

--
Andras Korn <korn at elan.rulez.org>

Warranty is void in case of nuclear war, whether caused by the program or not.

[Emmanuel Anne]

Wow that was a hard start. I expected you would start with something less dangerous, like starting on a test pool or something.

Anyway...

There was indeed a stupid possible crash in zfsfuse_getxattr, I trigered it very fast while trying this version as my normal zfs-fuse for my home. Well for me it stopped here, I fixed the bug, restarted and this time everything worked fine. Maybe it's because I didn't use bind, or maybe you were just unlucky, I don't know.

Anyway I pushed the fix, and since this time I really used it a bit it should be better.

For your recovery, check the load with top to see if the commands really do nothing. If so, try an export and then an import. Also be sure that your bound directory has been unmounted before trying to start the daemon (or reboot the machine).

Normaly it should be recoverable, I never heard of something like that which was not recoverable until now.

2011/9/16 Andras Korn <korn....@gmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Emmanuel Anne]

By the way : the core file appears in the directory where the program is started from. You can also run it directly in gdb, but you must use the -n command line option for zfs-fuse in this case to prevent it from beoming a daemon.

2011/9/16 Emmanuel Anne <emmanu...@gmail.com>

[Andras Korn]

On Fri, Sep 16, 2011 at 04:28:16AM +0200, Emmanuel Anne wrote:

> Wow that was a hard start. I expected you would start with something less
> dangerous, like starting on a test pool or something.

Did that, no problems there. :) So it goes...

> For your recovery, check the load with top to see if the commands really do
> nothing. If so, try an export and then an import. Also be sure that your bound
> directory has been unmounted before trying to start the daemon (or reboot the
> machine).

I did these things (except the export and the import). Top showed no
activity; zfs mount was waiting in recvfrom(); zfs-fuse was busily
futex()ing and reading a few characters from /dev/urandom a few times every
second, but that's all. No disk i/o.

I tried to ltrace it but then it died.

I ended up destroying the stuck fs and restoring it from backup. Luckily it
was small. :)

On Fri, Sep 16, 2011 at 04:30:53AM +0200, Emmanuel Anne wrote:
> By the way : the core file appears in the directory where the program is
> started from.

That's what I thought, so I added a cd /tmp before start-stop-daemon in the
initscript - but still the core went to /. Maybe start-stop-daemon chdir()s
to "/"? If so, I can write a wrapper that chdirs before execing zfs-fuse,
and call that using start-stop-daemon...

--
Andras Korn <korn at elan.rulez.org>

Don't vote, the government will get in anyway.

[Emmanuel Anne]

Well if you could recover from a backup I guess it's good news. A little weird though, I never had something like that, but when the daemon crashes, if you relaunch it without unmounting first what was mounted you can get this kind of behavior.

Anyway we'll never know what happened exactly then.

I hope you'll continue to test this anyway !

2011/9/16 Andras Korn <korn....@gmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[sgheeren]

On 09/16/2011 04:56 AM, Andras Korn wrote:
> Maybe start-stop-daemon chdir()s
> to "/"?

zfs-fuse chdir's to / as part of the daemoninzing step. Run it with -n
to avoid that (I think)

[Andras Korn]

start-stop-daemon certainly does it, but it has a -d switch which can be
used to select a different directory to chdir to.

I'll soon see whether this is sufficient.

--
Andras Korn <korn at elan.rulez.org>

Don't sweat the petty things and don't pet the sweaty things.

[sgheeren]

On 09/16/2011 10:04 PM, Andras Korn wrote:
>
> start-stop-daemon certainly does it, but it has a -d switch which can be
> used to select a different directory to chdir to.
>
> I'll soon see whether this is sufficient.
>

nah: grep the code:
http://gitweb.zfs-fuse.net/?p=emmanuel&a=search&h=refs%2Fheads%2Fposix_acl&st=grep&s=do_daemon
<http://gitweb.zfs-fuse.net/?p=emmanuel&a=search&h=refs%2Fheads%2Fposix_acl&st=grep&s=do_daemon>

first step in do_daemon:
http://gitweb.zfs-fuse.net/?p=emmanuel;a=blob;f=src/zfs-fuse/util.c#l114

As you can see, starting with -n (foreground mode) prevents it

[Andras Korn]

On Fri, Sep 16, 2011 at 10:49:26AM +0200, Emmanuel Anne wrote:

> Well if you could recover from a backup I guess it's good news. A little weird
> though, I never had something like that, but when the daemon crashes, if you
> relaunch it without unmounting first what was mounted you can get this kind of
> behavior.
> Anyway we'll never know what happened exactly then.
> I hope you'll continue to test this anyway !

Did that.

I managed to obtain this:

setup_acl_after_mode: no acl !
setup_acl_after_mode: no acl !
setup_acl_after_mode: no acl !
zfs-fuse: lib/libzpool/zfs_znode.c:1349: zfs_znode_free: Assertion `((zp)->z_vnode)->v_count == 0' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff4a8f700 (LWP 28295)]
0x00007ffff61f5405 in raise () from /lib/x86_64-linux-gnu/libc.so.6

(gdb) bt
#0 0x00007ffff61f5405 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff61f8680 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff61ee5b1 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00000000004f5667 in zfs_znode_free (zp=0x7fffd5a0f560) at lib/libzpool/zfs_znode.c:1349
#4 0x00000000004f5449 in zfs_znode_delete (zp=0x7fffd5a0f560, tx=0x7fffde620380) at lib/libzpool/zfs_znode.c:1286
#5 0x000000000042de2a in zfs_rmnode (zp=0x7fffd5a0f560) at zfs-fuse/zfs_dir.c:676
#6 0x00000000004f55cb in zfs_zinactive (zp=0x7fffd5a0f560) at lib/libzpool/zfs_znode.c:1332
#7 0x000000000044b0d3 in zfs_inactive (vp=0x7fffd591c330, cr=0x80bbf0, ct=0x0) at zfs-fuse/zfs_vnops.c:4327
#8 0x0000000000542e25 in fop_inactive (vp=0x7fffd591c330, cr=0x80bbf0, ct=0x0) at lib/libsolkerncompat/vnode.c:910
#9 0x0000000000542613 in vn_rele (vp=0x7fffd591c330) at lib/libsolkerncompat/vnode.c:652
#10 0x0000000000449fa6 in zfs_rename (sdvp=0x7fffd58b3340, snm=0x7fffebfbc040 "status.new", tdvp=0x7fffd58b3340,
tnm=0x7fffebfbc04b "status", cr=0x7ffff4a8ede0, ct=0x0, flags=0) at zfs-fuse/zfs_vnops.c:3667
#11 0x00000000005433ad in fop_rename (sdvp=0x7fffd58b3340, snm=0x7fffebfbc040 "status.new", tdvp=0x7fffd58b3340,
tnm=0x7fffebfbc04b "status", cr=0x7ffff4a8ede0, ct=0x0, flags=0) at lib/libsolkerncompat/vnode.c:1117
#12 0x0000000000453e9d in zfsfuse_rename (req=0x8588d0, parent=18086, name=0x7fffebfbc040 "status.new", newparent=18086,
newname=0x7fffebfbc04b "status") at zfs-fuse/zfs_operations.c:2104
#13 0x000000000044cb2a in zfsfuse_listener_loop (arg=0x0) at zfs-fuse/fuse_listener.c:297
#14 0x00007ffff79bfb40 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007ffff629836d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#16 0x0000000000000000 in ?? ()

This is with debug=2.

I'll try with debug=1 now (IIUC, the asserts won't be there then, so maybe
this particular problem wouldn't result in a crash).

--
Andras Korn <korn at elan.rulez.org>

Make analysis more anal.

[Emmanuel Anne]

By doing what ? This is a rename, so it was produced by something ?

2011/9/16 Andras Korn <korn....@gmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Emmanuel Anne]

I guess it's part of what a perfect daemon is supposed to do... 

There is a man page for this (man core), it says you can choose the name and the path of the core files in /proc/sys/kernel/core

2011/9/16 sgheeren <sghe...@hotmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Fri, Sep 16, 2011 at 11:39:36PM +0200, Emmanuel Anne wrote:

> By doing what ? This is a rename, so it was produced by something ?

Alas, I can't say exactly. What I did was stop a bunch of vservers (as in
linux-vserver.org), which resulted in a flurry of activity as logfiles were
closed, bind mounts unmounted and so on.

I don't know which specific action caused the assertion failure.

I'm now trying with debug=1 and only one vserver.

--
Andras Korn <korn at elan.rulez.org>

Nem megadni nekem a Nobel-dijat -- regi skandinav hagyomany.

[Andras Korn]

On Fri, Sep 16, 2011 at 11:50:16PM +0200, Andras Korn wrote:

> > By doing what ? This is a rename, so it was produced by something ?
>
> Alas, I can't say exactly. What I did was stop a bunch of vservers (as in
> linux-vserver.org), which resulted in a flurry of activity as logfiles were
> closed, bind mounts unmounted and so on.
>
> I don't know which specific action caused the assertion failure.
>
> I'm now trying with debug=1 and only one vserver.

I couldn't reproduce it with the first vserver I tried; I suppose I could
try them all individually and see if it's a specific one that causes the
problem.

Anyway.

With debug=1, stopping the bunch of vservers resulted in:

setup_acl_after_mode: no acl !
zfs-fuse: lib/libzpool/sa.c:1436: sa_lookup: Assertion `hdl' failed.

Program received signal SIGABRT, Aborted.

[Switching to Thread 0x7ffff4cd1700 (LWP 31732)]

0x00007ffff61f5405 in raise () from /lib/x86_64-linux-gnu/libc.so.6

(gdb) bt
#0 0x00007ffff61f5405 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff61f8680 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff61ee5b1 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6

#3 0x00000000004cd726 in sa_lookup (hdl=0x0, attr=9, buf=0x7ffff4cd0c90, buflen=<optimized out>) at lib/libzpool/sa.c:1436
#4 0x000000000042bb4f in zfs_rmnode (zp=0x7fffda5532f0) at zfs-fuse/zfs_dir.c:625
#5 0x0000000000440b13 in zfs_inactive (vp=<optimized out>, cr=<optimized out>, ct=<optimized out>) at zfs-fuse/zfs_vnops.c:4327
#6 0x0000000000448e92 in zfsfuse_getattr (req=0x7fffdc0008b0, ino=<optimized out>, fi=<optimized out>) at zfs-fuse/zfs_operations.c:354
#7 0x00007ffff7797105 in ?? () from /usr/lib/libfuse.so.2
#8 0x000000000044224a in zfsfuse_listener_loop (arg=<optimized out>) at zfs-fuse/fuse_listener.c:297
#9 0x00007ffff79bfb40 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007ffff629836d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()

--
Andras Korn <korn at elan.rulez.org>

If something goes without saying, let it.

[Emmanuel Anne]

This sounds a little crazy, you seem to use this thing as if it was tested code, and trying to make it do as much as possible. Well if you can't find me a way to reproduce this I won't be able to help you, and I don't intend to install vservers for that !

I have been using this version for my home since yesterday, no crash here, but so far I didn't do anything out of the oridnary anyway.

Maybe you'll just have to wait until the normal slow testing happens, but it will take much longer than that !

2011/9/17 Andras Korn <korn....@gmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[sgheeren]

On 09/16/2011 11:37 PM, Emmanuel Anne wrote:

I guess it's part of what a perfect daemon is supposed to do... 

There is a man page for this (man core), it says you can choose the name and the path of the core files in /proc/sys/kernel/core

Thx. I didn't know about that.

Relevant snippet

mkdir /tmp/corefiles
chmod 777 /tmp/corefiles
echo "/tmp/corefiles/core" > /proc/sys/kernel/core_pattern

[Andras Korn]

On Sat, Sep 17, 2011 at 12:14:54AM +0200, Emmanuel Anne wrote:

> This sounds a little crazy, you seem to use this thing as if it was tested
> code, and trying to make it do as much as possible. Well if you can't find me a

It's called experimenting. :) When I'm done, it _will_ be tested code. :)

> way to reproduce this I won't be able to help you, and I don't intend to
> install vservers for that !

I had hoped the backtrace would help; anyway, I'll play with it more and
perhaps find an easier way to reproduce the problem. I wasn't expecting you
to install vservers, just explaining what I did to trigger the crash.

--
Andras Korn <korn at elan.rulez.org>

A high IQ is like a Jeep; you will still get stuck, just farther from help.

[Andras Korn]

On Sat, Sep 17, 2011 at 12:03:28AM +0200, Andras Korn wrote:

> (gdb) bt
> #0 0x00007ffff61f5405 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007ffff61f8680 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #2 0x00007ffff61ee5b1 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
> #3 0x00000000004cd726 in sa_lookup (hdl=0x0, attr=9, buf=0x7ffff4cd0c90, buflen=<optimized out>) at lib/libzpool/sa.c:1436
> #4 0x000000000042bb4f in zfs_rmnode (zp=0x7fffda5532f0) at zfs-fuse/zfs_dir.c:625
> #5 0x0000000000440b13 in zfs_inactive (vp=<optimized out>, cr=<optimized out>, ct=<optimized out>) at zfs-fuse/zfs_vnops.c:4327
> #6 0x0000000000448e92 in zfsfuse_getattr (req=0x7fffdc0008b0, ino=<optimized out>, fi=<optimized out>) at zfs-fuse/zfs_operations.c:354
> #7 0x00007ffff7797105 in ?? () from /usr/lib/libfuse.so.2
> #8 0x000000000044224a in zfsfuse_listener_loop (arg=<optimized out>) at zfs-fuse/fuse_listener.c:297
> #9 0x00007ffff79bfb40 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
> #10 0x00007ffff629836d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #11 0x0000000000000000 in ?? ()

Got a similar one without stopping any vservers (just normal fs activity, so
I don't know what specifically triggered it):

(gdb) bt
#0 0x00002b0d51c73405 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00002b0d51c76680 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00002b0d51c6c5b1 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x000000000042bd75 in zfs_rmnode (zp=0x2b0d6b77d620) at zfs-fuse/zfs_dir.c:629
#4 0x0000000000440b13 in zfs_inactive (vp=<optimized out>, cr=<optimized out>, ct=<optimized out>) at zfs-fuse/zfs_vnops.c:4327
#5 0x0000000000448e92 in zfsfuse_getattr (req=0x1d2d160, ino=<optimized out>, fi=<optimized out>) at zfs-fuse/zfs_operations.c:354
#6 0x00002b0d507e0105 in ?? () from /usr/lib/libfuse.so.2
#7 0x000000000044224a in zfsfuse_listener_loop (arg=<optimized out>) at zfs-fuse/fuse_listener.c:297
#8 0x00002b0d505b5b40 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00002b0d51d1636d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Looking at zfs_dir.c around lines 625-629, can't this happen because some
entry supposedly has xattrs that aren't really there? (Just guessing.)

--
Andras Korn <korn at elan.rulez.org>

Does steel wool come from metal sheep?

[Andras Korn]

Aaaand got another (again, it occurred randomly; I wasn't even trying):

(gdb) bt
#0 0x00002af94b04e405 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00002af94b051680 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x000000000042ae76 in zfs_unlinked_add (zp=<optimized out>, tx=<optimized out>) at zfs-fuse/zfs_dir.c:456
#3 0x000000000042bc65 in zfs_rmnode (zp=0x2af9907797d0) at zfs-fuse/zfs_dir.c:669

#4 0x0000000000440b13 in zfs_inactive (vp=<optimized out>, cr=<optimized out>, ct=<optimized out>) at zfs-fuse/zfs_vnops.c:4327

#5 0x0000000000448e92 in zfsfuse_getattr (req=0x2af9500051c0, ino=<optimized out>, fi=<optimized out>) at zfs-fuse/zfs_operations.c:354
#6 0x00002af949bbb105 in ?? () from /usr/lib/libfuse.so.2

#7 0x000000000044224a in zfsfuse_listener_loop (arg=<optimized out>) at zfs-fuse/fuse_listener.c:297

#8 0x00002af949990b40 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00002af94b0f136d in clone () from /lib/x86_64-linux-gnu/libc.so.6

--
Andras Korn <korn at elan.rulez.org>

Does fuzzy logic tickle?

[Emmanuel Anne]

I can be wrong, but a few of these look like the bug related to parrallel rm reported in another thread (and which should be fixed already, but I was waiting for a better solution from Seth...).
Well I can always commit the fix I had for that, and see if it helps your cause.
If not, then you'll have to find a simpler way for me to reproduce this...

Ok it's done, the fix is in master, and I remerged it in posix_acl. Good luck !

2011/9/17 Andras Korn <korn....@gmail.com>

On Sat, Sep 17, 2011 at 01:15:11AM +0200, Andras Korn wrote:

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Sun, Sep 18, 2011 at 12:08:43AM +0200, Emmanuel Anne wrote:

> I can be wrong, but a few of these look like the bug related to parrallel rm
> reported in another thread (and which should be fixed already, but I was
> waiting for a better solution from Seth...).
> Well I can always commit the fix I had for that, and see if it helps your
> cause.
>
> If not, then you'll have to find a simpler way for me to reproduce this...
>
> Ok it's done, the fix is in master, and I remerged it in posix_acl. Good luck !

Great, thanks; I'll test it (with some luck, today; luck failing only next
weekend).

--
Andras Korn <korn at elan.rulez.org>

Mint orult, ki lehanyta lancat es most nem gyozi torolgetni...

[sgheeren]

On 09/18/2011 12:08 AM, Emmanuel Anne wrote:
> (and which should be fixed already, but I was waiting for a better
> solution from Seth...).

I don't know of a fix. I know of a workaround (to run zfs-fuse with -e
.00001 or greater)

see
http://groups.google.com/group/zfs-fuse/browse_thread/thread/47acc33319edfafa/d9362c9123b6c511?lnk=gst&q=Adding+a+meager#d9362c9123b6c511
<http://groups.google.com/group/zfs-fuse/browse_thread/thread/47acc33319edfafa/d9362c9123b6c511?lnk=gst&q=Adding+a+meager#d9362c9123b6c511>
for extended discussion

[Andras Korn]

On Sun, Sep 18, 2011 at 01:43:16PM +0200, sgheeren wrote:

> On 09/18/2011 12:08 AM, Emmanuel Anne wrote:
> > (and which should be fixed already, but I was waiting for a better
> > solution from Seth...).
> I don't know of a fix. I know of a workaround (to run zfs-fuse with -e
> .00001 or greater)

I'm running with

fuse-attr-timeout = 3600
fuse-entry-timeout = 3600

so that should be covered, iiuc. (I think these are the .deb defaults.)

--
Andras Korn <korn at elan.rulez.org>

You are only young once, but you can be immature forever.

[sgheeren]

On 09/18/2011 01:43 PM, sgheeren wrote:
> On 09/18/2011 12:08 AM, Emmanuel Anne wrote:
>> (and which should be fixed already, but I was waiting for a better
>> solution from Seth...).
> I don't know of a fix. I know of a workaround (to run zfs-fuse with -e
> .00001 or greater)

arrrrg duh: -a, not -e of course

[sgheeren]

On 09/18/2011 01:49 PM, Andras Korn wrote:
>
> I'm running with
>
> fuse-attr-timeout = 3600
> fuse-entry-timeout = 3600
>
> so that should be covered, iiuc

True

> . (I think these are the .deb defaults.)
>

True

[Emmanuel Anne]

Anyway this is the same kind of problem though.

So in this case, you still have a possibility : try to change the last parameter to zfs_zget to B_FALSE everywhere in zfs-fuse/zfs_operations.c

Still 3 occurences to change normally.

If with everything set to false you can still produce 1 of these weird crashes, I'll be lost !

(these are parts of an old fix which tried to fix 2 things :

 - issue 43 : write to a rolled back directory. This has been fixed totally differently since then, so it's not relevant anymore.

 - issue 45 : stat to unlinked file fails. it's the one you can work around with some cache.

If using some cache, it should be safe to put all these parameters to false, which is the way the original zfs code works anyway).

Again : good luck !

2011/9/18 sgheeren <sghe...@hotmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Sun, Sep 18, 2011 at 02:26:50PM +0200, Emmanuel Anne wrote:

> Anyway this is the same kind of problem though.
> So in this case, you still have a possibility : try to change the last
> parameter to zfs_zget to B_FALSE everywhere in zfs-fuse/zfs_operations.c
> Still 3 occurences to change normally.
> If with everything set to false you can still produce 1 of these weird crashes,
> I'll be lost !

Instead of weird crashes, I'm getting behaviour like this:

open("/tmp/user/1000/mutt-hellgate-1000-13073-4451243542118151558", O_RDONLY) = 3
stat("/tmp/user/1000/mutt-hellgate-1000-13073-4451243542118151558", {st_mode=S_IFREG|0600, st_size=168, ...}) = 0
unlink("/tmp/user/1000/mutt-hellgate-1000-13073-4451243542118151558") = 0
write(1, "\33[?25l", 6) = 6
fstat(3, 0x7fff79ad8940) = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f572633f000
read(3, 0x7f572633f000, 8192) = -1 ENOENT (No such file or directory)

So, unlinked files can no longer be read. Not good.

Which of the B_TRUEs should have stayed B_FALSE? :)

--
Andras Korn <korn at elan.rulez.org>

Ein Computerspiel ist wie ein Steak. Wenn mans blutig will nimmt mans englisch.

[Andras Korn]

On Sun, Sep 18, 2011 at 12:08:43AM +0200, Emmanuel Anne wrote:

> I can be wrong, but a few of these look like the bug related to parrallel rm
> reported in another thread (and which should be fixed already, but I was
> waiting for a better solution from Seth...).
> Well I can always commit the fix I had for that, and see if it helps your
> cause.
> If not, then you'll have to find a simpler way for me to reproduce this...
>
> Ok it's done, the fix is in master, and I remerged it in posix_acl. Good luck !

And yet another similar crash (this is with the supposed fix in posix_acl,
and without changing B_{TRUE|FALSE} in zfs_operations.c):

#0 0x00002b052787d405 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00002b0527880680 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00002b05278765b1 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00000000004cd716 in sa_lookup (hdl=0x0, attr=9, buf=0x2b052918bc90, buflen=<optimized out>) at lib/libzpool/sa.c:1436
#4 0x000000000042bb4f in zfs_rmnode (zp=0x2b055bbf0e20) at zfs-fuse/zfs_dir.c:625

#5 0x0000000000440b13 in zfs_inactive (vp=<optimized out>, cr=<optimized out>, ct=<optimized out>) at zfs-fuse/zfs_vnops.c:4327

#6 0x0000000000448e82 in zfsfuse_getattr (req=0x92aab0, ino=<optimized out>, fi=<optimized out>) at zfs-fuse/zfs_operations.c:356
#7 0x00002b05263ea105 in ?? () from /usr/lib/libfuse.so.2

#8 0x000000000044224a in zfsfuse_listener_loop (arg=<optimized out>) at zfs-fuse/fuse_listener.c:297

#9 0x00002b05261bfb40 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00002b052792036d in clone () from /lib/x86_64-linux-gnu/libc.so.6

#11 0x0000000000000000 in ?? ()

--

Andras Korn <korn at elan.rulez.org>

A vizsgaismetles meg a tudas K* Anyja!

[Andras Korn]

On Sun, Sep 18, 2011 at 09:09:44PM +0200, Andras Korn wrote:

> > Anyway this is the same kind of problem though.
> > So in this case, you still have a possibility : try to change the last
> > parameter to zfs_zget to B_FALSE everywhere in zfs-fuse/zfs_operations.c
> > Still 3 occurences to change normally.
> > If with everything set to false you can still produce 1 of these weird crashes,
> > I'll be lost !
>
> Instead of weird crashes, I'm getting behaviour like this:
>
> open("/tmp/user/1000/mutt-hellgate-1000-13073-4451243542118151558", O_RDONLY) = 3
> stat("/tmp/user/1000/mutt-hellgate-1000-13073-4451243542118151558", {st_mode=S_IFREG|0600, st_size=168, ...}) = 0
> unlink("/tmp/user/1000/mutt-hellgate-1000-13073-4451243542118151558") = 0
> write(1, "\33[?25l", 6) = 6
> fstat(3, 0x7fff79ad8940) = -1 ENOENT (No such file or directory)
> mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f572633f000
> read(3, 0x7f572633f000, 8192) = -1 ENOENT (No such file or directory)
>
> So, unlinked files can no longer be read. Not good.
>
> Which of the B_TRUEs should have stayed B_FALSE? :)

Sorry, vice versa of course: which B_FALSE should have stayed B_TRUE?

--
Andras Korn <korn at elan.rulez.org>

Amit ma megtehetsz, azt tegnap kellett volna.

[Emmanuel Anne]

Sorry a backtrace is not some magical formula bringing miracles, at this state without a proper way to reproduce this, or you really trying to debug it, it will come to an end.

There is no way to make any more wild guess from here.

That was a nice try !

(I am almost certain you'll have these crashes with the normal version as well, almost all the backtraces you sent were not related to xattr at all).

2011/9/18 Andras Korn <korn....@gmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Sun, Sep 18, 2011 at 09:43:03PM +0200, Emmanuel Anne wrote:

> Sorry a backtrace is not some magical formula bringing miracles, at this state
> without a proper way to reproduce this, or you really trying to debug it, it
> will come to an end.
>
> There is no way to make any more wild guess from here.
> That was a nice try !
> (I am almost certain you'll have these crashes with the normal version as well,
> almost all the backtraces you sent were not related to xattr at all).

The crashes seem to occur with a higher frequency (i.e. with near certainty)
when my dirvish jobs run; and so far there have been none with the 0.7 deb
Seth publishes (but that version has the cp -a bug when xattrs are enabled).

What can I do to "really try to debug it"?

--
Andras Korn <korn at elan.rulez.org>

Healthy people live longer - but they spend more time jogging.

[Emmanuel Anne]

I can't tell you step by step, it's not an easy thing. The general idea is read the code, and try to understand why it crashes here, and try things to avoid it. Not easy, but unless you can explain clearly how to reproduce that you'll be the only one able to do it for now.

2011/9/18 Andras Korn <korn....@gmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Sun, Sep 18, 2011 at 10:15:58PM +0200, Emmanuel Anne wrote:

> I can't tell you step by step, it's not an easy thing. The general idea is read
> the code, and try to understand why it crashes here, and try things to avoid
> it. Not easy, but unless you can explain clearly how to reproduce that you'll
> be the only one able to do it for now.

I'm afraid my C coding skills aren't up to that.

I'll try to create a test case, but don't hold your breath.

Meanwhile, the 0.7 deb version once more hangs on mounting one of the
filesystems after the posix_acl branch crashed.

zpool export/import doesn't help; once a mount of that fs is attempted,
zpool export results in 'pool is busy' (even though all filesystems in it
are unmounted), and it becomes impossible to stop zfs-fuse in any normal
way.

I'll have to destroy it and restore from backup again (it's another fs this
time, fwiw).

--
Andras Korn <korn at elan.rulez.org>

Yo dawg we heard you like Macs so we put only one pedal in your car.

[Emmanuel Anne]

Except that here too it's weird, in all the crashes I had, I could never experience this, and belive me I had more than a few crashes in all this time. More than weird in fact !

There is something really wrong somewhere, but well, what can I say ?

Maybe it's just better you forget about this idea then.

Eventually give me a ssh access (with sudo rights on the zfs-fuse daemon) to have a look by private mail, that's all I can say.

2011/9/18 Andras Korn <korn....@gmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Sun, Sep 18, 2011 at 09:43:03PM +0200, Emmanuel Anne wrote:

> Sorry a backtrace is not some magical formula bringing miracles, at this state
> without a proper way to reproduce this, or you really trying to debug it, it
> will come to an end.

Oh btw, you replied this not to a backtrace but to an strace log showing
that your suggestion of changing the last parameter of zfs_zget to B_FALSE
in zfs_operations.c resulted in an inability to read (or stat) unlinked
files.

This breaks everything so much that I can't even see whether I can reproduce
the crash with this modified version.

I think we've been overzealous with the boolean changes, but I can't decide
on my own which occurrence should've been left at B_TRUE.

--
Andras Korn <korn at elan.rulez.org>

When two men in business always agree, one of them is unnecessary.

[Emmanuel Anne]

Well I have used false everywhere here for a few months without problems, it really depends what you are doing with it. Anyway...

2011/9/18 Andras Korn <korn....@gmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Sun, Sep 18, 2011 at 10:36:33PM +0200, Emmanuel Anne wrote:

> Except that here too it's weird, in all the crashes I had, I could never
> experience this, and belive me I had more than a few crashes in all this time.
> More than weird in fact !
> There is something really wrong somewhere, but well, what can I say ?

I'm not expecting you to say anything - I'm just reporting what I see, in
the hope it helps. If it doesn't, it doesn't.

> Maybe it's just better you forget about this idea then.

I'm not giving up on it so soon. :)

The notion that the crashes I saw were related to parallel rm's seems
plausible to me - dirvish certainly does a lot of rm'ing (recursively, of
huge trees), and it's very likely something else did an rm meanwhile too.

I confess I haven't read the earlier discussion on this issue.

However, I think it's an interesting data point that Seth's deb version
doesn't crash (well, at least not nearly as certainly), not even when I run
dirvish.

> Eventually give me a ssh access (with sudo rights on the zfs-fuse daemon) to
> have a look by private mail, that's all I can say.

Sorry, I can't do that - however, if I'm able to reproduce this in some
virtualised environment, I'll give you the image (or something).

--
Andras Korn <korn at elan.rulez.org>

Some say the sun rises in the east, some say it rises in the west;
the truth probably lies somewhere in between.

[Andras Korn]

On Sun, Sep 18, 2011 at 10:43:55PM +0200, Emmanuel Anne wrote:

> Well I have used false everywhere here for a few months without problems, it
> really depends what you are doing with it. Anyway...

I was trying to run mutt, and what I saw was that all mail I opened seemed
empty. I then went on to strace mutt and saw that it copied the mail to a
tempfile, opened it, then unlinked it, fstat()ed it and tried to read from
it. The latter two calls failed with ENOENT.

--
Andras Korn <korn at elan.rulez.org>

Make love not war - see me for details.

[Emmanuel Anne]

Another weird point. I use mutt too from time to time, so I jumped on the opportunity to reproduce this simple problem here. Except that here it worked like a charm.

So :

1) I set all the last parameters of zfs_zget to B_FALSE

2) tried reading from mbox outside zfs -> no problem

3) tried reading from mbox in zfs dir -> no problem

4) tried reading from Maildir format in zfs dir -> no problem.

Well ! I am puzzled !

2011/9/18 Andras Korn <korn....@gmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Sun, Sep 18, 2011 at 11:15:21PM +0200, Emmanuel Anne wrote:

> Another weird point. I use mutt too from time to time, so I jumped on the
> opportunity to reproduce this simple problem here. Except that here it worked
> like a charm.
> So :
> 1) I set all the last parameters of zfs_zget to B_FALSE
> 2) tried reading from mbox outside zfs -> no problem
> 3) tried reading from mbox in zfs dir -> no problem
> 4) tried reading from Maildir format in zfs dir -> no problem.
>
> Well ! I am puzzled !

The point is that your TMP dir ('set tmpdir="/path/to/dir"' in muttrc) must
point to zfs to be able to reproduce it.

--
Andras Korn <korn at elan.rulez.org>

OK, so what's the speed of dark?

[Emmanuel Anne]

Ok !!!
Sorry for the delay, I needed time to cool down on this, and to find a better fix for the zfs_zget problem too.

So... For your side of the problem, there is a new posix_acl commit, get it, test it. If you can crash it, there's no need to try anything on it, you'll just have to find a way to tell us how to reproduce this that's all.

For those who are interested in the zfs_zget side of the problem : I never liked this fix anyway, that's why I lived without it for a very long time. The other way to make mutt work in this case (except having tmp outside the zfs dir which I had) is simply to disable the block cache.
Anyway what happens is this :
1) open fie for reading
2) then delete the file
3) then read from the handle from 1
in our code, at 2) the file was correctly placed in a kind of delete queue for later, so 3 should have worked without any fix at all.
Except that it didn't. I don't know why it can work like that on solaris, maybe they have some kind of cache for getattr so that it never comes here to get its info. In fuse anyway when 2 is done the cache is cleared for the entry, and the next thing we get is a getattr call, whcih calls zfs_zget. At this point, the zfs code just tests if the file has been deleted (z_unlinked) which is the case even if it's queued for later, and it returns ENOENT without trying to understand.
At his point it should just have checked if the file was queued for deletion by testing v_count. So this big intrusive fix we had before is just changed by a 1 line fix, which just tests v_count before returning ENOENT.
So far I couldn't crash it at all, even with entry_timeout = attr_timeout = 0.

It's committed in master and in posix_acl, I use it as my default zfs-fuse version now.
Maybe these crashes were totally unrelated to this problem after all, but at least this way we have some simpler code for this problem !

2011/9/18 Andras Korn <korn....@gmail.com>

On Sun, Sep 18, 2011 at 11:15:21PM +0200, Emmanuel Anne wrote:

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Tue, Sep 20, 2011 at 02:40:14PM +0200, Emmanuel Anne wrote:

> Sorry for the delay, I needed time to cool down on this, and to find a better
> fix for the zfs_zget problem too.
>
> So... For your side of the problem, there is a new posix_acl commit, get it,
> test it. If you can crash it, there's no need to try anything on it, you'll
> just have to find a way to tell us how to reproduce this that's all.

OK, thanks a lot; I'll test it (likely this Friday).

--
Andras Korn <korn at elan.rulez.org>

Don't steal. The government hates competition.

[Andras Korn]

On Tue, Sep 20, 2011 at 02:44:45PM +0200, Andras Korn wrote:

> > Sorry for the delay, I needed time to cool down on this, and to find a better
> > fix for the zfs_zget problem too.
> >
> > So... For your side of the problem, there is a new posix_acl commit, get it,
> > test it. If you can crash it, there's no need to try anything on it, you'll
> > just have to find a way to tell us how to reproduce this that's all.
>
> OK, thanks a lot; I'll test it (likely this Friday).

Well, well, well - it's looking a lot better! I wasn't able to crash it so
far in the usual ways. (For the record: this is
http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=snapshot;h=9960f2f8883a41621b2a2aae6489445f5648395e;sf=tgz.)

I'm now stressing it a bit just to make sure, but the previous version
would've crashed thrice over by now.

Good work!

--
Andras Korn <korn at elan.rulez.org>

If you're not rejected at least three times a week you're not really trying.

[Emmanuel Anne]

Good news thanks !

tapé sur le téléphone

[Andras Korn]

On Fri, Sep 23, 2011 at 10:05:30PM +0200, Emmanuel Anne wrote:

> Good news thanks !

Somthing is slightly broken though, because link() can take up to 6 seconds
(not always, but frequently):

link("/path/to/file1", "/path/to/file2") = 0 <5.945103>

meanwhile, zpool iostat -v 5 gives me suspiciously low numbers:

capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
backup 798G 98.0G 0 49 16.2K 711K
encrypted/backup_zfs 798G 98.0G 0 49 16.2K 711K
ssd/zfs_backup_zil 4K 250M 0 0 0 0
cache - - - - - -
ssd/zfs_backup_cache 666M 353M 0 2 5.60K 108K

(further output in same ballpark)

iostat -x 5 /dev/sd? looks like this:

avg-cpu: %user %nice %system %iowait %steal %idle
3.03 0.00 3.13 29.44 0.00 64.40

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 40.20 114.40 21.40 77.60 246.40 762.40 20.38 1.41 14.32 19.91 12.78 6.20 61.40
sdb 35.40 158.80 20.20 93.40 223.80 1002.40 21.59 2.03 17.90 17.52 17.99 8.01 91.00
sdc 32.40 152.40 24.40 77.40 244.90 912.00 22.73 1.16 11.30 12.62 10.88 4.91 50.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sde 29.80 141.00 24.20 80.20 236.80 877.60 21.35 1.18 11.30 13.22 10.72 4.98 52.00
sdf 43.60 105.20 22.20 73.60 256.50 706.40 20.10 1.48 14.93 17.66 14.10 5.49 52.60
sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdh 0.00 1.80 4.40 15.60 14.70 166.80 18.15 0.00 0.20 0.00 0.26 0.20 0.40
sdi 0.00 1.80 0.00 15.60 0.00 166.80 21.38 0.00 0.26 0.00 0.26 0.26 0.40

(ditto)

top(1) shows:

top - 23:18:22 up 2:39, 0 users, load average: 1.63, 1.64, 1.80
Tasks: 900 total, 1 running, 886 sleeping, 8 stopped, 5 zombie
Cpu0 : 3.6%us, 1.3%sy, 0.0%ni, 74.3%id, 20.7%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.7%us, 1.0%sy, 0.3%ni, 60.0%id, 37.9%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 1.3%us, 1.0%sy, 0.3%ni, 54.3%id, 41.4%wa, 0.0%hi, 1.6%si, 0.0%st
Cpu3 : 0.3%us, 1.0%sy, 0.0%ni, 42.7%id, 55.6%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 8150960k total, 4908140k used, 3242820k free, 87288k buffers
Swap: 10485752k total, 0k used, 10485752k free, 1288576k cached

PID USER PR NI VIRT RES SHR CODE DATA SWAP S P WCHAN %MEM %CPU TIME+ COMMAND
10578 root 20 0 1802m 1.3g 2128 1572 1.7g 500m S 1 futex_wai 16.4 1.9 19:57.23 /sbin/zfs-fuse --pidfile /var/run/zfs-fuse.pid
[...]

I think it used to be faster with Seth's .deb (but I don't have numbers to
back that up).

Explanation of layout:

encrypted/backup_zfs is an LVM LV on top of a LUKS encrypted software RAID6
array composed of sdb4[0] sde4[4] sdf4[3] sda4[2] sdc4[1]. ssd is a VG that
consists of a soft RAID1 array of sdg and sdh (two relatively decent SSDs).

The first 2G of swap would also go on the SSD.

The CPU is an AMD Phenom 9850 Quad-Core.

I haven't yet noticed other syscalls being slow, but I haven't investigated
this thoroughly.

--
Andras Korn <korn at elan.rulez.org>

Excuse me for butting in, but I'm interrupt-driven.

[Emmanuel Anne]

Well speed is or at least will be the main problem of this branch.
acls are read from xattr which are separate files in a virtual directory for each entry of the filesystem from zfs point of view, which means quite a lot more operations.
Now it's still surprising that you get such slow downs, there is nothing specific to the link function for acls/xattrs, but maybe it's the result of some other functions.

I didn't notice any significant loss of speed for my usage, but you seem to do some heavy i/o operations, so it's probably not comparable.

Well in the long term the idea is to speed up acl access by having acls directly accessible, but it's not a light change, and for now I didn't find enough courage to start on this. Even with this anyway it will be slower than a fs with no acls at all of course (but hopefully the difference won't be noticeable).

For info, the bug fixed last time was common to the posix_acl branch and to the master branch, maybe you trigger it more easily in the posix_acl branch because there are more i/o operations, but it's some common code to all the zfs-fuse versions (since version 0.5 or something !).

2011/9/23 Andras Korn <korn....@gmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Sat, Sep 24, 2011 at 02:29:19PM +0200, Emmanuel Anne wrote:

> Well speed is or at least will be the main problem of this branch.
> acls are read from xattr which are separate files in a virtual directory for
> each entry of the filesystem from zfs point of view, which means quite a lot
> more operations.
> Now it's still surprising that you get such slow downs, there is nothing
> specific to the link function for acls/xattrs, but maybe it's the result of
> some other functions.

I'm getting this behaviour on a filesystem with xattr=off (but xattrs are
otherwise enabled in zfsrc).

--
Andras Korn <korn at elan.rulez.org>

I think I'd kill for a Nobel Peace Prize.

[Andras Korn]

On Sat, Sep 24, 2011 at 03:25:25PM +0200, Andras Korn wrote:

> I'm getting this behaviour on a filesystem with xattr=off (but xattrs are
> otherwise enabled in zfsrc).

I'm just running bonnie++ to quantify the slowness, but it's taking hours.
The "writing intelligently..." part has taken about 4 hours so far to
produce a 2G file, with compression set to gzip-9 and dedup on. The CPU is
mostly idle and there is little enough i/o as well; however, as what bonnie
writes is mostly duplicate and very compressible, I'd have expected this
test to be CPU bound. I honestly don't know what it's doing that's neither
disk i/o nor CPU intensive.

I also suspect zfs-fuse is leaking memory. Top reports

PID USER PR NI VIRT RES SHR CODE DATA SWAP S P WCHAN %MEM %CPU TIME+ COMMAND

10578 root 20 0 2181m 1.5g 1828 1572 2.1g 695m S 1 futex_wai 18.7 4.5 191:43.96 /sbin/zfs-fuse --pidfile /var/run/zfs-fuse.pid

This is with:

vdev-cache-size = 10
max-arc-size = 1024
enable-xattr

fuse-attr-timeout = 3600
fuse-entry-timeout = 3600

fuse-mount-options = default_permissions
stack-size = 64

bonnie's writes take ages; output from strace -tttT:

1316879700.977377 write(3, "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"..., 8192) = 8192 <7.679347>
1316879708.657251 write(3, "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"..., 8192) = 8192 <0.049343>
1316879708.706666 write(3, "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"..., 8192) = 8192 <0.005410>
1316879708.712136 write(3, "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"..., 8192) = 8192 <5.971123>
1316879714.683344 write(3, "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"..., 8192) = 8192 <0.043716>
1316879714.727154 write(3, "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"..., 8192) = 8192 <5.579526>
1316879720.306773 write(3, "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"..., 8192) = 8192 <26.339473>
1316879746.646337 write(3, "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"..., 8192) = 8192 <6.686165>
1316879753.332573 write(3, "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"..., 8192) = 8192 <0.006655>
1316879753.339283 write(3, "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"..., 8192) = 8192 <2.417077>
1316879755.756786 write(3, "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"..., 8192) = 8192 <0.019757>
1316879755.776660 write(3, "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"..., 8192) = 8192 <2.951753>
1316879758.728502 write(3, "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"..., 8192) = 8192 <4.618213>

26 seconds for writing 8k just doesn't seem right.

Maybe there is some internal lock contention? FWIW, many futex() calls of
zfs-fuse are taking 4-7 seconds. I don't know if that's normal.

--
Andras Korn <korn at elan.rulez.org>

A verbal contract isn't worth the paper it's printed on.

[Emmanuel Anne]

This is crazy to try bonnie++ on a dedup filesystem, it's just irrelevant.

2011/9/24 Andras Korn <korn....@gmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Sat, Sep 24, 2011 at 08:06:05PM +0200, Emmanuel Anne wrote:

> This is crazy to try bonnie++ on a dedup filesystem, it's just irrelevant.

Please take a step back and assume for a moment that I'm not a moron. :)

Of course the results can't be meaningfully compared to other filesystems,
but that doesn't mean we can't have rational expectations on how such a
benchmark should go. Either i/o or CPU should be saturated; if neither is
(by a long shot), there is some delay somewhere and I'd like to know what
causes it.

It just shouldn't take 6+ hours to complete a bonnie run no matter what. A
single async 8k write operation shouldn't take 26 seconds; the average time
for an async 8k write operation shouldn't be over 1 second.

And none of this is related to the memory leak.

--
Andras Korn <korn at elan.rulez.org>

Always address your elders with respect; they could leave you a fortune.

[sgheeren]

On 09/24/2011 08:06 PM, Emmanuel Anne wrote:
> This is crazy to try bonnie++ on a dedup filesystem, it's just irrelevant.

Emmanuel, could you spare a second to argue your assertions? It doesn't
seem helpful to say 'it's your fault'[1] all too often.

Don't you think there is _anything_ strange about the figures? If it's
just (like me) that you don't have time/energy to chase it, I'm sure you
can just say it.

Oh, and I'm sorry about that, Andras, because it means I can't help you
getting zfs-fuse to do what you need a.t.m.

Seth

[sgheeren]

On 09/24/2011 02:29 PM, Emmanuel Anne wrote:
> Well speed is or at least will be the main problem of this branch.

> [...]

> Well in the long term the idea is to speed up acl access by having
> acls directly accessible, but it's not a light change, and for now I
> didn't find enough courage to start on this. Even with this anyway it
> will be slower than a fs with no acls at all of course (but hopefully
> the difference won't be noticeable).

---- OT:

I'm watching all this with minor interest and thinking that we better
hedge our efforts to keep a core functional zfs-fuse running ever more
stable than to include the bells and whistles (nfs, samba, zvol, acls
what not).

I think the zfsonlinux project is just better equipped - at least at the
moment.
They are even taking feature requests that are innovative compared to
what Sun was planning (COW cp is being discussed lately) and I might
humbly add that they _appear_ to understand the code a lot better than
the two of us do (I understand mostly the tooling, not ZFS
implementation itself...)

> For info, the bug fixed last time was common to the posix_acl branch
> and to the master branch, maybe you trigger it more easily in the
> posix_acl branch because there are more i/o operations, but it's some
> common code to all the zfs-fuse versions (since version 0.5 or
> something !).

That said, I too feel that the infamous zget patch[1] needs to be
'fixed' up. I never liked that patch (even though I made it myself, it
was actually a blind revert of some changes you did. I didn't know why
it was better/worse not to have them, I just observed that it fixed the
issue (#45) at that time (I mentioned that on this list _well before_
you have been trying various permutations of partial reverts).

This part is my interest: getting the _core_ functionality (even) more
stable.

----

Well, TL/DR: in short I believe that the last thing we need at this
moment in time is to embark on any destabilizing features; that would
merely send people to zfs-on-linux in a hurry (not that I'm trying to
keep users with zfs-fuse for any reason). As we have confirmed multiple
times on this list, there will always be a (warm :)) place for zfs-fuse
alongside zfsonlinux, but that won't be because of xattr, acl,
encryption, virusscanning, DTrace and proper zero-copy paths support :)

[1] it's lineage all the way back to this:

commit 0b2a939a6ca64f956279c8be3ad3bf9164e0437b
Author: Ricardo Correia <rcor...@wizy.org>
Date: Wed May 9 04:36:26 2007 +0100

Refactor zfs_zget() to work with unlinked files. Fix operations that
are supposed to work with unlinked files.

[Andras Korn]

On Sat, Sep 24, 2011 at 09:33:44PM +0200, sgheeren wrote:

> I'm watching all this with minor interest and thinking that we better
> hedge our efforts to keep a core functional zfs-fuse running ever more
> stable than to include the bells and whistles (nfs, samba, zvol, acls
> what not).

I'd be perfectly fine with that - I don't need working ACL support, I'd just
like to see the cp -a permission mangling issue fixed. :)

--
Andras Korn <korn at elan.rulez.org>

What we call work the Mexicans call siesta.

[Andras Korn]

On Sat, Sep 24, 2011 at 08:48:29PM +0200, Andras Korn wrote:

> > This is crazy to try bonnie++ on a dedup filesystem, it's just irrelevant.
>
> Please take a step back and assume for a moment that I'm not a moron. :)
>
> Of course the results can't be meaningfully compared to other filesystems,
> but that doesn't mean we can't have rational expectations on how such a
> benchmark should go. Either i/o or CPU should be saturated; if neither is
> (by a long shot), there is some delay somewhere and I'd like to know what
> causes it.
>
> It just shouldn't take 6+ hours to complete a bonnie run no matter what. A
> single async 8k write operation shouldn't take 26 seconds; the average time
> for an async 8k write operation shouldn't be over 1 second.
>
> And none of this is related to the memory leak.

The bonnie++ run didn't finish in 12 hours either; after some further time
the box rebooted (it reboots on panic as well as kernel-detected lockups and
I also have a hardware watchdog; alas, I don't know what it was this time).

I went back to the 0.7 deb, and for reference, bonnie's write()s look like
this with that version (to the same fs, same options):

1316941642.612677 write(3, "pppppppppppppppppppppppppppppppp"..., 8192) = 8192 <0.009550>
1316941642.622662 write(3, "pppppppppppppppppppppppppppppppp"..., 8192) = 8192 <0.009822>
1316941642.632608 write(3, "pppppppppppppppppppppppppppppppp"..., 8192) = 8192 <0.009402>
1316941642.642281 write(3, "pppppppppppppppppppppppppppppppp"..., 8192) = 8192 <0.008919>
1316941642.651446 write(3, "pppppppppppppppppppppppppppppppp"..., 8192) = 8192 <0.009776>
1316941642.661390 write(3, "pppppppppppppppppppppppppppppppp"..., 8192) = 8192 <0.009427>

The memory usage of zfs-fuse also hasn't increased substantially over 1G.

The values printed by zpool iostat -v 5 are also closer to what I'd expect:

capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----

backup 798G 97.5G 198 112 501K 87.0K
encrypted/backup_zfs 798G 97.5G 198 112 501K 87.0K
ssd/zfs_backup_zil 132K 250M 0 0 0 0

cache - - - - - -

ssd/zfs_backup_cache 1019M 80K 33 79 136K 9.56M

--
Andras Korn <korn at elan.rulez.org>

[Seth Heeren]

It panic-ed? That would not likely be related to zfs-fuse at all: or
you would actually have found a bug in fuse (the kernel module)

For the record, the retest with 0.7 was with dedup enabled as well?

[Andras Korn]

On Sun, Sep 25, 2011 at 05:13:45AM -0700, Seth Heeren wrote:

> It panic-ed?

I don't know. Could just as well have been the watchdog, or the soft/hard
lockup detection, or this:

CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=300
CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=1

Unfortunately the kernel log wasn't flushed to disk and nothing showed up in
netconsole either.

> For the record, the retest with 0.7 was with dedup enabled as well?

Yes. dedup=verify, compression=gzip-9, xattr=on, and the result is:

Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
jayhawk 16G 6 8 3279 1 2126 0 3524 89 104025 7 260.9 6
Latency 3117ms 84822ms 79369ms 79687us 37782ms 4846ms
Version 1.96 ------Sequential Create------ --------Random Create--------
jayhawk -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 1177 4 +++++ +++ 1933 5 1247 4 +++++ +++ 2446 6
Latency 40220us 2717us 2840us 75747us 148us 4213us
1.96,1.96,jayhawk,1,1316943363,16G,,6,8,3279,1,2126,0,3524,89,104025,7,260.9,6,16,,,,,1177,4,+++++,+++,1933,5,1247,4,+++++,+++,2446,6,3117ms,84822ms,79369ms,79687us,37782ms,4846ms,40220us,2717us,2840us,75747us,148us,4213us

Took a bit more than 3 hours and 49 minutes (so nothing like 12+ hours).

The write performance is still suspiciously low; disk bandwidth was
certainly not saturated at 2126k/s, and with ~0% CPU usage, neither was the
CPU. So I wonder where the bottleneck was.

I'll retest with xattr=off to see if it makes a difference (maybe it matters
for create/delete).

Oh, the memory leak (if there is one) appears to affect the 0.7 deb version
as well:

PID USER PR NI VIRT RES SHR CODE DATA SWAP S P WCHAN %MEM %CPU TIME+ COMMAND

21844 root 20 0 2335m 1.7g 1664 1116 2.3g 576m S 3 futex_wai 22.1 0.2 84:13.22 /sbin/zfs-fuse --pidfile /var/run/zfs-fuse.pid

zfsrc:

vdev-cache-size = 10
max-arc-size = 1024
enable-xattr
fuse-attr-timeout = 3600
fuse-entry-timeout = 3600
fuse-mount-options = default_permissions
stack-size = 64

--

Andras Korn <korn at elan.rulez.org>

A WHERE clause is a guy who turns into Santa every full moon.

[Emmanuel Anne]

Weather too warm here to look deeply into this but in short :
 - bonnie on dedup is irrelevant because dedup write speed is horrible and highly dependant on what is written (wether it finds the same block or not, plus also wether all the metadata fits in the arc cache or not, but I'd assume it does because otherwise it's just useless - by the way when using dedup it's a good idea to have primarycache=metadata and secondarycache=none, because you don't want anything in arc to overwrite this precious metadata from dedup).
So well to be short again : this branch is separated from my master branch because it will need quite a few changes in the core, making things unstable probably so it will remain like this for long. For now it's in an intermediate state, just before the changes to the core, so it's actually stable for now, and the performance impact should be very low in normal usage scenarios.
So for now dedup is limited to very special usages, if you want to use dedup on an experimental branch, you are on your own, I won't even try to reproduce this performance issue for now, I have many things which are higher on the priority list.

About zfsonlinux : If zfs on linux becomes stable enough I'll be very happy and switch to it and even stop trying to improve the zfs code at all, good for everyone.

2011/9/25 Seth Heeren <sghe...@hotmail.com>

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Andras Korn]

On Sun, Sep 25, 2011 at 02:59:46PM +0200, Andras Korn wrote:

> > For the record, the retest with 0.7 was with dedup enabled as well?
>
> Yes. dedup=verify, compression=gzip-9, xattr=on, and the result is:
>
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> jayhawk 16G 6 8 3279 1 2126 0 3524 89 104025 7 260.9 6
> Latency 3117ms 84822ms 79369ms 79687us 37782ms 4846ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> jayhawk -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 1177 4 +++++ +++ 1933 5 1247 4 +++++ +++ 2446 6
> Latency 40220us 2717us 2840us 75747us 148us 4213us
> 1.96,1.96,jayhawk,1,1316943363,16G,,6,8,3279,1,2126,0,3524,89,104025,7,260.9,6,16,,,,,1177,4,+++++,+++,1933,5,1247,4,+++++,+++,2446,6,3117ms,84822ms,79369ms,79687us,37782ms,4846ms,40220us,2717us,2840us,75747us,148us,4213us
>
> Took a bit more than 3 hours and 49 minutes (so nothing like 12+ hours).

With xattr=off (but enable-xattr in zfsrc), it took just under 3 hours and
15 minutes. Results:

Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

jayhawk 16G 10 8 5757 1 2228 0 3538 91 79567 6 58.6 1
Latency 818ms 21111ms 78516ms 38825us 17741ms 12414ms

Version 1.96 ------Sequential Create------ --------Random Create--------
jayhawk -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

16 596 2 +++++ +++ 1908 4 29 0 +++++ +++ 1203 3
Latency 9767ms 5114us 19623us 513s 1592us 6947us
1.96,1.96,jayhawk,1,1316972599,16G,,10,8,5757,1,2228,0,3538,91,79567,6,58.6,1,16,,,,,596,2,+++++,+++,1908,4,29,0,+++++,+++,1203,3,818ms,21111ms,78516ms,38825us,17741ms,12414ms,9767ms,5114us,19623us,513s,1592us,6947us

As there is no reason for blockwise input or sequential create to be any
slower in this case, I suppose the margin of error on these measurements is
very large.

Is there any interest in results with dedup off?

> Oh, the memory leak (if there is one) appears to affect the 0.7 deb version
> as well:
>
> PID USER PR NI VIRT RES SHR CODE DATA SWAP S P WCHAN %MEM %CPU TIME+ COMMAND
> 21844 root 20 0 2335m 1.7g 1664 1116 2.3g 576m S 3 futex_wai 22.1 0.2 84:13.22 /sbin/zfs-fuse --pidfile /var/run/zfs-fuse.pid

Increased even further:

PID USER PR NI VIRT RES SHR CODE DATA SWAP S P WCHAN %MEM %CPU TIME+ COMMAND

21844 root 20 0 3056m 2.0g 1464 1116 3.0g 1.0g S 3 futex_wai 25.8 4.0 226:12.40 /sbin/zfs-fuse --pidfile /var/run/zfs-fuse.pid

--
Andras Korn <korn at elan.rulez.org>

Set laser printers to "stun".

[Fajar Priyanto]

Sorry to bother you guys.

Is xattr the same as acl (as in setfacl)?

I need to setup samba share where some users can write and some can read only.

[Andras Korn]

On Sat, Oct 15, 2011 at 04:10:47PM +0800, Fajar Priyanto wrote:

> Sorry to bother you guys.
> Is xattr the same as acl (as in setfacl)?

Almost. POSIX ACLs as currently implemented in zfs-fuse are a subset of
xattr. You need xattr support if you want setfacl to work.

While in the 0.7 deb version you can set and retrieve ACLs, they are only stored
and don't take effect. This is useful for backups where storing them is all
you need, but not much else.

> I need to setup samba share where some users can write and some can read only.

Use smb.conf for that for now, or a different filesystem (or, if it's just
one group, even traditional unix access rights).

smb.conf pointers: write list, force group, force create mode, force
security mode, force directory security mask and so on.

Don't enable xattr support globally, or things will break (due to the cp -a
bug).

Andras

--
Andras Korn <korn at elan.rulez.org>

Uriember az, aki igeretehez hiven megmutatja a belyeggyujtemenyet.