ZFS-FUSE segfaulting

[exabyte]

ZFS-FUSE has been segfaulting for me for quite some time, at a rate of
once per week. Two hours ago it did it again, and this time it began
segfaulting at zfs-mount -a.

I have no idea if the "once per week" segfaults have anything to do
with the segfault I got now, but here's a backtrace. I fixed it by
downgrading to fuse 2.7.x.

#0 fuse_req_getgroups (req=0x0, size=65537, list=0x7fcf562ae010)
at fuse_lowlevel.c:1676
#1 0x000000000049aeb6 in groupmember (gid=1000, cr=0x73e8a0)
at lib/libsolkerncompat/policy.c:157
#2 0x0000000000408c6f in zfs_acl_ids_create (dzp=0x7fcf5a04edd0,
flag=0,
vap=0x7fcf7ff62890, cr=0x73e8a0, vsecp=<value optimized
out>,
acl_ids=0x7fcf7ff627a0) at zfs-fuse/zfs_acl.c:
1828
#3 0x0000000000419973 in zfs_create
(dvp=0x7fcfa33664c0,
name=0x7fcf56df7068 "sessionstore-4.js", vap=0x7fcf7ff62890,
excl=NONEXCL,
mode=0, vpp=0x7fcf7ff62990, cr=0x73e8a0, flag=0, ct=0x0,
vsecp=0x0)
at zfs-fuse/zfs_vnops.c:
1267
#4 0x00000000004a0be1 in fop_create
(dvp=0x0,
name=0x7fcfa2b419d8 "/proc/%lu/task/%lu/status",
vap=0x7fcf562ae010,
excl=2720531178, mode=-1, vpp=0x0, cr=0x73e8a0, flags=0, ct=0x0,
vsecp=0x0)
at lib/libsolkerncompat/vnode.c:1002
#5 0x0000000000411ec2 in zfs_replay_create (zfsvfs=0x7fcf5817b000,
lr=0x7fcf56df7000, byteswap=<value optimized out>)
at zfs-fuse/zfs_replay.c:479
#6 0x0000000000478435 in zil_replay_log_record (zilog=0x7fcf5abfd7c0,
lr=0x7fcf5dd273f0, zra=0x7fcf7ff62b90, claim_txg=<value optimized
out>)
at lib/libzpool/zil.c:1532
#7 0x00000000004797c4 in zil_parse (zilog=0x7fcf5abfd7c0,
parse_blk_func=0x477e30 <zil_incr_blks>,
parse_lr_func=0x4782b0 <zil_replay_log_record>,
arg=0x7fcf7ff62b90,
txg=1697814) at lib/libzpool/zil.c:271
#8 0x0000000000479eee in zil_replay (os=0x7fcfa335ef20,
arg=0x7fcf5817b000,
replay_func=0x721720) at lib/libzpool/zil.c:1596
#9 0x0000000000414552 in zfsvfs_setup (zfsvfs=0x7fcf5817b000,
mounting=<value optimized out>) at zfs-fuse/zfs_vfsops.c:1043
#10 0x0000000000414f75 in zfs_mount (vfsp=0x7fcf58f46840,
mvp=0x729fa0,
uap=<value optimized out>, cr=<value optimized out>)
at zfs-fuse/zfs_vfsops.c:1164
#11 0x000000000040506b in do_mount (spec=0x13cc230 "pool/home",
dir=0x13ce4b0 "/home", mflag=<value optimized out>, opt=0x13d80f0
"")
at zfs-fuse/util.c:155
#12 0x00000000004048f4 in cmd_mount_req (sock=11, cmd=<value optimized
out>)
at zfs-fuse/cmd_listener.c:80
#13 0x0000000000404baa in listener_loop (arg=<value optimized out>)
at zfs-fuse/cmd_listener.c:167
#14 0x00007fcfa2d60017 in start_thread () from /lib/libpthread.so.0
#15 0x00007fcfa228348d in clone () from /lib/libc.so.6
#16 0x0000000000000000 in ?? ()

[Emmanuel Anne]

You are not in a lucky day it would seem, but thanks for the backtrace, it enabled me to fix the problem.

So apparently when zfs replays some requests (probably after it was stopped abnormally, it replays them with a special credential, kcred, which is supposed to mean "all credentials", but which of course doesn't contain any fuse request. That's why it crashed here, cr->req == NULL. So the fix was easy, groupmember always returns 1 in this case.
It's commited, thanks for the report, it's quite a rare case, I never had this problem.

(indeed switching to fuse 2.7 changed groupmember to not use fuse request, it makes groupmember less precise in some cases, but at least you avoided this crash).

2009/10/11 exabyte <som...@gmail.com>

[exabyte]

Since it crashed two times yesterday after my post here, I ran it in
gdb the whole day, and I got another backtrace. However, I'm using
rudd-o's git repository right now. Should I be using yours? I'm not a
regular reader here, and I don't know what's the status of the
repositories are.

Here's the assertion error and the backtrace:

run-zfs-fuse: lib/libzpool/zfs_znode.c:576: zfs_znode_dmu_fini:
Assertion `zp->z_dbuf != ((void *)0)' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fc12dea6950 (LWP 12241)]
0x00007fc15392f205 in raise () from /lib/libc.so.6
(gdb) bt
#0 0x00007fc15392f205 in raise () from /lib/libc.so.6
#1 0x00007fc153930723 in abort () from /lib/libc.so.6
#2 0x00007fc153928229 in __assert_fail () from /lib/libc.so.6
#3 0x00000000004777bc in zfs_znode_dmu_fini (zp=0x7fc0eb157540)
at lib/libzpool/zfs_znode.c:576
#4 0x000000000040a130 in zfs_rmnode (zp=0x7fc0eb157540)
at zfs-fuse/zfs_dir.c:623
#4 0x000000000040a130 in zfs_rmnode (zp=0x7fc0eb157540)
at zfs-fuse/zfs_dir.c:623
#5 0x000000000041c13f in zfs_inactive (vp=0x7fc0dddd6e10,
cr=<value optimized out>, ct=<value optimized out>)
at zfs-fuse/zfs_vnops.c:3925
#6 0x0000000000420ac3 in zfsfuse_getattr_helper (req=0x89ac50,
ino=<value optimized out>, fi=<value optimized out>)
at zfs-fuse/zfs_operations.c:176
#7 0x000000000041cb4b in zfsfuse_listener_loop (arg=<value optimized
out>)
at zfs-fuse/fuse_listener.c:267
#8 0x00007fc154496017 in start_thread () from /lib/libpthread.so.0
#9 0x00007fc1539cd48d in clone () from /lib/libc.so.6
#10 0x0000000000000000 in ?? ()

[Emmanuel Anne]

2009/10/12 exabyte <som...@gmail.com>

Since it crashed two times yesterday after my post here, I ran it in
gdb the whole day, and I got another backtrace. However, I'm using
rudd-o's git repository right now. Should I be using yours? I'm not a
regular reader here, and I don't know what's the status of the
repositories are.

Well I was tired to run the diff command all the time to send patches, so I finally created my own repository, now he can take the patches from mine, so mine has the newest stuff, I try to make it as stable as possible though.

Anyway if you want to revert to fuse 2.8, you should use mine or you'll get your crash from yesterday back !
you can get it by git clone http://git.rudd-o.com/zfs/

Here's the assertion error and the backtrace:

run-zfs-fuse: lib/libzpool/zfs_znode.c:576: zfs_znode_dmu_fini:
Assertion `zp->z_dbuf != ((void *)0)' failed.

This one is VERY different from the other one.
It seems your filesystem is in a very special condition right now.
In the function just before the crash, it says  there is not enough space to delete the file, which seems paradoxal !!!

So could you give some more details here if you can, a df for example, if you enabled special parameters like compression, what setup for the pool (raidz or normal volume), this kind of stuff...
This time it's a sun assertion which fails, which means something which shouldn't have happened has happeed anyway.
Maybe there are issues if the filesystem is filled at 100%, I don't know I never filled mine to this point until now.
Also if the fs is full, how did you fill it : big files or plenty of small files ?

I'll wait for more details because I'd like to try to rerpoduce this problem if possible before trying to fix it...

Thanks to take the time to try to fix it anyway, it's probably going to be helpfull...

[Emmanuel Anne]

And more questions after taking a look :
did you use a snapshot on this filesystem ?
Compression ?
which architecture : 32 or 64 bits ?

Apparently this block was freed twice, the 2nd time trigered the crash, but how could it be freed twice ?
Some particular circunstances for the crash ?

2009/10/12 Emmanuel Anne <emmanu...@gmail.com>

[Rudd-O]

I also get crashes with freeing blocks sometimes. I have not been
able to track them.

HOWEVER, I need the fix that you posted on your repo, and possibly
others. Repo URL? I haven't been able to set your URL up to pull
from you. And I also need to nudge you into merging a performance fix
in the exact same groupmember function that I wrote just about a few
moments ago (it slashes lots of open() calls to files in /proc by just
setting up the max_ngroups variable ONCE per execution).

So, let's go on guys. And we NEED MORE BACKTRACES about the double
free program. Indeed, ZFS also segfaults for me every two to three
days, and I haven't been able to get a solid backtrace on the
problem. We NEED to fix this before the release.

On Oct 12, 6:32 pm, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:
> And more questions after taking a look :
> did you use a snapshot on this filesystem ?
> Compression ?
> which architecture : 32 or 64 bits ?
>
> Apparently this block was freed twice, the 2nd time trigered the crash, but
> how could it be freed twice ?
> Some particular circunstances for the crash ?
>

> 2009/10/12 Emmanuel Anne <emmanuel.a...@gmail.com>

>
>
>
> > 2009/10/12 exabyte <som...@gmail.com>
>
> >> Since it crashed two times yesterday after my post here, I ran it in
> >> gdb the whole day, and I got another backtrace. However, I'm using
> >> rudd-o's git repository right now. Should I be using yours? I'm not a
> >> regular reader here, and I don't know what's the status of the
> >> repositories are.
>
> > Well I was tired to run the diff command all the time to send patches, so I
> > finally created my own repository, now he can take the patches from mine, so
> > mine has the newest stuff, I try to make it as stable as possible though.
>
> > Anyway if you want to revert to fuse 2.8, you should use mine or you'll get
> > your crash from yesterday back !

> > you can get it by git clonehttp://git.rudd-o.com/zfs/

[Rudd-O]

I also get this exact backtrace from zfsfuse_getattr_helper. It is
always during times of heavy small file activity. I have a shitload
of disk space free, compression is on, atime is off. I will post a
backtrace as soon as I get one (I have enabled coredumps).

On Oct 12, 3:16 pm, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:
> 2009/10/12 exabyte <som...@gmail.com>
>
>
>
> > Since it crashed two times yesterday after my post here, I ran it in
> > gdb the whole day, and I got another backtrace. However, I'm using
> > rudd-o's git repository right now. Should I be using yours? I'm not a
> > regular reader here, and I don't know what's the status of the
> > repositories are.
>
> Well I was tired to run the diff command all the time to send patches, so I
> finally created my own repository, now he can take the patches from mine, so
> mine has the newest stuff, I try to make it as stable as possible though.
>
> Anyway if you want to revert to fuse 2.8, you should use mine or you'll get
> your crash from yesterday back !

> you can get it by git clonehttp://git.rudd-o.com/zfs/

[exabyte]

It's a RAID-Z pool with three partitions in it. Yes, I have snapshots
(on all filesystems that were in use, 8 for each). It's a 64 bit
system. I can't tell anything useful about the circumstances at the
time of the crash, and I still can't find a way to reproduce it
reliably myself -- I'm running many of my apps in strace now to see
what they're doing at the time of the crash, if it happens to crash
again. So far nothing.

One particular problem I wrote about in a previous post here is that I
have a hardware problem -- the speed to one of the drives in the array
for no apparant reasons becomes really slow for not-so-short periods,
usually when under load. This might be a contributing factor if the
problem is a race condition.

I think this was was present in most if not all the cases where the
crash happened -- apps were hanging, waiting for some I/O to complete
(Kopete running fdatasync on its statistics, MySQL reading or writing
something to a database, some app trying to stat a bunch files and one
stat taking half a second, etc.) But nothing peculiar I can attribute
the crash to (or at least not one that I can reproduce now.)

And this is the info about the free space:

# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
pool 2,61T 1,04T 1,57T 40% ONLINE


I prefer not to include a detailed info about the filesystems, as I
don't see how it would be useful. But it's not secret, so if it's
really needed, I would. The least empty filesystem is 82% full, and
it's an old backup -- not in use by anything, so it's not a free space
problem.

On 13 Окт, 02:32, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:
> And more questions after taking a look :
> did you use a snapshot on this filesystem ?
> Compression ?
> which architecture : 32 or 64 bits ?
>
> Apparently this block was freed twice, the 2nd time trigered the crash, but
> how could it be freed twice ?
> Some particular circunstances for the crash ?
>

> 2009/10/12 Emmanuel Anne <emmanuel.a...@gmail.com>

>
>
>
> > 2009/10/12 exabyte <som...@gmail.com>
>
> >> Since it crashed two times yesterday after my post here, I ran it in
> >> gdb the whole day, and I got another backtrace. However, I'm using
> >> rudd-o's git repository right now. Should I be using yours? I'm not a
> >> regular reader here, and I don't know what's the status of the
> >> repositories are.
>
> > Well I was tired to run the diff command all the time to send patches, so I
> > finally created my own repository, now he can take the patches from mine, so
> > mine has the newest stuff, I try to make it as stable as possible though.
>
> > Anyway if you want to revert to fuse 2.8, you should use mine or you'll get
> > your crash from yesterday back !

> > you can get it by git clonehttp://git.rudd-o.com/zfs/

[Emmanuel Anne]

2009/10/13 Rudd-O <rud...@rudd-o.com>

I also get crashes with freeing blocks sometimes.  I have not been
able to track them.

HOWEVER, I need the fix that you posted on your repo, and possibly
others.  Repo URL?

I post it all the time, I should probably put it in a signature or something... once again :
git clone  http://rainemu.swishparty.co.uk/git/zfs

 I haven't been able to set your URL up to pull
from you.

I just double checked it by running a git clone in another directory and it works.
 

 And I also need to nudge you into merging a performance fix
in the exact same groupmember function that I wrote just about a few
moments ago (it slashes lots of open() calls to files in /proc by just
setting up the max_ngroups variable ONCE per execution).

Ok, played with the mergetool, had some trouble with it, that's 2 commits for this simple thing but it was the 1st time I used it, anyway it's merged now.

So, let's go on guys.  And we NEED MORE BACKTRACES about the double
free program.  Indeed, ZFS also segfaults for me every two to three
days, and I haven't been able to get a solid backtrace on the
problem.  We NEED to fix this before the release.

From what I have seen so far, the problem is probably because
zfs_dirlook sometimes doesn't return a locked object as expected.
More precisely I suspect the central part here :
                 rw_enter(&dzp->z_parent_lock, RW_READER);
                error = zfs_zget(zfsvfs, dzp->z_phys->zp_parent, &zp, B_FALSE);
                if (error == 0) {
                        *vpp = ZTOV(zp);

I would add a
                        VN_HOLD(*vpp);
after this but I don't know when this part of the code is called, so I can't reproduce the problem for now...
Hard to test and to be sure here, and it's better not to change it in this case.

Also maybe dual cores reproduce this crash more easily (my main computer is still single core).
If someone has a way to reproduce this, at least adding some log here to check the lock state of vpp would be a good idea.

[Emmanuel Anne]

2009/10/13 exabyte <som...@gmail.com>

It's a RAID-Z pool with three partitions in it. Yes, I have snapshots
(on all filesystems that were in use, 8 for each). It's a 64 bit
system. I can't tell anything useful about the circumstances at the
time of the crash, and I still can't find a way to reproduce it
reliably myself -- I'm running many of my apps in strace now to see
what they're doing at the time of the crash, if it happens to crash
again. So far nothing.

There is a 99% probability that it's a locking problem, so you have to be "lucky" to reproduce it (2 events must occur at the same time).
 

One particular problem I wrote about in a previous post here is that I
have a hardware problem -- the speed to one of the drives in the array
for no apparant reasons becomes really slow for not-so-short periods,
usually when under load. This might be a contributing factor if the
problem is a race condition.

I have noticed while experimenting with a filled up volume that accesses are slowing down when there is very little space left. The big surprise here is that you still have plenty of spece left... Don't know for sure.
 

I think this was was present in most if not all the cases where the
crash happened -- apps were hanging, waiting for some I/O to complete
(Kopete running fdatasync on its statistics, MySQL reading or writing
something to a database, some app trying to stat a bunch files and one
stat taking half a second, etc.) But nothing peculiar I can attribute
the crash to (or at least not one that I can reproduce now.)

Yeah if things are slowing down then there are more chances that the 2 events occur together and crash zfs-fuse.
If you think you have a good chance to reproduce it, I would like to 1st just add some log info to it and if the log problem is indeed in this place then really patch it.
Or you can add the log + the patch at the same time, but it's considered experimental in this case.

This time it's in the middle of the code from sun, and found a forum post about a crash which looked a lot like this one in opensolaris forum, but it had no solution so far...

I attach the patch to this mail, it's not commited to my repo. Try to apply it to your source, and leave at least the syslog line.
It's about the VN_HOLD just after the syslog line that I am not sure at all...

[Rudd-O]

The fix that I wrote a few hours ago and you merged in, it seems to
reveal a deadlock locking problem. During periods of heavy disk
activity (scanning my entire media partition for media files), zfs
with the latest fix I wrote just HANGS. An inspection with gdb shows
that every single thread is waiting on a mutex idly. To get the
corefile I just did killall -ABRT zfs-fuse. Now, I don't understand
why changing a single line to make the groupmember function faster
would result in this deadlock problem, but I strongly suspect it's
because the function actually takes much, much less time to finish
than before, thus exposing a locking problem that has been in there
somewhere but not being exposed before.

Should I try with fuse 2.8.1?

BTW commit the fix you sent to this list into your repo. I will merge
carefully back into my repo, test it out. Also, my machine is
multicore, but I never see ZFS use the two cores.

>  patch_zfs_experimental
> 1KViewDownload

[Rudd-O]

BTW your patch VN_HOLDS the vpp, but doesn't seem to release the vpp.
Where are you releasing the vpp?

On Oct 13, 5:56 am, Emmanuel Anne <emmanuel.a...@gmail.com> wrote:

>  patch_zfs_experimental
> 1KViewDownload

[Mike Hommey]

On Tue, Oct 13, 2009 at 04:53:23AM -0700, Rudd-O wrote:
>
> The fix that I wrote a few hours ago and you merged in, it seems to
> reveal a deadlock locking problem. During periods of heavy disk
> activity (scanning my entire media partition for media files), zfs
> with the latest fix I wrote just HANGS. An inspection with gdb shows
> that every single thread is waiting on a mutex idly. To get the
> corefile I just did killall -ABRT zfs-fuse.

You may use gcore $(pidof zfs-fuse) instead, as depending on your
ulimits, killing the process may not dump a core.

Mike

[Rudd-O]

my initscript is specially tuned to get me a corefile in all cases :-)

[exabyte]

I think I managed to crash it. Once.

During my normal desktop load (which is when one of my disks stops
responding), I ran two find instances to scan my filesystem to load it
even more. Then I ran:
while :; do touch aaa; (rm aaa & rm aaa & rm aaa & rm aaa & rm aaa &
rm aaa); done

On a ZFS filesystem and had an immediate crash. However, I then tried
to synthetically load my system with only dd, find and friends,
nothing. I can't reproduce a crash that way. And I tried a lot of
times. (I don't think I managed to load my filesystem at all that
way.)

So I think there is something in my desktop file system access that
loads my file systems really bad, and then in turn leads to more
occurrences of the problem. Unfortunately, I can't really test anymore
now.

Another idea for testing: sending SIGSTOP and SIGCONT to zfs-fuse
while running something similar to what I ran above.

However, I noticed that my command doesn't hit the disk at all, so I'm
a bit unsure how slow disk access would affect it at all.

2. I couldn't actually load it

>  patch_zfs_experimental
> 1KПрегледИзтегляне

[sghe...@hotmail.com]

>
> This one is VERY different from the other one.
> > > It seems your filesystem is in a very special condition right now.
> > > In the function just before the crash, it says there is not enough space
> > > to delete the file, which seems paradoxal !!!

Deleting costs space in any logstructured fs. The very nature of btrfs,
NILFS, ZFS and the likes is such that deletion is merely 'appending a
new version of the metadata'. This is also the way in which snapshots
can be 'kept'. You simply mark the old tree of node versions as 'not to
be garbage collected' and keep a pointer to it's root as 'the snapshot'.

These filesystems invariably handle the cleanup using some kind of
garbage collector

[sghe...@hotmail.com]

Rudd-O wrote:
> Now, I don't understand
> why changing a single line to make the groupmember function faster
> would result in this deadlock problem, but I strongly suspect it's
> because the function actually takes much, much less time to finish
> than before, thus exposing a locking problem that has been in there
> somewhere but not being exposed before.
>
>

Or, as most deadlocks are caused by inconsistent ordering of lock
acquisition, it may be that due the new ordering of fs calls (changed,
at least), some locks are (implicitely) acquired/released in a subtly
different order than before. This may result in deadlocks when other
threads are acquiring the same locks in different orders at the same moment.

I know, farfetched, and yes, I didn't look at the refactored function
when phantasizing this one

[Emmanuel Anne]

Yeah I have tried all that too here to ty to reproduce the crash, and so far no luck at all !
Too bad the idea of Ctrl-Z on zfs-fuse to have all the threads waiting seemed good anyway, but it didn't work neither.
I even tried to add a sleep in the code to be sure I'll get more threads waiting, but it didn't help.

Notice : if it's just a locking problem, then decreasing the number of concurrent threads might help (even if it's not a fix).

I still need to find a way to reproduce this at least once...
As somebody said on a freebsd list : zfs lacks something : simplicity !!! Too many threads running at the same time create too many locking problems...

For now all I could get is the funny message about no place left on device while trying to rm something.
In this case all you can do is export / import again the pool and sometimes you win a little space.
More efficiently : destroy some snapshots, it creates free space immediately.

Anyway still no idea to reproduce this crash for now...

While doing my tests I corrupted a disk image on a file, which prevented zfs-fuse from even starting, but since it's a disk image and not a normal disk zfs-fuse had no control on its cache, so it's unlikely this kind of corruption can happen on a real disk (but there is an easy work around for this particular crash, I might commit anyway, it's just returning an error instead of calling abort on a failed assertion for zap_magic test).

Anyway if somebody has another idea for this crash... I'll come back to it later...

2009/10/13 exabyte <som...@gmail.com>

[sghe...@hotmail.com]

[mild rant alert]

Emmanuel Anne wrote:

As somebody said on a freebsd list : zfs lacks something : simplicity !!! Too many threads running at the same time create too many locking problems...

I beg to differ. The catch phrase "ZFS: the last word in file systems" was never intended as a metaphor of a pile of crap (every single [bad] feature ever invented on a stinking heap). If anything, the ZFS design goals have clearly been to distill the top 10 (or so) *major* ideas and implementing them _in the simplest way imaginable_

In the same way you don't criticize the minimalist design of a micro-kernel for being decentralized (ha-ha) and creating too many agents [1] to do the job that _one kernel_ could do...

In the same way you don't criticize a program written in in pure functional style for not exposing clear centralized data objects for external consumption...

Even in the same way you don't criticize the UNIX spirit for being unfathomably clumsy (who can tell how this cut, head, grep and sed business pipes into xargs?! "Way too complex!"? In fact, you should be saying how elegantly trivial these components are. How exactly clearcut their interface contracts are. How cheap to implement them. How eminently testable (and therefore reliable) they are. etc.

----
I think it is the other way around. ZFS-es design is very simple (or SUN would never have pulled it off in the timespan they did; they would never have reached the reliability that they have notwithstanding the fact that ZFS has the ambition to _replace_ the complexities of volume management, block device abstraction, raid, versioning and transaction control - notions that in their own right have taken companies decades to implement and perfect; I could go on...). This is immediately visible from the outside: simple CLI and raw _power_ that simply proves that there is no logical barrier to the things they wanted to be able to implement (when the complexity wins the designers out, the compromises leak to the outside and show as the rotten limitations [4]).

Now the major issue is with GROKKING the design. And the key is simple. You _do not_ grok the code. You _do not_ grok the implementation(s). You must grok the design. If it looks complex, you are looking at the code. You should be looking at the concepts and things like: "what is the responsability of this thread/procedure/class" and "what processes are running to cater for one pool; idem for filesystems and lastly for the zfs kernel". It will become clearer.

Now the problems for us arise on the practical level:

-  there is little in the way of documentation at that level.

-  Also, SUN has a legacy of being extremely good at designing systems to be massively scalable[5]. You can still see ext2/3 failing to utilize all cores of the CPU to do its job (I know precious few file systems that actually scale sideways in that respect[2]). It's trivial to see why upgrading to 16-core will enhance your ZFS performance. This is not complexity... it is simplicity (every process has such a clearcut and small task, it is easy to just open a can of drones and benefit from the cores in your system, flexible scheduling etc.). To the SUN engineers the mapping of their 'normal design patterns' into straight C code is - well - trivial. It is obvious where the locking goes and why. They perceive the building as the chess grandmaster perceives a middle game position: in logical chunks, not a thousand and coincidences.

- That borders on the next contributing factor: for performance reasons[3], everything has been coded in C. For size and deployment reasons everything is part of some 5 separate 'components'. This doesn't do justice to the separation of responsabilities. But the fact that it is not immediately visible to the untrained eye does not mean that such organization is not there.

----
In short: I love ZFS enough to baulk at the statement that it be too complex. I'd like us to humbly admit that we
1. simply aren't smart enough
2. haven't got enough inside information/coaching
3. don't have the patience
4. have let ourselves be tempted and carried away by a thing of technical beauty, thinking we can attack it with our normal pragmatic programmer skills and attitudes; Excited we are jumping at hacking the code. We assuming any code base should be hackable in days. In a way, all traditional open source projects (perhaps even OO as a concept?) seem to be that way and confirm the assumption. Thing is, ZFS is not just any software. It is different. It makes sense that is not just different on the outside.

I think we should be spending way more time analyzing the code. We should be reading all (I mean all) the documentation out there. We should be talking to SUN engineers. We should be reimplementing ZFS in lisp. We should be designing new features. _Then_ we can see how elegant ZFS is.

============ End of love letter ==============

I'd actually love to see what an implementation of zfs in haskell, lisp or (why not) even Erlang would look like. I'm suspecting it would suprise us how much 'simpler' the implementation would be, simply because the implementation would mirror the design and concepts.

Put another way: ZFS might be a demonstration of Greenspuns 10th Rule; we may be looking at an implementation of ZFS _on top of_/_intermingled with_ a [6] lisp interpreter [7].

TL;DR: Put even more to the point for this rant: I'll happily accept people complaining that this particular implementation of ZFS has a steep learning curve (lacking the background of its creators). I refuse to say that ZFS is complex. It's just not true;

Thanks for listening. I feel much better now :) Now if you'll excuse me, I'll have to go and mock-up a be-all, end-all filesystem driver in C# [7b] I've got the sneaking suspicion the performance could be okay [8]

Seth

[1] cleverly avoiding the choice between threads, coroutines, processes or your favourite alternative here (fibers?!?! eeck)

[2] I can't actually think of one now; I'm suspecting xtreemfs, cluster filesystems, ufs and maybe btrfs might be examples; (the distributed filesystems seem unfair examples as they have been forced into this property by boundary conditions)

[3] and thank God they had portability in mind when writing ZFS

[4] splendid example: lvm2 (see e.g. ZFS vs XFS at serverfault recently)

[5] IBM on the other has had the focus on reliability and control; performance suffers

[6] you don't say XML is complex because you couldn't even understand the assembly listing of the non-validating parser.

[7] to keep the advertising police happy: but you can also shop at F#, OcamL, Scheme, Lambda, Ruby, Arc, or have fun with Ruby, Javascript, C#, Fortran or assembly. Whatever floats your boat. Just be prepared: expressing elegant design in Cobol is gonna more of a challenge than doing so in Scheme.  However, if you are up to the job in Lisp, you can do it in Assembly too (the complexity is not in the details). Jeff Bronwick: that is you

[7b] yes LINQ queries are monads in disguise. Just don't tell anyone you got it from me. You didn't.

[8] Oh well, maybe I won't. I'll only end up confirming that it will be a disappointment compared to ZFS. For the same reason that the Bazaar codebase has much better coherence and "intent" than the Git codebase... Still most developers will pick Git because... well it's like the Ferrari next to a BMW. The BMW is comfortable, stylish and accessible (and cheap). The Ferrari is a beast and harder to handle but ... well it is way faster and more fun. Confession: I still use Bazaar because (a) it is more portable (yes it is) (b) it is good enough (c) I haven't gotten around to get over the learning curve to switch to git. Still I just _know_ when I do, I'll never look back. I do value power over convenience.

[Emmanuel Anne]

2009/10/13 sghe...@hotmail.com <sghe...@hotmail.com>

Or, as most deadlocks are caused by inconsistent ordering of lock

acquisition, it may be that due the new ordering of fs calls (changed,
at least), some locks are (implicitely) acquired/released in a subtly
different order than before. This may result in deadlocks when other
threads are acquiring the same locks in different orders at the same moment.

Yes good idea, except that we didn't change anything in this area (I mean the area close to the error).
The problem here is that (as for a lot of places in zfs code) lots of functions from lots of areas are involved here.
The backtrace goes as high as zfsfuse_getattr_helper which is quite fightening...

exabyte since you seem to be able to reproduce this too often for you, and I can't, could you try to compile your binary with "scons debug=2" to get a full backtrace (a few functions were optimized out here).
From what I understand, the problem is in VN_RELE -> zfs_inactive -> zfs_rmnode
and here an error occurs in dmu_tx_assign, it would be usefull to add a log at this point to know which error occurs, but anyway I simulated some errors here and nothing happened except failed operations (no crash).
So dmu_tx_assign fails for some reason and so it tries to free everything and when it reaches zfs_znode_dmu_fini, the node has already been released by another thread and it crashes.
The big question is how the other thread managed to free it at the same time ?

No clear idea for now, maybe it will be better tomorrow !

Notice that there is a work around for this bug if we don't find a solution : just avoid the crash if the node is already freed and simply return without doing anything in this case. It's not the ideal solution, but if we don't find anything better it could be interesting.

[sghe...@hotmail.com]

It must be in the air tonite:

intrinsic, accidental complexities; entropy; whoa the philosophical underpinnings of what I've been trying to say :)

This is actually in response to a post by Ted Neward, which[1] I don't particularly like.

Cheers

[1] note the which, not whom

[sghe...@hotmail.com]

Ok the final answer: we shall do a reimplementation of ZFS in Blub. That'll fix it!

[Emmanuel Anne]

At least you started another thread for that ! ;-)

But here, it's simply denying the obvious, at the code level, zfs is really complex, and there are quite a few examples of "accidental complexity" inside, and these were choices which were done willingly. Since you compared to the basic unix tools at one point, you could say that unix philosophy is to do complex things by using simple tools together, here all the parts of zfs are complex, put together it's even more complex, and you can add to that way more threads that what is necessary, even on a top architecture with 32 cpus and redundant disk controllers you couldn't possibly use all these threads at the same time.

I should copy and paste one day a few pieces of code to illustrate all this. I have only 2 examples in mind : the piece of code which is responsible for reading. They have put a comment there, saying that it could be done linearly, but it's more fun to do it with threads. It doesn't add anything in performance terms, it might be fun for sun programmers who love to put threads everywhere, but as a final result it makes things way more complex.
And 2nd example : simply the initial memory allocation. Even if they imported their own memory allocator (probably borrowed from some solaris lib), there are tons of example of efficient memory allocation on the web. Well they chose to use such a complex recursion that standard stack sizes are not enough to handle it in 32 bits ! Well of course we can make it work anyway, but saying that it's not unneeeded complexity would really be to deny the obvious.

It's not a criticism of zfs, it's simply a fact, reading this code even reminded me of something I have read about the way men and women program : women tend to add lots of comments to make their code as easy to read and to maintain as possible. Men tend to make very complex code, with as little comments as possible (it's not me who say it, it was written in the study). They say it's probably to show they were able to write such complex code... Well if you apply what this study says, zfs programmers were men, without any doubt ! ;-)

Anyway it's probably enough on the subject, zfs is complex, but we still like it, and it would be cool to continue to improve it anyway. At least it's a way to learn new things...

2009/10/14 sghe...@hotmail.com <sghe...@hotmail.com>