Serious bug in Emmanuel / vvatashki's work

[Rudd-O]

The latest code drop that vvatashki dropped and Emmanuel fixed here
has a serious bug and I think I found a way to reproduce it. For the
purposes of this mail, I'm using the code here:

http://git.rudd-o.com/zfs/shortlog/8169168334b7

which contains the latest enhancements that vvatashki and Emmanuel
have made. You can diff that tree with the code drop -3.tar.bz2 that
Emmanuel left and you will see no significant differences (except for
whitespace changes and compile commenting fixes to appease gas, the
gnu assembler, on debian).

I'm on a 64-bit architecture, using fuse 2.8pre(1 and 3, tested both).

----------------------------------------------------------

In essence, if you run that code, most of your datasets will appear to
be missing -- however, if you know the full dataset pathname, you can
access its properties with zfs get, set properties with zfs set and
addtionally mount it with zfs mount. So the datasets don't really
vanish -- they just aren't displayed.

I have spent close to an hour trying to determine what makes a dataset
vanish, and this is what I have discovered.

1. If you set the secondarycache property to off on your root dataset
(the zpool) there is a 50% chance that any dataset inside that one
will not show up in zfs list anymore. I was looking forward
2. If your dataset has the compression property on, it might vanish as
well.
3. If your dataset has a local mountpoint property not inherited from
its parent dataset, it may very well not show up.
4. Other properties that may be set and cause the datasets to vanihs
are: canmount, primarycache, atime.

This behavior is presnt WHETHER YOU HAVE UPGRADED YOUR POOLS TO
VERSION 16 OR NOT.

-----------------------------------------------------

You can reproduce this bug easily:

- create a new pool (root dataset)
- create a few datasets, themselves with children perhaps
- set the canmount property to noauto on some of them
- set the atime=off property on some of them
- put some of the datasets in different mount points, not inheriting,
of course
- set the compression property=on to some of them, making sure that
some of the children explicitly override it to off
- set the primarycache and secondarycache properties to none in the
root datasets

Do a zfs list, bang, most of your datasets are gone except for the
root ones. You'll note that the amount of disk space used in the
pools themselves does not vary, because the datasets are really not
gone, but not displayed by the zfs command.

I have a serious suspicion that this is a bug in the upstream code of
the zfs command, or some interaction between FUSE, zfs and zfs-fuse.
Regardless of what it is, I tested this with both fuse 2.8 pre1 and
fuse 2.8 pre3 (which I built explicitly for these tests).

------------------------------------------

Here is a very lousy workaround:

- boot your zfs-fuse with ricardo's code (from the tip of his tree)
- zfs unmount -a
- list all your datasets' properties -- this works fine regardless of
how many times you used the latest code or not, as long as you have
not upgraded your pools to version 16
- for every property that diverges from the default, in every dataset,
set it to inherit with zfs inherit
- set a mountpoint in each root dataset, let the child datasets
inherit that mountpoint

The last step may be problematic because zfs may insist on mounting
datasets and there is a weird interaction with fuse in which if zfs
attempts to move a mount or something (i cannot exactly pinpoint what
happens), the mount command (spawned by zfs-fuse) just hangs there and
the operation never finishes.

You may not mix and match binaries from one code drop with another
code drop's zfs-fuse. It causes aborts in the binaries.

------------------------------------------

Needless to say, Emmanuel's code drop is not useful at least to me
(and probably to anybody else at this point) because I have several
datasets with different properties and I cannot afford to turn them
off willy-nilly and have my datasets vanish or not be mounted at
startup.

Ricardo, Jonathan, you're the experts -- what say you?

[Rudd-O]

Oh, yeah, and I tested creating and setting properties on the datasets
in two different ways:

- creating them with ricardo's tip and accessing them with vvatashki's
code
- creating and accessing them both with vvatashki's code

It's indifferent -- the datasets vanish as soon as you set one of the
properties in the root dataset or one of its children.

[Emmanuel Anne]

Thanks for the very long bug report ! ;-)

The fix this time is 3 characters long ! ;-)
Anyway, to give more details than just 3 characters : when listing the available datasets, it allocates a structure at a given place, but forgets to count the size of the path (don't ask me why !). So it doesn't allocate enough, gets an "out of memory error" which you never see anywhere if you don't debug it (we should probably change the code to make it more verbose in case of error one day...).
So I just added 1024 to the default allocated size, and it seems to work everywhere.

Attached the very short patch to fix the very big bug ! ;-)

2009/7/14 Rudd-O <rud...@rudd-o.com>

[Rudd-O]

Can I respectfully request that you please make it so that ZFS blarghs
very very loudly to syslog when out of memory? In low-memory
conditions or when swap is unavailable, ZFS tends to just HANG there
with absolutely no messages.

Thanks in advance. And thanks for the patch. It's going into my
mercurial repo as we speak.

>  diff4
> < 1KViewDownload

[Rudd-O]

btw you should know that i committed the patch but with a 4k buffer
instead of a 3k buffer, for byte alignment and "to-be-on-the-safe-
side" reasons.

[Rudd-O]

Sorry, it doesn't work. I tested with 4k and 8k buffers, and still
some datasets vanish. Of course, they can be listed with zfs list and
can be modified with zfs get and zfs set if you manually put the path
to the dataset in the command line, but that's cold comfort.

More details. These are my datasets:


external
external/backup
external/backup/home
external/backup/root
external/backup/shared
external/backup/shared/fun
external/backup/shared/fun/movies
external/backup/shared/fun/music
external/backup/shared/fun/tv
external/backup/shared/knowledge
external/backup/shared/software
external/backup/tobey
external/lowimportance
internal
internal/home
internal/shared
internal/shared/fun
internal/shared/fun/movies
internal/shared/fun/music
internal/shared/fun/tv
internal/shared/knowledge
internal/shared/software

when I started the new zfs-fuse, the entire tree external/backup/
shared/fun vanished, as well as internal/shared too. I made external/
backup/shared/fun "reappear" by making the compression property
inherit from its parent. That was the ONLY property that was local
instead of inherited. So this bug is definitely related to inherited
vs. local properties.

HOWEVER, I could NOT make my internal/shared datasets reappear at all,
because they have local mount properties (unlike the external dataset,
which is mounted in a tree and every dataset just inherits the
mountpoint property). And I cannot afford to change the mount
properties in the internal/shared dataset without breaking my system.
I am, however, pretty confident that, had I inherited the mount points
in the datasets in internal/shared, I would have seen them reappear.

Mountpoints:

/home/rud...@karen.dragonfear Ω:
zfs get mountpoint
NAME PROPERTY
VALUE SOURCE
external mountpoint /mnt/
external local
external/backup mountpoint /mnt/external/
backup inherited from external
external/backup/home mountpoint /mnt/external/backup/
home inherited from external
external/backup/root mountpoint /mnt/external/backup/
root inherited from external
external/backup/shared mountpoint /mnt/external/backup/
shared inherited from external
external/backup/shared/fun mountpoint /mnt/external/backup/
shared/fun inherited from external
external/backup/shared/fun/movies mountpoint /mnt/external/backup/
shared/fun/movies inherited from external
external/backup/shared/fun/music mountpoint /mnt/external/backup/
shared/fun/music inherited from external
external/backup/shared/fun/tv mountpoint /mnt/external/backup/
shared/fun/tv inherited from external
external/backup/shared/knowledge mountpoint /mnt/external/backup/
shared/knowledge inherited from external
external/backup/shared/software mountpoint /mnt/external/backup/
shared/software inherited from external
external/backup/tobey mountpoint /mnt/external/backup/
tobey inherited from external
external/lowimportance mountpoint /mnt/external/
lowimportance inherited from external
internal mountpoint
none local
internal/home mountpoint /
home local
internal/shared mountpoint /var/
shared local
internal/shared/fun mountpoint /var/shared/
Entertainment local
internal/shared/fun/movies mountpoint /var/shared/
Entertainment/Movies local
internal/shared/fun/music mountpoint /var/shared/
Entertainment/Music local
internal/shared/fun/tv mountpoint /var/shared/
Entertainment/TV local
internal/shared/knowledge mountpoint /var/shared/
Knowledge local
internal/shared/software mountpoint /var/shared/
Software local


------------------------------

In conclusion: the bug persists, but "less gravely" (hey, at least
some datasets appeared now). I was suspicious that just an out-of-
memory error would not be the root cause of the bug, but only an
exacerbant.

Ideas?

[Rudd-O]

In the interest of reproduceability, here is a list of all my
properties in all my datasets (I have now restored the compression
property to the pertinent volumes since I reverted to the old zfs-
fuse). Anyone who creates pools according to these specifications
should be able to reproduce the bug.

----------------------------------------------------------

NAME PROPERTY
VALUE SOURCE
external quota
none default
external reservation
none default
external recordsize
128K default

external mountpoint /mnt/
external local

external sharenfs
off default
external checksum
on default
external compression
on local
external atime
off local
external devices
on default
external exec
on default
external setuid
on default
external readonly
off default
external zoned
off default
external snapdir
hidden default
external aclmode
groupmask default
external aclinherit
restricted default
external canmount
on default
external shareiscsi
off default
external xattr
on default
external copies
1 default
external vscan
off default
external nbmand
off default
external sharesmb
off default
external refquota
none default
external refreservation
none default
external primarycache
none local
external secondarycache
none local
external/backup quota
none default
external/backup reservation
none default
external/backup recordsize
128K default

external/backup mountpoint /mnt/external/
backup inherited from external

external/backup sharenfs
off default
external/backup checksum
on default
external/backup compression
on inherited from external
external/backup atime
off inherited from external
external/backup devices
on default
external/backup exec
on default
external/backup setuid
on default
external/backup readonly
off default
external/backup zoned
off default
external/backup snapdir
hidden default
external/backup aclmode
groupmask default
external/backup aclinherit
restricted default
external/backup canmount
on default
external/backup shareiscsi
off default
external/backup xattr
on default
external/backup copies
1 default
external/backup vscan
off default
external/backup nbmand
off default
external/backup sharesmb
off default
external/backup refquota
none default
external/backup refreservation
none default
external/backup primarycache
none inherited from external
external/backup secondarycache
none inherited from external
external/backup/home quota
none default
external/backup/home reservation
none default
external/backup/home recordsize
128K default

external/backup/home mountpoint /mnt/external/
backup/home inherited from external

external/backup/home sharenfs
off default
external/backup/home checksum
on default
external/backup/home compression
on inherited from external
external/backup/home atime
off inherited from external
external/backup/home devices
on default
external/backup/home exec
on default
external/backup/home setuid
on default
external/backup/home readonly
off default
external/backup/home zoned
off default
external/backup/home snapdir
hidden default
external/backup/home aclmode
groupmask default
external/backup/home aclinherit
restricted default
external/backup/home canmount
on default
external/backup/home shareiscsi
off default
external/backup/home xattr
on default
external/backup/home copies
1 default
external/backup/home vscan
off default
external/backup/home nbmand
off default
external/backup/home sharesmb
off default
external/backup/home refquota
none default
external/backup/home refreservation
none default
external/backup/home primarycache
none inherited from external
external/backup/home secondarycache
none inherited from external
external/backup/root quota
none default
external/backup/root reservation
none default
external/backup/root recordsize
128K default

external/backup/root mountpoint /mnt/external/
backup/root inherited from external

external/backup/root sharenfs
off default
external/backup/root checksum
on default
external/backup/root compression
on inherited from external
external/backup/root atime
off inherited from external
external/backup/root devices
on default
external/backup/root exec
on default
external/backup/root setuid
on default
external/backup/root readonly
off default
external/backup/root zoned
off default
external/backup/root snapdir
hidden default
external/backup/root aclmode
groupmask default
external/backup/root aclinherit
restricted default
external/backup/root canmount
on default
external/backup/root shareiscsi
off default
external/backup/root xattr
on default
external/backup/root copies
1 default
external/backup/root vscan
off default
external/backup/root nbmand
off default
external/backup/root sharesmb
off default
external/backup/root refquota
none default
external/backup/root refreservation
none default
external/backup/root primarycache
none inherited from external
external/backup/root secondarycache
none inherited from external
external/backup/shared quota
none default
external/backup/shared reservation
none default
external/backup/shared recordsize
128K default

external/backup/shared mountpoint /mnt/external/
backup/shared inherited from external

external/backup/shared sharenfs
off default
external/backup/shared checksum
on default
external/backup/shared compression
on inherited from external
external/backup/shared atime
off inherited from external
external/backup/shared devices
on default
external/backup/shared exec
on default
external/backup/shared setuid
on default
external/backup/shared readonly
off default
external/backup/shared zoned
off default
external/backup/shared snapdir
hidden default
external/backup/shared aclmode
groupmask default
external/backup/shared aclinherit
restricted default
external/backup/shared canmount
on default
external/backup/shared shareiscsi
off default
external/backup/shared xattr
on default
external/backup/shared copies
1 default
external/backup/shared vscan
off default
external/backup/shared nbmand
off default
external/backup/shared sharesmb
off default
external/backup/shared refquota
none default
external/backup/shared refreservation
none default
external/backup/shared primarycache
none inherited from external
external/backup/shared secondarycache
none inherited from external
external/backup/shared/fun quota
none default
external/backup/shared/fun reservation
none default
external/backup/shared/fun recordsize
128K default

external/backup/shared/fun mountpoint /mnt/external/
backup/shared/fun inherited from external

external/backup/shared/fun sharenfs
off default
external/backup/shared/fun checksum
on default
external/backup/shared/fun compression
off local
external/backup/shared/fun atime
off inherited from external
external/backup/shared/fun devices
on default
external/backup/shared/fun exec
on default
external/backup/shared/fun setuid
on default
external/backup/shared/fun readonly
off default
external/backup/shared/fun zoned
off default
external/backup/shared/fun snapdir
hidden default
external/backup/shared/fun aclmode
groupmask default
external/backup/shared/fun aclinherit
restricted default
external/backup/shared/fun canmount
on default
external/backup/shared/fun shareiscsi
off default
external/backup/shared/fun xattr
on default
external/backup/shared/fun copies
1 default
external/backup/shared/fun vscan
off default
external/backup/shared/fun nbmand
off default
external/backup/shared/fun sharesmb
off default
external/backup/shared/fun refquota
none default
external/backup/shared/fun refreservation
none default
external/backup/shared/fun primarycache
none inherited from external
external/backup/shared/fun secondarycache
none inherited from external
external/backup/shared/fun/movies quota
none default
external/backup/shared/fun/movies reservation
none default
external/backup/shared/fun/movies recordsize
128K default

external/backup/shared/fun/movies mountpoint /mnt/external/
backup/shared/fun/movies inherited from external

external/backup/shared/fun/movies sharenfs
off default
external/backup/shared/fun/movies checksum
on default
external/backup/shared/fun/movies compression
off inherited from external/backup/
shared/fun
external/backup/shared/fun/movies atime
off inherited from external
external/backup/shared/fun/movies devices
on default
external/backup/shared/fun/movies exec
on default
external/backup/shared/fun/movies setuid
on default
external/backup/shared/fun/movies readonly
off default
external/backup/shared/fun/movies zoned
off default
external/backup/shared/fun/movies snapdir
hidden default
external/backup/shared/fun/movies aclmode
groupmask default
external/backup/shared/fun/movies aclinherit
restricted default
external/backup/shared/fun/movies canmount
on default
external/backup/shared/fun/movies shareiscsi
off default
external/backup/shared/fun/movies xattr
on default
external/backup/shared/fun/movies copies
1 default
external/backup/shared/fun/movies vscan
off default
external/backup/shared/fun/movies nbmand
off default
external/backup/shared/fun/movies sharesmb
off default
external/backup/shared/fun/movies refquota
none default
external/backup/shared/fun/movies refreservation
none default
external/backup/shared/fun/movies primarycache
none inherited from external
external/backup/shared/fun/movies secondarycache
none inherited from external
external/backup/shared/fun/music quota
none default
external/backup/shared/fun/music reservation
none default
external/backup/shared/fun/music recordsize
128K default

external/backup/shared/fun/music mountpoint /mnt/external/
backup/shared/fun/music inherited from external

external/backup/shared/fun/music sharenfs
off default
external/backup/shared/fun/music checksum
on default
external/backup/shared/fun/music compression
off inherited from external/backup/
shared/fun
external/backup/shared/fun/music atime
off inherited from external
external/backup/shared/fun/music devices
on default
external/backup/shared/fun/music exec
on default
external/backup/shared/fun/music setuid
on default
external/backup/shared/fun/music readonly
off default
external/backup/shared/fun/music zoned
off default
external/backup/shared/fun/music snapdir
hidden default
external/backup/shared/fun/music aclmode
groupmask default
external/backup/shared/fun/music aclinherit
restricted default
external/backup/shared/fun/music canmount
on default
external/backup/shared/fun/music shareiscsi
off default
external/backup/shared/fun/music xattr
on default
external/backup/shared/fun/music copies
1 default
external/backup/shared/fun/music vscan
off default
external/backup/shared/fun/music nbmand
off default
external/backup/shared/fun/music sharesmb
off default
external/backup/shared/fun/music refquota
none default
external/backup/shared/fun/music refreservation
none default
external/backup/shared/fun/music primarycache
none inherited from external
external/backup/shared/fun/music secondarycache
none inherited from external
external/backup/shared/fun/tv quota
none default
external/backup/shared/fun/tv reservation
none default
external/backup/shared/fun/tv recordsize
128K default

external/backup/shared/fun/tv mountpoint /mnt/external/
backup/shared/fun/tv inherited from external

external/backup/shared/fun/tv sharenfs
off default
external/backup/shared/fun/tv checksum
on default
external/backup/shared/fun/tv compression
off inherited from external/backup/
shared/fun
external/backup/shared/fun/tv atime
off inherited from external
external/backup/shared/fun/tv devices
on default
external/backup/shared/fun/tv exec
on default
external/backup/shared/fun/tv setuid
on default
external/backup/shared/fun/tv readonly
off default
external/backup/shared/fun/tv zoned
off default
external/backup/shared/fun/tv snapdir
hidden default
external/backup/shared/fun/tv aclmode
groupmask default
external/backup/shared/fun/tv aclinherit
restricted default
external/backup/shared/fun/tv canmount
on default
external/backup/shared/fun/tv shareiscsi
off default
external/backup/shared/fun/tv xattr
on default
external/backup/shared/fun/tv copies
1 default
external/backup/shared/fun/tv vscan
off default
external/backup/shared/fun/tv nbmand
off default
external/backup/shared/fun/tv sharesmb
off default
external/backup/shared/fun/tv refquota
none default
external/backup/shared/fun/tv refreservation
none default
external/backup/shared/fun/tv primarycache
none inherited from external
external/backup/shared/fun/tv secondarycache
none inherited from external
external/backup/shared/knowledge quota
none default
external/backup/shared/knowledge reservation
none default
external/backup/shared/knowledge recordsize
128K default

external/backup/shared/knowledge mountpoint /mnt/external/
backup/shared/knowledge inherited from external

external/backup/shared/knowledge sharenfs
off default
external/backup/shared/knowledge checksum
on default
external/backup/shared/knowledge compression
on inherited from external
external/backup/shared/knowledge atime
off inherited from external
external/backup/shared/knowledge devices
on default
external/backup/shared/knowledge exec
on default
external/backup/shared/knowledge setuid
on default
external/backup/shared/knowledge readonly
off default
external/backup/shared/knowledge zoned
off default
external/backup/shared/knowledge snapdir
hidden default
external/backup/shared/knowledge aclmode
groupmask default
external/backup/shared/knowledge aclinherit
restricted default
external/backup/shared/knowledge canmount
on default
external/backup/shared/knowledge shareiscsi
off default
external/backup/shared/knowledge xattr
on default
external/backup/shared/knowledge copies
1 default
external/backup/shared/knowledge vscan
off default
external/backup/shared/knowledge nbmand
off default
external/backup/shared/knowledge sharesmb
off default
external/backup/shared/knowledge refquota
none default
external/backup/shared/knowledge refreservation
none default
external/backup/shared/knowledge primarycache
none inherited from external
external/backup/shared/knowledge secondarycache
none inherited from external
external/backup/shared/software quota
none default
external/backup/shared/software reservation
none default
external/backup/shared/software recordsize
128K default

external/backup/shared/software mountpoint /mnt/external/
backup/shared/software inherited from external

external/backup/shared/software sharenfs
off default
external/backup/shared/software checksum
on default
external/backup/shared/software compression
on inherited from external
external/backup/shared/software atime
off inherited from external
external/backup/shared/software devices
on default
external/backup/shared/software exec
on default
external/backup/shared/software setuid
on default
external/backup/shared/software readonly
off default
external/backup/shared/software zoned
off default
external/backup/shared/software snapdir
hidden default
external/backup/shared/software aclmode
groupmask default
external/backup/shared/software aclinherit
restricted default
external/backup/shared/software canmount
on default
external/backup/shared/software shareiscsi
off default
external/backup/shared/software xattr
on default
external/backup/shared/software copies
1 default
external/backup/shared/software vscan
off default
external/backup/shared/software nbmand
off default
external/backup/shared/software sharesmb
off default
external/backup/shared/software refquota
none default
external/backup/shared/software refreservation
none default
external/backup/shared/software primarycache
none inherited from external
external/backup/shared/software secondarycache
none inherited from external
external/backup/tobey quota
none default
external/backup/tobey reservation
none default
external/backup/tobey recordsize
128K default

external/backup/tobey mountpoint /mnt/external/
backup/tobey inherited from external

external/backup/tobey sharenfs
off default
external/backup/tobey checksum
on default
external/backup/tobey compression
on inherited from external
external/backup/tobey atime
off inherited from external
external/backup/tobey devices
on default
external/backup/tobey exec
on default
external/backup/tobey setuid
on default
external/backup/tobey readonly
off default
external/backup/tobey zoned
off default
external/backup/tobey snapdir
hidden default
external/backup/tobey aclmode
groupmask default
external/backup/tobey aclinherit
restricted default
external/backup/tobey canmount
on default
external/backup/tobey shareiscsi
off default
external/backup/tobey xattr
on default
external/backup/tobey copies
1 default
external/backup/tobey vscan
off default
external/backup/tobey nbmand
off default
external/backup/tobey sharesmb
off default
external/backup/tobey refquota
none default
external/backup/tobey refreservation
none default
external/backup/tobey primarycache
none inherited from external
external/backup/tobey secondarycache
none inherited from external
external/lowimportance quota
none default
external/lowimportance reservation
none default
external/lowimportance recordsize
128K default

external/lowimportance mountpoint /mnt/external/
lowimportance inherited from external

external/lowimportance sharenfs
off default
external/lowimportance checksum
on default
external/lowimportance compression
off local
external/lowimportance atime
off inherited from external
external/lowimportance devices
on default
external/lowimportance exec
on default
external/lowimportance setuid
on default
external/lowimportance readonly
off default
external/lowimportance zoned
off default
external/lowimportance snapdir
hidden default
external/lowimportance aclmode
groupmask default
external/lowimportance aclinherit
restricted default
external/lowimportance canmount
on default
external/lowimportance shareiscsi
off default
external/lowimportance xattr
on default
external/lowimportance copies
1 default
external/lowimportance vscan
off default
external/lowimportance nbmand
off default
external/lowimportance sharesmb
off default
external/lowimportance refquota
none default
external/lowimportance refreservation
none default
external/lowimportance primarycache
none inherited from external
external/lowimportance secondarycache
none inherited from external
internal quota
none default
internal reservation
none default
internal recordsize
128K default
internal mountpoint
none local
internal sharenfs
off default
internal checksum
on default
internal compression
on local
internal atime
off local
internal devices
on default
internal exec
on default
internal setuid
on default
internal readonly
off default
internal zoned
off default
internal snapdir
hidden default
internal aclmode
groupmask default
internal aclinherit
restricted default
internal canmount
on default
internal shareiscsi
off default
internal xattr
on default
internal copies
1 default
internal vscan
off default
internal nbmand
off default
internal sharesmb
off default
internal refquota
none default
internal refreservation
none default
internal primarycache
none local
internal secondarycache
none local
internal/home quota
none default
internal/home reservation
none default
internal/home recordsize
128K default

internal/home mountpoint /
home local

internal/home sharenfs
off default
internal/home checksum
on default
internal/home compression
on inherited from internal
internal/home atime
off inherited from internal
internal/home devices
on default
internal/home exec
on default
internal/home setuid
on default
internal/home readonly
off default
internal/home zoned
off default
internal/home snapdir
hidden default
internal/home aclmode
groupmask default
internal/home aclinherit
restricted default
internal/home canmount
on default
internal/home shareiscsi
off default
internal/home xattr
on default
internal/home copies
1 default
internal/home vscan
off default
internal/home nbmand
off default
internal/home sharesmb
off default
internal/home refquota
none default
internal/home refreservation
none default
internal/home primarycache
none inherited from internal
internal/home secondarycache
none inherited from internal
internal/shared quota
none default
internal/shared reservation
none default
internal/shared recordsize
128K default

internal/shared mountpoint /var/
shared local

internal/shared sharenfs
off default
internal/shared checksum
on default
internal/shared compression
on inherited from internal
internal/shared atime
off inherited from internal
internal/shared devices
on default
internal/shared exec
on default
internal/shared setuid
on default
internal/shared readonly
off default
internal/shared zoned
off default
internal/shared snapdir
hidden default
internal/shared aclmode
groupmask default
internal/shared aclinherit
restricted default
internal/shared canmount
on default
internal/shared shareiscsi
off default
internal/shared xattr
on default
internal/shared copies
1 default
internal/shared vscan
off default
internal/shared nbmand
off default
internal/shared sharesmb
off default
internal/shared refquota
none default
internal/shared refreservation
none default
internal/shared primarycache
none inherited from internal
internal/shared secondarycache
none inherited from internal
internal/shared/fun quota
none default
internal/shared/fun reservation
none default
internal/shared/fun recordsize
128K default

internal/shared/fun mountpoint /var/shared/
Entertainment local

internal/shared/fun sharenfs
off default
internal/shared/fun checksum
on default
internal/shared/fun compression
off local
internal/shared/fun atime
off inherited from internal
internal/shared/fun devices
on default
internal/shared/fun exec
on default
internal/shared/fun setuid
on default
internal/shared/fun readonly
off default
internal/shared/fun zoned
off default
internal/shared/fun snapdir
hidden default
internal/shared/fun aclmode
groupmask default
internal/shared/fun aclinherit
restricted default
internal/shared/fun canmount
on default
internal/shared/fun shareiscsi
off default
internal/shared/fun xattr
on default
internal/shared/fun copies
1 default
internal/shared/fun vscan
off default
internal/shared/fun nbmand
off default
internal/shared/fun sharesmb
off default
internal/shared/fun refquota
none default
internal/shared/fun refreservation
none default
internal/shared/fun primarycache
none inherited from internal
internal/shared/fun secondarycache
none inherited from internal
internal/shared/fun/movies quota
none default
internal/shared/fun/movies reservation
none default
internal/shared/fun/movies recordsize
128K default

internal/shared/fun/movies mountpoint /var/shared/
Entertainment/Movies local

internal/shared/fun/movies sharenfs
off default
internal/shared/fun/movies checksum
on default
internal/shared/fun/movies compression
off inherited from internal/shared/
fun
internal/shared/fun/movies atime
off inherited from internal
internal/shared/fun/movies devices
on default
internal/shared/fun/movies exec
on default
internal/shared/fun/movies setuid
on default
internal/shared/fun/movies readonly
off default
internal/shared/fun/movies zoned
off default
internal/shared/fun/movies snapdir
hidden default
internal/shared/fun/movies aclmode
groupmask default
internal/shared/fun/movies aclinherit
restricted default
internal/shared/fun/movies canmount
on default
internal/shared/fun/movies shareiscsi
off default
internal/shared/fun/movies xattr
on default
internal/shared/fun/movies copies
1 default
internal/shared/fun/movies vscan
off default
internal/shared/fun/movies nbmand
off default
internal/shared/fun/movies sharesmb
off default
internal/shared/fun/movies refquota
none default
internal/shared/fun/movies refreservation
none default
internal/shared/fun/movies primarycache
none inherited from internal
internal/shared/fun/movies secondarycache
none inherited from internal
internal/shared/fun/music quota
none default
internal/shared/fun/music reservation
none default
internal/shared/fun/music recordsize
128K default

internal/shared/fun/music mountpoint /var/shared/
Entertainment/Music local

internal/shared/fun/music sharenfs
off default
internal/shared/fun/music checksum
on default
internal/shared/fun/music compression
off inherited from internal/shared/
fun
internal/shared/fun/music atime
off inherited from internal
internal/shared/fun/music devices
on default
internal/shared/fun/music exec
on default
internal/shared/fun/music setuid
on default
internal/shared/fun/music readonly
off default
internal/shared/fun/music zoned
off default
internal/shared/fun/music snapdir
hidden default
internal/shared/fun/music aclmode
groupmask default
internal/shared/fun/music aclinherit
restricted default
internal/shared/fun/music canmount
on default
internal/shared/fun/music shareiscsi
off default
internal/shared/fun/music xattr
on default
internal/shared/fun/music copies
1 default
internal/shared/fun/music vscan
off default
internal/shared/fun/music nbmand
off default
internal/shared/fun/music sharesmb
off default
internal/shared/fun/music refquota
none default
internal/shared/fun/music refreservation
none default
internal/shared/fun/music primarycache
none inherited from internal
internal/shared/fun/music secondarycache
none inherited from internal
internal/shared/fun/tv quota
none default
internal/shared/fun/tv reservation
none default
internal/shared/fun/tv recordsize
128K default

internal/shared/fun/tv mountpoint /var/shared/
Entertainment/TV local

internal/shared/fun/tv sharenfs
off default
internal/shared/fun/tv checksum
on default
internal/shared/fun/tv compression
off inherited from internal/shared/
fun
internal/shared/fun/tv atime
off inherited from internal
internal/shared/fun/tv devices
on default
internal/shared/fun/tv exec
on default
internal/shared/fun/tv setuid
on default
internal/shared/fun/tv readonly
off default
internal/shared/fun/tv zoned
off default
internal/shared/fun/tv snapdir
hidden default
internal/shared/fun/tv aclmode
groupmask default
internal/shared/fun/tv aclinherit
restricted default
internal/shared/fun/tv canmount
on default
internal/shared/fun/tv shareiscsi
off default
internal/shared/fun/tv xattr
on default
internal/shared/fun/tv copies
1 default
internal/shared/fun/tv vscan
off default
internal/shared/fun/tv nbmand
off default
internal/shared/fun/tv sharesmb
off default
internal/shared/fun/tv refquota
none default
internal/shared/fun/tv refreservation
none default
internal/shared/fun/tv primarycache
none inherited from internal
internal/shared/fun/tv secondarycache
none inherited from internal
internal/shared/knowledge quota
none default
internal/shared/knowledge reservation
none default
internal/shared/knowledge recordsize
128K default

internal/shared/knowledge mountpoint /var/shared/
Knowledge local

internal/shared/knowledge sharenfs
off default
internal/shared/knowledge checksum
on default
internal/shared/knowledge compression
on inherited from internal
internal/shared/knowledge atime
off inherited from internal
internal/shared/knowledge devices
on default
internal/shared/knowledge exec
on default
internal/shared/knowledge setuid
on default
internal/shared/knowledge readonly
off default
internal/shared/knowledge zoned
off default
internal/shared/knowledge snapdir
hidden default
internal/shared/knowledge aclmode
groupmask default
internal/shared/knowledge aclinherit
restricted default
internal/shared/knowledge canmount
on default
internal/shared/knowledge shareiscsi
off default
internal/shared/knowledge xattr
on default
internal/shared/knowledge copies
1 default
internal/shared/knowledge vscan
off default
internal/shared/knowledge nbmand
off default
internal/shared/knowledge sharesmb
off default
internal/shared/knowledge refquota
none default
internal/shared/knowledge refreservation
none default
internal/shared/knowledge primarycache
none inherited from internal
internal/shared/knowledge secondarycache
none inherited from internal
internal/shared/software quota
none default
internal/shared/software reservation
none default
internal/shared/software recordsize
128K default

internal/shared/software mountpoint /var/shared/
Software local

internal/shared/software sharenfs
off default
internal/shared/software checksum
on default
internal/shared/software compression
on inherited from internal
internal/shared/software atime
off inherited from internal
internal/shared/software devices
on default
internal/shared/software exec
on default
internal/shared/software setuid
on default
internal/shared/software readonly
off default
internal/shared/software zoned
off default
internal/shared/software snapdir
hidden default
internal/shared/software aclmode
groupmask default
internal/shared/software aclinherit
restricted default
internal/shared/software canmount
on default
internal/shared/software shareiscsi
off default
internal/shared/software xattr
on default
internal/shared/software copies
1 default
internal/shared/software vscan
off default
internal/shared/software nbmand
off default
internal/shared/software sharesmb
off default
internal/shared/software refquota
none default
internal/shared/software refreservation
none default
internal/shared/software primarycache
none inherited from internal
internal/shared/software secondarycache
none inherited from internal

[Rudd-O]

As for the no memory condition hanging ZFS, this in zfs_util.c
apparently never gets executed:

/*
* Display an out of memory error message and abort the current
program.
*/
int
no_memory(libzfs_handle_t *hdl)
{
return (zfs_error(hdl, EZFS_NOMEM, "internal error"));
}

Try it. Run zfs-fuse without swap and with less than 512 MB RAM and
see it hang. I have no idea where it's hanging but it is hanging.

[Emmanuel Anne]

Wow ! Long list, and quite hard to manage because of the line breaks inserted everywhere...
Anyway I have tried to reproduce your problem, but I couldn't do it so far. So.... I have created a shell script to try to reproduce your datasets (without the data in them of course, just the metadata). I attach it to this mail, please run it as root, it should create an image file of 100 Mb, create a pool named "external" (you might want to change the name to something else like external2 if you test it on the same system), with a mountpoint of /mnt/external, compress=on, primarycache/secondarycache=none, then it creates all the datasets in it... and it works, no problem.

Notice : in your listed properties, the mountpoint for internal is "none", so it can't be mounted...

Sorry but I'll need more help here to be able to reproduce this...
(I created the datasets with version 0.5.0 but it still works when mounted with my latest version).

2009/7/15 Rudd-O <rud...@rudd-o.com>

[Rudd-O]

That script won't let you reproduce the problem I am experiencing.

Do this:

# 100 Mb file for testing
dd if=/dev/zero of=image bs=1024 count=102400
zpool create external -m /mnt/external `pwd`/image
zfs set compression=on external
zfs set primarycache=none external
zfs set secondarycache=none external
zfs create external/backup
zfs create external/backup/home
zfs create external/backup/root
zfs create external/backup/shared
zfs create external/backup/shared/fun
zfs create external/backup/shared/fun/movies
zfs create external/backup/shared/fun/music
zfs create external/backup/shared/fun/tv
zfs create external/backup/shared/knowledge
zfs create external/backup/shared/software
zfs create external/backup/tobey
zfs create external/lowimportance
zfs set mountpoint=/mnt/test external/backup
zfs set mountpoint=/mnt/test/Entertainment external/backup/fun
zfs set mountpoint=/mnt/test/Entertainment/Movies external/backup/fun/
movies
zfs set mountpoint=/mnt/test/Entertainment/Music external/backup/fun/
music
zfs set mountpoint=/mnt/test/Entertainment/TV external/backup/fun/tv
zfs set compression=off external/backup/shared

I bet you a case of beer that you will be able to reproduce the bug
now. Do a zfs list and you will see that external/backup/fun or
external/backup vanished.

[Rudd-O]

Found it

RUN THIS SCRIPT AND YOU CAN REPRODUCE IT! WOO HOO!


------------------------------------------------------------------


set -e
set -x
killall -9 zfs-fuse
rm -f image
zfs-fuse --no-daemon &
sleep 2

dd if=/dev/zero of=image bs=1024 count=102400

zpool list | grep -q external && zpool destroy external
rm -rf /mnt/external
zpool create external -m /mnt/external $PWD/image
zfs umount -a

zfs set compression=on external
zfs set primarycache=none external
zfs set secondarycache=none external
zfs create external/backup
zfs create external/backup/home
zfs create external/backup/root
zfs create external/backup/shared
zfs create external/backup/shared/fun
zfs create external/backup/shared/fun/movies
zfs create external/backup/shared/fun/music
zfs create external/backup/shared/fun/tv
zfs create external/backup/shared/knowledge
zfs create external/backup/shared/software
zfs create external/backup/tobey
zfs create external/lowimportance

zfs set mountpoint=/mnt/test external/backup/shared
zfs set mountpoint=/mnt/test/Entertainment external/backup/shared/fun
zfs set mountpoint=/mnt/test/Entertainment/Movies external/backup/
shared/fun/movies
zfs set mountpoint=/mnt/test/Entertainment/Music external/backup/
shared/fun/music
zfs set mountpoint=/mnt/test/Entertainment/TV external/backup/shared/
fun/tv
zfs set compression=off external/backup/shared
sleep 1
zfs list

-------------------------------


tested on a CentOS VM. Thanks for the initial idea. This script can
let you reproduce the bug exactly.

[Rudd-O]

More details

If you run the script above, you will note that either setting a local
compression attribute in one of the child datasets, or setting a local
mountpoint attribute in one of the child datasets, is enough to have
zfs not list the datasets anymore. Most importantly, only ONE of the
two attribute changes need to be performed -- it does not require you
to make multiple changes... for example, just setting the compression
attribute makes the dataset disappear.

But the key thing is that the property must be set to a CHILD
dataset. If you set properties in the root dataset, nothing bad
happens.

[Rudd-O]

I have added the test-datasets script with some documentation to the
contrib/ directory in my repo:

http://git.rudd-o.com/zfs/rev/b910abf83202

[Emmanuel Anne]

Ok I confirm I get something, and I get this in syslog :

Jul 16 09:32:43 athlon64 zfs-fuse: put_nvlist: error Cannot allocate memory
Jul 16 09:32:43 athlon64 zfs-fuse: put_nvlist: error Cannot allocate memory

Usefull message after all !

2009/7/16 Rudd-O <rud...@rudd-o.com>

[Emmanuel Anne]

Ok, it's fixed now (hopefully for good this time !). Thanks for your findings.

To sum up the problem was indeed in put_nvlist, except that when you have inherited properties for datasets, then this function is called recursively. And at each turn of the recursion, the size of the buffer shrinks ! The person who wrote that probably thought about some kind of optimization since a buffer which is sent to the fuse socket, but he totally forgot about the recursion.
So the only way I found this is to prevent it from shrinking, just 1 line to comment out for that. Your script works now (I just changed it on my side to not kill zfs-fuse nor launch another one, nor unmounting anything, but it still does the same tests except that).

I also improved the precision of the error messages to have the values which triggered the out of memory, and added 2 sync() calls to work around these stupid message we get sometimes "cannot unmount, dataset is busy" or "cannot export pool is busy", and you just retype the same command and it works the 2nd time. The sync calls should make the messages to go away !

The patch is against the current mercurial tree.

2009/7/16 Emmanuel Anne <emmanu...@gmail.com>

[Rudd-O]

Holy.... you're a ninja. You've outdone yourself. I'm going to try
this now.

[Rudd-O]

I have pushed your patch into the repo. Pertinent revision:

http://git.rudd-o.com/zfs/rev/c1e602c8165f

[Rudd-O]

With this patch, I have successfully upgraded to the latest compiled
ZFS from my tree, upgraded my volumes to version 16 and my datasets to
version 4.

Everything is running normally.

Thanks, Emmanuel! You rock.