Zfs-fuse: Mount is off but it wont mont to my mount point.

[Lachlan Holmes]

Hey Guys,

I can't access my data anymore.

ZFS-Fuse mounts my pool called "storage" but after a reboot the pool no longer mounts to m=/storage

# sudo zfs get all storage

NAME     PROPERTY              VALUE                  SOURCE

storage  type                  filesystem             -

storage  creation              Wed Dec 14  9:15 2011  -

storage  used                  4.16T                  -

storage  available             1.16T                  -

storage  referenced            4.16T                  -

storage  compressratio         1.00x                  -

storage  mounted               no                     -

storage  quota                 none                   default

storage  reservation           none                   default

storage  recordsize            128K                   default

storage  mountpoint            /storage               local

storage  sharenfs              off                    default

storage  checksum              on                     default

storage  compression           off                    default

storage  atime                 on                     default

storage  devices               on                     default

storage  exec                  on                     default

storage  setuid                on                     default

storage  readonly              off                    default

storage  zoned                 off                    default

storage  snapdir               hidden                 default

storage  aclmode               groupmask              default

storage  aclinherit            restricted             default

storage  canmount              on                     default

storage  xattr                 on                     default

storage  copies                1                      default

storage  version               4                      -

storage  utf8only              off                    -

storage  normalization         none                   -

storage  casesensitivity       sensitive              -

storage  vscan                 off                    default

storage  nbmand                off                    default

storage  sharesmb              off                    default

storage  refquota              none                   default

storage  refreservation        none                   default

storage  primarycache          all                    default

storage  secondarycache        all                    default

storage  usedbysnapshots       0                      -

storage  usedbydataset         4.16T                  -

storage  usedbychildren        1.17G                  -

storage  usedbyrefreservation  0                      -

storage  logbias               latency                default

storage  dedup                 on                     local

storage  mlslabel              on                     -

When i do a sudo zfs mount i get the following.

cannot mount 'storage': mountpoint or dataset is busy

This is history of the pool.

History for 'storage':

2011-12-14.09:15:17 zpool create storage -m /storage raidz2 disk/by-id/scsi-SATA_Hitachi_HDS7230_MN1220F30PHJED disk/by-id/scsi-SATA_Hitachi_HDS7230_MN1220F30RUXLD disk/by-id/scsi-SATA_Hitachi_HDS7230_MN1220F30RW2LD disk/by-id/scsi-SATA_Hitachi_HDS7230_MN1220F30SAHAD disk/by-id/scsi-SATA_Hitachi_HDS7230_MN1220F30SAT8D spare disk/by-id/scsi-SATA_Hitachi_HDS7230_MN1220F32M0J1D -f

2012-02-06.19:55:54 zpool scrub -s storage

2012-02-06.19:56:21 zpool scrub -s storage

2012-03-07.21:56:08 zpool scrub -s storage

2012-04-15.08:08:46 zpool scrub -s storage

2012-04-15.08:08:49 zpool scrub -s storage

2012-05-31.20:03:45 zpool scrub -s storage

2012-05-31.20:03:47 zpool scrub -s storage

2012-08-04.18:16:02 zpool scrub storage

2012-08-08.18:36:26 zpool scrub storage

2012-08-11.09:47:00 zpool scrub storage

2012-09-05.18:27:27 zpool scrub storage

2012-09-05.18:34:00 zfs set dedup=on storage

can anyone explain why the mounted is set to no and then i can't mount it? or should i give up?

regards

lachlan

[Emmanuel Anne]

Did you check that /storage is empty ?

(enabling dedup after some files were written to the fs is most of the time a bad idea, dedup requires a lot of metadata and slows things a lot usually, it's better to have specialized filesystems for this).

2012/9/10 Lachlan Holmes <lachlan....@gmail.com>

lachlan

--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

--
my zfs-fuse git repository : http://rainemu.swishparty.co.uk/cgi-bin/gitweb.cgi?p=zfs;a=summary

[Lachlan Holmes]

Hey mate yes the /storage folder is empty... 

but the folder in question shows this. which is to say that the /storage folder doesnt seem to me associated to the zfs pool

# sudo df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/sdg7              37G  3.7G   32G  11% /

none                  1.9G  264K  1.9G   1% /dev

none                  1.9G  176K  1.9G   1% /dev/shm

none                  1.9G  328K  1.9G   1% /var/run

none                  1.9G     0  1.9G   0% /var/lock

/dev/sdg5              92M   63M   24M  73% /boot

/dev/sdg6              30G  1.3G   27G   5% /home

# sudo zpool list

NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT

storage  9.06T  6.97T  2.09T    76%  1.01x  ONLINE  -

# sudo zfs list

NAME      USED  AVAIL  REFER  MOUNTPOINT

storage  4.15T  1.17T  4.15T  /storage

if this doesnt help let me know what i need to do. I still feel like the data is there fro mthe zfs list but its not working.

regards,

lachlan

[Emmanuel Anne]

So you are saying that zfs thinks /storage is mounted, but your os doesn't ?

Well then just try sudo zfs umount storage followed by sudo zfs mount storage, and see what happens... !

[Lachlan Holmes]

lachlan@#########:~$ sudo zfs umount storage

[sudo] password for lachlan: 

cannot unmount 'storage': not currently mounted

lachlan@#########:~$ sudo zfs mount storage

cannot mount 'storage': mountpoint or dataset is busy

so yeah i dont know what to do here.

regards,

lachlan

[sehe]

This problem started immediately after enabling dedup?

Dedup eats a lot of RAM. 

Can you try to give zfs-fuse a lot more RAM? Use swap if you have to... I know that's not workable but you can at least see whether it will load your dataset if it can use more RAM for DDT

$0.02

[Emmanuel Anne]

Or easier, just disable dedup to test, if it works then you can always change your settings if you really want to enable it after that.

2012/9/10 sehe <sghe...@hotmail.com>

--

To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/

[Lachlan Holmes]

It occurred after removing 1.4 TV of data from the pool. Which hung in the process. I rebooted the server and no mounted pool but it's set to auto mount according to the set all setting in the logs I've provided 

Thanks Lachlan

--

[Lachlan Holmes]

lachlan@#######:~$ sudo zpool set dedup=off storage
cannot set property for 'storage': 'dedup' is readonly

am I doing it right?  Thanks Lachlan

[Emmanuel Anne]

Find some documentation about dedup, it will obviously be useful, it's not an option you can use without first learning a lot of things !

2012/9/10 Lachlan Holmes <lachlan....@gmail.com>

lachlan@#######:~$ sudo zpool set dedup=off storage

[sehe]

That won't work. Once the DDT tables are there, they'll have to be loaded even after you disable dedup. A pool with dedup **ever** enabled will

always stay dedup-enabled on the storage level.

[sehe]

Okay then, you're probably running into the wellknown problem of excessively slow deletion under dedup. 

It didn't get 'hung in the process': it just takes forever to complete. Rebooting doesn't help, in fact it makes it worse: the pending IO transactions will be completed at pool import, causing the datasets to not-mount, precisely as you have found out.

Some later pool versions have alleviated some of the trouble with this. Also, adding more RAM may speed up considerably/immensely. If you care to try, you might bring the pool to a ZFS/zfsonlinux server box with ample RAM [1]. Let the transaction(s) run to completion and all should be fine.

I recommend against using dedup unless you absolutely need it. If you do, run on server hardware and learn lots about tuning and dimensioning up front as was suggested by others.

Cheers,

Seth

[1] (to benefit from more RAM and possibly later versions; chances are there won't be benefits from the ZFS version, since the pool is from another version. Beware: don't upgrade the pool version unless you don't need to access it using older zfs(-fuse) versions anymore!)

[Lachlan Holmes]

hey sehe,

this server is my personal nas box.

I let the server sit over night and today during work and when i came back home.... MY DATA IS BACK!!!!!

you were right about the excessively slow deletion under dedup and that's what caused it to hang...

i guess this issue is now well known to me know.

moving forward what settings should i put on the pool to not cause this again.

please note im not a linux admin im a windows admin by trade so be nice to me.

Im very happy its back... now to start the backup project!

lachlan. :)

[sehe]

> i guess this issue is now well known to me know.

I laughed :)

It's a sad state of affairs, really. Not undocumented, but still.

The best way is: create a pool and never enable dedup. Never at all.
(To recreate an existing pool, `zfs send` to backup media and
`receive` after creating a pool from scratch. Yes this is a painful
process).

Emmanuels repo contains a newer pool version (26), but IIRC zfs-
onlinux (pool version 28?) contains the real improvements that make
this problem less rough to deal with. (Note that it may also occurr
when a send/receive was aborted midflight, leaving many temporary
dataset holds; in particular this problem should be alleviated with
newer pool versions).

Good luck