OpenZFS ZFS-OSX (osx.zfs-signed-20131011.dmg): testing

[Graham Perrin]

Follow-up to <https://groups.google.com/d/msg/maczfs-devel/o11uAjEcHlE/EJzgJVAQYlIJ> (osx.zfs-20130930.dmg, tested 2013-10-05)

Tested today: 

a) pre-release golden master seeded build 13A598 of OS X 10.9

b) today's ZFS-OSX and a variety of other third party software

c) a busy ZFS home directory, Core Storage for encryption, sshd

Results today were initially comparable to results on the 5th. 

Again, the combination of a+b+c was unusable for me. Today, as on the 5th, one of the symptoms was similar to a symptom mentioned in the following issue: 

Desktop freezes while using zfs volume as home directory

<https://github.com/zfs-osx/zfs/issues/57>

– but as explained at <https://groups.google.com/d/msg/maczfs-devel/o11uAjEcHlE/mdPHV8OfnXUJ> it's inappropriate to reveal the details in my case. Sorry. 

Ultimately it became impossible to export the pool with ZFS-OSX so with reluctance, I applied force to a restart (or shutdown) of the Mac. 

All subsequent attempts to use the pool with ZEVO Community Edition 1.1.1, on OS X 10.8.5 Mountain Lion, resulted in kernel panics. 

After suppressing dynamic import, with ZEVO I forced an import without mounts then rolled back the file system that's my home directory. Not long afterwards, before logging in to that home directory, there was another panic and so I rolled back all three file systems within the pool. 

I'm now logged in to the affected home directory with ZEVO, scrubbing the pool. Sipping Amaretto and preparing myself for OZOO :-)

Reference material

==================

Panics after a test of ZFS-OSX osx.zfs-signed-20131011.dmg

<http://pastebin.com/9UGvFN09>

Suppression of dynamic import

<http://zevo.getgreenbytes.com/forum/viewtopic.php?p=5193#p5193>

– I'll ask a moderator to pin that topic

Attempting zpool xtreme_rewind - neither mount nor rewind

<http://pastebin.com/zwfxzesP>

– some uncertain experimentation by me, I doubt that  

    zpool import -fFnNX

  was a good combination of options

Experimenting with zdb

<http://pastebin.com/Nsvr16V4>

Comparison with a good pool that was last used with ZEVO

<http://pastebin.com/U8ZHyr5d>

A successful import without mounts

<http://pastebin.com/VRSSUMgv>

Home directory rollback with ZEVO, mounted without a panic

<http://pastebin.com/q1rfKf5A>

Another panic, and so: rollbacks of all three file systems

<http://pastebin.com/CSZApHvE>

[ilov...@icloud.com]

Cross-posting: http://zevo.getgreenbytes.com/forum/viewtopic.php?f=4&t=2178&p=5427#p5427

This was likely a corruption of your ZIL.

Similar issues and solutions described here:

https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-discuss/bD83Q1GR98k

https://github.com/zfsonlinux/zfs/issues/527

https://forums.oracle.com/thread/2295691

If you choose, you can resume testing now that you have dealt with your ZIL issue, as whatever other issues you encountered are likely entirely unrelated to the ZIL corruption, but will not resolve themselves. In particular, you need to run

sysctl -w zfs.vnops_osx_debug=1

and provide your Console logs.

[Jorgen Lundman]

> c) a busy ZFS home directory, Core Storage for encryption, sshd
>

I too have noticed that it can hang when used as home directory. The
challenge now is to try to find the minimal amount of steps required to
trigger this issue.

Lund

[Graham Perrin]

Thank you both. 

From the comparison <http://zevo.getgreenbytes.com/forum/viewtopic.php?p=5426#p5426> with the kernel panic under <http://zevo.getgreenbytes.com/forum/viewtopic.php?t=2218> (wonkywonky used a separate device for ZIL, I did not) I realised that the panics were ZIL-related, but I can't tell whether there's –

i) an issue with ZFS-OSX that allows corruption in exceptional circumstances

ii) an issue with ZEVO Community Edition 1.1.1 that may affect its ability to replay in exceptional circumstances

– or both. 

As I never before encountered panics of this nature, I *assume* an issue with ZFS-OSX. If so, then "this sort of damage should be impossible" but whilst 13A598 is pre-release I'll not raise an issue in the tracker for ZFS-OSX. 

Force

=====

<http://zevo.getgreenbytes.com/forum/viewtopic.php?p=4544#p4544> (2013-04-04) arose from discussion of panics that involved the DMU (not the ZIL). I may be wrong to generalise, but my take-home from topics such as these is: 

* whenever applying force to a restart or shut down of the Mac, there's a _slight_ risk of corruption to a pool

– and in ZEVO CE 1.1.1 cases we may expect kernel panics in response to assertion failures. 

Multiple issues

===============

It does make sense to treat as separate the issue(s) that affect uses of home directories. 

(There was a sequence of actions, but I don't assume that corruption was a direct result of attempting to log in to a ZFS home directory. Sequence, not consequence.)

Next steps

==========

Realistically, I'll not find the resources (hardware, time and so on) to attempt reproduction of the issues before OS X 10.9 is released. In the meantime, logs and so on are subject to the usual confidentiality agreement. 

I do have thoughts on how we might refine the issue(s) affecting home directories. I shouldn't add clutter to <https://github.com/zfs-osx/zfs/issues/57> so I'll spin off to a separate topic. 

Thanks again

Graham