zfs_enable vs zfs_load in loader.conf (but neither works)

[J David]

After setting up a new machine to boot from a ZFS root using the 9.1
install, it worked fine, but when the kernel & world was updated to
releng/9.2, it stopped booting. The pool is called "data" and the
root partition is "data/root."

Under 9.1 it had in loader.conf:

zfs_load="YES"
vfs.root.mountfrom="zfs:data/root"

Under 9.2-RC3, the same config results in a panic:

Trying to mount root from zfs:data/root []…
init: not found in path
/sbin/init:/sbin/oinit:/sbin/init.bak:/rescue/init:/stand/sysinstall
panic: no init

If this is changed (as many Google hits recommend) to:

zfs_enable="YES"
vfs.root.mountfrom="zfs:data/root"

It seems like ZFS doesn't get loaded, so it fails instead with:

Trying to mount root from zfs:data/root []…
Mounting from zfs:data/root failed with error 2: unknown file system.

If the "?" mountroot> option is used, 50 devices are listed, none of
which are ZFS. And the "unknown file system" response comes from
vfs_byname returning NULL for zfs.

(If both zfs_enable and zfs_load are set to "YES" then it fails as the
zfs_load case.)

The system is using update-to-date zpool (v5000 / feature flags), and
all the updated bootblocks from the releng/9.2 build. zpool.cache is
correct, the zpool imports fine from the 9.2-RC3 live cd. The zpool's
bootfs is set correctly, the zfs mountpoint of data/root is / . And,
of course, init is present and health in data/root. The system booted
fine until updating to 9.2.

Which loader.conf entry is actually correct for ZFS roots on 9.2, and
what (else) needs to happen to make this system bootable again?

Thanks for any advice!
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"

[Darren Pilgrim]

On 9/8/2013 6:02 PM, J David wrote:
> Trying to mount root from zfs:data/root []…
> Mounting from zfs:data/root failed with error 2: unknown file system.

Did you build and install new boot blocks?

[J David]

On Sun, Sep 8, 2013 at 10:22 PM, Darren Pilgrim
<list_f...@bluerosetech.com> wrote:
> Did you build and install new boot blocks?

Yes.

Oddly, setting:

zfs set mountpoint=legacy data/root (plus the appropriate fstab entry)

instead of

zfs set mountpoint=/ data/root

seems to produce a bootable system, although it absolutely should not
be necessary to do things that way anymore.

Weird.

[Darren Pilgrim]

On 9/8/2013 7:52 PM, J David wrote:
> On Sun, Sep 8, 2013 at 10:22 PM, Darren Pilgrim
> <list_f...@bluerosetech.com> wrote:
>> Did you build and install new boot blocks?
>
> Yes.
>
> Oddly, setting:
>
> zfs set mountpoint=legacy data/root (plus the appropriate fstab entry)

You can use zfs.root.mountfrom="zfs:data/root" in /boot/loader.conf
instead of an fstab entry. Mountpoint=legacy is required either way.

> instead of
>
> zfs set mountpoint=/ data/root

This only applies to Solaris, IIRC.

> seems to produce a bootable system, although it absolutely should not
> be necessary to do things that way anymore.

I ran into that problem as well. The instructions for root-on-zfs for
9.x (at least as of 9.1) are wrong--you need to use the 8.x-style
instructions with mountpoint=legacy for / and, for fresh installs,
leaving the pool imported and copying over /boot/zfs/zpool.cache.

[krad]

once you have it all working and understood have a look at the following
port ''usr/ports/sysutils/beadm'' It may make things a little easier to
manage in the future. In my experience BE's on zfs rock.

> freeb...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-...@freebsd.org"

[J David]

On Sun, Sep 8, 2013 at 11:11 PM, Darren Pilgrim
<list_f...@bluerosetech.com> wrote:
> You can use zfs.root.mountfrom="zfs:data/root" in /boot/loader.conf instead
> of an fstab entry.

That has been in loader.conf the whole time.

> Mountpoint=legacy is required either way.

It isn't. There is another machine right next to it running 9.2-RC1
and it works fine with the mountpoint=/ setting and an empty fstab.

And it worked fine on *this* machine on 9.1. It's not clear what
changed to stop it from working after upgrading to 9.2-RC3.

If time permits, we will update our PXE install environment to 9.2 and
rebuild the whole thing. That would eliminate the upgrade step and
hopefully give a result others could reproduce.

> This only applies to Solaris, IIRC.

[citation needed]

Setting mountpoint=/ is necessary in order for "zpool import -o
altroot=/mnt data" to do the right thing at maintenance time.

Thanks!

[Shane Ambler]

On 09/09/2013 21:20, J David wrote:
> On Sun, Sep 8, 2013 at 11:11 PM, Darren Pilgrim
> <list_f...@bluerosetech.com> wrote:
>> You can use zfs.root.mountfrom="zfs:data/root" in /boot/loader.conf
>> instead of an fstab entry.
>
> That has been in loader.conf the whole time.
>
>> Mountpoint=legacy is required either way.
>
> It isn't. There is another machine right next to it running 9.2-RC1
> and it works fine with the mountpoint=/ setting and an empty fstab.
>

I installed 9.0 onto my machine booting from zfs about a year and a half
ago and remember having issues getting it bootable. As I recall
mounpoint=legacy and mountpoint=/ effectively point to two different
filesystems.

Changing the mounpoint after installing hides the / filesystem.

So it isn't so much which mountpoint to use but which mountpoint *was*
used when you installed the system.

[Andriy Gapon]

First, a note that below I try to reply not only to this specific message but to
the whole thread.

on 09/09/2013 04:02 J David said the following:

> After setting up a new machine to boot from a ZFS root using the 9.1
> install, it worked fine, but when the kernel & world was updated to
> releng/9.2, it stopped booting. The pool is called "data" and the
> root partition is "data/root."
>
> Under 9.1 it had in loader.conf:
>
> zfs_load="YES"
> vfs.root.mountfrom="zfs:data/root"
>
> Under 9.2-RC3, the same config results in a panic:
>
> Trying to mount root from zfs:data/root []…
> init: not found in path
> /sbin/init:/sbin/oinit:/sbin/init.bak:/rescue/init:/stand/sysinstall
> panic: no init

This is a very weird error. It means that kernel was able to mount data/root as
a root filesystem, but couldn't find /sbin/init in it.
Which can mean at least two different things: (1) some other filesystem was
mounted instead of data/root because of some bug; (2) your data/root didn't
actually contain valid FreeBSD installation.

I set up a test system exactly the way you described above and I can not
reproduce this behavior.
Just in case, I used mfsbsd zfsinstall and that's how it creates and configures
a pool by default.

> If this is changed (as many Google hits recommend) to:
>
> zfs_enable="YES"

I think that this was discussed enough in the thread and the right conclusions
have been already reached.
I just have two general comments:
- you don't have to trust everything that is written "on the internet". Prefer
to use more or less authoritative sources: FreeBSD documentation, FreeBSD wiki,
posts by FreeBSD developers and alike
- it surprises me how many people who don't understand how the code works feel
that they can give advices to other people
- when *I* used the following query
https://www.google.com/search?q=%22zfs_enable%22+%22loader.conf%22
I could not find a single suggestion to put zfs_enable into loader.conf in the
first dozen of results (references to this thread excluded)

> vfs.root.mountfrom="zfs:data/root"
>
> It seems like ZFS doesn't get loaded, so it fails instead with:
>
> Trying to mount root from zfs:data/root []…
> Mounting from zfs:data/root failed with error 2: unknown file system.
>
> If the "?" mountroot> option is used, 50 devices are listed, none of
> which are ZFS. And the "unknown file system" response comes from
> vfs_byname returning NULL for zfs.

Obvious (as already established).

> (If both zfs_enable and zfs_load are set to "YES" then it fails as the
> zfs_load case.)

Obvious (as already established).

> The system is using update-to-date zpool (v5000 / feature flags), and
> all the updated bootblocks from the releng/9.2 build. zpool.cache is
> correct, the zpool imports fine from the 9.2-RC3 live cd. The zpool's
> bootfs is set correctly, the zfs mountpoint of data/root is / . And,
> of course, init is present and health in data/root. The system booted
> fine until updating to 9.2.

I just wish that I could reproduce this problem using exactly the same setup...
But I can't.
Perhaps there are any other special things about your configuration - like
having other pools or other disks/partitions that are not used by 'data' pool.
Any other non-standard things...


--
Andriy Gapon

[Andriy Gapon]

Now some high level information on how ZFS boot works and a little bit more
detailed information on how a root filesystem is chosen in ZFS case. The
information is applicable to recent versions of FreeBSD in head, stable/9
(including upcoming 9.2) and stable/8 (including 8.4).

- boot0-like stage always takes boot2-like stage from the same disk using simple
rules
- boot2-like stage probes all disks and partitions it can understand for ZFS pools
- default pool is the first pool detected by probing which starts at boot disk
- default filesystem is determined by bootfs property
- boot2-like stage allows to select a different pool, a specific filesystem in
the pool and a specific loader

boot0-like stage is pmbr in the case of GPT partitioning.
boot0-like stage is the first block of zfsboot in the case of whole-disk ZFS.
boot2-like stage is either gptzfsboot or zfsboot correspondingly.

- loader uses boot pool and filesystem information passed by boot2-like stage
- loader exposes loaddev and currdev variables, initially they point to the pool
and filesystem obtained from boot2-like stage
- currdev can be changed (e.g. at the prompt) while loaddev is read only
- kernel and modules are loaded from currdev by default

- kernel mounts root from a filesystem specified by vfs.root.mountfrom variable
that is passed by loader to kenv
- value of the variable is determined as follows:
- loader tries to set this variable based on "/" entry, if any, in /etc/fstab,
if any, in the filesystem specified by currdev
- the variable can be explicitly set in loader.conf or at the prompt; the
explicit assignment overrides the fstab-based auto-detected value
- for ZFS, if the above methods do not produce any value, vfs.root.mountfrom is
set based on currdev

So, you can see that all three methods mentioned in this thread can work equally
well. You can either specify a root entry in fstab, or set vfs.root.mountfrom
in loader.conf, or simply set bootfs property. The above information also
describes precedence rules if more than one knob is used: vfs.root.mountfrom is
the most significant, fstab is after it, bootfs plays role in root filesystem
selection only if neither of the previous is set.
Thus, it's completely up to you which method to use. Whichever is more
convenient. I prefer to just set bootfs.

Another piece of information is that neither mountpoint nor canmount property
affects ZFS root mounting. They of course have their usual effect in other
contexts like importing a pool on a different system or when a different
filesystem is selected to be a root filesystem.
So, again, you can set these properties to whatever is most convenient for you.

[J David]

Thanks very much for the info Andriy.

On Fri, Sep 13, 2013 at 9:22 AM, Andriy Gapon <a...@freebsd.org> wrote:
> Another piece of information is that neither mountpoint nor canmount property
> affects ZFS root mounting.

It is mountpoint=legacy that boots on this machine and mountpoint=/
that can't find init, with no other changes. So clearly under some
obscure edge case, this is not strictly correct.

Did your test include zpool root != filesystem root? Because as you
describe one possible cause of the problem is mounting the wrong
filesystem as root, one wonders if somehow with the mountpoint=/
setting the zpool root (which has no files at all) is incorrectly
being chosen as fsroot with mountpoint=/ on data/root. I.e. perhaps
somewhere in the code is looking for "legacy" (or skipping anything
with a mountpoint set) and defaulting back to the zpool root if it is
not found?

Unfortunately there is no way I know of to check and see what the root
filesystem turned out to be after the failure to find init. That
would reduce speculation about what is happening quite a bit. Can the
kernel debugger extract this info?

If it matters, the pools in question are fairly complex, not just
throwing everything possible on one drive. Example:

NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da0p2 ONLINE 0 0 0
da4p2 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
da1p2 ONLINE 0 0 0
da5p2 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
da2p2 ONLINE 0 0 0
da6p2 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
da3p2 ONLINE 0 0 0
da7p2 ONLINE 0 0 0
logs
ada0p2 ONLINE 0 0 0
cache
ada1p2 ONLINE 0 0 0

The boot order for that system begins at either ada0 or da0, not sure
which without checking the BIOS.

Thanks!

[Andriy Gapon]

on 15/09/2013 17:16 J David said the following:

> Thanks very much for the info Andriy.
>
> On Fri, Sep 13, 2013 at 9:22 AM, Andriy Gapon <a...@freebsd.org> wrote:
>> Another piece of information is that neither mountpoint nor canmount property
>> affects ZFS root mounting.
>
> It is mountpoint=legacy that boots on this machine and mountpoint=/
> that can't find init, with no other changes. So clearly under some
> obscure edge case, this is not strictly correct.

I am sure that the same can be said about almost anything documented about any
software...

> Did your test include zpool root != filesystem root? Because as you
> describe one possible cause of the problem is mounting the wrong
> filesystem as root, one wonders if somehow with the mountpoint=/
> setting the zpool root (which has no files at all) is incorrectly
> being chosen as fsroot with mountpoint=/ on data/root. I.e. perhaps
> somewhere in the code is looking for "legacy" (or skipping anything
> with a mountpoint set) and defaulting back to the zpool root if it is
> not found?

I tried to reproduce with exactly the same configuration as you described.
To be specific, yes I used data/root as a root filesystem and I set its
mountpoint to "/".

> Unfortunately there is no way I know of to check and see what the root
> filesystem turned out to be after the failure to find init. That
> would reduce speculation about what is happening quite a bit. Can the
> kernel debugger extract this info?

Unfortunately, I am not sure if it is possible to obtain anu useful information
from ddb and saving a crash dump is not possible in pre-init environment.
I could write a patch that would print some useful debugging info.
Will you able to use it?

--
Andriy Gapon

[Andriy Gapon]

on 16/09/2013 07:42 kpn...@pobox.com said the following:
> What happens if mountpoint is inherited instead of being set to one of
> those two values?

Would you like to test this and tell us?

I am 99.9% confident that mountpoint and canmount properties are never examined
in kernel. They are honored only by zfs(8) utility. Thus, they can not
possibly influence root mounting. Modulo very very very obscure bugs, of course.

--
Andriy Gapon

[J David]

On Sun, Sep 15, 2013 at 11:13 AM, Andriy Gapon <a...@freebsd.org> wrote:
> Unfortunately, I am not sure if it is possible to obtain anu useful information
> from ddb and saving a crash dump is not possible in pre-init environment.
> I could write a patch that would print some useful debugging info.
> Will you able to use it?

Yes, if it applies cleanly to releng/9.2.

Thanks!