Can't boot on ZFS -- /boot/zfsloader not found

[Jeremie Le Hen]

Hi,

I'm in the process of transferring my home server to an Intel NUC. I've zfs
send/receive'd the whole pool to the new drive and first followed
[ZfsOnGpt] but it fails in the first stage boot loader. The USB keyboard
doesn't seem to be recognized so I can't really debug anything.

Then I tried both [ZfsInMbrSlice]
but it complains that it can't find /boot/zfsloader. Again USB keyboard
doesn't work.

Both times I've been very careful to install the bootcodes at the right
place.

1. Any idea why the USB keyboard doesn't work?
2. Any idea what can be wrong in the setups I tried? How can I debug this?

[ZfsOnGpt] https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot
[ZfsInMbrSlice] https://wiki.freebsd.org/RootOnZFS/ZFSBootPartition

-jlh
_______________________________________________
freebsd...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"

[Steven Hartland]

As your not at the boot loader stage yet for keyboard enabling legacy
USB keyboard / mouse support in BIOS may help.

If you see issues with keyboard after the kernel takes over setting
hint.ehci.0.disabled=1 may help.

If your installing 11.x then the guides boot partition size is out of
date and you should use 512K for your freebsd-boot partition.

For the GPT version (I've never used MBR one) what actual error do you get?

[Jeremie Le Hen]

(Sorry I had to copy-paste this email from the archives to a new
thread, because I'm not subscribed to -stable@. Would you mind cc:ing
me next time please?)

> As your not at the boot loader stage yet for keyboard enabling legacy
> USB keyboard / mouse support in BIOS may help.

That worked, thanks for the tip.

> If you see issues with keyboard after the kernel takes over setting
> hint.ehci.0.disabled=1 may help.
>
> If your installing 11.x then the guides boot partition size is out of
> date and you should use 512K for your freebsd-boot partition.

Oh, really? I'll update it once I'll have figure it all out. That's
weird though because /boot/gptzfsboot is less than 90 KB.

> For the GPT version (I've never used MBR one) what actual error do you get?

When I just start up the NUC, I get:
"""
\Can't find /boot/zfsloader

FreeBSD/x86 boot
Default: zroot:/boot/kernel/kernel
boot:
\
Can't find /boot/kernel/kernel
"""

I though it was the first stage boot loader, but it may be the second stage.

-jlh

>
> On 11/01/2017 19:22, Jeremie Le Hen wrote:
> > Hi,
> >
> > I'm in the process of transferring my home server to an Intel NUC. I've zfs
> > send/receive'd the whole pool to the new drive and first followed
> > [ZfsOnGpt] but it fails in the first stage boot loader. The USB keyboard
> > doesn't seem to be recognized so I can't really debug anything.
> >
> > Then I tried both [ZfsInMbrSlice]
> > but it complains that it can't find /boot/zfsloader. Again USB keyboard
> > doesn't work.
> >
> > Both times I've been very careful to install the bootcodes at the right
> > place.
> >
> > 1. Any idea why the USB keyboard doesn't work?
> > 2. Any idea what can be wrong in the setups I tried? How can I debug this?
> >
> > [ZfsOnGpt] https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot
> > [ZfsInMbrSlice] https://wiki.freebsd.org/RootOnZFS/ZFSBootPartition
> >
> > -jlh
> > _______________________________________________

> > freebsd-stable at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"


--
Jeremie Le Hen
j...@FreeBSD.org

[Steven Hartland]

On 11/01/2017 22:58, Jeremie Le Hen wrote:
> (Sorry I had to copy-paste this email from the archives to a new
> thread, because I'm not subscribed to -stable@. Would you mind cc:ing
> me next time please?)
>
>> As your not at the boot loader stage yet for keyboard enabling legacy
>> USB keyboard / mouse support in BIOS may help.
> That worked, thanks for the tip.
>
>
>> If you see issues with keyboard after the kernel takes over setting
>> hint.ehci.0.disabled=1 may help.
>>
>> If your installing 11.x then the guides boot partition size is out of
>> date and you should use 512K for your freebsd-boot partition.
> Oh, really? I'll update it once I'll have figure it all out. That's
> weird though because /boot/gptzfsboot is less than 90 KB.

11.x
du -h /boot/gptzfsboot
97K /boot/gptzfsboot

12.x
du -h /boot/gptzfsboot
113K /boot/gptzfsboot

The reason I'd recommend 512k for boot is to provide room for expansion
moving forward, as repartitioning to upgrade is a scary / hard thing to
do. Remember it wasn't long ago when it was well under 64k and that's
what was recommend, its not like with disk sizes these days you'll miss
the extra 384k ;-)

>
>
>> For the GPT version (I've never used MBR one) what actual error do you get?
> When I just start up the NUC, I get:
> """
> \Can't find /boot/zfsloader
>
> FreeBSD/x86 boot
> Default: zroot:/boot/kernel/kernel
> boot:
> \
> Can't find /boot/kernel/kernel
> """
>
> I though it was the first stage boot loader, but it may be the second stage.

Boot to a live cd, I'd recommend mfsbsd, and make sure the boot loader
was written to ALL boot disks correctly e.g.
if you have a mirrored pool with ada0 and ada1:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

If this doesn't help the output from gpart show, uname -a and zpool
status would also be helpful.

This is all assuming standard BIOS mode and not UEFI which is done
differently.

>
> -jlh
>
>> On 11/01/2017 19:22, Jeremie Le Hen wrote:
>>> Hi,
>>>
>>> I'm in the process of transferring my home server to an Intel NUC. I've zfs
>>> send/receive'd the whole pool to the new drive and first followed
>>> [ZfsOnGpt] but it fails in the first stage boot loader. The USB keyboard
>>> doesn't seem to be recognized so I can't really debug anything.
>>>
>>> Then I tried both [ZfsInMbrSlice]
>>> but it complains that it can't find /boot/zfsloader. Again USB keyboard
>>> doesn't work.
>>>
>>> Both times I've been very careful to install the bootcodes at the right
>>> place.
>>>
>>> 1. Any idea why the USB keyboard doesn't work?
>>> 2. Any idea what can be wrong in the setups I tried? How can I debug this?
>>>
>>> [ZfsOnGpt] https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot
>>> [ZfsInMbrSlice] https://wiki.freebsd.org/RootOnZFS/ZFSBootPartition
>>>
>>> -jlh
>>> _______________________________________________
>>> freebsd-stable at freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
>>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
>

[Jeremie Le Hen]

Hey Steven,

(Please cc: me on reply)

On Thu, Jan 12, 2017 at 1:32 AM, Steven Hartlan

> The reason I'd recommend 512k for boot is to provide room for expansion
> moving forward, as repartitioning to upgrade is a scary / hard thing to do.
> Remember it wasn't long ago when it was well under 64k and that's what was
> recommend, its not like with disk sizes these days you'll miss the extra
> 384k ;-)

Yeah, that's wise you're right.

> Boot to a live cd, I'd recommend mfsbsd, and make sure the boot loader was
> written to ALL boot disks correctly e.g.
> if you have a mirrored pool with ada0 and ada1:
> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
>
> If this doesn't help the output from gpart show, uname -a and zpool status
> would also be helpful.
>
> This is all assuming standard BIOS mode and not UEFI which is done
> differently.

I just use the installation media on an USB key and then drop to the
shell. This is a full FreeBSD running, so that's fine.

% # gpart show ada0
% => 40 312581728 ada0 GPT (149G)
% 40 1024 1 freebsd-boot (512K)
% 1064 8387840 2 freebsd-swap (4.0G)
% 8388904 304192864 3 freebsd-zfs (145G)
%
% # uname -a
% FreeBSD 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0 r306420: Thu Sep
29 01:43:23 UTC 2016 % %
ro...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
%
% # zpool status
% pool: zroot
% state: ONLINE
% scan: none requested
% config:
%
% NAME STATE READ
WRITE CKSUM
% zroot ONLINE 0
0 0
% gptid/1c387d3b-d892-11e6-944b-f44d30620eeb ONLINE 0
0 0
%
% errors: No known data errors

Here are the steps to write the bootloader:

% # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
% partcode written to ada0p1
% bootcode written to ada0
% # zpool get bootfs zroot
% NAME PROPERTY VALUE SOURCE
% zroot bootfs zroot local

--
Jeremie Le Hen
j...@FreeBSD.org

[Steven Hartland]

Two things spring to mind

Idea 1:
Is your root fs actually your direct pool or is it actually /root off
your pool.
If so you want to run:
zpool set bootfs=zroot/root zroot

Idea 2:
You mentioned in your original post and you used zfs send / recv to
restore the pool, so I wonder if your cache file is out of date.

Try the following:
|zpool export zroot
zpool import -R /mnt -o cachefile=/boot/zfs/zpool.cache zroot
cp /boot/zfs/zpool.cache /mnt/boot/zfs/zpool.cache
zpool set bootfs=zroot/root zroot

Regards
Steve

[Jeremie Le Hen]

That was it. My boot volume is "zroot/root" and I was just following
brainlessly the doc which said to use "zroot". This didn't shock me
because this at least points to the right pool

>
> Idea 2:
> You mentioned in your original post and you used zfs send / recv to restore
> the pool, so I wonder if your cache file is out of date.
>
> Try the following:
> zpool export zroot
> zpool import -R /mnt -o cachefile=/boot/zfs/zpool.cache zroot
> cp /boot/zfs/zpool.cache /mnt/boot/zfs/zpool.cache
> zpool set bootfs=zroot/root zroot

I think that was wrong too, but this would probably have caused
problems later in the boot I think.

Anyway, thanks a lot for your help. I'll give a quick lifting to the wiki page.

-jlh

>
> Regards
> Steve

--
Jeremie Le Hen
j...@FreeBSD.org