[Bug 209096] zfsroot bricked on 10.3-RELEASE

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

Mark Linimon <lin...@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Assignee|freebs...@FreeBSD.org |freeb...@FreeBSD.org

--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
freeb...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-...@freebsd.org"

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

Kubilay Kocak <ko...@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
Flags| |mfc-stable10?,
| |mfc-stable11?
Priority|--- |Normal
Severity|Affects Only Me |Affects Many People
Keywords| |needs-patch, needs-qa
CC| |ko...@FreeBSD.org
Status|New |Open

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

Kubilay Kocak <ko...@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------

See Also| |https://bugs.freebsd.org/bu
| |gzilla/show_bug.cgi?id=2049
| |76

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

Steven Hartland <s...@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |s...@FreeBSD.org

--- Comment #1 from Steven Hartland <s...@FreeBSD.org> ---
Can you check to messages during the boot with the new kernel to see if you see
mfid0 created?

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #2 from Daniel Ylitalo <dan...@blodan.se> ---
Hmm, is there anyway to enable verbose booting or something?

Or where would I look for this message?

Nothing is outputted by the boot loader before the "ZFS: i/o error all block
copies unavailable" message and I'm dropped to boot:

(This is before the kernel is being loaded)

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #3 from Steven Hartland <s...@FreeBSD.org> ---
So this was a loader message not a kernel message?

If so can you confirm what boot method your using I'm guessing bios not efi?

If so then did you update your boot blocks e.g.
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid0

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #4 from Daniel Ylitalo <dan...@blodan.se> ---
No, I havent tried updating the bootcode, should that be necessary between
release upgrades?

(10.2 is running fine)

If you want I can give that a try from the 10.3 live cd in about two weeks
because right now the failover server for that cluster is offline so I can't
take this server down to test with until the failover server is back online.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #5 from Daniel Ylitalo <dan...@blodan.se> ---
Sorry, missed answering your first question, yes bios boot and not efi

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #6 from Steven Hartland <s...@FreeBSD.org> ---
Yep give that a go.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #7 from Daniel Ylitalo <dan...@blodan.se> ---
That kind of made things worse, now it won't boot even if I copy back the /boot
folder from the live cd

I can still import the zroot pool from both the 10.2 and 10.3 live cd

I'm also attaching a screenshot if that tells you anything.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #8 from Daniel Ylitalo <dan...@blodan.se> ---
Created attachment 173189
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=173189&action=edit
screenshot of boot error

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #9 from Steven Hartland <s...@FreeBSD.org> ---
Is your pool configured with blocks > 128K as that isn't supported for booting
from.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #10 from Daniel Ylitalo <dan...@blodan.se> ---
How can I see that?

This pool is created from the 10.2 installer using the zfsroot option (so it
should not have anything not supported)

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #11 from Steven Hartland <s...@FreeBSD.org> ---
The following should show you
zpool get feature@large_blocks

Note that setting recordsize on dataset is the trigger between enabled and
active for this.

It may be the message in your screenshot is invalid but worth checking.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #12 from Steven Hartland <s...@FreeBSD.org> ---
Also whats the output of:
zdb -C
and:
zpool status

For zdb if you're booted from an alternative medium you may need to use -U to
ensure it uses the cache file from your main pool.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #13 from Andriy Gapon <a...@FreeBSD.org> ---
Can you also capture a full output of lsdev -v command issued at the loader
boot prompt? Your problem may have to do with the firmware giving a wrong
value for disk sizes (especially given that your disk seems to be larger than 2
TB).

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #14 from Daniel Ylitalo <dan...@blodan.se> ---
This is what it has;

NAME PROPERTY VALUE SOURCE
zroot feature@large_blocks enabled local

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #15 from Daniel Ylitalo <dan...@blodan.se> ---
Created attachment 173192
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=173192&action=edit
zdb -C output

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #16 from Daniel Ylitalo <dan...@blodan.se> ---
Hmm, the boot loader wont let me run lsdev, attaching screenshot

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #17 from Daniel Ylitalo <dan...@blodan.se> ---
Created attachment 173193
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=173193&action=edit
lsdev -v not found

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #18 from Daniel Ylitalo <dan...@blodan.se> ---
You have the zpool status output in my initial post

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #19 from Steven Hartland <s...@FreeBSD.org> ---
Whats the content of /boot/loader.conf from the pool?

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #20 from Daniel Ylitalo <dan...@blodan.se> ---
Here is the loader.conf;

kern.geom.label.gptid.enable="0"
zfs_load="YES"
vfs.zfs.arc_max="4294967296"

net.inet.tcp.tcbhashsize="32768"
net.inet.tcp.hostcache.bucketlimit="128"
net.inet.tcp.hostcache.hashsize="16384"
net.inet.tcp.syncache.hashsize="16384"
net.inet.tcp.syncache.bucketlimit="128"
net.link.ifqmaxlen="2048"

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #21 from Daniel Ylitalo <dan...@blodan.se> ---
Any suggestions? :)

Or am I better off just reinstalling this machine tomorrow with regular ufs as
it's a single drive anyway

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #22 from Andriy Gapon <a...@FreeBSD.org> ---
(In reply to Daniel Ylitalo from comment #17)
Ah, sorry, you are not even getting to the loader which does support lsdev.
You are stuck at zfsboot stage.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #23 from Andriy Gapon <a...@FreeBSD.org> ---
(In reply to Daniel Ylitalo from comment #21)

You can check current location of zroot's MOS like this:
zdb -dddd zroot 0
If this command fails, try prepending -e before other options.

You can also try to check the pool with zfsboottest tool if you have /usr/src
checkout:
cd /usr/src/tools/tools/zfsboottest/
make
./zfsboottest.sh zroot

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #24 from Daniel Ylitalo <dan...@blodan.se> ---
Created attachment 173211
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=173211&action=edit
zdb mos output

I have no idea what the output is telling me but it looks good atleast :)

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #25 from Daniel Ylitalo <dan...@blodan.se> ---
Created attachment 173212
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=173212&action=edit
zfsboottest output

zfsboottest sais everything is ok :(

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #26 from Daniel Ylitalo <dan...@blodan.se> ---
Anyone of you have a verbose version of the boot loader i can install/apply so
that we can see whats going on perhaps?

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #27 from Andriy Gapon <a...@FreeBSD.org> ---
(In reply to Daniel Ylitalo from comment #26)
Based on the latest information that you have provided and the history of your
problem I think that the problem is caused by the BIOS or other firmware
components in your system.

So, the next step would be to check if there are newer versions of BIOS and
firmware for the disk controller and see if upgrading fixes the issue.

If that does not help, or just as an alternative solution, you can break up
your freebsd-zfs partition in two such that the first one does not cross 2 TiB
boundary (from the start of the disk). Then you can create a boot / root pool
on top of that partition. You were prepared to go back to UFS, so this could
allow you to keep using ZFS, but at the cost of moving data.
BTW, if you go with this solution and you get a bootable system, then I still
would like to see output of lsdev -v from the loader prompt (which you get by
selecting "escape to propmpt" option from the loader menu).

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #28 from Daniel Ylitalo <dan...@blodan.se> ---
I'll do the split setup just for testing as the server is going to be
reinstalled anyhow so can reinstall it two times :P

Strange that the zroot works fine on 10.2 though and not on 10.3

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

ka...@denninger.net changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |ka...@denninger.net

--- Comment #29 from ka...@denninger.net ---
(In reply to Daniel Ylitalo from comment #28)

Not really if the disk is > 2Tb. On original install the kernel was likely in
the first 2Tb, but once you start filling the disk the new write of the new
/boot/kernel directory may go into blocks beyond that boundary, and now it's
unreadable at boot time.

This can happen on a UFS disk too.... not sure if that's what's going on but
it's certainly possible.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #30 from Daniel Ylitalo <dan...@blodan.se> ---
hehe, good to know, then I will stop updating our content servers that doesnt
run freebsd on a sata dom if thats the case :)

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #31 from ka...@denninger.net ---
(In reply to Daniel Ylitalo from comment #30)

This has ALWAYS been dangerous on a disk that is >2Tb; the solution is to place
the boot (or boot/root, if they're on the same filesystem) within the first 2Tb
and do *not* allow the size of that filesystem to cross the 2Tb boundary. This
is an especially nasty problem because it appears that all is ok and is for a
while, right up until the system starts allocating space beyond the boundary
when you do an update... then you are suddenly hosed as the machine will not
boot.

It is safe to have root/boot on a disk larger than 2Tb, but *not* to allow the
size of that filesystem to extend across that boundary.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #32 from Daniel Ylitalo <dan...@blodan.se> ---
Looks very much like it was the "outside 2tb" issue as I just wiped the
existing partitions and did a zfsroot reinstall with the 10.3 installer and it
booted up fine.

Sorry for hassling you guys about it.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #33 from Daniel Ylitalo <dan...@blodan.se> ---
Created attachment 173256
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=173256&action=edit
Working boot lsdev -v

Attaching requested lsdev -v output of a working boot as requested

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #34 from Steven Hartland <s...@FreeBSD.org> ---
Thanks for confirming think we should consider adding checks / warning in for
this as the messages provided are particularly poor in helping to identify the
real issue at hand.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

Daniel Ylitalo <dan...@blodan.se> changed:

What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |Works As Intended
Status|Open |Closed

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #35 from Andriy Gapon <a...@FreeBSD.org> ---
(In reply to Daniel Ylitalo from comment #33)
Hmm, so it seems that lsdev -v does not report the disk size as I've expected.
So it's not that useful :-(

BTW, it seems that you are still using a single huge partition for ZFS...
If you split it into the under 2TB and above 2TB partitions, then you could
easily test the theory by making the >2TB partition a root ZFS pool and
checking if you can boot to it.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096

--- Comment #36 from Daniel Ylitalo <dan...@blodan.se> ---
(In reply to Andriy Gapon from comment #35)

Yeah, I couldn't really figure out how to split the partitions in the installer
so I just went with the full disk install just to see if that would even work
and it did.

I've put a sata dom 32gb in the server to install freebsd on to just make it
simple (thats how our newer servers operate anyhow), so I will be going with
regular ufs on the large mfi array as it's running through a raid controller
already, we were just testing zfsroot to see how well it worked out on that
server.

But if you really want me to I can give it a go if you could give me a few
pointers on how to create the two partitions and then tell the installer to
install on the second one

[Bruce Evans]

On Wed, 3 Aug 2016 a bug that doesn't want rep...@freebsd.org wrote:

> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209096
>
> --- Comment #35 from Andriy Gapon <a...@FreeBSD.org> ---
> (In reply to Daniel Ylitalo from comment #33)
> Hmm, so it seems that lsdev -v does not report the disk size as I've expected.
> So it's not that useful :-(
>
> BTW, it seems that you are still using a single huge partition for ZFS...
> If you split it into the under 2TB and above 2TB partitions, then you could
> easily test the theory by making the >2TB partition a root ZFS pool and
> checking if you can boot to it.

Old BIOSes break at 128GB. Of course, you shouldn't use zfs on systems
with such a small limit.

I have a 10 year old laptop, which is not very old for me. Changing its
disk from 100 GB to 750 GB worked well except when I tried to move
partitions above the 128GB boundary on it. This was confusing, although
I have a lot of experience keeping partitions below 8GB so that they were
bootable with the 1989 boot0 that I used to use, and had rewritten this
boot0 to remove its limit, and was switching systems to use it.

Bruce