Re: bin/144214: zfsboot fails on gang block after upgrade to zfs v14
Martin Matuska
From: Martin Matuska <m...@FreeBSD.org>
To: bug-fo...@FreeBSD.org, c.k...@gmail.com
Cc:
Subject: Re: bin/144214: zfsboot fails on gang block after upgrade to zfs
v14
Date: Thu, 29 Apr 2010 02:16:24 +0200
This bug is still actual.
We had to forcibly reboot a server with a 97% full zpool (~2GB free space)
and we came to a ZFS: gang block detected
The workaround with re-creating and re-populating /boot worked.
_______________________________________________
freeb...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-...@freebsd.org"
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-...@muc.de
Daniel Gerzo
From: Daniel Gerzo <dan...@FreeBSD.org>
To: bug-fo...@FreeBSD.org, c.k...@gmail.com
Cc:
Subject: Re: bin/144214: zfsboot fails on gang block after upgrade to zfs
v14
Date: Thu, 29 Apr 2010 02:11:16 +0200
Hello,
this bug seems to be still present in stable/8. The proposed workaround
seems to work. You may find the console screenshot at
http://danger.rulez.sk/dockdrop/144214.png
--
S pozdravom / Best regards
Daniel Gerzo, FreeBSD committer
Daniel Geržo
From: =?windows-1252?Q?Daniel_Ger=9Eo?= <dan...@rulez.sk>
To: bug-fo...@FreeBSD.org, c.k...@gmail.com
Cc:
Subject: Re: bin/144214: zfsboot fails on gang block after upgrade to zfs
v14
Date: Thu, 29 Apr 2010 02:13:42 +0200
Hi,
note that the HDD has been almost full (97%) when the box died (ca. 2GB
free).
--
S pozdravom / Best regards
Daniel Gerzo
Andriy Gapon
From: Andriy Gapon <a...@icyb.net.ua>
To: bug-fo...@FreeBSD.org
Cc:
Subject: Re: bin/144214: zfsboot fails on gang block after upgrade to zfs
v14
Date: Thu, 29 Apr 2010 15:10:28 +0300
Just to be on the sure side: have you guys actually updated bootblocks on your system?
I.e. the code that runs before loader and that resides beyond filesystems.
--
Andriy Gapon
Volodymyr Kostyrko
> Just to be on the sure side: have you guys actually updated bootblocks on your system?
> I.e. the code that runs before loader and that resides beyond filesystems.
Yes, I do.
Actually I was hit by the same bug recently one more time. After rebuild
machine refuses to boot because of some gang blocks. I've tried to use
i386 for the restoration purpose and set up a virtualbox in which I more
then ten times tried to rewrite /boot and reboot from pool. Possible
results were:
* instant crash with colorful junk on the screen, just after the
wiggling dash appear - seems like corrupted zfsloader;
* refusing to load kernel showing the error;
* lotsa gibberish about forth words unknown - seems like loader or
some config files corrupted.
Having played for 1 day with the disk I find that amount of luck for me
to boot should be heavy, so I switched virtualbox to amd64 and just
after the first rewrite of /boot system was up and running.
1. I used the same FreeBSD version to rewrite the data:
ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/201004/FreeBSD-8.0-STABLE-201004-amd64-livefs.iso
ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/201004/FreeBSD-8.0-STABLE-201004-i386-livefs.iso
2. I used the same boot code and haven't changed it.
So it looks like ZFS is writing data to the disks differently on i386.
Just "the other way".
--
Sphinx of black quartz judge my vow.
Andriy Gapon
> On 29.04.2010 15:20, Andriy Gapon wrote:
>> your system? I.e. the code that runs before loader and that resides beyond
>> filesystems.
>
Pardon me, but please clarify what you mean by this answer.
See below for explanation of my confusion.
> Actually I was hit by the same bug recently one more time. After rebuild
> machine refuses to boot because of some gang blocks. I've tried to use i386
> for the restoration purpose and set up a virtualbox in which I more then ten
> times tried to rewrite /boot and reboot from pool. Possible results were:
>
> * instant crash with colorful junk on the screen, just after the wiggling
> dash appear - seems like corrupted zfsloader; * refusing to load kernel
> showing the error; * lotsa gibberish about forth words unknown - seems like
> loader or some config files corrupted.
>
> Having played for 1 day with the disk I find that amount of luck for me to
> boot should be heavy, so I switched virtualbox to amd64 and just after the
> first rewrite of /boot system was up and running.
>
> 1. I used the same FreeBSD version to rewrite the data:
> ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/201004/FreeBSD-8.0-STABLE-201004-amd64-livefs.iso
>
>
> ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/201004/FreeBSD-8.0-STABLE-201004-i386-livefs.iso
>
>
> 2. I used the same boot code and haven't changed it.
This is what I am confused about.
What do you mean by the same 'boot code'?
I originally asked if you have the *latest* bootcode installed, but you seem to
be giving contradicting answers...
You detailed your magic workaround of re-writing data, but you haven't mentioned
how you updated your boot code.
Could you please report the actual commands that you use to update the boot
blocks? Just so we are sure that the problem is indeed present in the latest code.
Andriy Gapon
From: Andriy Gapon <a...@icyb.net.ua>
To: bug-fo...@FreeBSD.org, c.k...@gmail.com
Cc:
Subject: Re: bin/144214: zfsboot fails on gang block after upgrade to zfs
v14
Date: Thu, 13 May 2010 08:59:28 +0300
I had a private conversation with Daniel Gerzo (danger@) and neither him nor mm@
are sure that the system for which they reported the problem had the latest boot
blocks that are supposed to actually support zfs gang blocks.
P.S. gang block support seems to have been added to stable/8 by rnoland@ on 21
Nov 2009 in r199634, so anything before that is not expected to work.
Volodymyr Kostyrko
> This is what I am confused about.
> What do you mean by the same 'boot code'?
> I originally asked if you have the *latest* bootcode installed, but you seem to
> be giving contradicting answers...
>
> You detailed your magic workaround of re-writing data, but you haven't mentioned
> how you updated your boot code.
> Could you please report the actual commands that you use to update the boot
> blocks? Just so we are sure that the problem is indeed present in the latest code.
1. I have updated bootcode just in month or so before the failure.
dd if=/boot/zfsboot of=/dev/slice count=1 bs=512
dd if=/boot/zfsboot of=/dev/slice skip=1 seek=1024 bs=512
Yes, this means revision 206815 was not there.
2. After the computer fails this time I haven't touched bootcode.
I see your point and I'll try to retest on recent STABLE. This changes
nothing though.
http://svn.freebsd.org/viewvc/base/stable/8/sys/boot/zfs/zfs.c?r1=199634&r2=206815&pathrev=206815
I'll try to retest writing on recent STABLE on i386. Maybe not the
bootcode is culprit here.
--
Sphinx of black quartz judge my vow.
Andriy Gapon
From: Andriy Gapon <a...@icyb.net.ua>
To: bug-fo...@FreeBSD.org, c.k...@gmail.com
Cc: Robert Noland <rno...@FreeBSD.org>
Subject: Re: bin/144214: zfsboot fails on gang block after upgrade to zfs
v14
Date: Thu, 13 May 2010 09:59:00 +0300
This is a multi-part message in MIME format.
--------------010603020208000201000600
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
It seems that I have been misunderstanding the problem.
"ZFS: gang block detected" won't even appear if boot code is too old.
Having briefly glanced over the code and comparing it to the code in osol and in
zio_gang_tree_issue(), I think the following change is needed.
But I am not sure if it is a real fix for the issue at hand.
If anyone can reproduce the problem, could you please test this change?
Thanks!
--
Andriy Gapon
--------------010603020208000201000600
Content-Type: text/plain;
name="zfs-boot-gang.patch"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
filename="zfs-boot-gang.patch"
ZGlmZiAtLWdpdCBhL3N5cy9ib290L3pmcy96ZnNpbXBsLmMgYi9zeXMvYm9vdC96ZnMvemZz
aW1wbC5jCmluZGV4IGRjMjllMGUuLmVhZTAwMjMgMTAwNjQ0Ci0tLSBhL3N5cy9ib290L3pm
cy96ZnNpbXBsLmMKKysrIGIvc3lzL2Jvb3QvemZzL3pmc2ltcGwuYwpAQCAtOTYyLDggKzk2
MiwxMyBAQCB6aW9fcmVhZF9nYW5nKHNwYV90ICpzcGEsIGNvbnN0IGJsa3B0cl90ICpicCwg
Y29uc3QgZHZhX3QgKmR2YSwgdm9pZCAqYnVmKQogCQlyZXR1cm4gKEVJTyk7CiAKIAlmb3Ig
KGkgPSAwOyBpIDwgU1BBX0dCSF9OQkxLUFRSUzsgaSsrKSB7Ci0JCWlmICh6aW9fcmVhZChz
cGEsICZ6aW9fZ2IuemdfYmxrcHRyW2ldLCBidWYpKQorCQlibGtwdHJfdCAqZ2JwID0gJnpp
b19nYi56Z19ibGtwdHJbaV07CisKKwkJaWYgKEJQX0lTX0hPTEUoZ2JwKSkKKwkJCWNvbnRp
bnVlOworCQlpZiAoemlvX3JlYWQoc3BhLCBnYnAsIGJ1ZikpCiAJCQlyZXR1cm4gKEVJTyk7
CisJCWJ1ZiA9IChjaGFyKilidWYgKyBCUF9HRVRfUFNJWkUoZ2JwKTsKIAl9CiAgCiAJcmV0
dXJuICgwKTsK
--------------010603020208000201000600--
Bartosz Stec
> 2010/5/13 Andriy Gapon<a...@icyb.net.ua>:
>
>> This is what I am confused about.
>> What do you mean by the same 'boot code'?
>> I originally asked if you have the *latest* bootcode installed, but you seem to
>> be giving contradicting answers...
>>
>> You detailed your magic workaround of re-writing data, but you haven't mentioned
>> how you updated your boot code.
>> Could you please report the actual commands that you use to update the boot
>> blocks? Just so we are sure that the problem is indeed present in the latest code.
>>
>>
that my raidz pool was hit by this bug also on CURRENT on april 30. I
did make buildword; make buildkernel; make installkernel; reboot and I
didn't realized that at this moment zpool was 98% full (about 300-400MB
free space left). Loader catched gang block while trying to loading
kernel and after a moment it give up, informing that it can't find from
what device is booting from.
So at this moment my system was in non-bootable state, with installed
kernel but without new world installed. Zpool however was fine, I didn't
have any problems accesing it from fixit environment. So what I did at
first time was installing world and updating bootblocks from freshly
builded /boot using fixit dvd. Of course it left my zpool with even less
free space. After reboot loader text show itself for about half a second
and after that I could only watch blank screen.
I also tried some other things from fixit environment: installing
bootcode from 8.0-release (it failed because zpool v13 vs v14 mismatch),
revert back from snapshot from time before new world and kernel was
build (gang block again), and finally I fixed issue by deleting all
snapshots and deleted as max files as possible (so I had about 5GB free
space left) and then:
cp -pR /boot /boot2 ; mv /boot /boot.bak ; mv /boot2 /boot ; reboot
As my system was bootable again, I have csupped new sources, rebuilded
world and kernel, updated bootcodes and everything was fine. It seems
that bug can be catched only with almost full zpool.
--
Bartosz Stec
Bartosz Stec
> It seems that bug can be catched only with almost full zpool.
>
understand correctly they are straight realted to insufficient space so
my conclusion was obvious and useless ;) Fact is - zfs loader doesn't
work well with gang blocks, at least it didnt 2 weeks ago on -CURRENT.
Robert Noland
From: Robert Noland <rno...@FreeBSD.org>
To: Andriy Gapon <a...@icyb.net.ua>
Cc: bug-fo...@FreeBSD.org, c.k...@gmail.com
Subject: Re: bin/144214: zfsboot fails on gang block after upgrade to zfs
v14
Date: Thu, 13 May 2010 08:52:14 -0500
Andriy Gapon wrote:
> It seems that I have been misunderstanding the problem.
> "ZFS: gang block detected" won't even appear if boot code is too old.
>
> Having briefly glanced over the code and comparing it to the code in osol and in
> zio_gang_tree_issue(), I think the following change is needed.
> But I am not sure if it is a real fix for the issue at hand.
>
> If anyone can reproduce the problem, could you please test this change?
> Thanks!
This looks sane. I was never actually able to test it, since
reproducing the issue is rather tricky.
robert.
Volodymyr Kostyrko
From: Volodymyr Kostyrko <c.k...@gmail.com>
To: Andriy Gapon <a...@icyb.net.ua>
Cc: bug-followup <bug-fo...@freebsd.org>, Robert Noland <rno...@freebsd.org>
Subject: Re: bin/144214: zfsboot fails on gang block after upgrade to zfs v14
Date: Fri, 14 May 2010 17:12:23 +0300
2010/5/13 Andriy Gapon <a...@icyb.net.ua>:
>
> It seems that I have been misunderstanding the problem.
> "ZFS: gang block detected" won't even appear if boot code is too old.
>
> Having briefly glanced over the code and comparing it to the code in osol and in
> zio_gang_tree_issue(), I think the following change is needed.
> But I am not sure if it is a real fix for the issue at hand.
>
> If anyone can reproduce the problem, could you please test this change?
> Thanks!
Tested it. Same problem.
1. Rebuild and reinstall on i386. Filling disk up (600M free of 120G, 0.5%).
2. Immediately after starting boot screen bursts into psychic colors.
Computer reboots.
3. Booted from ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/201004/FreeBSD-8.0-STABLE-201004-i386-livefs.iso
in VirtualBox i386. Boot code updated with dd.
4. Same as p2. in vBox i386 takes looong time to rotate dash then
spits "ZFS: gang block detected" and hangs.
5. Booted from amd64 install, updated boot code with dd.
6. Booted on amd64. Immediately after starting boot spits out "ZFS:
gang block detected" and hangs.
7. Booted from amd64 install. /boot transferred transferred to/from other disk.
8. Booted on amd64. Immediately after starting boot spits out "ZFS:
gang block detected" and hangs.
9. Booted from amd64 install. Some files deleted (800M free, files
were written contiguously). /boot transferred transferred to/from
other disk.
10. Booted on amd64.
Results:
1. Patch changes something. However zfsloader(?) still can't be read completely.
2. Bug can happen on amd64. More extreme conditions needed(?).
3. I'll post a follow-up on successfully booting on original i386 hardware.
--
Sphinx of black quartz judge my vow.
Andriy Gapon
From: Andriy Gapon <a...@icyb.net.ua>
To: Volodymyr Kostyrko <c.k...@gmail.com>
Cc: bug-followup <bug-fo...@freebsd.org>,
Robert Noland <rno...@freebsd.org>
Subject: Re: bin/144214: zfsboot fails on gang block after upgrade to zfs
v14
Date: Fri, 14 May 2010 21:15:31 +0300
on 14/05/2010 17:12 Volodymyr Kostyrko said the following:
> 2010/5/13 Andriy Gapon <a...@icyb.net.ua>:
>> It seems that I have been misunderstanding the problem.
>> "ZFS: gang block detected" won't even appear if boot code is too old.
>>
>> Having briefly glanced over the code and comparing it to the code in osol and in
>> zio_gang_tree_issue(), I think the following change is needed.
>> But I am not sure if it is a real fix for the issue at hand.
>>
>> If anyone can reproduce the problem, could you please test this change?
>> Thanks!
>
> Tested it. Same problem.
Sigh. I almost do not see any other obvious differences with other code that is
supposed to support gang blocks.
> 1. Rebuild and reinstall on i386. Filling disk up (600M free of 120G, 0.5%).
> 2. Immediately after starting boot screen bursts into psychic colors.
> Computer reboots.
With unpatched boot code I presume?
> 3. Booted from ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/201004/FreeBSD-8.0-STABLE-201004-i386-livefs.iso
> in VirtualBox i386. Boot code updated with dd.
Have you updated both both part of zfsboot and loader?
Are you sure that you used patched versions? (asking just in case)
> 4. Same as p2. in vBox i386 takes looong time to rotate dash then
> spits "ZFS: gang block detected" and hangs.
Nothing else get printed?
Asking because of this screenshot:
http://danger.rulez.sk/dockdrop/144214.png
> 5. Booted from amd64 install, updated boot code with dd.
> 6. Booted on amd64. Immediately after starting boot spits out "ZFS:
> gang block detected" and hangs.
> 7. Booted from amd64 install. /boot transferred transferred to/from other disk.
> 8. Booted on amd64. Immediately after starting boot spits out "ZFS:
> gang block detected" and hangs.
amd64 has exactly the same boot code that i386 has, perhaps some difference
could arise during compilation, but even if so, it should not matter much in our
case.
> 9. Booted from amd64 install. Some files deleted (800M free, files
> were written contiguously). /boot transferred transferred to/from
> other disk.
> 10. Booted on amd64.
Not interested much in the workarounds - if they work, then OK, but mainly we
are trying to fix the boot code. Only behavior of installed zfsboot and
zfsloader are interesting to us.
> Results:
> 1. Patch changes something. However zfsloader(?) still can't be read completely.
> 2. Bug can happen on amd64. More extreme conditions needed(?).
> 3. I'll post a follow-up on successfully booting on original i386 hardware.
Can you please also share output of 'zfs get all' for the boot filesystem?
Thank you for your help!
And one last thing that I could think of:
--- a/sys/boot/zfs/zfsimpl.c
+++ b/sys/boot/zfs/zfsimpl.c
@@ -1001,7 +1001,7 @@ zio_read(spa_t *spa, const blkptr_t *bp, void *buf)
if (DVA_GET_GANG(dva)) {
printf("ZFS: gang block detected!\n");
if (zio_read_gang(spa, bp, dva, buf))
- return (EIO);
+ continue;
} else {
vdevid = DVA_GET_VDEV(dva);
offset = DVA_GET_OFFSET(dva);
This should be applied in addition to the previous patch.
If this still doesn't work, the it would make sense to add printfs in various
places of zio_read_gang() function to try to see what happens there.
--
Andriy Gapon