gptzfsboot problem on HP P410i Smart Array

[Bjorn Larsson]

Dear


We are having problems with gptzfsboot on a HP DL360 G7 using the P410i
Smart Array Controller.


I’ve some information on this in the archive on this mailing list that this
has supposedly been fixed with revision 227400, by implementing the edd
data structure.

However we still see the same problem and by adding a printf() in zfsboot.c
fixes the problem.


Please, let me know if anyone have seen this problem and if there is a fix
for it?


Best regards,


Björn Larsson
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"

[Andrey V. Elsukov]

On 19.08.2012 11:22, Bjorn Larsson wrote:
> We are having problems with gptzfsboot on a HP DL360 G7 using the P410i
> Smart Array Controller.
> I’ve some information on this in the archive on this mailing list that this
> has supposedly been fixed with revision 227400, by implementing the edd
> data structure.
> However we still see the same problem and by adding a printf() in zfsboot.c
> fixes the problem.
> Please, let me know if anyone have seen this problem and if there is a fix
> for it?

Hi,

what exactly do you have?

--
WBR, Andrey V. Elsukov

[Bjorn Larsson]

Hi Andrey,

We are installing freeBSD using ZFS as root filesystem using the GPT
method as described on the freeBSD ZFS wiki. We are creating a GPT
boot partition with the gptzfsboot program embedded and then a zroot
partition with the freeBSD binaries. This works well and boots on
every system we tested except HP P410i smart array systems. The
problem is that no disks are identified in gptzfsboot and the
following error code is displayed:

gptzfsboot: error 1 lba 32
gptzfsboot: error 1 lba 1

It appears that gptzfsboot is not identifying the drives properly.
However, when we insert a printf() command in main() in zfsboot.c to
troubleshoot the identification problem, the system boots perfectly
fine.

This is a problem that I believe was fixed last year in 9-CURRENT by
implementing a proper struct for edd rather than using a char array
for BIOS communication. However, it doesn't seems to have fixed the on
the p410i smart arrays.

Best regards,

Björn Larsson

[Sergey Dyatko]

2012/8/20 Bjorn Larsson <bjw...@gmail.com>

> Hi Andrey,
>
> We are installing freeBSD using ZFS as root filesystem using the GPT
> method as described on the freeBSD ZFS wiki. We are creating a GPT
> boot partition with the gptzfsboot program embedded and then a zroot
> partition with the freeBSD binaries. This works well and boots on
> every system we tested except HP P410i smart array systems. The
> problem is that no disks are identified in gptzfsboot and the
> following error code is displayed:
>
> gptzfsboot: error 1 lba 32
> gptzfsboot: error 1 lba 1
>
> It appears that gptzfsboot is not identifying the drives properly.
> However, when we insert a printf() command in main() in zfsboot.c to
> troubleshoot the identification problem, the system boots perfectly
> fine.
>
> This is a problem that I believe was fixed last year in 9-CURRENT by
> implementing a proper struct for edd rather than using a char array
> for BIOS communication. However, it doesn't seems to have fixed the on
> the p410i smart arrays.
>
>

I was faced with same problem on my laptop. Adding printf() into main()
before dsk = malloc(sizeof(struct dsk)); fix boot. Yesterday, avg@ proposed
patch:
Index: /usr/src/sys/boot/i386/zfsboot/zfsboot.c
===================================================================
--- /usr/src/sys/boot/i386/zfsboot/zfsboot.c (revision 248421)
+++ /usr/src/sys/boot/i386/zfsboot/zfsboot.c (working copy)
@@ -302,6 +302,7 @@
* region in the SMAP, use the last 3MB of 'extended' memory as a
* high heap candidate.
*/
+ high_heap_size = 0;
if (bios_extmem >= HEAP_MIN && high_heap_size < HEAP_MIN) {
high_heap_size = HEAP_MIN;
high_heap_base = bios_extmem + 0x100000 - HEAP_MIN;

it works for me, without printf() :) Can you test it ?

[Andriy Gapon]

on 19/03/2013 07:41 Sergey Dyatko said the following:

> I was faced with same problem on my laptop. Adding printf() into main()
> before dsk = malloc(sizeof(struct dsk)); fix boot. Yesterday, avg@ proposed
> patch:
> Index: /usr/src/sys/boot/i386/zfsboot/zfsboot.c
> ===================================================================
> --- /usr/src/sys/boot/i386/zfsboot/zfsboot.c (revision 248421)
> +++ /usr/src/sys/boot/i386/zfsboot/zfsboot.c (working copy)
> @@ -302,6 +302,7 @@
> * region in the SMAP, use the last 3MB of 'extended' memory as a
> * high heap candidate.
> */
> + high_heap_size = 0;
> if (bios_extmem >= HEAP_MIN && high_heap_size < HEAP_MIN) {
> high_heap_size = HEAP_MIN;
> high_heap_base = bios_extmem + 0x100000 - HEAP_MIN;
>
> it works for me, without printf() :) Can you test it ?

A comment about a nature of this patch.

Based on the previous investigation by Christoph Hoffmann and jhb:
http://thread.gmane.org/gmane.os.freebsd.current/134199/focus=134309
I made a guess that either BIOS/firmware provides incorrect memory map or some
agent in the BIOS/firmware (e.g. SMM handler) or controller firmware writes
outside of a memory range reserved for it.
I think that jhb made a similar guess at the time while Christoph conjectured
that memory corruption was related to CPU caches or some such.
My conjecture is that it is simply a combination of timing and a particular
memory range.

Just in case, here is how the memory map looks on the Sergey's system:
SMAP type=01 base=0000000000000000 end=000000000009fc00 len=000000000009fc00
SMAP type=02 base=000000000009fc00 end=00000000000a0000 len=0000000000000400
SMAP type=02 base=00000000000e0000 end=0000000000100000 len=0000000000020000
SMAP type=01 base=0000000000100000 end=00000000bc1a1000 len=00000000bc0a1000
SMAP type=04 base=00000000bc1a1000 end=00000000bc1a4000 len=0000000000003000
SMAP type=01 base=00000000bc1a4000 end=00000000bdf04000 len=0000000001d60000
SMAP type=04 base=00000000bdf04000 end=00000000bdf3f000 len=000000000003b000
SMAP type=01 base=00000000bdf3f000 end=00000000bdf6a000 len=000000000002b000
SMAP type=02 base=00000000bdf6a000 end=00000000bdfbf000 len=0000000000055000
SMAP type=01 base=00000000bdfbf000 end=00000000bdfeb000 len=000000000002c000
SMAP type=03 base=00000000bdfeb000 end=00000000bdfff000 len=0000000000014000
SMAP type=01 base=00000000bdfff000 end=00000000be000000 len=0000000000001000
SMAP type=02 base=00000000be000000 end=00000000c0000000 len=0000000002000000
SMAP type=02 base=00000000f8000000 end=00000000fc000000 len=0000000004000000
SMAP type=02 base=00000000fec00000 end=00000000fec01000 len=0000000000001000
SMAP type=02 base=00000000fed10000 end=00000000fed14000 len=0000000000004000
SMAP type=02 base=00000000fed18000 end=00000000fed1a000 len=0000000000002000
SMAP type=02 base=00000000fed1c000 end=00000000fed20000 len=0000000000004000
SMAP type=02 base=00000000fee00000 end=00000000fee01000 len=0000000000001000
SMAP type=02 base=00000000ffe00000 end=0000000100000000 len=0000000000200000
SMAP type=01 base=0000000100000000 end=0000000140000000 len=0000000040000000

The algorithm for placing the heap picks up a range at bc1a4000, which is
between two ranges of type '4' (ACPI NVS memory).
So my idea was just to try a different memory range. Seems that it worked.

P.S. I am not sure why our algorithm for selecting heap location is what it is.
On all systems that I have I see that the "bios_extmem" range (the one starting
at 0x100000) is usually the largest one and has more than enough space for both
the heap and other things that are placed there.
Additionally, in the case of zfsboot I think that we do not use memory above 1MB
for anything else besides the heap.

--
Andriy Gapon

[John Baldwin]

Yes, we likely could start using that, we would just need to ensure it has some
sort of minimum size. However, maybe it would always have that minimum size as you
are only going to have additional ranges if you have more than 4GB of RAM anyway.
I think though that there were some odd BIOSes that would place a hole at 15-16MB,
and for those the memory region at 1MB is too small. A minimum size of 16MB might
handle that case correctly while using the first extended region in the common
case.

> Additionally, in the case of zfsboot I think that we do not use memory above 1MB
> for anything else besides the heap.

You load either /boot/zfsloader or the kernel there.

--
John Baldwin

[Andriy Gapon]

I think it could be useful if those who have the hardware and have a support
contract would try to get in touch with the _technical_ support and give them some
technical details. Maybe it's something that could be easily fixed by the vendor
if they know about the problem.

--
Andriy Gapon

[Andriy Gapon]

on 19/03/2013 18:20 John Baldwin said the following:

> Yes, we likely could start using that, we would just need to ensure it has some
> sort of minimum size. However, maybe it would always have that minimum size as you
> are only going to have additional ranges if you have more than 4GB of RAM anyway.
> I think though that there were some odd BIOSes that would place a hole at 15-16MB,
> and for those the memory region at 1MB is too small. A minimum size of 16MB might
> handle that case correctly while using the first extended region in the common
> case.

How about something like this?

Author: Andriy Gapon <a...@icyb.net.ua>
Date: Wed Apr 3 11:48:34 2013 +0300

[test] zfsboot: bios_getmem prefer extmem over another random chunk of high memory

If the extended memory region is sufficiently large.

diff --git a/sys/boot/i386/zfsboot/zfsboot.c b/sys/boot/i386/zfsboot/zfsboot.c
index 82402b6..12ceeb0 100644
--- a/sys/boot/i386/zfsboot/zfsboot.c
+++ b/sys/boot/i386/zfsboot/zfsboot.c
@@ -374,6 +374,16 @@ bios_getmem(void)
}

/*
+ * If extended memory is at least twice as large as the largest
+ * region of higher memory, then carve the high heap out of
+ * extended memory.
+ */
+ if (bios_extmem > 2 * high_heap_size) {
+ high_heap_base = 0x100000 + bios_extmem / 2;
+ high_heap_size = bios_extmem / 2;
+ }
+
+ /*
* If we have extended memory and did not find a suitable heap

* region in the SMAP, use the last 3MB of 'extended' memory as a
* high heap candidate.

--
Andriy Gapon

[John Baldwin]

We should really use the same algorithm in boot2 and gptboot as well.

I think though that in this case you can just use the last 3MB of heap
rather than half of the extended memory as heap.

--
John Baldwin

[Andriy Gapon]

on 04/04/2013 20:16 John Baldwin said the following:

> On Thursday, April 04, 2013 12:16:32 pm Andriy Gapon wrote:
>> diff --git a/sys/boot/i386/zfsboot/zfsboot.c
> b/sys/boot/i386/zfsboot/zfsboot.c
>> index 82402b6..12ceeb0 100644
>> --- a/sys/boot/i386/zfsboot/zfsboot.c
>> +++ b/sys/boot/i386/zfsboot/zfsboot.c
>> @@ -374,6 +374,16 @@ bios_getmem(void)
>> }
>>
>> /*
>> + * If extended memory is at least twice as large as the largest
>> + * region of higher memory, then carve the high heap out of
>> + * extended memory.
>> + */
>> + if (bios_extmem > 2 * high_heap_size) {
>> + high_heap_base = 0x100000 + bios_extmem / 2;
>> + high_heap_size = bios_extmem / 2;
>> + }
>> +
>> + /*
>> * If we have extended memory and did not find a suitable heap
>> * region in the SMAP, use the last 3MB of 'extended' memory as a
>> * high heap candidate.
>>
>
> We should really use the same algorithm in boot2 and gptboot as well.

Yes, this is just something to start with.

BTW, all other components use bios_getmem from sys/boot/i386/libi386/biosmem.c ?

> I think though that in this case you can just use the last 3MB of heap
> rather than half of the extended memory as heap.

I thought the more the better? :-)
I've kept the block of code that tries to make high_heap_size at least 3MB.

--
Andriy Gapon

[Rainer Duffner]

Hi,

I found this old thread….

I can’t boot FreeBSD 10 installed with zfsroot on a DL380G7 (P410i controller).
I tried the installer and I tried installing with mfsbsd10se.
System has 48GB RAM.

Is there a PR for this?


Now, I’ve got to waste 2’600 GB disks (and 300-odd I/Os) for a boot-disk…..

[Matthew Seaman]

On 09/04/2014 22:17, Rainer Duffner wrote:
> Hi,
>
> I found this old thread….
>
> I can’t boot FreeBSD 10 installed with zfsroot on a DL380G7 (P410i controller).
> I tried the installer and I tried installing with mfsbsd10se.
> System has 48GB RAM.
>
> Is there a PR for this?
>
>
> Now, I’ve got to waste 2’600 GB disks (and 300-odd I/Os) for a boot-disk…..

You've got more than 8 drives in your zpool? I ran into a similar
problem a while back: the bios only tells the OS about the first 8
drives during boot, and that isn't enough to assemble a workin zpool from.

Solution I adopted was to have a USB mem stick with /boot on it --
enough to get the kernel up and running and to assemble the zpool, which
could then provide the root fs perfectly well.

Cheers,

Matthew

--
Dr Matthew J Seaman MA, D.Phil.
PGP: http://www.infracaninophile.co.uk/pgpkey

[Rainer Duffner]

Am 09.04.2014 um 23:48 schrieb Matthew Seaman <mat...@FreeBSD.org>:

> On 09/04/2014 22:17, Rainer Duffner wrote:
>> Hi,
>>
>> I found this old thread….
>>
>> I can’t boot FreeBSD 10 installed with zfsroot on a DL380G7 (P410i controller).
>> I tried the installer and I tried installing with mfsbsd10se.
>> System has 48GB RAM.
>>
>> Is there a PR for this?
>>
>>
>> Now, I’ve got to waste 2’600 GB disks (and 300-odd I/Os) for a boot-disk…..
>

> You've got more than 8 drives in your spool?



No, it’s actually just a single RAID6-0 disk created by the P410i…

And no, as the server is in a remote datacenter, an USB-stick is not an option.

It’s slow enough booting via a virtual USB-image over iLO...

[Matthew Seaman]

On 09/04/2014 22:52, Rainer Duffner wrote:
> And no, as the server is in a remote datacenter, an USB-stick is not an option.
>
> It’s slow enough booting via a virtual USB-image over iLO...

Uh... it only has to read the kernel+modules from the USB stick one time
while booting. Otherwise, there really shouldn't be any IO inside /boot
unless you login and do stuff in that directory manually. Your root
filesystem would be on the normal hard drives.

Anyhow the question is moot, since you don't have the same problem I did.

> No, it’s actually just a single RAID6-0 disk created by the P410i…

If you're going to use the RAID controller to generate a virtual drive,
do you really need to use ZFS on top of that? Couldn't you partition
your virtual drive and put / onto a small UFS partition and then make a
zpool on the rest?

[Rainer Duffner]

Am 10.04.2014 um 00:02 schrieb Matthew Seaman <mat...@FreeBSD.org>:

> On 09/04/2014 22:52, Rainer Duffner wrote:
>> And no, as the server is in a remote datacenter, an USB-stick is not an option.
>>
>> It’s slow enough booting via a virtual USB-image over iLO...
>
> Uh... it only has to read the kernel+modules from the USB stick one time
> while booting. Otherwise, there really shouldn't be any IO inside /boot
> unless you login and do stuff in that directory manually. Your root
> filesystem would be on the normal hard drives.
>
> Anyhow the question is moot, since you don't have the same problem I did.
>
>> No, it’s actually just a single RAID6-0 disk created by the P410i…
>
> If you're going to use the RAID controller to generate a virtual drive,
> do you really need to use ZFS on top of that? Couldn't you partition
> your virtual drive and put / onto a small UFS partition and then make a
> zpool on the rest?

I don’t want to sacrifice two disks for a RAID1 boot-disk.
Normally, I would actually do that, but in this case, the server is a MySQL-slave to a master that has 12 disks - and should the master die, this system has to take over its work.

[Andreas Nilsson]

On Thu, Apr 10, 2014 at 1:04 AM, Rainer Duffner <rai...@ultra-secure.de>wrote:

>
> Am 10.04.2014 um 00:02 schrieb Matthew Seaman <mat...@FreeBSD.org>:
>
> > On 09/04/2014 22:52, Rainer Duffner wrote:
> >> And no, as the server is in a remote datacenter, an USB-stick is not an
> option.
> >>
> >> It's slow enough booting via a virtual USB-image over iLO...
> >
> > Uh... it only has to read the kernel+modules from the USB stick one time
> > while booting. Otherwise, there really shouldn't be any IO inside /boot
> > unless you login and do stuff in that directory manually. Your root
> > filesystem would be on the normal hard drives.
> >
> > Anyhow the question is moot, since you don't have the same problem I did.
> >

> >> No, it's actually just a single RAID6-0 disk created by the P410i...

> >
> > If you're going to use the RAID controller to generate a virtual drive,
> > do you really need to use ZFS on top of that? Couldn't you partition
> > your virtual drive and put / onto a small UFS partition and then make a
> > zpool on the rest?
>
>
> I don't want to sacrifice two disks for a RAID1 boot-disk.
> Normally, I would actually do that, but in this case, the server is a
> MySQL-slave to a master that has 12 disks - and should the master die,
> this system has to take over its work.
>

> You never specified exactly how it fails. But I'll take a guess:

*Attempting Boot From Hard Drive (C:)*

*gptzfsboot: error 1 lba 32*

*gptzfsboot: error 1 lba 1*

*g**ptzfsboot: No ZFS pools located, can't boot*


A workaround is
http://lists.freebsd.org/pipermail/freebsd-current/2011-August/026624.html

Best regards
Andreas

[Rainer Duffner]

Am Thu, 10 Apr 2014 06:59:25 +0200
schrieb Andreas Nilsson <andr...@gmail.com>:

> > You never specified exactly how it fails. But I'll take a guess:
>
> *Attempting Boot From Hard Drive (C:)*
>
> *gptzfsboot: error 1 lba 32*
>
> *gptzfsboot: error 1 lba 1*
>
> *g**ptzfsboot: No ZFS pools located, can't boot*

True.

But as that was the failure-symptom of the original thread, I kind of
neglected to mention it

> A workaround is
> http://lists.freebsd.org/pipermail/freebsd-current/2011-August/026624.html

Yes, but that requires rebuilding FreeBSD.

Why has this never been patched "properly"?

[Andreas Nilsson]

On Thu, Apr 10, 2014 at 12:25 PM, Rainer Duffner <rai...@ultra-secure.de>wrote:

> Am Thu, 10 Apr 2014 06:59:25 +0200
> schrieb Andreas Nilsson <andr...@gmail.com>:
>
>
> > > You never specified exactly how it fails. But I'll take a guess:
> >
> > *Attempting Boot From Hard Drive (C:)*
> >
> > *gptzfsboot: error 1 lba 32*
> >
> > *gptzfsboot: error 1 lba 1*
> >
> > *g**ptzfsboot: No ZFS pools located, can't boot*
>
>
> True.
>
> But as that was the failure-symptom of the original thread, I kind of
> neglected to mention it
>

Sorry, I read on client without threading...

>
> > A workaround is
> >
> http://lists.freebsd.org/pipermail/freebsd-current/2011-August/026624.html
>
>
>
> Yes, but that requires rebuilding FreeBSD.
>

It does, yes.

>
> Why has this never been patched "properly"?
>

Ask HP, I guess. Whatever they are doing in that hardware is "strange".

As is some other HP stuff as well, but that has been ranted about before so
I wont.

Best regards
Andreas

[Andreas Nilsson]

On Thu, Apr 10, 2014 at 12:44 PM, Andreas Nilsson <andr...@gmail.com>wrote:

>
>
>
> On Thu, Apr 10, 2014 at 12:25 PM, Rainer Duffner <rai...@ultra-secure.de>wrote:
>
>> Am Thu, 10 Apr 2014 06:59:25 +0200
>> schrieb Andreas Nilsson <andr...@gmail.com>:
>>
>>
>> > > You never specified exactly how it fails. But I'll take a guess:
>> >
>> > *Attempting Boot From Hard Drive (C:)*
>> >
>> > *gptzfsboot: error 1 lba 32*
>> >
>> > *gptzfsboot: error 1 lba 1*
>> >
>> > *g**ptzfsboot: No ZFS pools located, can't boot*
>>
>>
>> True.
>>
>> But as that was the failure-symptom of the original thread, I kind of
>> neglected to mention it
>>
>
> Sorry, I read on client without threading...
>
>
>>
>> > A workaround is
>> >
>> http://lists.freebsd.org/pipermail/freebsd-current/2011-August/026624.html
>>
>>
>>
>> Yes, but that requires rebuilding FreeBSD.
>>
> It does, yes.
>

But only for a subdir. So it would be one binary to distribute to affected
systems.

>
>
>>
>> Why has this never been patched "properly"?
>>
> Ask HP, I guess. Whatever they are doing in that hardware is "strange"
>

[John Baldwin]

On Thursday, April 10, 2014 12:59:25 am Andreas Nilsson wrote:
> On Thu, Apr 10, 2014 at 1:04 AM, Rainer Duffner <rainer@ultra-

I believe the proper fix for that bug was committed here:

http://svnweb.freebsd.org/base?view=revision&revision=243025

--
John Baldwin

[Sergey V. Dyatko]

possible, but it doesn't help in my case (lenovo b450 laptop). I install
fbsd11-current ~month ago in it and problem still persist, proposed by avg@
patch works fine in my case:

Index: sys/boot/i386/zfsboot/zfsboot.c
===================================================================
--- sys/boot/i386/zfsboot/zfsboot.c (revision 263419)
+++ sys/boot/i386/zfsboot/zfsboot.c (working copy)
@@ -302,6 +302,7 @@

* region in the SMAP, use the last 3MB of 'extended' memory as a
* high heap candidate.

*/
+ high_heap_size = 0;
if (bios_extmem >= HEAP_MIN && high_heap_size < HEAP_MIN) {
high_heap_size = HEAP_MIN;
high_heap_base = bios_extmem + 0x100000 - HEAP_MIN;

--
wbr, tiger