Re: gptzfsboot problem on HP P410i Smart Array

[Sergey Dyatko]

2012/8/20 Bjorn Larsson <bjw...@gmail.com>

> Hi Andrey,
>
> We are installing freeBSD using ZFS as root filesystem using the GPT
> method as described on the freeBSD ZFS wiki. We are creating a GPT
> boot partition with the gptzfsboot program embedded and then a zroot
> partition with the freeBSD binaries. This works well and boots on
> every system we tested except HP P410i smart array systems. The
> problem is that no disks are identified in gptzfsboot and the
> following error code is displayed:
>
> gptzfsboot: error 1 lba 32
> gptzfsboot: error 1 lba 1
>
> It appears that gptzfsboot is not identifying the drives properly.
> However, when we insert a printf() command in main() in zfsboot.c to
> troubleshoot the identification problem, the system boots perfectly
> fine.
>
> This is a problem that I believe was fixed last year in 9-CURRENT by
> implementing a proper struct for edd rather than using a char array
> for BIOS communication. However, it doesn't seems to have fixed the on
> the p410i smart arrays.
>
>
I was faced with same problem on my laptop. Adding printf() into main()
before dsk = malloc(sizeof(struct dsk)); fix boot. Yesterday, avg@ proposed
patch:
Index: /usr/src/sys/boot/i386/zfsboot/zfsboot.c
===================================================================
--- /usr/src/sys/boot/i386/zfsboot/zfsboot.c (revision 248421)
+++ /usr/src/sys/boot/i386/zfsboot/zfsboot.c (working copy)
@@ -302,6 +302,7 @@
* region in the SMAP, use the last 3MB of 'extended' memory as a
* high heap candidate.
*/
+ high_heap_size = 0;
if (bios_extmem >= HEAP_MIN && high_heap_size < HEAP_MIN) {
high_heap_size = HEAP_MIN;
high_heap_base = bios_extmem + 0x100000 - HEAP_MIN;

it works for me, without printf() :) Can you test it ?

> Best regards,
>
> Björn Larsson
>
>
> On Sun, Aug 19, 2012 at 6:41 PM, Andrey V. Elsukov <bu7...@yandex.ru>
> wrote:
> >
> > On 19.08.2012 11:22, Bjorn Larsson wrote:
> > > We are having problems with gptzfsboot on a HP DL360 G7 using the P410i
> > > Smart Array Controller.
> > > I’ve some information on this in the archive on this mailing list that
> this
> > > has supposedly been fixed with revision 227400, by implementing the edd
> > > data structure.
> > > However we still see the same problem and by adding a printf() in
> zfsboot.c
> > > fixes the problem.
> > > Please, let me know if anyone have seen this problem and if there is a
> fix
> > > for it?
> >
> > Hi,
> >
> > what exactly do you have?
> >
> > --
> > WBR, Andrey V. Elsukov
> >
> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
>
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"

[Andriy Gapon]

on 19/03/2013 07:41 Sergey Dyatko said the following:

> I was faced with same problem on my laptop. Adding printf() into main()
> before dsk = malloc(sizeof(struct dsk)); fix boot. Yesterday, avg@ proposed
> patch:
> Index: /usr/src/sys/boot/i386/zfsboot/zfsboot.c
> ===================================================================
> --- /usr/src/sys/boot/i386/zfsboot/zfsboot.c (revision 248421)
> +++ /usr/src/sys/boot/i386/zfsboot/zfsboot.c (working copy)
> @@ -302,6 +302,7 @@
> * region in the SMAP, use the last 3MB of 'extended' memory as a
> * high heap candidate.
> */
> + high_heap_size = 0;
> if (bios_extmem >= HEAP_MIN && high_heap_size < HEAP_MIN) {
> high_heap_size = HEAP_MIN;
> high_heap_base = bios_extmem + 0x100000 - HEAP_MIN;
>
> it works for me, without printf() :) Can you test it ?

A comment about a nature of this patch.

Based on the previous investigation by Christoph Hoffmann and jhb:
http://thread.gmane.org/gmane.os.freebsd.current/134199/focus=134309
I made a guess that either BIOS/firmware provides incorrect memory map or some
agent in the BIOS/firmware (e.g. SMM handler) or controller firmware writes
outside of a memory range reserved for it.
I think that jhb made a similar guess at the time while Christoph conjectured
that memory corruption was related to CPU caches or some such.
My conjecture is that it is simply a combination of timing and a particular
memory range.

Just in case, here is how the memory map looks on the Sergey's system:
SMAP type=01 base=0000000000000000 end=000000000009fc00 len=000000000009fc00
SMAP type=02 base=000000000009fc00 end=00000000000a0000 len=0000000000000400
SMAP type=02 base=00000000000e0000 end=0000000000100000 len=0000000000020000
SMAP type=01 base=0000000000100000 end=00000000bc1a1000 len=00000000bc0a1000
SMAP type=04 base=00000000bc1a1000 end=00000000bc1a4000 len=0000000000003000
SMAP type=01 base=00000000bc1a4000 end=00000000bdf04000 len=0000000001d60000
SMAP type=04 base=00000000bdf04000 end=00000000bdf3f000 len=000000000003b000
SMAP type=01 base=00000000bdf3f000 end=00000000bdf6a000 len=000000000002b000
SMAP type=02 base=00000000bdf6a000 end=00000000bdfbf000 len=0000000000055000
SMAP type=01 base=00000000bdfbf000 end=00000000bdfeb000 len=000000000002c000
SMAP type=03 base=00000000bdfeb000 end=00000000bdfff000 len=0000000000014000
SMAP type=01 base=00000000bdfff000 end=00000000be000000 len=0000000000001000
SMAP type=02 base=00000000be000000 end=00000000c0000000 len=0000000002000000
SMAP type=02 base=00000000f8000000 end=00000000fc000000 len=0000000004000000
SMAP type=02 base=00000000fec00000 end=00000000fec01000 len=0000000000001000
SMAP type=02 base=00000000fed10000 end=00000000fed14000 len=0000000000004000
SMAP type=02 base=00000000fed18000 end=00000000fed1a000 len=0000000000002000
SMAP type=02 base=00000000fed1c000 end=00000000fed20000 len=0000000000004000
SMAP type=02 base=00000000fee00000 end=00000000fee01000 len=0000000000001000
SMAP type=02 base=00000000ffe00000 end=0000000100000000 len=0000000000200000
SMAP type=01 base=0000000100000000 end=0000000140000000 len=0000000040000000

The algorithm for placing the heap picks up a range at bc1a4000, which is
between two ranges of type '4' (ACPI NVS memory).
So my idea was just to try a different memory range. Seems that it worked.

P.S. I am not sure why our algorithm for selecting heap location is what it is.
On all systems that I have I see that the "bios_extmem" range (the one starting
at 0x100000) is usually the largest one and has more than enough space for both
the heap and other things that are placed there.
Additionally, in the case of zfsboot I think that we do not use memory above 1MB
for anything else besides the heap.

--
Andriy Gapon

[John Baldwin]

Yes, we likely could start using that, we would just need to ensure it has some
sort of minimum size. However, maybe it would always have that minimum size as you
are only going to have additional ranges if you have more than 4GB of RAM anyway.
I think though that there were some odd BIOSes that would place a hole at 15-16MB,
and for those the memory region at 1MB is too small. A minimum size of 16MB might
handle that case correctly while using the first extended region in the common
case.

> Additionally, in the case of zfsboot I think that we do not use memory above 1MB
> for anything else besides the heap.

You load either /boot/zfsloader or the kernel there.

--
John Baldwin

[Christoph Hoffmann]

Dear All,

At present I do not have access to a HP box with a SmartArray
but zfsboot works perfectly with FreeBSD 9.1-RELEASE #0 r245668
(including EDD fix) on:

</>hpiLO-> show system1
[...]
Properties
name=ProLiant DL160 Gen8
[...]

with the following firmware

</>hpiLO-> show system1/firmware1
[...]
/system1/firmware1
Targets
Properties
version=J03
date=08/20/2012
[...]

I know it is not much help, but hope it might give you
some input.

Thank you very much indeed for your work on this issue.

Best Regards,

Christoph

--
Christoph Hoffmann
e-mail: christoph...@me.com

[Andriy Gapon]

I think it could be useful if those who have the hardware and have a support
contract would try to get in touch with the _technical_ support and give them some
technical details. Maybe it's something that could be easily fixed by the vendor
if they know about the problem.

--
Andriy Gapon

[Andriy Gapon]

on 19/03/2013 18:20 John Baldwin said the following:

> Yes, we likely could start using that, we would just need to ensure it has some
> sort of minimum size. However, maybe it would always have that minimum size as you
> are only going to have additional ranges if you have more than 4GB of RAM anyway.
> I think though that there were some odd BIOSes that would place a hole at 15-16MB,
> and for those the memory region at 1MB is too small. A minimum size of 16MB might
> handle that case correctly while using the first extended region in the common
> case.

How about something like this?

Author: Andriy Gapon <a...@icyb.net.ua>
Date: Wed Apr 3 11:48:34 2013 +0300

[test] zfsboot: bios_getmem prefer extmem over another random chunk of high memory

If the extended memory region is sufficiently large.

diff --git a/sys/boot/i386/zfsboot/zfsboot.c b/sys/boot/i386/zfsboot/zfsboot.c
index 82402b6..12ceeb0 100644
--- a/sys/boot/i386/zfsboot/zfsboot.c
+++ b/sys/boot/i386/zfsboot/zfsboot.c
@@ -374,6 +374,16 @@ bios_getmem(void)
}

/*
+ * If extended memory is at least twice as large as the largest
+ * region of higher memory, then carve the high heap out of
+ * extended memory.
+ */
+ if (bios_extmem > 2 * high_heap_size) {
+ high_heap_base = 0x100000 + bios_extmem / 2;
+ high_heap_size = bios_extmem / 2;
+ }
+
+ /*
* If we have extended memory and did not find a suitable heap

* region in the SMAP, use the last 3MB of 'extended' memory as a
* high heap candidate.

[John Baldwin]

We should really use the same algorithm in boot2 and gptboot as well.

I think though that in this case you can just use the last 3MB of heap
rather than half of the extended memory as heap.

--
John Baldwin

[Andriy Gapon]

on 04/04/2013 20:16 John Baldwin said the following:

> On Thursday, April 04, 2013 12:16:32 pm Andriy Gapon wrote:
>> diff --git a/sys/boot/i386/zfsboot/zfsboot.c
> b/sys/boot/i386/zfsboot/zfsboot.c
>> index 82402b6..12ceeb0 100644
>> --- a/sys/boot/i386/zfsboot/zfsboot.c
>> +++ b/sys/boot/i386/zfsboot/zfsboot.c
>> @@ -374,6 +374,16 @@ bios_getmem(void)
>> }
>>
>> /*
>> + * If extended memory is at least twice as large as the largest
>> + * region of higher memory, then carve the high heap out of
>> + * extended memory.
>> + */
>> + if (bios_extmem > 2 * high_heap_size) {
>> + high_heap_base = 0x100000 + bios_extmem / 2;
>> + high_heap_size = bios_extmem / 2;
>> + }
>> +
>> + /*
>> * If we have extended memory and did not find a suitable heap
>> * region in the SMAP, use the last 3MB of 'extended' memory as a
>> * high heap candidate.
>>
>
> We should really use the same algorithm in boot2 and gptboot as well.

Yes, this is just something to start with.

BTW, all other components use bios_getmem from sys/boot/i386/libi386/biosmem.c ?

> I think though that in this case you can just use the last 3MB of heap
> rather than half of the extended memory as heap.

I thought the more the better? :-)
I've kept the block of code that tries to make high_heap_size at least 3MB.

--
Andriy Gapon