problem booting from 2nd drive

[Christos Chatzaras]

I used dump/restore commands to copy a custom freebsd 11.1 to a new server. After I restore the files I create gmirror (RAID-1) and copy the bootcode to both disks:

gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1

When I select from BIOS to boot using 1st disk (ada0) then it boots without issue. If I select to boot from 2nd disk (ada1) then it hangs and restarts again and again.


gpart show

=> 40 7814037088 ada0 GPT (3.6T)
40 472 1 freebsd-boot (236K)
512 8388608 2 freebsd-ufs (4.0G)
8389120 134217728 3 freebsd-swap (64G)
142606848 33554432 4 freebsd-ufs (16G)
176161280 134217728 5 freebsd-ufs (64G)
310379008 33554432 6 freebsd-ufs (16G)
343933440 1073741824 7 freebsd-ufs (512G)
1417675264 6396361856 8 freebsd-ufs (3.0T)
7814037120 8 - free - (4.0K)

=> 40 7814037088 ada1 GPT (3.6T)
40 472 1 freebsd-boot (236K)
512 8388608 2 freebsd-ufs (4.0G)
8389120 134217728 3 freebsd-swap (64G)
142606848 33554432 4 freebsd-ufs (16G)
176161280 134217728 5 freebsd-ufs (64G)
310379008 33554432 6 freebsd-ufs (16G)
343933440 1073741824 7 freebsd-ufs (512G)
1417675264 6396361856 8 freebsd-ufs (3.0T)
7814037120 8 - free - (4.0K)


The message I get during boot is:

gptboot: invalid primary GPT header
gptboot: invalid backup GPT header
gptboot: unable to load GPT


Any idea what the issue could be?
_______________________________________________
freebsd-...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questi...@freebsd.org"

[Frank Leonhardt]

I'd guess this is something to do with the BIOS and the way disks get
configured when you boot from the second one. You get this is the sector
read fails for some reason. Knowing the reason would be useful!

AIUI, gptboot goes off looking for a UFS partition to boot from - not
necessarily the disk it's just loaded from! Could it be looking at some
other drive (real, or imagined by the BIOS) that doesn't have a GPT
partition on it? Something to think about until Warran shows up :-)

Regards, Frank.

[Frank Leonhardt]

On 06/09/2017 21:55, Christos Chatzaras wrote:

P.S. Possible reasons for the invalid header message are that the sector
it reads in doesn't start with "EFI PART", the LBA the thing is read
from doesn't match the one stored in the GPT, the revision number is too
low or there there are too many entries in it. Looking at this again,
I'd guess that it WAS reading a sector from a valid drive, but something
about the sector was wrong. Either it IS the wrong sector, or it ended
up on a sector that didn't match on the second drive.

Why not dump out the GPT on both and take a look?

Are these drives identical?

[Frank Leonhardt]

On 06/09/2017 21:55, Christos Chatzaras wrote:

P.P.S. Dumb question - when you say you create a geom mirror, I assume
this isn't for the whole disk. If you did try this I'd expect it to
throw an error saying that the backup GPT was corrupt on ada0, but you
said it booted normally. If my assumption is wrong, you need to read up
on GPT and Geom Mirror compatibility.

[Christos Chatzaras]

>
> Why not dump out the GPT on both and take a look?
>

gpart show

=> 40 7814037088 ada0 GPT (3.6T)
40 472 1 freebsd-boot (236K)
512 8388608 2 freebsd-ufs (4.0G)
8389120 134217728 3 freebsd-swap (64G)
142606848 33554432 4 freebsd-ufs (16G)
176161280 134217728 5 freebsd-ufs (64G)
310379008 33554432 6 freebsd-ufs (16G)
343933440 1073741824 7 freebsd-ufs (512G)
1417675264 6396361856 8 freebsd-ufs (3.0T)
7814037120 8 - free - (4.0K)

=> 40 7814037088 ada1 GPT (3.6T)
40 472 1 freebsd-boot (236K)
512 8388608 2 freebsd-ufs (4.0G)
8389120 134217728 3 freebsd-swap (64G)
142606848 33554432 4 freebsd-ufs (16G)
176161280 134217728 5 freebsd-ufs (64G)
310379008 33554432 6 freebsd-ufs (16G)
343933440 1073741824 7 freebsd-ufs (512G)
1417675264 6396361856 8 freebsd-ufs (3.0T)
7814037120 8 - free - (4.0K)

dd if=/dev/ada0 of=./bootcode_ada0 count=1
1+0 records in
1+0 records out
512 bytes transferred in 0.013218 secs (38736 bytes/sec)

dd if=/dev/ada1 of=./bootcode_ada1 count=1
1+0 records in
1+0 records out
512 bytes transferred in 0.012451 secs (41120 bytes/sec)

diff bootcode_ada0 bootcode_ada1

diff shows the files have no differences

[Christos Chatzaras]

I use dump to take the backup from another server, then boot with mfsbsd and run:

gpart destroy -F ada0
gpart destroy -F ada1

gpart create -s gpt ada0
gpart add -b 40 -s 472 -t freebsd-boot ada0
gpart add -s 4G -t freebsd-ufs ada0
gpart add -s 64G -t freebsd-swap ada0
gpart add -s 16G -t freebsd-ufs ada0
gpart add -s 64G -t freebsd-ufs ada0
gpart add -s 16G -t freebsd-ufs ada0
gpart add -s 512G -t freebsd-ufs ada0
gpart add -t freebsd-ufs ada0

newfs -S 4096 -f 4096 -b 32768 ada0p2
newfs -S 4096 -f 4096 -b 32768 -U ada0p4
newfs -S 4096 -f 4096 -b 32768 -U ada0p5
newfs -S 4096 -f 4096 -b 32768 -U ada0p6
newfs -S 4096 -f 4096 -b 32768 -U ada0p7
newfs -S 4096 -f 4096 -b 32768 -U ada0p8

mount /dev/ada0p2 /mnt
cd /mnt
mkdir var tmp usr home home2
mount /dev/ada0p4 /mnt/tmp
mount /dev/ada0p5 /mnt/var
mount /dev/ada0p6 /mnt/usr
mount /dev/ada0p7 /mnt/home
mount /dev/ada0p8 /mnt/home2

cd /mnt/home2
fetch http://xxx.xxx.xxxx.xxx/FreeBSD-11.1-p1-20170905.tar.gz
tar xf FreeBSD-11.1-p1-20170905.tar.gz
rm FreeBSD-11.1-p1-20170905.tar.gz
mv FreeBSD-11.1-p1-20170905 dump

cd /mnt/tmp && restore -r -f /mnt/home2/dump/tmp.dump && rm restoresymtable
cd /mnt/usr && restore -r -f /mnt/home2/dump/usr.dump && rm restoresymtable
cd /mnt/var && restore -r -f /mnt/home2/dump/var.dump && rm restoresymtable
cd /mnt/home && restore -r -f /mnt/home2/dump/home.dump && rm restoresymtable
cd /mnt && restore -r -f /mnt/home2/dump/root.dump && rm restoresymtable
cd

gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptboot -i 1 ada0

umount /mnt/tmp
umount /mnt/var
umount /mnt/usr
umount /mnt/home
umount /mnt/home2
umount /mnt

gmirror label -vb load gm0p2 /dev/ada0p2
gmirror label -vb load gm0p3 /dev/ada0p3
gmirror label -vb load gm0p4 /dev/ada0p4
gmirror label -vb load gm0p5 /dev/ada0p5
gmirror label -vb load gm0p6 /dev/ada0p6
gmirror label -vb load gm0p7 /dev/ada0p7
gmirror label -vb load gm0p8 /dev/ada0p8

shutdown -r now

----

Then the server come back online and run:

sysctl kern.geom.debugflags=17
gpart backup ada0 | gpart restore -F ada1

gmirror insert gm0p2 /dev/ada1p2
gmirror insert gm0p3 /dev/ada1p3
gmirror insert gm0p4 /dev/ada1p4
gmirror insert gm0p5 /dev/ada1p5
gmirror insert gm0p6 /dev/ada1p6
gmirror insert gm0p7 /dev/ada1p7
gmirror insert gm0p8 /dev/ada1p8

gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0

shutdown -r now

-----

ada0 and ada1 are the same models and have the same bootcode.

BIOS has default settings, booting from UEFI, then Network boot, then USB disk, then hard disks.

If I don't run "gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1" and the disk boot priority is ada1 first and then ada0 it boots. If I write the bootcode to ada1 then the problem exist.

[Christos Chatzaras]

> P.P.S. Dumb question - when you say you create a geom mirror, I assume this isn't for the whole disk. If you did try this I'd expect it to throw an error saying that the backup GPT was corrupt on ada0, but you said it booted normally. If my assumption is wrong, you need to read up on GPT and Geom Mirror compatibility.

Yes see my previous message. I do seperate gmirror for each partition so GPT and gmirror don't write both at the last part of the disk.

[Christos Chatzaras]

> Sorry, I meant hex dump of GPT. I suspect you might find they are the same, but at different locations. See my post giving the possible reasons why the boot code gives the messages you are getting. Its got to be one of those, but which? I'd compile a new version of the loader that printed out the reason for failure, but this is not suitable for everyone.
>
> I'm just guessing. Mr Block of this parish is the expert on the GPT loader.

What command I can use for this?

[Christos Chatzaras]

>> Sorry, I meant hex dump of GPT. I suspect you might find they are the same, but at different locations. See my post giving the possible reasons why the boot code gives the messages you are getting. Its got to be one of those, but which? I'd compile a new version of the loader that printed out the reason for failure, but this is not suitable for everyone.
>>
>> I'm just guessing. Mr Block of this parish is the expert on the GPT loader.

hexdump -C -n 512 /dev/ada0p1

and

hexdump -C -n 512 /dev/ada1p1

show the same result.

I install the bootcode in another server in both disks again and the boot was successful from 2nd drive.

These 2 servers have only different motherboard (the server with the problem has Fujitsu D3401-H2 and the server without the problem has Fujitsu D3401-H1) and different disks (the server with the problem has Seagate ST4000NM0245-1Z2107 and the the server without the problem has Seagate ST4000NM0024-1HT178). The BIOS for server with problem is V5.0.0.12 R1.8.0 for D3401-H2x (Release Date: 05/15/2017) and BIOS for server without problem is V5.0.0.11 R1.21.0 for D3401-H1x (Release Date: 05/15/2017).

[Christos Chatzaras]

The datacenter swap the first with the second disk. Now it doesn't boot if the new first disk (previously was the second) is selected in BIOS. If I configure BIOS to boot from second disk (previously was the first) it boots. So bootcode is not installed correctly in previously second disk? It doesn't make sense.

[Christos Chatzaras]

Finally the problem is not related to this server. I did a fresh install in another server "that didn't have the same issue" and it finally has the same issue.

As you can see in my previous messages I install bootcode in first disk while in mfsbsd 11.0:

gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptboot -i 1 ada0

and when I finish the installation and reboot in FreeBSD 11.1 I install the bootcode for 2nd disk:

gpart bootcode -b /boot/pmbr -p /mnt/boot/gptboot -i 1 ada1

This way the problem exist.


But if I install bootcode for both disks while in mfsbsd 11.0 then I can boot from both disks:

gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptboot -i 1 ada0

gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptboot -i 1 ada1


So my conclusion is that I can install the bootcode during the installation from mfsbsd 11.0 but not from freebsd 11.1

The mfsbsd version is:

FreeBSD rescue 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0 r306420: Thu Sep 29 01:43:23 UTC 2016 ro...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64


But if I don't install the bootcode on 2nd disk during the installation from mfsbsd and after the installation boot on mfsbsd again and run:

mount /dev/ada0p2 /mnt
gpart bootcode -b /mnt/boot/pmbr -p /mnt/boot/gptboot -i 1 ada1

then it doesn't boot from second disk.

So the question is why it works if I install bootcode during the installation using mfsbsd but not after I reboot back in mfsbsd?

[Christos Chatzaras]

Another strange thing:

If I do the installation using mfsbsd 10.3 and write the bootcode to 1st disk during the installation and then reboot to freebsd 11.3 and write the bootcode for 2nd disk I can boot from both disks.

I suspect now the mfsbsd 11.0 provided by the datacenter. I will try to create a mfsbsd 11.1 on my own and boot from it. Then I will try to do the installation again.

[Christos Chatzaras]

I boot using mfsbsd 11.1 but the same issue happens. Any idea why it only works with mfsbsd 10.3?

Even with mfsbsd 10.3 used for installation and writing the bootcode for first disk, I write the bootcode for second disk after installation using FreeBSD 11.3.

Any idea why only works using mfsbsd 10.3 ?