failed to boot from system partition 1 after installing FreeBSD 14.0-current world

[Cheng Cui]

Hello,

I am using pc759 in Experiment: fbsd-transport/OS, and am using this node to upgrade a FreeBSD image. My goal is to create a FreeBSD 14-current (head) from a FBSD13-STABLE image. However, after "make installworld" and reboot, the console output hung at the following state:

"Loading disk 0x80 MBR and booting system partition 1
-"

I suspect I might have messed up with the existing MBR. But I am far from checking and fixing it myself. Would you please take a look or help direct me with a wiki/FAQ to dig up? Thanks.

Best Regards,

Cheng Cui

[Mike Hibler]

There is an unfortunate interaction between our ancient network boot code
(that is based on the FreeBSD PXE boot) and the on-disk boot loader. So
after you do an install, you have to link /boot/loader to /boot/loader_4th
rather than the LUA-based loader.

At the moment, I will have to fix it up for you. Can I take over pc759 for
a short time?

> --
> You received this message because you are subscribed to the Google Groups
> "emulab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to emulab-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/
> emulab-users/
> CAGaXui%2BO90iFDoAusyLkiKf8YeRpz%2BEQiNJC_H-VANRvg1CXgw%40mail.gmail.com.

[Cheng Cui]

On Wednesday, March 15, 2023 at 4:48:22 PM UTC-4 Mike Hibler wrote:

There is an unfortunate interaction between our ancient network boot code
(that is based on the FreeBSD PXE boot) and the on-disk boot loader. So
after you do an install, you have to link /boot/loader to /boot/loader_4th
rather than the LUA-based loader.

At the moment, I will have to fix it up for you. Can I take over pc759 for
a short time?

Yes, please go ahead. I just rebooted the node to keep it from swapping out. thanks

Cheng

[Mike Hibler]

Okay, that didn't work. Maybe something else going on. Still looking.

> --
> You received this message because you are subscribed to the Google Groups
> "emulab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to emulab-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/

> emulab-users/5c887250-14b1-4e30-934b-8ed994eef4afn%40googlegroups.com.

[Cheng Cui]

All I know is upgrading (installworld) from FBSD13.1/13.2 to FBSD13-stable works. Maybe something requires attention in FreeBSD 14.0.

If you need more time, please go ahead and keep the node. Thanks.

Cheng

[Mike Hibler]

It did work, I just didn't to it right...hard link, not sumlink.
It is back up now. A couple of things.

* The image you started with has a strange disk layout that we abandoned
after I made that image. A better starting point would be the new
FBSD132-64-STD image (which is actually 13.2-RC2). It has the GPT
partition layout we are using now that can be booted on both legacy
BIOS and UEFI machines.

* There is an FBSD-64-CURRENT image which is unfortunately, not that
current. I haven't updated it in about a year. It also has an older
MBR-based disk layout. So you probably don't want to use this one.

> emulab-users/3cb07341-4410-48d8-bef7-cae17e307581n%40googlegroups.com.

[Cheng Cui]

Thanks Mike. Let me make an image out of the current one, and then build a new one based on the FBSD132-64-STD.

Also thanks for the suggestion. :)

Cheng

[Cheng Cui]

Hi Mike,

Just want to confirm that after rebuild/install world of FreeBSD 14, I should make a copy of /boot/loader to /boot/loader_4th, right?

Cheng

[Mike Hibler]

Yes, make sure those two are hard linked together.

> > emulab-users/3cb07341-4410-48d8-bef7-cae17e307581n%40googlegroups.com

> .
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "emulab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to emulab-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/

> emulab-users/d482a486-f967-4922-9f47-2bf338dd1dbcn%40googlegroups.com.

[Cheng Cui]

Still hung at the booting phase. Experiment: fbsd-transport/OS using pc831. Here is what I hard linked. Please take a look.

root@n1:/boot # ln /boot/loader /boot/loader.4th
ln: /boot/loader.4th: File exists
root@n1:/boot # rm /boot/loader.4th
root@n1:/boot # ln /boot/loader /boot/loader.4th
root@n1:/boot # ll loader
-r-xr-xr-x  4 root  wheel  503808 Mar 16 14:59 loader*
root@n1:/boot # ll loader.4th
-r-xr-xr-x  4 root  wheel  503808 Mar 16 14:59 loader.4th*
root@n1:/boot # date
Thu Mar 16 15:04:03 MDT 2023

[Mike Hibler]

That should be loader_4th and not loader.4th. The latter is a file of
Forth code and not the binary for the boot loader.

> emulab-users/dc43070b-0cd4-4f8c-a61b-e12423b1d9d5n%40googlegroups.com.

[Cheng Cui]

Oh, my bad. Let me re-work on it. Thanks

[Cheng Cui]

Still not working. Not sure what went wrong.

root@n1:/boot # ll loader

-r-xr-xr-x  3 root  wheel  503808 Mar 16 17:40 loader*
root@n1:/boot # ll loader_4th
-r-xr-xr-x  1 root  wheel  442368 Mar 16 17:40 loader_4th*
root@n1:/boot # ln /boot/loader /boot/loader_4th
ln: /boot/loader_4th: File exists
root@n1:/boot # rm loader_4th
root@n1:/boot # ln /boot/loader /boot/loader_4th
root@n1:/boot # ll loader_4th
-r-xr-xr-x  4 root  wheel  503808 Mar 16 17:40 loader_4th*

[Mike Hibler]

Well, I hate to tell you this, but you linked the wrong way. You made
loader_4th a copy of loader. Do:

cd /boot
rm loader
ln loader_4th loader

> > emulab-users/dc43070b-0cd4-4f8c-a61b-e12423b1d9d5n%40googlegroups.com

> .
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "emulab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to emulab-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/

> emulab-users/6d70b7e5-c2d6-4d9b-ab17-9a68df272e9dn%40googlegroups.com.

[Cheng Cui]

Thanks. Hard link loader to loader_4th works. And the image can be booted in HW d430.

root@n1:/ # rm /boot/loader

root@n1:/ # ln /boot/loader_4th /boot/loader

I also tried LOADER_DEFAULT_INTERP=4th as in https://reviews.freebsd.org/rS338050, but it does not work after installing the world.

root@n1:/usr/src # "make -j32 buildworld -D LOADER_DEFAULT_INTERP=4th"

However, then I tried to boot this image FBSD14x on a HW d710 node: pc515, the booting stopped with the following error from its console.

Do you know how to fix these warnings and the mounting problem?

...

Setting hostuuid: 44454c4c-4300-1036-8038-b8c04f4b3731.
Setting hostid: 0xa01d88ee.
warning: total configured swap (16777216 pages) exceeds maximum recommended amount (12191136 pages).
warning: increase kern.maxswzone or reduce amount of swap.
Starting file system checks:
/dev/da0p3: 87893 files, 987950 used, 15260881 free (30513 frags, 1903796 blocks, 0.2% fragmentatio)
mount: /dev/da0p3: Operation not permitted
Mounting root filesystem rw failed, startup aborted
ERROR: ABORTING BOOT (sending SIGTERM to parent)!
2023-03-17T10:53:51.964582-06:00 - init 1 - - /bin/sh on /etc/rc terminated abnormally, going to sie
Enter full pathname of shell or RETURN for /bin/sh:

[Mike Hibler]

Do you still have a machine with that image running successfully?
I have to see exactly what the partition table layout looks like.
If it is what I think, then I need to fix up some state for your image.

> emulab-users/685f60fb-0892-4e36-933b-aad5c70bbfacn%40googlegroups.com.

[Cheng Cui]

You can use pc753 that runs this image.

[Mike Hibler]

So it is our new ondisk format. The image got properly identified in
the database, and the image load appeared to do the right thing, but
something is clearly amiss. I will look at it this weekend when I get
some time.

> emulab-users/50d537d8-c5e2-4b84-831c-b08226362152n%40googlegroups.com.

[Cheng Cui]

I will be out of my desk for the next two hours. And pc753 has been idle for almost one hour.

If it auto swapped out, please just let me know how to fix it, or you can keep it from auto swapping out.

Thanks,

Cheng

[Cheng Cui]

Thanks. In the mean time, I will just create a separate image based on d710.

[Cheng Cui]

I created a FBSD14-current image (FBSD14x_d710) based on HW d710. And verified it on HW d430. Interestingly, it works.

I remember when I created the image based on d710, the node didn't reboot immediately from console when I clicked the "clone" web button.

And I noticed the console prompted something like "ensure path for shell or return to enter the current shell /usr/bin/shell", so I returned and

it prompted to root command line, and I typed "reboot". Then, the emulab prepare process started immediately and the file system was

dismounted. So I guess this is the root cause for the other image FBSD14x (created from a d430 node) which does not work on d710,

because when I started that image creation process, I left and didn't notice the console, so it probably was rebooted forcefully and showed

file system was not dismounted correctly.

Cheers, now I have a FBSD14 image that works on both d710 and d430. Thanks!

Cheng

[Mike Hibler]

Yes, I discovered yesterday that the script setting up the prepare process
was not correct. You cloned this image originally from the FBSD13-STABLE
image? That image should go away now, I was just using it to test the
latest version of the "bnxt" driver on some of our nodes that had a
previously unsupported NIC. Since 13.2 has that driver, we will have an
image that works going forward.

What happened in your "bad" image is that the prepare script did not run
and the entry for swap in /etc/fstab did not get removed. When that image
got loaded on a new machine, the post-imaging customization script managed
to modify the swap entry to be the same as the root entry. So when the image
booted, it turned on swap on top of the root filesystem. The result was not
pretty...

Let me know when you have your image running on a node and I will make sure
the Emulab scripts are up to date and then you can save a new copy of the
image.

> > > > emulab-users/6d70b7e5-c2d6-4d9b-ab17-9a68df272e9dn%

> 40googlegroups.com.
> > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "emulab-users" group.
> > > To unsubscribe from this group and stop receiving emails from it,
> send an
> > email
> > > to emulab-users...@googlegroups.com.
> > > To view this discussion on the web visit https://groups.google.com/
> d/
> > msgid/

> > > emulab-users/685f60fb-0892-4e36-933b-aad5c70bbfacn%

> 40googlegroups.com.
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups
> > "emulab-users" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send an email
> > to emulab-users...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/
> msgid/

> > emulab-users/50d537d8-c5e2-4b84-831c-b08226362152n%40googlegroups.com

> .
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "emulab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to emulab-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/

> emulab-users/3a1a875b-a411-4a0c-8f67-324f12ad41e7n%40googlegroups.com.

[Cheng Cui]

I deleted the bad image. The good image is based on 13.2 with the correct loader and with finished prepare process.

Now the running node pc712 is using this good image FBSD14x_d710. Please take a look. Thanks,

Cheng

[Mike Hibler]

I fixed the one script that would have prevented image creation from
happening fully automatically. You should recapture the image from pc712.

If you are on the console when you remake the image, you will see it
shutting down, then running the prepare script, and then rebooting.

> emulab-users/e7c0b5f5-2204-4e58-b3ed-c4ce2aa1249an%40googlegroups.com.

[Cheng Cui]

Thanks Mike! I made a new image from pc712, and it looks good.

Cheng

[Cheng Cui]

Hi Mike,

A followup question. Given the script needs to be corrected in the image, is there anyway that Emulab client code can be synced to stable or good version from a user's perspective? I am asking because I may mess up with new images, so I guess I have to store multiple backup images.

Cheng

[Cheng Cui]

Looks I have to manually update the script in image according to the fixes in emulab-dev.

Cheng

[Mike Hibler]

[ Sorry, thought I had sent this. ]

No, we don't have a great way to do this. However, as long as you create
new images from one of the current standard images or from the image you
have fixed, this particular problem won't affect you again.

Whenever we introduce new features for the clientside we do encourage people
to update their images by re-customizing a current standard image if they
need those features. Automatically applying updates to existing custom images
could break those images, so we don't even try.

On Thu, Mar 23, 2023 at 07:42:17AM -0700, Cheng Cui wrote:
> Hi Mike,
>

> > > emulab-users/3a1a875b-a411-4a0c-8f67-324f12ad41e7n%

> 40googlegroups.com.
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups
> > "emulab-users" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send an email
> > to emulab-users...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/
> msgid/

> > emulab-users/e7c0b5f5-2204-4e58-b3ed-c4ce2aa1249an%40googlegroups.com

> .
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "emulab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to emulab-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/

> emulab-users/3f68effe-37ef-407d-8490-dab893bdb667n%40googlegroups.com.