Re: CTF: UEFI HTTP boot support

[Miguel C]

I've been trying out FreeBSD with raspberry Pi4 (4GB) and wanted to see
what the state of HTTP BOOT is in FreeBSD, so I bumped into this!

I'm curious if it should be possible to point to a img/iso directly (I
tried to use the img.xz unpacked it and make it available on a local web
server and that didn't seem to work for me) but maybe thats cause those
images miss something, so arm64 aside does that work for amd64? I.E. using
the bootonly.iso?


And on the other hand is there any doc on how to set up dhcp/http specific
to FreeBSD similar to https://en.opensuse.org/UEFI_HTTPBoot_Server_Setup?

I looked into https://www.freebsd.org/doc/handbook/network-diskless.html
but that doesn't seem to be up to date (or at least it focuses only on PXE
and TFTP).

For clarification my ultimate goal is to use a few pi4's as "thin clients",
so eventually I will have to setup an image of the system with the needed
software (freerdp) but for starters I just wanted to check if pointing
directly to a img/iso would work and that does not seem to be the case.

Thanks.
_______________________________________________
freebsd...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"

[Rodney W. Grimes]

> I've been trying out FreeBSD with raspberry Pi4 (4GB) and wanted to see
> what the state of HTTP BOOT is in FreeBSD, so I bumped into this!
>
> I'm curious if it should be possible to point to a img/iso directly (I
> tried to use the img.xz unpacked it and make it available on a local web
> server and that didn't seem to work for me) but maybe thats cause those
> images miss something, so arm64 aside does that work for amd64? I.E. using
> the bootonly.iso?

One problem you run into in attemtping this is even if you get an
image downloaded and started that image is being provided by some
memory device driver that emulates some type of iso device.
FreeBSD does not have a driver for that device so once the kernel
gets to the point of mounting its root file system it falls on
its face with a mountroot failure.

>
> And on the other hand is there any doc on how to set up dhcp/http specific
> to FreeBSD similar to https://en.opensuse.org/UEFI_HTTPBoot_Server_Setup?

Since Linux uses this idea of a kernel payload and an initrd payload
to boot with it is much easier to get these 2 things over the network
and then have a workable system. FreeBSD does not have the initrd
payload and that complicates things, you need a functionaly filesystem
avaliable at the end of kernel initilization.

>
> I looked into https://www.freebsd.org/doc/handbook/network-diskless.html
> but that doesn't seem to be up to date (or at least it focuses only on PXE
> and TFTP).

Yes, old but workable. I have a more advanced system that supports NFS
booting using NFS support in PXE. The only thing done via tftp is to
upgrade the PXE running on the client to one that speaks NFS, then the
kernel is loaded via NFS and the root file system is later provided
via NFS. The use of NFS provides very fast boots, and I do not need
a web server to do it :-).

> For clarification my ultimate goal is to use a few pi4's as "thin clients",
> so eventually I will have to setup an image of the system with the needed
> software (freerdp) but for starters I just wanted to check if pointing
> directly to a img/iso would work and that does not seem to be the case.

I would strongly suggest use of NFS instead of trying to provide an
ISO image, as you no longer need to store the ISO in memory on the
client box, and with a pi4 your already memory contrained.

> Thanks.
> _______________________________________________
> freebsd...@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"

--
Rod Grimes rgr...@freebsd.org

[Simon J. Gerraty]

Rodney W. Grimes <freeb...@gndrsh.dnsmgr.net> wrote:
> > I'm curious if it should be possible to point to a img/iso directly (I
> > tried to use the img.xz unpacked it and make it available on a local web
> > server and that didn't seem to work for me) but maybe thats cause those
> > images miss something, so arm64 aside does that work for amd64? I.E. using
> > the bootonly.iso?
>
> One problem you run into in attemtping this is even if you get an
> image downloaded and started that image is being provided by some
> memory device driver that emulates some type of iso device.
> FreeBSD does not have a driver for that device so once the kernel
> gets to the point of mounting its root file system it falls on
> its face with a mountroot failure.

Are you refering to something like:

vfs.root.mountfrom="cd9660:/dev/md0.uzip"

we boot that way all the time.

[Miguel C]

Thanks for the tips, but I was really looking for HTTP BOOT info no NFS,
that's why I replied to this thread.

I might look into that at some point if HTTP BOOT is not an option of
course, but this thread is about a Call for Testers for UEFI HTTP BOOT, not
NFS and I would like to help test, the pi4 project just conveniently
touches on the same use case (an it also does have support for http boot
using https://rpi4-uefi.dev/) so I'm curious if I can test that way.

Other than the iso I can ofc attempt the dhcp+dns+webserver setup but for
that I would need a bit more guidance as the linked URL here is linux
centric, hence why some docs would help.

[Rodney W. Grimes]

> Rodney W. Grimes <freeb...@gndrsh.dnsmgr.net> wrote:
> > > I'm curious if it should be possible to point to a img/iso directly (I
> > > tried to use the img.xz unpacked it and make it available on a local web
> > > server and that didn't seem to work for me) but maybe thats cause those
> > > images miss something, so arm64 aside does that work for amd64? I.E. using
> > > the bootonly.iso?
> >
> > One problem you run into in attemtping this is even if you get an
> > image downloaded and started that image is being provided by some
> > memory device driver that emulates some type of iso device.
> > FreeBSD does not have a driver for that device so once the kernel
> > gets to the point of mounting its root file system it falls on
> > its face with a mountroot failure.
>
> Are you refering to something like:
>
> vfs.root.mountfrom="cd9660:/dev/md0.uzip"
>
> we boot that way all the time.

What provides the cd9660 driver to FreeBSD? When you load the .iso
over a network card, aka PXE/HTTP, the code that does that usually
creates a ram disk and a "fake cd drive" that stops working as soon
as you stop running the PXE code, which means the device does not
show up while the kernel is probing. FreeBSD simply does not support
bios devices in this manner, unless something has changed I am totally
unaware of.


--
Rod Grimes rgr...@freebsd.org

[Andreas Nilsson]

Am I just misremembering or can't you get freebsd to load an mfsroot-image,
which can act as rw fs ?

I seem to remember pc-bsd DVDs using this say 7 years ago.

You would of course have to modify/build your own iso image with the
mfsroot-image on it.

Best regards
Andreas

[Simon J. Gerraty]

Rodney W. Grimes <freeb...@gndrsh.dnsmgr.net> wrote:
> > Are you refering to something like:
> >
> > vfs.root.mountfrom="cd9660:/dev/md0.uzip"
> >
> > we boot that way all the time.
>
> What provides the cd9660 driver to FreeBSD? When you load the .iso
> over a network card, aka PXE/HTTP, the code that does that usually
> creates a ram disk and a "fake cd drive" that stops working as soon

We don't use PXE much except in a bringup lab, and then I think we use
NFS for rootfs.

Normally if iso is comming from network it is to do an install
eg loader is doing 'install tftp://host/install.tar'

The "fake cd drive" is in the kernel, loader just copies the iso into
memory like any other module, and by the time that's done you just
reboot into the newly installed system, which again uses

vfs.root.mountfrom="cd9660:/dev/md0.uzip"

but in that case the rootfs is an iso image on local disk.

The rootfs iso is minimal - enough to fsck and mount real media
and initialize Verified Exec.
It improves our chances of being able to recover from severe disk
corruption after cleaning lady pulls the cord, to vaccuum ;-)

--sjg

[Miguel C]

I don't think it needs the cdrom driver does it? UEF accepts a iso (since
2.5) has a http boot URI and that is mounted as a ramdisk via the http boot
driver itself (I think, this is all a bit new to me too :)


Most of the info I found on it is from tiano core:
https://github.com/tianocore/tianocore.github.io/wiki/HTTP-Boot)



I did just find the presentation/demo of this working with the FreeBSD
amd64 iso that I bumped into before:

https://bhyvecon.org/bhyveconOttawa2019-DScott.pdf

[Rodney W. Grimes]

Yes, but that is *not* an .iso as was asked above.

> I seem to remember pc-bsd DVDs using this say 7 years ago.

Yes.

>
> You would of course have to modify/build your own iso image with the
> mfsroot-image on it.

Perhaps, if you can get the mfsroot image into memory by the loader
while you still have access to the bios code.

> Best regards
> Andreas

--
Rod Grimes rgr...@freebsd.org

[Rodney W. Grimes]

> Rodney W. Grimes <freeb...@gndrsh.dnsmgr.net> wrote:
> > > Are you refering to something like:
> > >
> > > vfs.root.mountfrom="cd9660:/dev/md0.uzip"
> > >
> > > we boot that way all the time.
> >
> > What provides the cd9660 driver to FreeBSD? When you load the .iso
> > over a network card, aka PXE/HTTP, the code that does that usually
> > creates a ram disk and a "fake cd drive" that stops working as soon
>
> We don't use PXE much except in a bringup lab, and then I think we use
> NFS for rootfs.

Probably much like what I do once my kernel is loaded.

> Normally if iso is comming from network it is to do an install
> eg loader is doing 'install tftp://host/install.tar'
>
> The "fake cd drive" is in the kernel, loader just copies the iso into
> memory like any other module, and by the time that's done you just
> reboot into the newly installed system, which again uses
>
> vfs.root.mountfrom="cd9660:/dev/md0.uzip"

^^^

Argh, the cd9660 confused me, I think your doing a
"root on mfs/md"?

> but in that case the rootfs is an iso image on local disk.
>
> The rootfs iso is minimal - enough to fsck and mount real media
> and initialize Verified Exec.
> It improves our chances of being able to recover from severe disk
> corruption after cleaning lady pulls the cord, to vaccuum ;-)
>
> --sjg
> _______________________________________________
> freebsd...@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
>

--
Rod Grimes rgr...@freebsd.org

[Rebecca Cran]

On 6/16/20 5:17 AM, Miguel C wrote:

> I've been trying out FreeBSD with raspberry Pi4 (4GB) and wanted to see
> what the state of HTTP BOOT is in FreeBSD, so I bumped into this!
>
> I'm curious if it should be possible to point to a img/iso directly (I
> tried to use the img.xz unpacked it and make it available on a local web
> server and that didn't seem to work for me) but maybe thats cause those
> images miss something, so arm64 aside does that work for amd64? I.E. using
> the bootonly.iso?

Unfortunately HTTP boot only works as far as the kernel: UEFI fetches
loader.efi, the loader fetches and runs the kernel over HTTP -- and then
you need to use NFS to mount the filesystem (or have a local root
filesystem).


UEFI also has RamDisk support, but I don't think that's for remote
ISO/disk files, just local files.


--
Rebecca Cran

[Miguel C]

On Tue, Jun 16, 2020 at 7:25 PM Rebecca Cran <reb...@bsdio.com> wrote:

> On 6/16/20 5:17 AM, Miguel C wrote:
>
> > I've been trying out FreeBSD with raspberry Pi4 (4GB) and wanted to see
> > what the state of HTTP BOOT is in FreeBSD, so I bumped into this!
> >
> > I'm curious if it should be possible to point to a img/iso directly (I
> > tried to use the img.xz unpacked it and make it available on a local web
> > server and that didn't seem to work for me) but maybe thats cause those
> > images miss something, so arm64 aside does that work for amd64? I.E.
> using
> > the bootonly.iso?
>
> Unfortunately HTTP boot only works as far as the kernel: UEFI fetches
> loader.efi, the loader fetches and runs the kernel over HTTP -- and then
> you need to use NFS to mount the filesystem (or have a local root
> filesystem).
>
>

Thanks for the reply , I can work with that for a live system still better
than tftp, http+nfs should be that hard.

> UEFI also has RamDisk support, but I don't think that's for remote
> ISO/disk files, just local files.
>

> As for the ISO it does seem to work for remote ISO files, the bhyve con
presentantion suggests the same and I was able to boot the ubuntu arm64
install iso using the direct link as a HTTP BOOT entry,.

�>Start HTTP Boot over IPv4....
> Station IP address is 172.16.50.62
>
> URI: http:/?172.16.50.106/uarm64.iso
> File Size: 916357120 Bytes
> Downloading...26%
>

But we also don't seem to have iso images for the raspberry pi, so it might
not work there, this does sugget just a link to the .efi file would work:
https://github.com/jljusten/tianocore/wiki/HTTP-Boot but I tired that with
no succes

In any case from what you're saying... for a live system I need http + nfs
for rootfs.

But I'm still clueless how to set that up for FreeBSD, the guide mentioned
here is linux centric, what need to live on the http server side?

[Miguel C]

Actually that was bad interpertation on my side for a ramdisk to be used it
needs o be iso/img format.

[Miguel C]

Update so I was trying the ubuntu boot again and noticed it required a few
tries (literary choose that boot entry ... fail... try again... 3 times and
it worked ) Could be a bug in rpi4-uefi.dev though but it proves it works.

I then tried openbsd miniroot img (just case its smaller honestly :D ) and
it also booted the img file, but not FreeBSD (using 12.1 )

I noticed the openbsd image includes startup.nsh inside ef/boot/ alongside
bootaa64.efi, unsure if that's what FreeBSD would need.

[Simon J. Gerraty]

Rodney W. Grimes <freeb...@gndrsh.dnsmgr.net> wrote:
> > The "fake cd drive" is in the kernel, loader just copies the iso into
> > memory like any other module, and by the time that's done you just
> > reboot into the newly installed system, which again uses
> >
> > vfs.root.mountfrom="cd9660:/dev/md0.uzip"
> ^^^
>
> Argh, the cd9660 confused me, I think your doing a
> "root on mfs/md"?

loader.conf says

rootfs_load="yes"
rootfs_name="contents.izo"
rootfs_type="md_image"
vfs.root.mountfrom="cd9660:/dev/md0.uzip"

contents.izo is uzip'd contents.iso which file(1)
describes as ISO 9660 CD-ROM filesystem data ''

That's for normal boot, for the loader 'install' command
it expects an uncompressed iso for rootfs.

[Rodney W. Grimes]

> On 6/16/20 5:17 AM, Miguel C wrote:
>
> > I've been trying out FreeBSD with raspberry Pi4 (4GB) and wanted to see
> > what the state of HTTP BOOT is in FreeBSD, so I bumped into this!
> >
> > I'm curious if it should be possible to point to a img/iso directly (I
> > tried to use the img.xz unpacked it and make it available on a local web
> > server and that didn't seem to work for me) but maybe thats cause those
> > images miss something, so arm64 aside does that work for amd64? I.E. using
> > the bootonly.iso?
>
> Unfortunately HTTP boot only works as far as the kernel: UEFI fetches
> loader.efi, the loader fetches and runs the kernel over HTTP -- and then
> you need to use NFS to mount the filesystem (or have a local root
> filesystem).
>
>
> UEFI also has RamDisk support, but I don't think that's for remote
> ISO/disk files, just local files.
>

Does FreeBSD kernel have a driver that can talk to the UEFI ramdisk?

I believe that is how many of the other distro's
are actually able to do this "load the stock .iso over the
network and just run from that in memory." I have booted
serveral things this way and it is a nice feature.

> --
> Rebecca Cran
--
Rod Grimes rgr...@freebsd.org

[Rodney W. Grimes]

> Rodney W. Grimes <freeb...@gndrsh.dnsmgr.net> wrote:
> > > The "fake cd drive" is in the kernel, loader just copies the iso into
> > > memory like any other module, and by the time that's done you just
> > > reboot into the newly installed system, which again uses
> > >
> > > vfs.root.mountfrom="cd9660:/dev/md0.uzip"
> > ^^^
> >
> > Argh, the cd9660 confused me, I think your doing a
> > "root on mfs/md"?
>
> loader.conf says
>
> rootfs_load="yes"
> rootfs_name="contents.izo"
> rootfs_type="md_image"
> vfs.root.mountfrom="cd9660:/dev/md0.uzip"
>
> contents.izo is uzip'd contents.iso which file(1)
> describes as ISO 9660 CD-ROM filesystem data ''
>
> That's for normal boot, for the loader 'install' command
> it expects an uncompressed iso for rootfs.

Ok, now the puzzle is how much work to get from a stock FreeBSD .iso
image to something that works with this. Obviously we need a non-stock
/boot/loader.conf file, or to type some commands manually at a loader
prompt. I believe the stock GENERIC kernel has the md_root support
for this already, so it may not be that hard to do.


--
Rod Grimes rgr...@freebsd.org

[Rodney W. Grimes]

> On Tue., Jun. 16, 2020, 8:35 a.m. Rodney W. Grimes, <

> freeb...@gndrsh.dnsmgr.net> wrote:
>
> > > I've been trying out FreeBSD with raspberry Pi4 (4GB) and wanted to see
> > > what the state of HTTP BOOT is in FreeBSD, so I bumped into this!
> > >
> > > I'm curious if it should be possible to point to a img/iso directly (I
> > > tried to use the img.xz unpacked it and make it available on a local web
> > > server and that didn't seem to work for me) but maybe thats cause those
> > > images miss something, so arm64 aside does that work for amd64? I.E.
> > using
> > > the bootonly.iso?
> >

> > One problem you run into in attemtping this is even if you get an
> > image downloaded and started that image is being provided by some
> > memory device driver that emulates some type of iso device.
> > FreeBSD does not have a driver for that device so once the kernel
> > gets to the point of mounting its root file system it falls on
> > its face with a mountroot failure.
> >
>

> I donno what you are talking about Rodney, frankly! You information might
> be way outdated, like 15 years outdated. :) FreeBSD comes with very decent
> compressed image support in MD(4)+geom_uzip, which could be just UFS
> snapshot or something created with mkimg utility. That said image could be
> then either loaded after the kernel or embedded into one. Using this
> approach we deploy our systems here, both kernel and all root + python
> interpreter + custom gui installer fit into 40mb ISO but apart from loader
> bits it's just two files.

Max,
Let me try explain what this user is actually experienceing,
and that is taking a box stock linux distro, sticking it on a webserver
and using PXE/HTTP booting to a running system. FreeBSD can not
achieve that today WITHOUT some additional work. All that
stuff like UFS snapshot, mkimg utlity, embeding the image is a ton
of work compared to what others are doing.

This person probably does not even have a running FreeBSD box to do
any of your suggested solutions on.

So give me some credit, I have only been doing "diskless" since 1982,
and actually do exactly what the OP is doing with Esxi, Ubuntu, Windows
Installers, etc. Just my choice of protocol at the PXE layer is NFS
instead of HTTP, but my config files can do HTTP with 1 variable change
and point a web server at the root of my boot images tree.

I even have a menu entry that sends me off to:
https://netboot.xyz/

Regards,
Rod

>
> -Max

> > > Thanks.

[Rebecca Cran]

> On Jun 17, 2020, at 11:40 AM, Rodney W. Grimes <freeb...@gndrsh.dnsmgr.net> wrote:
>
> Does FreeBSD kernel have a driver that can talk to the UEFI ramdisk?

I’m fairly sure UEFI generates it as a standard CD drive.

—
Rebecca Cran

[Warner Losh]

On Wed, Jun 17, 2020 at 11:53 AM Rodney W. Grimes <

freeb...@gndrsh.dnsmgr.net> wrote:

> > Rodney W. Grimes <freeb...@gndrsh.dnsmgr.net> wrote:
> > > > The "fake cd drive" is in the kernel, loader just copies the iso into
> > > > memory like any other module, and by the time that's done you just
> > > > reboot into the newly installed system, which again uses
> > > >
> > > > vfs.root.mountfrom="cd9660:/dev/md0.uzip"
> > > ^^^
> > >
> > > Argh, the cd9660 confused me, I think your doing a
> > > "root on mfs/md"?
> >
> > loader.conf says
> >
> > rootfs_load="yes"
> > rootfs_name="contents.izo"
> > rootfs_type="md_image"
> > vfs.root.mountfrom="cd9660:/dev/md0.uzip"
> >
> > contents.izo is uzip'd contents.iso which file(1)
> > describes as ISO 9660 CD-ROM filesystem data ''
> >
> > That's for normal boot, for the loader 'install' command
> > it expects an uncompressed iso for rootfs.
>
> Ok, now the puzzle is how much work to get from a stock FreeBSD .iso
> image to something that works with this. Obviously we need a non-stock
> /boot/loader.conf file, or to type some commands manually at a loader
> prompt. I believe the stock GENERIC kernel has the md_root support
> for this already, so it may not be that hard to do.
>

Looking at the code, I think MD_ROOT alone is insufficient here...

If there's no MD root provided, we look for the symbols mfs_root and
mfs_root_end, which I think means that rootfs_ in the above example needs
to be md_root_ instead so that we find it.

You may need to have a custom kernel with 'options MD_ROOT_READONLY'
because isofs is read-only.

And there's a small chance you may need to define ROOTDEVNAME in the build
as well to be "cd9660:/dev/md0.uzip"

Every time I do stuff like this I have to re-puzzle it out, alas, but these
should give you some guide posts. It should be better documented in md(4),
but isn't at the moment.

I'd honestly try to get this setup working first loading all the files off
a local disk before layering in the networking on top of that.

Warner

[Warner Losh]

On Wed, Jun 17, 2020 at 12:06 PM Rebecca Cran <reb...@bsdio.com> wrote:

>
> > On Jun 17, 2020, at 11:40 AM, Rodney W. Grimes <
> freeb...@gndrsh.dnsmgr.net> wrote:
> >
> > Does FreeBSD kernel have a driver that can talk to the UEFI ramdisk?
>
> I’m fairly sure UEFI generates it as a standard CD drive.
>

I missed the start of this thread, so maybe I'm missing a key detail.
However, I thought UEFI didn't have a RAM-disk, per se, but that we could
load memory areas and pass that into the kernel using freebsd-only methods.
But UEFI is a bit weird, so maybe it will generate a virtual cdrom...

Warner

[Rodney W. Grimes]

I was a bit worried about that, but hopefull. We do work out of the
box with a NFS root as long as the NIC is found during boot. And
given that I load the loader over NFS the loader can also find my
/boot/ directory and the files in it, so that part is already solved.

> If there's no MD root provided, we look for the symbols mfs_root and
> mfs_root_end, which I think means that rootfs_ in the above example needs
> to be md_root_ instead so that we find it.

Isnt this all handled by the loader? I think we have 2 slightly
different cases here. THe one sjg shows up, where you actually
load the md_image from a seperate file, and the case your talking
about where you actually embed the kernel and image into a single
file.

>
> You may need to have a custom kernel with 'options MD_ROOT_READONLY'
> because isofs is read-only.
>
> And there's a small chance you may need to define ROOTDEVNAME in the build
> as well to be "cd9660:/dev/md0.uzip"

I do not think that is necessesary but I'll keep it in mind, at present
I do over ride the vfs.root.mountfrom to point to my version specific
root file system using some ipxe variables.

>
> Every time I do stuff like this I have to re-puzzle it out, alas, but these
> should give you some guide posts. It should be better documented in md(4),
> but isn't at the moment.

Thanks for exposing what may be some pointy sticks to stumble on.

>
> I'd honestly try to get this setup working first loading all the files off
> a local disk before layering in the networking on top of that.

Probalby a good idea, as the usually failure mode in the diskless
word is black screens leaving little detail about what went wrong.

> Warner
--
Rod Grimes rgr...@freebsd.org

[Warner Losh]

On Wed, Jun 17, 2020 at 12:19 PM Rodney W. Grimes <

Yes. If you NFS mount root, it works. But if you are loading a full image,
the boot loader has to do all that and pass the right meta-data to the
kernel for the kernel to know what to use for root.

> > If there's no MD root provided, we look for the symbols mfs_root and
> > mfs_root_end, which I think means that rootfs_ in the above example needs
> > to be md_root_ instead so that we find it.
>
> Isnt this all handled by the loader? I think we have 2 slightly
> different cases here. THe one sjg shows up, where you actually
> load the md_image from a seperate file, and the case your talking
> about where you actually embed the kernel and image into a single
> file.
>

No. I'm talking about dynamically loading as well. The loader sets up the
symbols that point to the image, though I didn't thread my way all the way
through that. The loader sets it all up, and the kernel just uses the hints
the loader gives and the md driver does at least part of the heavy lifting
to make it happen.

>
> > You may need to have a custom kernel with 'options MD_ROOT_READONLY'
> > because isofs is read-only.
> >
> > And there's a small chance you may need to define ROOTDEVNAME in the
> build
> > as well to be "cd9660:/dev/md0.uzip"
>
> I do not think that is necessesary but I'll keep it in mind, at present
> I do over ride the vfs.root.mountfrom to point to my version specific
> root file system using some ipxe variables.
>

Usually that's enough, though when you want to use a MD device as root,
it's configured in weird ways that appear from the code to be driven by
ifdefs. And the default defines are for an uncompressed UFS filesystem
inside that MD image. Others are possible, but I believe require compile
time options.

> Every time I do stuff like this I have to re-puzzle it out, alas, but
> these
> > should give you some guide posts. It should be better documented in
> md(4),
> > but isn't at the moment.
>
> Thanks for exposing what may be some pointy sticks to stumble on.
>

Yea, There be a dragon or two here.

The ZRouter folks appear to have do a reroot to the
cdrom:9660:/dev/md0.uzip after dd'ing the installer image onto it.... but
otherwise they don't reply on md stuff running direclty out of flash that's
been uncompressed. I'd thought sure they'd worked this out for network
booting, but if so it's not in the copy of the scripts I have.

> I'd honestly try to get this setup working first loading all the files off
> > a local disk before layering in the networking on top of that.
>
> Probalby a good idea, as the usually failure mode in the diskless
> word is black screens leaving little detail about what went wrong.
>

Yea. Having both serial and video consoles connected wouldn't hurt :)

Good Luck!

Warner

[Simon J. Gerraty]

Warner Losh <i...@bsdimp.com> wrote:
> > loader.conf says
> >
> > rootfs_load="yes"
> > rootfs_name="contents.izo"
> > rootfs_type="md_image"
> > vfs.root.mountfrom="cd9660:/dev/md0.uzip"
> >
> > contents.izo is uzip'd contents.iso which file(1)
> > describes as ISO 9660 CD-ROM filesystem data ''
> >
> > That's for normal boot, for the loader 'install' command
> > it expects an uncompressed iso for rootfs.
>
> Ok, now the puzzle is how much work to get from a stock FreeBSD .iso
> image to something that works with this. Obviously we need a non-stock
> /boot/loader.conf file, or to type some commands manually at a loader
> prompt. I believe the stock GENERIC kernel has the md_root support
> for this already, so it may not be that hard to do.

So obviously we don't use the GENERIC kernel, but
I don't think we have any magic except in 4th files and loader.conf
and for the loader install command its all in the loader itself,
and I've been keeping head up todate on recent fixes/improvements there
since for UEFI I'm using loader.efi built from head.

Oh and all the scripts run by init during boot are custom.

> Looking at the code, I think MD_ROOT alone is insufficient here...
>
> If there's no MD root provided, we look for the symbols mfs_root and
> mfs_root_end, which I think means that rootfs_ in the above example
> needs to be md_root_ instead so that we find it.

FWIW our kernel options include

options CD9660
options MD_ROOT
options MD_ROOT_FSTYPE=\"cd9660\"

> You may need to have a custom kernel with 'options MD_ROOT_READONLY' because isofs is read-only.
>
> And there's a small chance you may need to define ROOTDEVNAME in the build as well to be "cd9660:/dev/md0.uzip"

> Every time I do stuff like this I have to re-puzzle it out, alas, but
> these should give you some guide posts. It should be better documented
> in md(4), but isn't at the moment.
>
> I'd honestly try to get this setup working first loading all the files
> off a local disk before layering in the networking on top of that.

Agreed! Booting from say tftp://host/install.tar
is far more fragile - the tar file needs to present all the files in
correct order since we cannot seek backwards (much), and tftp sucks ;-)

userboot is very handy for testing all this stuff, though building it to
run on host (the way we do) seems broken in head.

--sjg

[Warner Losh]

On Wed, Jun 17, 2020 at 12:56 PM Simon J. Gerraty <s...@juniper.net> wrote:

> Warner Losh <i...@bsdimp.com> wrote:
> > > loader.conf says
> > >
> > > rootfs_load="yes"
> > > rootfs_name="contents.izo"
> > > rootfs_type="md_image"
> > > vfs.root.mountfrom="cd9660:/dev/md0.uzip"
> > >
> > > contents.izo is uzip'd contents.iso which file(1)
> > > describes as ISO 9660 CD-ROM filesystem data ''
> > >
> > > That's for normal boot, for the loader 'install' command
> > > it expects an uncompressed iso for rootfs.
> >
> > Ok, now the puzzle is how much work to get from a stock FreeBSD .iso
> > image to something that works with this. Obviously we need a non-stock
> > /boot/loader.conf file, or to type some commands manually at a loader
> > prompt. I believe the stock GENERIC kernel has the md_root support
> > for this already, so it may not be that hard to do.
>
> So obviously we don't use the GENERIC kernel, but
> I don't think we have any magic except in 4th files and loader.conf
> and for the loader install command its all in the loader itself,
> and I've been keeping head up todate on recent fixes/improvements there
> since for UEFI I'm using loader.efi built from head.
>

Yea the loader will load the image, but the md driver looks for images of
type md_image or md_root and will create md devices for those devices it
finds.

> Oh and all the scripts run by init during boot are custom.
>
> > Looking at the code, I think MD_ROOT alone is insufficient here...
> >
> > If there's no MD root provided, we look for the symbols mfs_root and
> > mfs_root_end, which I think means that rootfs_ in the above example
> > needs to be md_root_ instead so that we find it.
>
> FWIW our kernel options include
>
> options CD9660
> options MD_ROOT
> options MD_ROOT_FSTYPE=\"cd9660\"
>

Yea, this lets you mout cd9660 images as root...

It also drives setting rootdevnames[0] which is used to create the default
mountroot script on loading the first md image.

> > You may need to have a custom kernel with 'options MD_ROOT_READONLY'
> because isofs is read-only.
> >
> > And there's a small chance you may need to define ROOTDEVNAME in the
> build as well to be "cd9660:/dev/md0.uzip"
>
> > Every time I do stuff like this I have to re-puzzle it out, alas, but
> > these should give you some guide posts. It should be better documented
> > in md(4), but isn't at the moment.
> >
> > I'd honestly try to get this setup working first loading all the files
> > off a local disk before layering in the networking on top of that.
>
> Agreed! Booting from say tftp://host/install.tar
> is far more fragile - the tar file needs to present all the files in
> correct order since we cannot seek backwards (much), and tftp sucks ;-)
>
> userboot is very handy for testing all this stuff, though building it to
> run on host (the way we do) seems broken in head.
>

Yea, of course. that makes sense. I'd forgotten userboot helps a lot. The
default build of it should be working, though...

Warner

[Rebecca Cran]

> On Jun 17, 2020, at 12:12 PM, Warner Losh <i...@bsdimp.com> wrote:
>
> I missed the start of this thread, so maybe I'm missing a key detail. However, I thought UEFI didn't have a RAM-disk, per se, but that we could load memory areas and pass that into the kernel using freebsd-only methods. But UEFI is a bit weird, so maybe it will generate a virtual cdrom...

See https://uefi.org/sites/default/files/resources/FINAL%20Pres4%20UEFI%20HTTP%20Boot.pdf

“ RAM Disk Standard

• UEFI 2.5 defined RAM Disk device path nodes
- Standard access to a RAM Disk in UEFI
- Supports Virtual Disk and Virtual CD (ISO image) in persistent or
volatile memory”

—
Rebecca Cran

[Rodney W. Grimes]

>
> > On Jun 17, 2020, at 12:12 PM, Warner Losh <i...@bsdimp.com> wrote:
> >
> > I missed the start of this thread, so maybe I'm missing a key detail. However, I thought UEFI didn't have a RAM-disk, per se, but that we could load memory areas and pass that into the kernel using freebsd-only methods. But UEFI is a bit weird, so maybe it will generate a virtual cdrom...
>
> See https://uefi.org/sites/default/files/resources/FINAL%20Pres4%20UEFI%20HTTP%20Boot.pdf
>

> ? RAM Disk Standard
>
> ? UEFI 2.5 defined RAM Disk device path nodes

> - Standard access to a RAM Disk in UEFI
> - Supports Virtual Disk and Virtual CD (ISO image) in persistent or

> volatile memory?

Does freeBSD have any way to access these "Virtual Disk"
or Virtual CD images once we leave the world of the loader?
I believe we do not, as these are BIOS/UEFI devices that
require calls into the UEFI code, which, IIRC is gone
once we exit the loader and start the kernel proper,
or shortly there after.

As far as I know these devices well not be found by the FreeBSD
cam layer ATA or AHCI drivers as they do not present an actual
PCI device to find.

> Rebecca Cran
--
Rod Grimes rgr...@freebsd.org

[Dave Cottlehuber]

On Wed, 17 Jun 2020, at 17:52, Rodney W. Grimes wrote:
> > Rodney W. Grimes <freeb...@gndrsh.dnsmgr.net> wrote:
> > > > The "fake cd drive" is in the kernel, loader just copies the iso into
> > > > memory like any other module, and by the time that's done you just
> > > > reboot into the newly installed system, which again uses
> > > >
> > > > vfs.root.mountfrom="cd9660:/dev/md0.uzip"
> > > ^^^
> > >
> > > Argh, the cd9660 confused me, I think your doing a
> > > "root on mfs/md"?
> >
> > loader.conf says
> >
> > rootfs_load="yes"
> > rootfs_name="contents.izo"
> > rootfs_type="md_image"
> > vfs.root.mountfrom="cd9660:/dev/md0.uzip"
> >
> > contents.izo is uzip'd contents.iso which file(1)
> > describes as ISO 9660 CD-ROM filesystem data ''
> >
> > That's for normal boot, for the loader 'install' command
> > it expects an uncompressed iso for rootfs.
>
> Ok, now the puzzle is how much work to get from a stock FreeBSD .iso
> image to something that works with this. Obviously we need a non-stock
> /boot/loader.conf file, or to type some commands manually at a loader
> prompt. I believe the stock GENERIC kernel has the md_root support
> for this already, so it may not be that hard to do.

Hi Miguel, all,

I spent a bit of time on UEFI HTTP Boot earlier in the year in qemu, bhyve, and intel NUCs -- until everything in the world went to custard. I made some rough notes[1] and I'll go through them again tonight with a fresh build. Hopefully its useful.

What I got stuck on was the final pivot, I have never debugged this setup before and I'm still not clear at what point things fail. Olivier's PXE booting and BSDRP were a fantastic reference, and I assume they work in BSDRP already for him.

Worth noting that LE TLS certs didn't play well with the PXE UEFI implementation on my intel NUC, this comes up as a very unhelpful error. At least use plain HTTP to get started.

While my notes are amd64 oriented I'm very interested in using this for aarch64 locally & in the clowd.

My loader.conf follows:

boot_multicons="YES"
console="efi,comconsole"
comconsole_speed="115200"
boot_verbose="YES"
# make booting somewhat less painful
#entropy_cache_load="NO"
#kern.random.initial_seeding.bypass_before_seeding="0"
# entropy_cache_load="YES"
# boot_single="YES"
tmpfs_load="YES"
autoboot_delay="-1"
# dump net vars
# exec="show boot.netif.hwaddr"
# exec="show boot.netif.ip"
# exec="show boot.netif.netmask"
# exec="show boot.netif.gateway"
# ensure we have enough ram for our image
vm.kmem_size=2G
vfs.root.mountfrom="ufs:/dev/md0"
# vfs.root.mountfrom.options=ro
mfs_load="YES"
mfs_type="md_image"
mfs_name="/boot/mfs-miniroot"

interesting these are different from what's above in the thread.

references:

[1]: https://hackmd.io/@dch/H1X9RYEZr
[mfsBSD]: https://mfsbsd.vx.sk/ still 150% awesome
[olivier]: https://blog.cochard.me/2019/02/pxe-booting-of-freebsd-disk-image.html
[BSDRP]: https://github.com/ocochard/BSDRP

A+
Dave

[Rebecca Cran]

On 6/17/20 2:11 PM, Rodney W. Grimes wrote:

> Does freeBSD have any way to access these "Virtual Disk"
> or Virtual CD images once we leave the world of the loader?
> I believe we do not, as these are BIOS/UEFI devices that
> require calls into the UEFI code, which, IIRC is gone
> once we exit the loader and start the kernel proper,
> or shortly there after.
>
> As far as I know these devices well not be found by the FreeBSD
> cam layer ATA or AHCI drivers as they do not present an actual
> PCI device to find.

I just tried it on my Supermicro server and you're right, it doesn't
show up under FreeBSD. I'm not sure if there's a way to access them via
UEFI Runtime Services or if it's designed purely as a boot-time thing.


--
Rebecca Cran

[Miguel C]

Well actually I've been using opensbd to test not linux, I'm using
https://rpi4-uefi.dev/ (not the sotck firmware).

This firmware deos have UEFI HTTP support and as stated bellow UEFI 2.5
supports booting from just a iso or img file, ofc the OS needs to support
this too.

You can even point it to the internet FWIW but I was actually downloading
miniroot67.img (openbsd installer) and serving it locally!

Creating a HTTP Boot entry manually in UEFI works perfect with this and I
can get to the shell, and even use it to install to a USB drive or sdcad.

>
> This person probably does not even have a running FreeBSD box to do
> any of your suggested solutions on.
>
>

No really I do have FreeBSD boxes, I just wanted to clarify that; I'm not
too familiar with the process of making system images but I can learn.

[Miguel C]

Ah thanks a lot for this and for the references, especially the first one
with all the notes :D

[Maxim Sobolev]

This is what we have running in AWS right now, kinda proof of concept but
it's not that difficult to generalize:

[root@ip-172-31-10-188 /usr/local/etc/freeswitch]# mdconfig -lv
md0 preload 160M -

[root@ip-172-31-10-188 /usr/local/etc/freeswitch]# df
Filesystem 512-blocks Used Avail Capacity Mounted on
/dev/ufs/root_20200617071427 1300080 1220480 79600 94% /
devfs 2 2 0 100% /dev
/dev/ufs/etc_20200617071427 9912 6384 2736 70% /etc
/dev/ufs/local_20200617071427 2746992 2572144 174848 94% /usr/local
/dev/ufs/boot_20200617071427 389560 361208 28352 93% /boot
tmpfs 65536 624 64912 1% /tmp
tmpfs 20480 16 20464 0%
/usr/home/ssp-user
tmpfs 524288 336816 187472 64% /var

Root file system is untrimmed 1.2GB UFS, generated with mkuzip compressed
down to 160MB with the UZIP, and pre-loaded along with the kernel. The
/usr/local file system is read-only UFS+UZIP images placed directly onto
the GPT and probed out with GEOM_LABEL. Out of those only /etc is
read-write. The idea here is that the box should theoretically survive
total loss of connectivity to both root and the /usr/local storage (or we
can replace it on the fly with the new version).

[root@ip-172-31-10-188 /usr/local/etc/freeswitch]# mount
/dev/ufs/root_20200617071427 on / (ufs, local, read-only)
devfs on /dev (devfs, local, multilabel)
/dev/ufs/etc_20200617071427 on /etc (ufs, local, synchronous)
/dev/ufs/local_20200617071427 on /usr/local (ufs, local, read-only)
/dev/ufs/boot_20200617071427 on /boot (ufs, local, read-only)
tmpfs on /tmp (tmpfs, local)
tmpfs on /usr/home/ssp-user (tmpfs, local)
tmpfs on /var (tmpfs, local)

Configuration is dead simple:

vfs.root.mountfrom="ufs:ufs/root_20200617071427"
image_load="YES"
image_name="/root.uzp"
image_type="mfs_root"
autoboot_delay="-1"

It takes less than 100 lines of code I think to generate this out of
buildworld/buildkernel. 0 third party tools.

Replace loading root from disk with loading it from HTTP server and it
would work just as good with the only need to load 1 or two files.

There is only one catch there - with real UEFI hardware sometimes there is
small(ish) region followed by a hole and then the much bigger region.
Unfortunately our loader picks smaller region for its work area and
md_image loading mechanism is not good enough to either place it entirely
into a bigger segment, or do scatter-gather and split it out and make the
kernel do some VM trick to re-assemble later. But with some
post-installworld cleaning if you can compress down the full image to some
30-40MB that usually works and has everything one may need including
kitchen sink (i.e. python 3.7 with few modules). As I said, no woodo magic
like famous crunchgen, just a very liberal application of the rm -rf.

With regards to ro vs. rw, recipy is "don't do it" :) If you want hard RO
embedded into kernel - compress your image with mkuzip first and then embed
it into kernel. This what FreeBSD/MIPS guys have mastered a lot (platform
was very tight on flash), which is why geom_uzip is practically in every
MIPS kernel config file. But that works (or should work) just as well on
x64. Not only that would save lots of VM but also the RO proper attribute
will be provided by the geom_uzip for free.

I am by the way hacking some way to populate /var with something more
interesting that just stock /etc/mtree/BSD.var.dist, there will be a review
request soon. :)

-Max

[Rodney W. Grimes]

I think your understating several of the stumbling blocks
that exist here. As Warner pointed out there are some
pokey sticks around doing this over the net fs doing this
from a local disk.

>
> There is only one catch there - with real UEFI hardware sometimes there is
> small(ish) region followed by a hole and then the much bigger region.
> Unfortunately our loader picks smaller region for its work area and
> md_image loading mechanism is not good enough to either place it entirely
> into a bigger segment, or do scatter-gather and split it out and make the
> kernel do some VM trick to re-assemble later. But with some
> post-installworld cleaning if you can compress down the full image to some
> 30-40MB that usually works and has everything one may need including
> kitchen sink (i.e. python 3.7 with few modules). As I said, no woodo magic
> like famous crunchgen, just a very liberal application of the rm -rf.
>
> With regards to ro vs. rw, recipy is "don't do it" :) If you want hard RO
> embedded into kernel - compress your image with mkuzip first and then embed
> it into kernel. This what FreeBSD/MIPS guys have mastered a lot (platform
> was very tight on flash), which is why geom_uzip is practically in every
> MIPS kernel config file. But that works (or should work) just as well on
> x64. Not only that would save lots of VM but also the RO proper attribute
> will be provided by the geom_uzip for free.
>
> I am by the way hacking some way to populate /var with something more
> interesting that just stock /etc/mtree/BSD.var.dist, there will be a review
> request soon. :)

You may want to look at /etc/rc.initdiskless, the /var and /etc population
do have one existing solution.

--
Rod Grimes rgr...@freebsd.org

[Warner Losh]

BTW, I think this is an excellent area to move from kernel compile options
to kernel env vars that maybe default to the kernel build options for
compat.

And, of course, the sharp bits would be less surprising if well
documented...

Warner