Bhyve with physical partitions?

[David Chmelik]

Can one use physical partitions one multi-boots from in Bhyve in FreeBSD
UNIX and keep multi-booting? I'd like to use Slackware GNU/Linux (and
other OS) in Bhyve but still keep it on its partition.
How come newbies--even new computer programmers--in places like Libera
IRC (which most freenode moved to) and maybe the webforum deny FreeBSD is
a Unix and don't know it's a 'genetic Unix'?
Which of the following can you cross-post to without having to wait
for moderation (which I tried some months/years ago and my post never
appeared and I got no email why)?

comp.unix.freebsd.misc
list.freebsd.questions
mailing.freebsd.questions
(others that would be good?)

[Andreas Kempe]

Den 2023-12-21 skrev David Chmelik <dchm...@gmail.com>:
> Can one use physical partitions one multi-boots from in Bhyve in FreeBSD
> UNIX and keep multi-booting? I'd like to use Slackware GNU/Linux (and
> other OS) in Bhyve but still keep it on its partition.

You can pass drives/partitions to bhyve just as you would image files.
Passing -s 2,ahci-hd,/dev/nvd0p1 to bhyve makes the first partition on
my NVME show up as a block device in the VM. I have never tried it
myself with an existing installation, but I think it should be
possible.

If only passing partitions through I suspect you could have some
issues with the bootloader since each partition becomes its own block
device. You might have to either adjust the bootloader's settings or
create a separate bootloader for that VM only.

[Grant Taylor]

On 12/21/23 03:54, Andreas Kempe wrote:
> You can pass drives/partitions to bhyve just as you would image files.
> Passing -s 2,ahci-hd,/dev/nvd0p1 to bhyve makes the first partition
> on my NVME show up as a block device in the VM. I have never tried
> it myself with an existing installation, but I think it should be
> possible.

The problem that I've had doing things like this -- admittedly with
other hypervisors and other OSs -- is that it a partition is not an
entire disk.

Thus when the VM thinks it's working with an entire disk and slices and
partitions therein, it's actually doing it inside of another partition
as seen from the outside of the VM.

To pull this off I've had to pass the entire drive to the VM so that
it's inside view of the drive / block device matches the outside view of
the drive / block device.

> If only passing partitions through I suspect you could have some
> issues with the bootloader since each partition becomes its own block
> device. You might have to either adjust the bootloader's settings or
> create a separate bootloader for that VM only.

IMHO the boot loader issue is a second order problem based on the first
order problem of nesting (slice & partitions inside a slice & partition).



--
Grant. . . .

[Andreas Kempe]

Den 2023-12-21 skrev Grant Taylor <gta...@tnetconsulting.net>:
> On 12/21/23 03:54, Andreas Kempe wrote:
>> You can pass drives/partitions to bhyve just as you would image files.
>> Passing -s 2,ahci-hd,/dev/nvd0p1 to bhyve makes the first partition
>> on my NVME show up as a block device in the VM. I have never tried
>> it myself with an existing installation, but I think it should be
>> possible.
>
> The problem that I've had doing things like this -- admittedly with
> other hypervisors and other OSs -- is that it a partition is not an
> entire disk.
>
> Thus when the VM thinks it's working with an entire disk and slices and
> partitions therein, it's actually doing it inside of another partition
> as seen from the outside of the VM.
>
> To pull this off I've had to pass the entire drive to the VM so that
> it's inside view of the drive / block device matches the outside view of
> the drive / block device.
>

What kind of issues have you been having? When I pass a partition
through, it appears as a block device without a partition table with a
file system directly on the block device. Once Linux/FreeBSD is
booted, I have never had an issue mounting and using the storage, ex.
mount /dev/sda /mnt inside a Linux VM.

That said, in the scenario where you multi-boot, passing the entire
drive and selecting the right OS in the boot loader is probably the
easiest solution, but I imagine things can go horribly wrong if you
accidentally boot from the host's drive.

Only tangentially related to the matter at hand, but I usually forego
the partition table on things like USB sticks when I only want a
single file system and have never had an issue with it.

>> If only passing partitions through I suspect you could have some
>> issues with the bootloader since each partition becomes its own block
>> device. You might have to either adjust the bootloader's settings or
>> create a separate bootloader for that VM only.
>
> IMHO the boot loader issue is a second order problem based on the first
> order problem of nesting (slice & partitions inside a slice & partition).
>

Considering I've never tried doing this, the bootloader issue was
mostly me speculating because it was really the only thing I could
imagine having issues with considering both FreeBSD/Linux, in my
experience, handle file systems directly on block devices just fine,
but I don't know what Grub or other common bootloaders think of
booting a drive without a partition table.

[Grant Taylor]

On 12/21/23 10:14, Andreas Kempe wrote:
> What kind of issues have you been having? When I pass a partition
> through, it appears as a block device without a partition table with
> a file system directly on the block device.

You answered your own question. "When you pass a partition through, it

appears as a block device without a partition table with a file system
directly on the block device."

> Once Linux/FreeBSD is booted, I have never had an issue mounting and
> using the storage, ex. mount /dev/sda /mnt inside a Linux VM.

Even if it works, which Linux and FreeBSD probably can / will, it is an
atypical arrangement to use a file system on a raw partitioned block device.

What's more is that is different behavior inside vs outside of the
virtualization. Ergo inconsistent behavior. Now you are dependent on
an OS that can tolerate the inconsistency.

There's also a decent chance that the actual device names will be
different inside vs outside. How big of a problem that is will be
dependent on the OS.

I'll say it this way, the difference in accessing the block device
inside vs outside will be different and non-transparent.

> That said, in the scenario where you multi-boot, passing the entire
> drive and selecting the right OS in the boot loader is probably the
> easiest solution, but I imagine things can go horribly wrong if you
> accidentally boot from the host's drive.

I would suggest a different drive to avoid the issue that you mention.

> Only tangentially related to the matter at hand, but I usually forego
> the partition table on things like USB sticks when I only want a
> single file system and have never had an issue with it.

Most *nix will work with that fairly well. Any time I've tried to have
Windows access that flash drive (different computer I'm working on for a
family member) things don't go well, if at all. Far more times than I
care to count I've had to partition the drive and format the partition
vs formatting the raw drive.

> Considering I've never tried doing this, the bootloader issue was
> mostly me speculating because it was really the only thing I could
> imagine having issues with considering both FreeBSD/Linux, in my
> experience, handle file systems directly on block devices just fine,
> but I don't know what Grub or other common bootloaders think of
> booting a drive without a partition table.

Valid concerns.

I've tried to do something similar with Solaris and the one time I did,
the device names were significantly different and it was a non-starter.



Grant. . . .

[Andreas Kempe]

Den 2023-12-22 skrev Grant Taylor <gta...@tnetconsulting.net>:
> On 12/21/23 10:14, Andreas Kempe wrote:
>> Once Linux/FreeBSD is booted, I have never had an issue mounting and
>> using the storage, ex. mount /dev/sda /mnt inside a Linux VM.
>
> Even if it works, which Linux and FreeBSD probably can / will, it is an
> atypical arrangement to use a file system on a raw partitioned block device.
>
> What's more is that is different behavior inside vs outside of the
> virtualization. Ergo inconsistent behavior. Now you are dependent on
> an OS that can tolerate the inconsistency.
>
> There's also a decent chance that the actual device names will be
> different inside vs outside. How big of a problem that is will be
> dependent on the OS.
>
> I'll say it this way, the difference in accessing the block device
> inside vs outside will be different and non-transparent.
>

A very good point. I only thought about the root device, which for
Linux is passed by the bootloader so in my mind that probably wouldn't
be an issue if one got the bootloader working. If you have any
additional storage set up in the fstab, you will need to modify it
for use as a VM.

>> That said, in the scenario where you multi-boot, passing the entire
>> drive and selecting the right OS in the boot loader is probably the
>> easiest solution, but I imagine things can go horribly wrong if you
>> accidentally boot from the host's drive.
>
> I would suggest a different drive to avoid the issue that you mention.
>

Naturally. I was writing with David Chmelik's scenario in mind with a
multi-boot and partitions.

[Andreas Kempe]

Den 2023-12-21 skrev David Chmelik <dchm...@gmail.com>:

> Can one use physical partitions one multi-boots from in Bhyve in FreeBSD
> UNIX and keep multi-booting? I'd like to use Slackware GNU/Linux (and
> other OS) in Bhyve but still keep it on its partition.

To not just talk out of my ass, I tried this with my Gentoo
installation and got it to work.

Create an image to use for a custom EFI:
dd if=/dev/zero of=/path/to/efi.img
mdconfig /path/to/efi.img
gpart create -s GPT md0
gpart add -t efi md0
newfs_msdos /dev/md0p1

Boot the VM using a live CD medium:
bhyve -A -H -P -c 2 -m 2G -s 0,hostbridge \
-s 1,lpc -s 2:0,ahci,hd:/path/to/efi.img,hd:/dev/gentoodevicep2 \
-s 3,ahci-cd,/path/to/install-amd64-minimal-20231210T170356Z.iso \
-s 4,virtio-net,tap4 -s 29,fbuf,tcp=0.0.0.0:5901,w=800,h=600,wait \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd testvm

The iso is the Gentoo minimal install ISO. /path/to/efi.img is the
custom EFI file system created earlier and /dev/gentoop2 is the
partition where the Gentoo install resides.

When the VM boots, /dev/sdb is the gentoo file system and /dev/sda1 is
the EFI partition. Mount the file systems and chroot.

mount /dev/sdb /mnt/gentoo
mount /dev/sda1 /mnt/gentoo/efi
arch-chroot /mnt/gentoo

In the chroot, backup the old grub config and generate a new one:
cp -r /boot/grub /root
grub-install --target=x86_64-efi --efi-directory=/efi --removable
grub-mkconfig -o /boot/grub/grub.cfg

I did not need to adjust the fstab since my fstab was empty. Make sure
to check it if you have mounts in there.

Exit the chroot and shut down the VM:
exit
shutdown -hP now

The VM can now be booted without the installation medium and boots
just fine:
bhyve -A -H -P -c 2 -m 2G -s 0,hostbridge \
-s 1,lpc -s 2:0,ahci,hd:/path/to/efi.img,hd:/dev/gentoodevicep2 \
-s 4,virtio-net,tap4 -s 29,fbuf,tcp=0.0.0.0:5901,w=800,h=600,wait \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd testvm

It is doable (since I did it), but how easy it is will depend on your
distribution. I have not used Slackware so I am afraid I don't really
know how easily the equivalent of what I did for my Gentoo
installation would be.

[Andreas Kempe]

Den 2023-12-22 skrev Andreas Kempe <ke...@lysator.liu.se>:
> Den 2023-12-21 skrev David Chmelik <dchm...@gmail.com>:
>> Can one use physical partitions one multi-boots from in Bhyve in FreeBSD
>> UNIX and keep multi-booting? I'd like to use Slackware GNU/Linux (and
>> other OS) in Bhyve but still keep it on its partition.
>
> To not just talk out of my ass, I tried this with my Gentoo
> installation and got it to work.
>
> Create an image to use for a custom EFI:
> dd if=/dev/zero of=/path/to/efi.img
> mdconfig /path/to/efi.img
> gpart create -s GPT md0
> gpart add -t efi md0
> newfs_msdos /dev/md0p1

mdconfig -u 0 -d

The md0 device should be removed before using the image.

[Grant Taylor]

On 12/22/23 03:59, Andreas Kempe wrote:
> A very good point. I only thought about the root device, which
> for Linux is passed by the bootloader so in my mind that probably
> wouldn't be an issue if one got the bootloader working.

I think Linux has some flexibility that makes this a little bit easier
in that you can use UUID and / or a label that's part of the file system
meta-data to identify the file system for root and / or other mount
points. The kernel will then check all available block devices to
locate the file system based on that. It's nice and can be quite handy
in a dynamic hardware environment.

I don't know if any other OSs have any capability like that.

> If you have any additional storage set up in the fstab, you will need
> to modify it for use as a VM.

Most likely.

My stance is that this may be technically possible to do, but that it
will likely be ... problematic and not conducive for the average system
remotely supported.

> Naturally. I was writing with David Chmelik's scenario in mind with
> a multi-boot and partitions.

Fair enough.

[Grant Taylor]

Pre-script: I'm sick and tired of the UI->UX changes made in
contemporary Thunderbird. Different buttons must now be used between
email and news to direct replies where I want them to go. >:-|

On 12/22/23 05:41, Andreas Kempe wrote:
> To not just talk out of my ass, I tried this with my Gentoo
> installation and got it to work.

Testing is good.

Aside: Are you running Gentoo (user space) on top of a FreeBSD kernel?
-- I believe it's an officially supported configuration, but I think
that most will assume Linux in association with Gentoo. The "bhyve" and
"mdconfig" seem very *BSD to me.

> Create an image to use for a custom EFI:
> dd if=/dev/zero of=/path/to/efi.img
> mdconfig /path/to/efi.img
> gpart create -s GPT md0
> gpart add -t efi md0
> newfs_msdos /dev/md0p1
>
> Boot the VM using a live CD medium:
> bhyve -A -H -P -c 2 -m 2G -s 0,hostbridge \
> -s 1,lpc -s 2:0,ahci,hd:/path/to/efi.img,hd:/dev/gentoodevicep2 \
> -s 3,ahci-cd,/path/to/install-amd64-minimal-20231210T170356Z.iso \
> -s 4,virtio-net,tap4 -s 29,fbuf,tcp=0.0.0.0:5901,w=800,h=600,wait \
> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd testvm

Would you please break down what that command is doing, what virtual
hardware it's creating, how the disks are being backed, etc.

I'm not familiar with bhyve. I know of it, but not much more.

> The iso is the Gentoo minimal install ISO. /path/to/efi.img is the
> custom EFI file system created earlier and /dev/gentoop2 is the
> partition where the Gentoo install resides.
>
> When the VM boots, /dev/sdb is the gentoo file system and /dev/sda1 is
> the EFI partition. Mount the file systems and chroot.
>
> mount /dev/sdb /mnt/gentoo
> mount /dev/sda1 /mnt/gentoo/efi
> arch-chroot /mnt/gentoo

The arch-chroot is a new one on me. A quick search makes me think it's
just a wrapper around chroot with some creature features. Cool.

> In the chroot, backup the old grub config and generate a new one:
> cp -r /boot/grub /root

What is your goal of copying /boot/grub to root's home directory; /root?

> grub-install --target=x86_64-efi --efi-directory=/efi --removable
> grub-mkconfig -o /boot/grub/grub.cfg
>
> I did not need to adjust the fstab since my fstab was empty. Make sure
> to check it if you have mounts in there.

fstab is one of those things that's going to be very pointy if you use
device paths. Less so if you use UUID and / or labels.

> Exit the chroot and shut down the VM:
> exit
> shutdown -hP now
>
> The VM can now be booted without the installation medium and boots
> just fine:
> bhyve -A -H -P -c 2 -m 2G -s 0,hostbridge \
> -s 1,lpc -s 2:0,ahci,hd:/path/to/efi.img,hd:/dev/gentoodevicep2 \
> -s 4,virtio-net,tap4 -s 29,fbuf,tcp=0.0.0.0:5901,w=800,h=600,wait \
> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd testvm
>
> It is doable (since I did it), but how easy it is will depend on your
> distribution.

Sure. Lots of things are possible with Linux.

Dr. Ian Malcolm from Jurassic Park comes to mind: ... your scients were
so preoccupied with whether or not they could that they didn't stop to
think if they should.

I'm all for a proof of concept for academic reasons. But I don't think
I'd want to do many of the academic things that I've done in production.
At least not as I did them.

> I have not used Slackware so I am afraid I don't really know how easily
> the equivalent of what I did for my Gentoo installation would be.

:-)

[Andreas Kempe]

Den 2023-12-22 skrev Grant Taylor <gta...@tnetconsulting.net>:

> Pre-script: I'm sick and tired of the UI->UX changes made in
> contemporary Thunderbird. Different buttons must now be used between
> email and news to direct replies where I want them to go. >:-|
>
> On 12/22/23 05:41, Andreas Kempe wrote:
>> To not just talk out of my ass, I tried this with my Gentoo
>> installation and got it to work.
>
> Testing is good.
>
> Aside: Are you running Gentoo (user space) on top of a FreeBSD kernel?
> -- I believe it's an officially supported configuration, but I think
> that most will assume Linux in association with Gentoo. The "bhyve" and
> "mdconfig" seem very *BSD to me.
>

I have Gentoo with the Linux kernel on a block device. Gentoo with the
FreeBSD kernel was abandoned a few years ago, if I'm not mistaken.

mdconfig is a FreeBSD utility. I should have specified, but I made the
experiment on my FreeBSD daily driver with a "normal" Gentoo
installation on a block device with a proper partition table.

Creating the partition table and file system in the EFI image was done
from FreeBSD using mdconfig, gpart and newfs_msdos, but could just as
well have been done using Linux tools from the installtion medium
instead.

>> Create an image to use for a custom EFI:
>> dd if=/dev/zero of=/path/to/efi.img
>> mdconfig /path/to/efi.img
>> gpart create -s GPT md0
>> gpart add -t efi md0
>> newfs_msdos /dev/md0p1
>>
>> Boot the VM using a live CD medium:
>> bhyve -A -H -P -c 2 -m 2G -s 0,hostbridge \
>> -s 1,lpc -s 2:0,ahci,hd:/path/to/efi.img,hd:/dev/gentoodevicep2 \
>> -s 3,ahci-cd,/path/to/install-amd64-minimal-20231210T170356Z.iso \
>> -s 4,virtio-net,tap4 -s 29,fbuf,tcp=0.0.0.0:5901,w=800,h=600,wait \
>> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd testvm
>
> Would you please break down what that command is doing, what virtual
> hardware it's creating, how the disks are being backed, etc.
>
> I'm not familiar with bhyve. I know of it, but not much more.
>

-A - provide ACPI tables, not strictly needed, but I added it on by
reflex

-H - make the virtual CPU yield when idle to avoid constantly spinning

-P - force the guest virtual CPU to exit when a PAUSE instruction is
detected

-c 2 - two virtual cores

-m 2G - 2GB of memory

-s 1,lpc - add an LPC PCI-ISA bridge for serial ports

-s 2:0,ahci,hd:/path/to/efi.img,hd:/dev/gentoodevicep2 - provide an
AHCI controller with two block devices backed by efi.img, a
file, and gentoodevicep2, partition 2 of a GPT partitioned
block device. They show up as /dev/{sda,sdb}.

-s 3,ahci-cd,/path/to/install-amd64-minimal-20231210T170356Z.iso -
provide an AHCI CD controller backed by an iso file.

-s 4,virtio-net,tap4 - provide a network interface, I manually bridge
it on the host

-s 29,fbuf,tcp=0.0.0.0:5901,w=800,h=600,wait - graphics controller
listening for a VNC connection, wait means that the VM waits for an
incoming VNC connection before booting

-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd - bhyve UEFI
firmware needed to boot an UEFI VM

testvm - name of the virtual machine

>> The iso is the Gentoo minimal install ISO. /path/to/efi.img is the
>> custom EFI file system created earlier and /dev/gentoop2 is the
>> partition where the Gentoo install resides.
>>
>> When the VM boots, /dev/sdb is the gentoo file system and /dev/sda1 is
>> the EFI partition. Mount the file systems and chroot.
>>
>> mount /dev/sdb /mnt/gentoo
>> mount /dev/sda1 /mnt/gentoo/efi
>> arch-chroot /mnt/gentoo
>
> The arch-chroot is a new one on me. A quick search makes me think it's
> just a wrapper around chroot with some creature features. Cool.
>

The Gentoo install process uses a chroot and previously you had to
manually mount devfs, proc, sysfs etc in the chroot. arch-chroot does
the mounting for you so there is less to type.

>> In the chroot, backup the old grub config and generate a new one:
>> cp -r /boot/grub /root
>
> What is your goal of copying /boot/grub to root's home directory; /root?
>

It's just a backup. I don't want to use my Gentoo installation this
way so I wanted a simple way of restoring the bootloader config
without having to boot the installation medium to generate a new one.

[David Chmelik]

On Fri, 22 Dec 2023 11:41:41 -0000 (UTC), Andreas Kempe wrote:

> Den 2023-12-21 skrev David Chmelik <dchm...@gmail.com>:
>> Can one use physical partitions one multi-boots from in Bhyve in
>> FreeBSD UNIX and keep multi-booting? I'd like to use Slackware
>> GNU/Linux (and other OS) in Bhyve but still keep it on its partition.
>
> To not just talk out of my ass, I tried this with my Gentoo installation
> and got it to work.

Good to see some discussion ending up with people testing this and getting
it working and explaining that for Gentoo GNU/Linux (which I'd also try
this with, without UEFI).

I looked for documentation on FreeBSD.org and BHyVe.org but didn't find
much. Is there a manual somewhere in (e)textbook format other than just
the manual page, or does one have to learn everything from that? I've
found textbooks to be a bit more descriptive than many of the manual pages
(some cryptic, some historically removed with the AT&T/whatever fiasco).

[Andreas Kempe]

I'm afraid I don't know of any really good bhyve resource. What I
know, I've mostly learnt from reading man bhyve which at least has
some good examples.

For my daily behyve use, I use virt-manager and libvirtd since it now
has a bhyve back-end. Unfortunately, the interactive configuration in
virt-manager doesn't really support all the bhyve stuff so I have had
to resort to XML editing and reading libvirt's wiki page[0] on the
bhyve driver.

[0]: https://libvirt.org/drvbhyve.html