Boot device/order

[Marcus Sorensen]

Hi, I think it would be interesting if users could boot from ISO and install to PV, using KubeVirt for template preparation. It seems the only thing preventing this is the ability to provide the Libvirt domain XML with the option to boot from cdrom. I looked through the APIs and code, it doesn’t seem to be exposed.

It occurs to me as well that one might want to provide explicit boot device and or order. Any suggestions as to how this might look in the customer facing API? Expose an “os” object under the domain spec?

As an aside, this is a use case where the VirtualMachine object is useful, a temporary VM used only to fulfill a task.

[Roman Mohr]

On Tuesday, May 15, 2018 at 7:51:43 AM UTC+2, Marcus Sorensen wrote:

Hi, I think it would be interesting if users could boot from ISO and install to PV, using KubeVirt for template preparation. It seems the only thing preventing this is the ability to provide the Libvirt domain XML with the option to boot from cdrom. I looked through the APIs and code, it doesn’t seem to be exposed.

For that use-case we could add boot-order directives ourselves based on the disk order on the VirtualMachine, would that be sufficient?

 

It occurs to me as well that one might want to provide explicit boot device and or order. Any suggestions as to how this might look in the customer facing API? Expose an “os” object under the domain spec?

I wonder what the use-cases for that are. If the boot-order is derived from the disk order then the user can change it by swapping elements there. Is it valuable to exclude certain disks from the boot menu? Maybe for preventing booting from the wrong disk?

 

As an aside, this is a use case where the VirtualMachine object is useful, a temporary VM used only to fulfill a task.

Good point. 

Best Regards,

Roman 

[Fabian Deutsch]

On Tue, May 15, 2018 at 9:11 AM, Roman Mohr <rm...@redhat.com> wrote:

On Tuesday, May 15, 2018 at 7:51:43 AM UTC+2, Marcus Sorensen wrote:

Hi, I think it would be interesting if users could boot from ISO and install to PV, using KubeVirt for template preparation. It seems the only thing preventing this is the ability to provide the Libvirt domain XML with the option to boot from cdrom. I looked through the APIs and code, it doesn’t seem to be exposed.

For that use-case we could add boot-order directives ourselves based on the disk order on the VirtualMachine, would that be sufficient?

 

It occurs to me as well that one might want to provide explicit boot device and or order. Any suggestions as to how this might look in the customer facing API? Expose an “os” object under the domain spec?

I wonder what the use-cases for that are. If the boot-order is derived from the disk order then the user can change it by swapping elements there. Is it valuable to exclude certain disks from the boot menu? Maybe for preventing booting from the wrong disk?

One issue I am seeing with this approach is that it might also impact the disk addresses and bus assignment.

I.e. if boot priority, device address and/or bus assignment are dervied (somehow) from the disk order, then we might get into an issue.

I'm also seeing an issue if we want to define the boot order for a mix of net and disk devices.

Looking at how libvirt does it, then they set a boot order on _devices_ not a specific class of them https://libvirt.org/formatdomain.html#elementsNICSBoot

Not saying we should adopt this, but it at least does not change topology and works cross device class.

 

 

As an aside, this is a use case where the VirtualMachine object is useful, a temporary VM used only to fulfill a task.

Good point. 

+1

- fabian

 

Best Regards,

Roman 

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubevi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/1ce728d9-40a9-4bd6-b9b8-34aab4bba27b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Roman Mohr]

On Tuesday, May 15, 2018 at 10:07:34 AM UTC+2, Fabian Deutsch wrote:

On Tue, May 15, 2018 at 9:11 AM, Roman Mohr <rm...@redhat.com> wrote:

On Tuesday, May 15, 2018 at 7:51:43 AM UTC+2, Marcus Sorensen wrote:

Hi, I think it would be interesting if users could boot from ISO and install to PV, using KubeVirt for template preparation. It seems the only thing preventing this is the ability to provide the Libvirt domain XML with the option to boot from cdrom. I looked through the APIs and code, it doesn’t seem to be exposed.

For that use-case we could add boot-order directives ourselves based on the disk order on the VirtualMachine, would that be sufficient?

 

It occurs to me as well that one might want to provide explicit boot device and or order. Any suggestions as to how this might look in the customer facing API? Expose an “os” object under the domain spec?

I wonder what the use-cases for that are. If the boot-order is derived from the disk order then the user can change it by swapping elements there. Is it valuable to exclude certain disks from the boot menu? Maybe for preventing booting from the wrong disk?

One issue I am seeing with this approach is that it might also impact the disk addresses and bus assignment.

I.e. if boot priority, device address and/or bus assignment are dervied (somehow) from the disk order, then we might get into an issue.

Indeed that may happen. However I am still waiting for cases where we have the issues. I was pretty surprised that e.g. in oVirt we do not care about these aspects in many scenarios. So it can even happen there on importing, cluster-upgrade, ... that these aspects change. Please someone correct me if I am wrong here (Itamar, Yaniv, Michal?). So before we play safe here, I would want to hear if we really have to.

 

I'm also seeing an issue if we want to define the boot order for a mix of net and disk devices.

Looking at how libvirt does it, then they set a boot order on _devices_ not a specific class of them https://libvirt.org/formatdomain.html#elementsNICSBoot

Not saying we should adopt this, but it at least does not change topology and works cross device class.

Right. That would be interesting to solve. So this would then be for instance PXE boot, right? I wonder if we would even want to mix that with the disk device boot order ...

 

 

As an aside, this is a use case where the VirtualMachine object is useful, a temporary VM used only to fulfill a task.

Good point. 

+1

- fabian

 

Best Regards,

Roman 

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To post to this group, send email to kubevi...@googlegroups.com.

[Roman Mohr]

On Tuesday, May 15, 2018 at 9:11:21 AM UTC+2, Roman Mohr wrote:

On Tuesday, May 15, 2018 at 7:51:43 AM UTC+2, Marcus Sorensen wrote:

Hi, I think it would be interesting if users could boot from ISO and install to PV, using KubeVirt for template preparation. It seems the only thing preventing this is the ability to provide the Libvirt domain XML with the option to boot from cdrom. I looked through the APIs and code, it doesn’t seem to be exposed.

For that use-case we could add boot-order directives ourselves based on the disk order on the VirtualMachine, would that be sufficient?

 

It occurs to me as well that one might want to provide explicit boot device and or order. Any suggestions as to how this might look in the customer facing API? Expose an “os” object under the domain spec?

I wonder what the use-cases for that are. If the boot-order is derived from the disk order then the user can change it by swapping elements there. Is it valuable to exclude certain disks from the boot menu? Maybe for preventing booting from the wrong disk?

Another thing: since we also derive the ordering hints "vda", "vdb", ... from the order, we would change the order inside the vm, which is probably confusing and not what we would want.

[michal.s...@redhat.com]

On Tuesday, May 15, 2018 at 11:03:43 AM UTC+2, Roman Mohr wrote:

Indeed that may happen. However I am still waiting for cases where we have the issues. I was pretty surprised that e.g. in oVirt we do not care about these aspects in many scenarios. So it can even happen there on importing, cluster-upgrade, ... that these aspects change. Please someone correct me if I am wrong here (Itamar, Yaniv, Michal?). So before we play safe here, I would want to hear if we really have to. 

This is just a historical thing. We wanted to change that couple times already, just never got to it. It's not intentional, if we were refactoring this area we would have changed it to allow per-device boot order. It has been requested and required couple times, especially on multiple NICs

[Roman Mohr]

Ok, so adding a boot device order property may work on the API. Probably worth prototyping. 

[Fabian Deutsch]

On Tue, May 15, 2018 at 11:38 AM, Roman Mohr <rm...@redhat.com> wrote:

On Tuesday, May 15, 2018 at 9:11:21 AM UTC+2, Roman Mohr wrote:

On Tuesday, May 15, 2018 at 7:51:43 AM UTC+2, Marcus Sorensen wrote:

Hi, I think it would be interesting if users could boot from ISO and install to PV, using KubeVirt for template preparation. It seems the only thing preventing this is the ability to provide the Libvirt domain XML with the option to boot from cdrom. I looked through the APIs and code, it doesn’t seem to be exposed.

For that use-case we could add boot-order directives ourselves based on the disk order on the VirtualMachine, would that be sufficient?

 

It occurs to me as well that one might want to provide explicit boot device and or order. Any suggestions as to how this might look in the customer facing API? Expose an “os” object under the domain spec?

I wonder what the use-cases for that are. If the boot-order is derived from the disk order then the user can change it by swapping elements there. Is it valuable to exclude certain disks from the boot menu? Maybe for preventing booting from the wrong disk?

Another thing: since we also derive the ordering hints "vda", "vdb", ... from the order, we would change the order inside the vm, which is probably confusing and not what we would want.

In general I'd suggest to stick away from trying to predict the guest sided device names (i.e. vda, vdb, …) as this relies on capabilities of the guest themselfes.

In general I think that it's not possible to predefine names of devices within guests for all guests.

Thus for boot ordering by priority sounds good. Compared to an index this would be more relaxed, i.e. would allow gaps and allow the same prio twice.

- fabian

 

 

 

As an aside, this is a use case where the VirtualMachine object is useful, a temporary VM used only to fulfill a task.

Good point. 

Best Regards,

Roman 

--

You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com.

To post to this group, send email to kubevi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/48d19c66-37e0-4dee-bfd7-51706e53c46c%40googlegroups.com.

[Daniel P. Berrangé]

Yes, the boot index is a value that sets an ordering across all devices
in the guest. It is currently supported for disks, nics and host devices.
So you can setup a boot priority across a variety of devices

idx=0 -> 2nd disk (which is a cdrom)
idx=1 -> 3rd disk (which is a floppy)
idx=2 -> 1st nic
idx=3 -> 1st hostdev (which is an assigned SRIOV nic)
idx=4 -> 2nd nic
idx=5 -> 1st disk (the main harddisk)

> Not saying we should adopt this, but it at least does not change topology
> and works cross device class.

I'd rather strongly recommend against trying to invent a different way
to set device ordering, and especially don't try and set it based on the
disk name or disk address, as you'll just paint yourself into a corner


Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

[Fabian Deutsch]

No - not name or address.

The only variation I saw is to consider a priority vs index.

- fabian

[Daniel P. Berrangé]

Libvirt exposes the device name in its XML config, but if it matches what
appears in the guest, this is simply luck. Especially if using SCSI and
the guest enumerates devices in parallel all bets are off wrt naming.

Libvirt just uses the device name as an ordering when assigning addresses
to the disks. So if you list vda, vdb, vdc that just means they'll be
assigned PCI address 3, 4, 5, or sda, sdb, sdc, just influence the
order in which libvirt will assign SCSI addresses.

Even then, the device name can be omitted, and let libvirt give it based
on the order of the devices in the XML.

If users need to be able to identify disks in the guest, and correlate
to how the host side was configured, the "serial" string is a better
choice.

[Daniel P. Berrangé]

What do you mean by a "priority vs index" ? The boot index is a priority
value.

[Daniel P. Berrangé]

On Tue, May 15, 2018 at 12:11:21AM -0700, Roman Mohr wrote:
>
>
> On Tuesday, May 15, 2018 at 7:51:43 AM UTC+2, Marcus Sorensen wrote:
> >
> > Hi, I think it would be interesting if users could boot from ISO and
> > install to PV, using KubeVirt for template preparation. It seems the only
> > thing preventing this is the ability to provide the Libvirt domain XML with
> > the option to boot from cdrom. I looked through the APIs and code, it
> > doesn’t seem to be exposed.
> >
>
> For that use-case we could add boot-order directives ourselves based on the
> disk order on the VirtualMachine, would that be sufficient?
>
>
> >
> > It occurs to me as well that one might want to provide explicit boot
> > device and or order. Any suggestions as to how this might look in the
> > customer facing API? Expose an “os” object under the domain spec?
>
> I wonder what the use-cases for that are. If the boot-order is derived from
> the disk order then the user can change it by swapping elements there. Is
> it valuable to exclude certain disks from the boot menu? Maybe for
> preventing booting from the wrong disk?

You definitely want ability to prevent disks being considerd for boot, as
security protection. Consider a VM has many disks, and some disks are given
out to unprivileged application process for use as a raw volume. You don't
want that unpriv app to be able to write a valid partition table to the disk,
which the KVM BIOS then considers as a valid boot target on next reboot of
the OS.

[Marcus Sorensen]

On Tuesday, May 15, 2018 at 1:11:21 AM UTC-6, Roman Mohr wrote:

On Tuesday, May 15, 2018 at 7:51:43 AM UTC+2, Marcus Sorensen wrote:

Hi, I think it would be interesting if users could boot from ISO and install to PV, using KubeVirt for template preparation. It seems the only thing preventing this is the ability to provide the Libvirt domain XML with the option to boot from cdrom. I looked through the APIs and code, it doesn’t seem to be exposed.

For that use-case we could add boot-order directives ourselves based on the disk order on the VirtualMachine, would that be sufficient?

I did a little experimentation, and while adjusting the order of the disks in the YAML does rearrange the disks, it seems when I just have an ISO it will boot it, but when I have an ISO and a PV it will not ever boot the ISO, regardless of order. So while rearranging the disks may be sufficient for some cases, it seems at a minimum we need to be able add "boot dev='cdrom'" or turn on the boot menu.

[Marcus Sorensen]

On Tuesday, May 15, 2018 at 6:51:57 AM UTC-6, Daniel P. Berrangé wrote:

On Tue, May 15, 2018 at 12:11:21AM -0700, Roman Mohr wrote:
>
>
> On Tuesday, May 15, 2018 at 7:51:43 AM UTC+2, Marcus Sorensen wrote:
> >
> > Hi, I think it would be interesting if users could boot from ISO and
> > install to PV, using KubeVirt for template preparation. It seems the only
> > thing preventing this is the ability to provide the Libvirt domain XML with
> > the option to boot from cdrom. I looked through the APIs and code, it
> > doesn’t seem to be exposed.
> >
>
> For that use-case we could add boot-order directives ourselves based on the
> disk order on the VirtualMachine, would that be sufficient?
>  
>
> >
> > It occurs to me as well that one might want to provide explicit boot
> > device and or order. Any suggestions as to how this might look in the
> > customer facing API? Expose an “os” object under the domain spec?
>
> I wonder what the use-cases for that are. If the boot-order is derived from
> the disk order then the user can change it by swapping elements there. Is
> it valuable to exclude certain disks from the boot menu? Maybe for
> preventing booting from the wrong disk?

You definitely want ability to prevent disks being considerd for boot, as
security protection. Consider a VM has many disks, and some disks are given
out to unprivileged application process for use as a raw volume. You don't
want that unpriv app to be able to write a valid partition table to the disk,
which the KVM BIOS then considers as a valid boot target on next reboot of
the OS.

That's an interesting point, but is that even possible currently with Libvirt? You can control boot order and boot devices, but can you actually blacklist devices?

[Marcus Sorensen]

Ok. With Libvirt you either have a 'boot order' property on the disk itself, or a 'boot dev' entry in the 'os' section of the domain. They are mutually exclusive. I think I prefer having the boot order on the disk, it seems more explicit and clear. 

However I think someone down the road might ask for network booting, and while network devices also have a 'boot order' property, I'm not sure where that will fit in the KubeVirt API. From that perspective it may may sense to just add the ability to switch between 'net', 'cdrom', and 'hd' to the domain and rely on the device ordering in the YAML.

[Daniel P. Berrangé]

On Tue, May 15, 2018 at 07:24:09AM -0700, Marcus Sorensen wrote:
> Ok. With Libvirt you either have a 'boot order' property on the disk
> itself, or a 'boot dev' entry in the 'os' section of the domain. They are
> mutually exclusive. I think I prefer having the boot order on the disk, it
> seems more explicit and clear.

Never use the 'boot dev' entry in the 'os' section - it exists solely
for compat with older libvirts. The boot order index against the individual
devices offers much better functionality.

> However I think someone down the road might ask for network booting, and
> while network devices also have a 'boot order' property, I'm not sure where
> that will fit in the KubeVirt API. From that perspective it may may sense
> to just add the ability to switch between 'net', 'cdrom', and 'hd' to the
> domain and rely on the device ordering in the YAML.

This simple "net", "cdrom", and "hd" selection was the way QEMU did things
before boot index support was added and it is horribly limited in what it
can express. This forces users to mess around with device order to attempt
to get what they need. Reordering devices in the XML has negative impact on
the guest by changing the hardware ABI, which in turns will cause Windows
to complain about changing disks and possibly even fail to boot.

[Daniel P. Berrangé]

You can't explicitly blacklist devices, but by using boot index you can
guarantee something that is known bootable will be picked first. If you
use the legacy boot dev=cdrom|floppy|disk approach, you can't have that
direct control, forcing you to re-arrange disks to prevent undesirable
disks being tried for boot first.

[Marcus Sorensen]

On Tuesday, May 15, 2018 at 8:33:26 AM UTC-6, Daniel P. Berrangé wrote:

On Tue, May 15, 2018 at 07:24:09AM -0700, Marcus Sorensen wrote:
> Ok. With Libvirt you either have a 'boot order' property on the disk
> itself, or a 'boot dev' entry in the 'os' section of the domain. They are
> mutually exclusive. I think I prefer having the boot order on the disk, it
> seems more explicit and clear.

Never use the 'boot dev' entry in the 'os' section - it exists solely
for compat with older libvirts. The boot order index against the individual
devices offers much better functionality.

 

 I'll prototype something with a boot order field on the disks, then, and leave it up to future work to determine how that will translate for network devices.

[Roman Mohr]

On Tue, May 15, 2018 at 4:53 PM, Marcus Sorensen <shad...@gmail.com> wrote:

On Tuesday, May 15, 2018 at 8:33:26 AM UTC-6, Daniel P. Berrangé wrote:

On Tue, May 15, 2018 at 07:24:09AM -0700, Marcus Sorensen wrote:
> Ok. With Libvirt you either have a 'boot order' property on the disk
> itself, or a 'boot dev' entry in the 'os' section of the domain. They are
> mutually exclusive. I think I prefer having the boot order on the disk, it
> seems more explicit and clear.

Never use the 'boot dev' entry in the 'os' section - it exists solely
for compat with older libvirts. The boot order index against the individual
devices offers much better functionality.

 

 I'll prototype something with a boot order field on the disks, then, and leave it up to future work to determine how that will translate for network devices.

 

Sounds good.

 

> However I think someone down the road might ask for network booting, and
> while network devices also have a 'boot order' property, I'm not sure where
> that will fit in the KubeVirt API. From that perspective it may may sense
> to just add the ability to switch between 'net', 'cdrom', and 'hd' to the
> domain and rely on the device ordering in the YAML.

This simple "net", "cdrom", and "hd" selection was the way QEMU did things
before boot index support was added and it is horribly limited in what it
can express. This forces users to mess around with device order to attempt
to get what they need. Reordering devices in the XML has negative impact on
the guest by changing the hardware ABI, which in turns will cause Windows
to complain about changing disks and possibly even fail to boot.


Regards,
Daniel
--
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--

You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev+unsubscribe@googlegroups.com.

To post to this group, send email to kubevi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/eb2d8b0e-96df-4ceb-affd-cae7cb44424e%40googlegroups.com.