BHyVe hypervisor support for OSv

[Takuya ASADA]

Hi,

I'm working on BHyVe hypervisor support for OSv.

BHyVe is a hypervisor for FreeBSD, will be installed by default on up coming release of FreeBSD(10.0-RELEASE).

Web site link: http://bhyve.org/

Currently, BHyVe hasn't wide support of guest OSes, because:

- No BIOS/UEFI support

- Not able to execute 16bit/32bit code

- Very few device support(UART, virtio-net, virtio-blk only)

- Required to implement custom OS loader

For OSv, few driver support is not a problem, but we have to implement OS loader for OSv to use BHyVe.

So I implemented a OSv loader for BHyVe, and modified OSv kernel a little bit for it.

Here's a initial version of a patch:

https://gist.github.com/syuu1228/7207615/raw/34193e12d606cf3b76800454eba36c7d8581eba1/osv-bhyve-20131029.diff

Right now, OSv starts boot up on BHyVe, but dies with following message: 

alloc_page(): out of memory

Aborted

I guess it's because e820 structure processing is failed or something, but not yet debug into there.

[Pekka Enberg]

Hi,

On Tue, Oct 29, 2013 at 3:21 AM, Takuya ASADA <sy...@dokukino.com> wrote:
> Hi,
>
> I'm working on BHyVe hypervisor support for OSv.
> BHyVe is a hypervisor for FreeBSD, will be installed by default on up coming
> release of FreeBSD(10.0-RELEASE).
> Web site link: http://bhyve.org/
>
> Currently, BHyVe hasn't wide support of guest OSes, because:
> - No BIOS/UEFI support
> - Not able to execute 16bit/32bit code
> - Very few device support(UART, virtio-net, virtio-blk only)
> - Required to implement custom OS loader
>
> For OSv, few driver support is not a problem, but we have to implement OS
> loader for OSv to use BHyVe.
> So I implemented a OSv loader for BHyVe, and modified OSv kernel a little
> bit for it.
> Here's a initial version of a patch:
> https://gist.github.com/syuu1228/7207615/raw/34193e12d606cf3b76800454eba36c7d8581eba1/osv-bhyve-20131029.diff

Looks really interesting. Is the loader something that's should live
in osv.git or in BHyVe tree? I don't see any fundamental problem in
having it in osv.git but I wonder if it's easier to maintain in BHyVe?

Pekka

[Takuya ASADA]

Hi,

e820 bug had fixed.
https://gist.github.com/syuu1228/7210108/raw/a23bf85b72e9b219d9abb04c7ab2d372277ffe51/osv-bhyve-20131029-1.diff

Then I got new error:
Assertion failed: st == AE_OK (../../drivers/hpet.cc: hpet_init: 171)

Aborted

This is because OSv only have paravirt clock drivers + HPET driver, and BHyVe only have PIT device.
I'll try to look into clock driver..

2013年10月29日火曜日 10時21分28秒 UTC+9 Takuya ASADA:

[Takuya ASADA]

2013年10月29日火曜日 15時43分32秒 UTC+9 Pekka Enberg:

I think maintain in OSv git tree is better, at least for FreeBSD 10.0-RELEASE users.

Here're some reasons:

- There's no chance to merge this loader into FreeBSD 10.0 tree, because it's stable tree

- BHyVe developers are porting UEFI firmware for BHyVe, maybe it will release with 10.1

- So 10.0 user need the loader, but we have no chance to merge it. We have chance to merge the loader to 10.1 or later release, but it won't need the loader because it will have the firmware.

- The loader tightly dependent on OSv kernel implementation(at least for now). If you change implementation of OSv boot stable, the loader may also need to change. In that case, loader should develop with the kernel itself.

[Takuya ASADA]

I forgot to notice that, I'm trying to boot OSv on FreeBSD & BHyVe, but not trying to build OSv on FreeBSD (Except OSv loader for BHyVe).

I'm using rsync to transfer OSv directory for now. 

I'll check it possible to build or not, but maybe we need some work.

2013年10月29日火曜日 10時21分28秒 UTC+9 Takuya ASADA:

Hi,

[Pekka Enberg]

On Tue, Oct 29, 2013 at 9:20 AM, Takuya ASADA <sy...@dokukino.com> wrote:
> I'll check it possible to build or not, but maybe we need some work.

FreeBSD is using Clang, right? There are definitely some build issues with Clang
that are fixable. The biggest pain point is actually assembly code but hopefully
you still have GNU Assembler on FreeBSD.

OSv pulling libgcc and the C++ standard library are also an issue for Clang but
probably also fixable.

Pekka

[Pekka Enberg]

On Tue, Oct 29, 2013 at 9:13 AM, Takuya ASADA <sy...@dokukino.com> wrote:
> I think maintain in OSv git tree is better, at least for FreeBSD

OK, makes sense. The loader very non-intrusive and clean so it's
probably not a problem.

Pekka

[Avi Kivity]

On 10/29/2013 03:21 AM, Takuya ASADA wrote:
> Hi,
>
> I'm working on BHyVe hypervisor support for OSv.
> BHyVe is a hypervisor for FreeBSD, will be installed by default on up
> coming release of FreeBSD(10.0-RELEASE).
> Web site link: http://bhyve.org/
>
> Currently, BHyVe hasn't wide support of guest OSes, because:
> - No BIOS/UEFI support
> - Not able to execute 16bit/32bit code

How do you boot extra vcpus?

> - Very few device support(UART, virtio-net, virtio-blk only)

What do you use for clocks?

> - Required to implement custom OS loader
>
> For OSv, few driver support is not a problem, but we have to implement
> OS loader for OSv to use BHyVe.
> So I implemented a OSv loader for BHyVe, and modified OSv kernel a
> little bit for it.
> Here's a initial version of a patch:
> https://gist.github.com/syuu1228/7207615/raw/34193e12d606cf3b76800454eba36c7d8581eba1/osv-bhyve-20131029.diff
>

Nice. I imagine you need to build the loader on FreeBSD?

Is there a way to include th loader in the image? It is awkward to
distribute a VM image in two pieces.

An alternative is to make osv conform to the same ABI that the FreeBSD
loader uses to load FreeBSD (assuming there is such an ABI).

> Right now, OSv starts boot up on BHyVe, but dies with following message:
> alloc_page(): out of memory
> Aborted
>
> I guess it's because e820 structure processing is failed or something,
> but not yet debug into there.
>

> --
> You received this message because you are subscribed to the Google
> Groups "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to osv-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Takuya ASADA]

Clang is default compiler, but you can install gcc/g++ from pkg/ports.

Even make is incompatible because it's BSD make, it also can install GNU make from pkg/ports too.

So I think OSv just need to require these packages before starting build.

# GNU as is still in base, so not a problem

2013年10月29日火曜日 16時48分04秒 UTC+9 Pekka Enberg:

[Takuya ASADA]

Hi,

2013年10月29日火曜日 17時02分04秒 UTC+9 Avi Kivity:

On 10/29/2013 03:21 AM, Takuya ASADA wrote:
> Hi,
>
> I'm working on BHyVe hypervisor support for OSv.
> BHyVe is a hypervisor for FreeBSD, will be installed by default on up
> coming release of FreeBSD(10.0-RELEASE).
> Web site link: http://bhyve.org/
>
> Currently, BHyVe hasn't wide support of guest OSes, because:
> - No BIOS/UEFI support
> - Not able to execute 16bit/32bit code

How do you boot extra vcpus? 

First of all, "not able to execute 16bit/32bit code" was wrong. Sorry about that.

It was true in 2012, when I was tried to execute real mode code on BHyVe, there's no such a code.

(There was no unrestricted mode support, and LME/LMA flag was hardcoded at that time)

I just find they merged real mode spinup code in 10 month ago:

https://github.com/freebsd/freebsd/commit/01173b0b4a9b00c153489a51f2cba1b3d0cfc119

Original spinup code was something like this:

- all initialization had done in VMM, not in guest mode

- AP in guest mode will directly start at 64bit guest kernel function

https://github.com/freebsd/freebsd/commit/c34be7b811ad199e64f66db339e7f64c773ca0a7 

So I was tried to boot directly to start64() function in OSv because I thought BHyVe doesn't support 32bit boot up, actually it possible to start at start32().

# Even BHyVe supports real mode, executing start16() is not possible, because it still doesn't have BIOS.

# OSv doesn't need full BIOS implementation, probably int13 and e820 call support is enough to run start16(), but still direct boot is lot more easier.

> - Very few device support(UART, virtio-net, virtio-blk only)

What do you use for clocks?

I think it has ACPI Timer, PIT, RTC.

But there's no HPET.

 

> - Required to implement custom OS loader
>
> For OSv, few driver support is not a problem, but we have to implement
> OS loader for OSv to use BHyVe.
> So I implemented a OSv loader for BHyVe, and modified OSv kernel a
> little bit for it.
> Here's a initial version of a patch:
> https://gist.github.com/syuu1228/7207615/raw/34193e12d606cf3b76800454eba36c7d8581eba1/osv-bhyve-20131029.diff
>

Nice.  I imagine you need to build the loader on FreeBSD?

Is there a way to include th loader in the image? It is awkward to
distribute a VM image in two pieces.

 

Unless they release UEFI firmware, we need to provide original loader.

An alternative is to make osv conform to the same ABI that the FreeBSD
loader uses to load FreeBSD (assuming there is such an ABI).

Umm, maybe you can cheat FreeBSD boot loader to boot up another binary, but I'm not sure how it's difficult(or not).

[Takuya ASADA]

> - Required to implement custom OS loader
>
> For OSv, few driver support is not a problem, but we have to implement
> OS loader for OSv to use BHyVe.
> So I implemented a OSv loader for BHyVe, and modified OSv kernel a
> little bit for it.
> Here's a initial version of a patch:
> https://gist.github.com/syuu1228/7207615/raw/34193e12d606cf3b76800454eba36c7d8581eba1/osv-bhyve-20131029.diff
>

Nice.  I imagine you need to build the loader on FreeBSD?

Is there a way to include th loader in the image? It is awkward to
distribute a VM image in two pieces.

 

Unless they release UEFI firmware, we need to provide original loader.

I just got a mail from a BHyVe developer, he sent a link which is new userland OS loader:

https://github.com/grehan-freebsd/grub2-bhyve

It's userland GRUB.

OSv is currently boot from original boot code, the disk format is not compatible with GRUB, but it already has multiboot header so I guess there's a way to solve it.

Maybe we can put OSv's ELF image(contains multiboot header) on ZFS, GRUB should able to read the kernel.

[Avi Kivity]

Yes, that is possible, and grub supports zfs.  I am worried that grub will slow the boot time though.

It will be much better if bhyve adds support for enough bios (or bios emulation) to boot osv directly.  All that is needed is int 15/e820 (for which the information is already available) and int 13 (for which a small virtio driver is needed).

[Takuya ASADA]

2013年10月30日水曜日 Avi Kivity a...@cloudius-systems.com:

On 10/30/2013 05:01 AM, Takuya ASADA wrote:

> - Required to implement custom OS loader
>
> For OSv, few driver support is not a problem, but we have to implement
> OS loader for OSv to use BHyVe.
> So I implemented a OSv loader for BHyVe, and modified OSv kernel a
> little bit for it.
> Here's a initial version of a patch:
> https://gist.github.com/syuu1228/7207615/raw/34193e12d606cf3b76800454eba36c7d8581eba1/osv-bhyve-20131029.diff
>

Nice.  I imagine you need to build the loader on FreeBSD?

Is there a way to include th loader in the image? It is awkward to
distribute a VM image in two pieces.

 

Unless they release UEFI firmware, we need to provide original loader.

I just got a mail from a BHyVe developer, he sent a link which is new userland OS loader:

https://github.com/grehan-freebsd/grub2-bhyve

It's userland GRUB.

OSv is currently boot from original boot code, the disk format is not compatible with GRUB, but it already has multiboot header so I guess there's a way to solve it.

Maybe we can put OSv's ELF image(contains multiboot header) on ZFS, GRUB should able to read the kernel.

Yes, that is possible, and grub supports zfs.  I am worried that grub will slow the boot time though.

You mean OSv already able to boot from GRUB?

It will be much better if bhyve adds support for enough bios (or bios emulation) to boot osv directly.  All that is needed is int 15/e820 (for which the information is already available) and int 13 (for which a small virtio driver is needed).

 

According to BHyVe developer who sent a link, the user land GRUB try to boot Linux/x86_64 from protected mode entry point directly, works without BIOS emu.

(And because of this, i386 kernel doesn't works. It calls BIOS after entering protected mode.)

Userland GRUB runs on Host OS, setup register and everything protected mode needs, just like my OSv Loader.

Actually, he also said Multiboot binary is not supported yet, it only supports Linux bzImage and OpenBSD ELF.

Maybe it can fix soon though.

[Avi Kivity]

On 10/30/2013 12:21 PM, Takuya ASADA wrote:

2013年10月30日水曜日 Avi Kivity a...@cloudius-systems.com:

On 10/30/2013 05:01 AM, Takuya ASADA wrote:

> - Required to implement custom OS loader
>
> For OSv, few driver support is not a problem, but we have to implement
> OS loader for OSv to use BHyVe.
> So I implemented a OSv loader for BHyVe, and modified OSv kernel a
> little bit for it.
> Here's a initial version of a patch:
> https://gist.github.com/syuu1228/7207615/raw/34193e12d606cf3b76800454eba36c7d8581eba1/osv-bhyve-20131029.diff
>

Nice.  I imagine you need to build the loader on FreeBSD?

Is there a way to include th loader in the image? It is awkward to
distribute a VM image in two pieces.

 

Unless they release UEFI firmware, we need to provide original loader.

I just got a mail from a BHyVe developer, he sent a link which is new userland OS loader:

https://github.com/grehan-freebsd/grub2-bhyve

It's userland GRUB.

OSv is currently boot from original boot code, the disk format is not compatible with GRUB, but it already has multiboot header so I guess there's a way to solve it.

Maybe we can put OSv's ELF image(contains multiboot header) on ZFS, GRUB should able to read the kernel.

Yes, that is possible, and grub supports zfs.  I am worried that grub will slow the boot time though.

You mean OSv already able to boot from GRUB?

Well, we didn't test it, but osv is multiboot compliant (boots from qemu -kernel loader.elf).

It will be much better if bhyve adds support for enough bios (or bios emulation) to boot osv directly.  All that is needed is int 15/e820 (for which the information is already available) and int 13 (for which a small virtio driver is needed).

 

According to BHyVe developer who sent a link, the user land GRUB try to boot Linux/x86_64 from protected mode entry point directly, works without BIOS emu.

(And because of this, i386 kernel doesn't works. It calls BIOS after entering protected mode.)

Userland GRUB runs on Host OS, setup register and everything protected mode needs, just like my OSv Loader.

Actually, he also said Multiboot binary is not supported yet, it only supports Linux bzImage and OpenBSD ELF.

Maybe it can fix soon though.

Ah.

One thing we can do is put the osv binary in a parition, and enhance bhyve-grub to be able to treat the paritition as a file system with just one file.  This way we can boot from an osv image without installing grub on it.

[Takuya ASADA]

2013年10月30日水曜日 19時29分16秒 UTC+9 Avi Kivity:

I think it's good idea. I'll look into bhyve-grub code to make it. 

[Takuya ASADA]

Hi,

I could boot-up unmodified OSv disk-image(usr.img) on BHyVe, with my custom OS loader(bhyveosvload).

# Actually OSv still dies while initializing clock driver, since there's no HPET on BHyVe. I'll work on it soon.

# And BHyVe doesn't support QCOW2, I needed to convert QCOW2 to RAW before loading the image.

And I made independent repository for it: https://github.com/syuu1228/bhyveosvload

I think distributing this loader via FreeBSD Ports is one of good solution to provide OSv for FreeBSD users for now.

Because we won't need any changes on disk layout, boot-strap code etc (grub2-bhyve requires it).

2013年10月31日木曜日 4時56分45秒 UTC+9 Takuya ASADA:

[Glauber Costa]

On 11/06/2013 04:37 PM, Takuya ASADA wrote:
> Hi,

Hi Takuya

>
> I could boot-up unmodified OSv disk-image(usr.img) on BHyVe, with my
> custom OS loader(bhyveosvload).
> # Actually OSv still dies while initializing clock driver, since there's
> no HPET on BHyVe. I'll work on it soon.

Which clock device does BHyVe uses?

> # And BHyVe doesn't support QCOW2, I needed to convert QCOW2 to RAW
> before loading the image.
>

We also need to do this for Amazon, so we'll likely have both images.

> And I made independent repository for it:
> https://github.com/syuu1228/bhyveosvload
> I think distributing this loader via FreeBSD Ports is one of good
> solution to provide OSv for FreeBSD users for now.
> Because we won't need any changes on disk layout, boot-strap code etc
> (grub2-bhyve requires it).
>

It makes sense.

[Takuya ASADA]

2013年11月6日水曜日 21時56分15秒 UTC+9 Glauber Costa:

On 11/06/2013 04:37 PM, Takuya ASADA wrote:
> Hi,

Hi Takuya

>
> I could boot-up unmodified OSv disk-image(usr.img) on BHyVe, with my
> custom OS loader(bhyveosvload).
> # Actually OSv still dies while initializing clock driver, since there's
> no HPET on BHyVe. I'll work on it soon.

Which clock device does BHyVe uses?

I'm not sure there's limitation of use or not, but it looks like there're ACPI PM Timer, RTC, PIT(8254), LAPIC Timer, TSC.

 

> # And BHyVe doesn't support QCOW2, I needed to convert QCOW2 to RAW
> before loading the image.
>
We also need to do this for Amazon, so we'll likely have both images.

Good to hear that.

If it'll publicly available on download page, I can provide download script with the loader.

[Glauber Costa]

On 11/06/2013 05:17 PM, Takuya ASADA wrote:
>
>
> 2013年11月6日水曜日 21時56分15秒 UTC+9 Glauber Costa:
>
> On 11/06/2013 04:37 PM, Takuya ASADA wrote:
> > Hi,
>
> Hi Takuya
>
> >
> > I could boot-up unmodified OSv disk-image(usr.img) on BHyVe, with my
> > custom OS loader(bhyveosvload).
> > # Actually OSv still dies while initializing clock driver, since
> there's
> > no HPET on BHyVe. I'll work on it soon.
>
> Which clock device does BHyVe uses?
>
>
> I'm not sure there's limitation of use or not, but it looks like
> there're ACPI PM Timer, RTC, PIT(8254), LAPIC Timer, TSC.
>

It would be fairly trivial to provide a TSC timer for OSv. Only problem
is that this would rule out the use of OSv in machines that doesn't have
stable TSC. That said, Those machines were quite common 2 years ago, but
should be less of an issue today.

Avi, what do you think ?

[Takuya ASADA]

I just quick hacked hpet.cc to skip accessing HPET, and tried to use rdtsc() instead.

Maybe it's returns incorrect value, but at least it able to boot on KVM.

So I tried to boot this image on BHyVe.

Here's a result.

=====

[syuu@fbsd3 ~/bhyveosvload]$ sudo sh boot-osv.sh /zpool0/usr.img.raw 

Preparing bridged networking

Okay if you get the error "does not exist"

Okay if you get the error "does not exist"

May report that File exists

ifconfig: BRDGADD em0: File exists

Destroying the guest if already running

Loading the guest kernel with /usr/sbin/bhyveload

sizeof e820data=48

cmdline=java.so -jar /usr/mgmt/web-1.0.0.jar app prod

start64:0x208ee3

Booting the guest kernel with /usr/sbin/bhyve

FreeBSD 8.* guests may exhibit a few second delay

ACPI: RSDP 0xf0400 00024 (v02 BHYVE )

ACPI: XSDT 0xf0480 00034 (v01 BHYVE  BVXSDT   00000001 INTL 20130823)

ACPI: APIC 0xf0500 0004A (v01 BHYVE  BVMADT   00000001 INTL 20130823)

ACPI: FACP 0xf0600 0010C (v05 BHYVE  BVFACP   00000001 INTL 20130823)

ACPI: DSDT 0xf0800 000F2 (v02 BHYVE  BVDSDT   00000001 INTL 20130823)

ACPI: FACS 0xf0780 00040

acpi 0 apic 0

APIC base fee00000

OSv v0.02-86-g31fee0f Copyright 2013 Cloudius Systems

locale works

ACPI: All ACPI Tables successfully acquired

Unhandled inw 0x0003

=====

I got inw 0x0003, which is not supported device on BHyVe.

I have no ideas what is this device, I just set mode which ignoring unsupported IO port access and tried again.

Then I got different output.

=====

[syuu@fbsd3 ~/bhyveosvload]$ sudo sh boot-osv.sh /zpool0/usr.img.raw     

Preparing bridged networking

Okay if you get the error "does not exist"

Okay if you get the error "does not exist"

May report that File exists

ifconfig: BRDGADD em0: File exists

Destroying the guest if already running

Loading the guest kernel with /usr/sbin/bhyveload

sizeof e820data=48

cmdline=java.so -jar /usr/mgmt/web-1.0.0.jar app prod

start64:0x208ee3

Booting the guest kernel with /usr/sbin/bhyve

FreeBSD 8.* guests may exhibit a few second delay

ACPI: RSDP 0xf0400 00024 (v02 BHYVE )

ACPI: XSDT 0xf0480 00034 (v01 BHYVE  BVXSDT   00000001 INTL 20130823)

ACPI: APIC 0xf0500 0004A (v01 BHYVE  BVMADT   00000001 INTL 20130823)

ACPI: FACP 0xf0600 0010C (v05 BHYVE  BVFACP   00000001 INTL 20130823)

ACPI: DSDT 0xf0800 000F2 (v02 BHYVE  BVDSDT   00000001 INTL 20130823)

ACPI: FACS 0xf0780 00040

acpi 0 apic 0

APIC base fee00000

OSv v0.02-86-g31fee0f Copyright 2013 Cloudius Systems

locale works

ACPI: All ACPI Tables successfully acquired

VFS: mounting ramfs at /

VFS: mounting devfs at /dev

RAM disk at 0x0xffffc0003e735000 (4096K bytes)

Initializing network stack...

Done!

page fault outside application

=====

Now I've got page fault. Looks like this is just before "VFS: mounting zfs at /usr".

Maybe that's because I constructed page table by bhyveosvload, or e820 table is still incorrect, I'm not really sure what it is.

2013年11月6日水曜日 22時20分55秒 UTC+9 Glauber Costa:

[Glauber Costa]

On 11/06/2013 06:01 PM, Takuya ASADA wrote:
> I just quick hacked hpet.cc to skip accessing HPET, and tried to use
> rdtsc() instead.
> Maybe it's returns incorrect value, but at least it able to boot on KVM.
> So I tried to boot this image on BHyVe.
> Here's a result.

We first need to decide if TSC is really the best option (It is fast,
and in 2013 it may even work). Maybe it would be better to go for a more
conservative option that works all the time.

But just to guide you in the right direction, if it is indeed, that is
not how you should implement it. The way to do it would be to have it
live in its own file, and implement all the clock interface. Take a look
at kvmclock.cc and xenclock.cc for that matter.

If the BSD Hypervisor exports all the relevant cpuflags even better. We
should test it in cpuid and make sure it is sane. If it is not, bail out.

However, I understand that for now you're just testing it. So let's go ahead

Me neither, but I wonder if this isn't a PCI access where the pci base
addr is misconfigured to be 0 ?

That is a good theory.

Attaching GDB to it when it breaks and looking at the backtrace may give
you more indication about how to proceed.

> 2013年11月6日水曜日 22時20分55秒 UTC+9 Glauber Costa:
>
> On 11/06/2013 05:17 PM, Takuya ASADA wrote:
> >
> >
> > 2013年11月6日水曜日 21時56分15秒 UTC+9 Glauber Costa:
> >
> > On 11/06/2013 04:37 PM, Takuya ASADA wrote:
> > > Hi,
> >
> > Hi Takuya
> >
> > >
> > > I could boot-up unmodified OSv disk-image(usr.img) on
> BHyVe, with my
> > > custom OS loader(bhyveosvload).
> > > # Actually OSv still dies while initializing clock driver,
> since
> > there's
> > > no HPET on BHyVe. I'll work on it soon.
> >
> > Which clock device does BHyVe uses?
> >
> >
> > I'm not sure there's limitation of use or not, but it looks like
> > there're ACPI PM Timer, RTC, PIT(8254), LAPIC Timer, TSC.
> >
>
> It would be fairly trivial to provide a TSC timer for OSv. Only problem
> is that this would rule out the use of OSv in machines that doesn't
> have
> stable TSC. That said, Those machines were quite common 2 years ago,
> but
> should be less of an issue today.
>
> Avi, what do you think ?
>