GSoC'20 Interested Student: Adding support to Jailhouse Hypervisor

[PRAKHAR BANSAL]

Hello All,

My name is Prakhar Bansal and I am a graduate student in Computer Engineering at Iowa State University, US.

I have experience with Analysing Performance of Applications running inside multiple virtual machines hosted by the libvirt QEMU-KVM through virt-manager.

I am interested in working on the project to develop a Libvirt driver for the Jailhouse hypervisor. I looked into the initial attempt on the Jailhouse driver which seems to be based on the Jailhouse command-line interface. I am currently looking into learning and understanding the kernel APIs for jailhouse hypervisor.

I followed the below articles mentioned by Valentine Sinitsyn to begin learning about the Jailhouse hypervisor.

https://lwn.net/Articles/578295/

https://lwn.net/Articles/578852/

I have a few questions regarding this project, please let me know if someone can help me out.

Thanks & Regards,

Prakhar Bansal

[Jan Kiszka]

Hi Prakhar,

On 23.03.20 07:53, PRAKHAR BANSAL wrote:
> Hello All,
>
> My name is Prakhar Bansal and I am a graduate student in Computer
> Engineering at Iowa State University, US.
> I have experience with Analysing Performance of Applications running
> inside multiple virtual machines hosted by the libvirt QEMU-KVM through
> virt-manager.
>
> I am interested in working on the project to develop a Libvirt driver
> for the Jailhouse hypervisor. I looked into the initial attempt on the
> Jailhouse driver which seems to be based on the Jailhouse command-line
> interface. I am currently looking into learning and understanding the
> kernel APIs for jailhouse hypervisor.

Thanks for your interest!

> I followed the below articles mentioned by Valentine Sinitsyn to begin
> learning about the Jailhouse hypervisor.
>
> https://lwn.net/Articles/578295/
> https://lwn.net/Articles/578852/
>
> I have a few questions regarding this project, please let me know if
> someone can help me out.

Sure, go ahead. Depending on the scope of the question, libvirt might be
the better community to ask. Therefore, I'm adding its list to this thread.

Jan

>
> Thanks & Regards,
> Prakhar Bansal
>

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[PRAKHAR BANSAL]

Hi Jan,

Thanks for the reply. I looked deeper into the libvirt and Jailhouse source code and found following two things that seem relevant to the project I am interested in.

- Libvirt driver interface at [libvirt.git] / src / driver.h

- Jailhouse tool, which is using the ioctl API of the Jailhouse, available at https://github.com/siemens/jailhouse/blob/master/tools/jailhouse.c.

With the help of the above two, it looks like, a libvirt driver for the Jailhouse can be implemented. Let me know if I am moving in the right direction so far.

I have been looking at the other libvirt driver implementations for hypervisors like HyperV and VMware to understand their implementation and learn from there.

Thanks & Regards,

Prakhar Bansal

Graduate Student' 20, Computer Engineering

Iowa State University, Ames, IA

[PRAKHAR BANSAL]

Hi All,

Gentle Reminder,

Please have a look at my email below. It would be great if anyone can help me with this project.

Thanks & Regards,

Prakhar Bansal

Graduate Student' 20, Computer Engineering

Iowa State University, Ames, IA

[Jan Kiszka]

Hi Prakhar,

On 25.03.20 05:36, PRAKHAR BANSAL wrote:
> Hi Jan,
>
> Thanks for the reply. I looked deeper into the libvirt and Jailhouse
> source code and found following two things that seem relevant to the
> project I am interested in.
>
> - Libvirt driver interface at [libvirt.git]

> <https://libvirt.org/git/?p=libvirt.git;a=tree;hb=HEAD> / src
> <https://libvirt.org/git/?p=libvirt.git;a=tree;f=src;hb=HEAD> / driver.h
> <https://libvirt.org/git/?p=libvirt.git;a=blob_plain;f=src/driver.h;hb=HEAD>

> - Jailhouse tool, which is using the ioctl API of the Jailhouse,
> available at
> https://github.com/siemens/jailhouse/blob/master/tools/jailhouse.c.
>
> With the help of the above two, it looks like, a libvirt driver for the
> Jailhouse can be implemented. Let me know if I am moving in the right
> direction so far.

From the Jailhouse perspective, it is important to not consider the
command line tool an interface anymore (like in the first prototype) but
build on top of the Linux driver API (ioctls, sysfs). There is already a
Python library which started to abstract this interface for
Jailhouse-internal use cases. However, I strongly suspect libvirt will
rather want a native binding.

>
> I have been looking at the other libvirt driver implementations for
> hypervisors like HyperV and VMware to understand their implementation
> and learn from there.

As Jailhouse is a static partitioning hypervisor without abstraction of
the underlying hardware, your starting point for the libvirt binding
should be a given set of Jailhouse cell configurations describing a
complete partitioned system. So rather than instantiating on demand a
domain (Jailhouse cell) with, say, a network adapter, the driver should
match a user request for a domain against the configuration set and use
what is there - or report the mismatch. What it could organize, though,
is interconnecting cells that have a (preconfigured) virtual network
link to the root cell.

Due to this different concept, there will be no 1:1 mapping for
commodity hypervisor drivers to the Jailhouse scenario. Still, studying
what they do is useful and needed in order to understand what "normally"
happens and find a reasonable translation. This is probably the most
challenging part of the project.

Jan

[PRAKHAR BANSAL]

Hi Jan,

Thanks for the reply!

I was only considering the command-line tool "code" for reference to the Jailhouse kernel API(ioctl calls) because I didn't find a documentation of the Jailhouse kernel APIs.

For the second part as you mentioned that Jailhouse can only create cells with the constraints defined in the root cell configuration. I have a few questions regarding that.

1. Is there a way to know if Jailhouse is enabled on the host and get the root cell configuration(s) from Jailhouse through an API? This can be used while binding the libvirt to the Jailhouse hypervisor.

2. If Jailhouse is not enabled(again can we know this using some API) then, can libvirt enable/disable Jailhouse during the libvirt binding of the Jailhouse driver with a given set of Jailhouse cell configurations describing a complete partitioned system?

3. I was wondering, as you mentioned that libvirt driver should check for mismatch of the cell configuration with the root cell configuration, the question is, isn't that done by Jailhouse itself? If yes, then libvirt can just pass on the cell creation requests to Jailhouse and return the response to the user as it is, rather than driver doing any such mismatch check.

-Prakhar

[Jan Kiszka]

On 28.03.20 08:47, PRAKHAR BANSAL wrote:
> Hi Jan,
>

> Thanks for the reply!
>
> I was only considering the command-line tool "code" for reference to the
> Jailhouse kernel API(ioctl calls) because I didn't find a documentation
> of the Jailhouse kernel APIs.

Right, the IOCTL API is not documented so far. It is currently only used
inside the Jailhouse project. This needs to be formalized when there
shall be external users like a libvirt driver.

That might be a nice small contribution task: Create some
Documentation/driver-interfaces.md that describes the IOCTLs along with
their parameter structures and that also includes the current
sysfs-entries.txt as a section. Then send this as patch here. I'll help
out when details are not clear from reading the code.

>
> For the second part as you mentioned that Jailhouse can only create
> cells with the constraints defined in the root cell configuration. I
> have a few questions regarding that.
>
> 1. Is there a way to know if Jailhouse is enabled on the host and get
> the root cell configuration(s) from Jailhouse through an API? This can
> be used while binding the libvirt to the Jailhouse hypervisor.

Look at
https://github.com/siemens/jailhouse/blob/master/Documentation/sysfs-entries.txt
for what is reported as runtime information. Full configurations can't
be read back at this point. This might be reconsidered in the light of
[1], but I wouldn't plat for that yet.

>
> 2. If Jailhouse is not enabled(again can we know this using some API)
> then, can libvirt enable/disable Jailhouse during the libvirt binding of
> the Jailhouse driver with a given set of Jailhouse cell configurations
> describing a complete partitioned system?

With the API above and a given configuration set, yes. The config set
would have to be provided to the libvirt driver in some to-be-defined
way (e.g. /etc/libvirt/jailhouse.conf -> /etc/libvirt/jailhouse/*.cell).

>
> 3. I was wondering, as you mentioned that libvirt driver should check
> for mismatch of the cell configuration with the root cell configuration,
> the question is, isn't that done by Jailhouse itself? If yes, then
> libvirt can just pass on the cell creation requests to Jailhouse and
> return the response to the user as it is, rather than driver doing any
> such mismatch check.

With matching I'm referring to a libvirt user request like "create a
domain with 2 CPUs", while there are no cells left that have more than
one CPU. Or "give the domain 1G RAM", and you need to find an available
cell with that much memory. Those are simple examples. A request that
states "connect the domain to the host network A" implies that a cell
has a shared-memory link to, say, the root cell that can be configured
to bridge this. But let's keep that for later and start as simple as
possible.

Jan

[1]
https://groups.google.com/d/msgid/jailhouse-dev/CADiTV-1QiRhSWZnw%2BkHhJMO-BoA4sAcOmTkQE7ZWbHkGh3Jexw%40mail.gmail.com

[PRAKHAR BANSAL]

Hi Jan,

On Sat, Mar 28, 2020 at 4:12 AM Jan Kiszka <jan.k...@web.de> wrote:

On 28.03.20 08:47, PRAKHAR BANSAL wrote:
> Hi Jan,
>
> Thanks for the reply!
>
> I was only considering the command-line tool "code" for reference to the
> Jailhouse kernel API(ioctl calls) because I didn't find a documentation
> of the Jailhouse kernel APIs.

Right, the IOCTL API is not documented so far. It is currently only used
inside the Jailhouse project. This needs to be formalized when there
shall be external users like a libvirt driver.

That might be a nice small contribution task: Create some
Documentation/driver-interfaces.md that describes the IOCTLs along with
their parameter structures and that also includes the current
sysfs-entries.txt as a section. Then send this as patch here. I'll help
out when details are not clear from reading the code.

  Sure. I will do that. 

>
> For the second part as you mentioned that Jailhouse can only create
> cells with the constraints defined in the root cell configuration. I
> have a few questions regarding that.
>
> 1. Is there a way to know if Jailhouse is enabled on the host and get
> the root cell configuration(s) from Jailhouse through an API? This can
> be used while binding the libvirt to the Jailhouse hypervisor.

Look at
https://github.com/siemens/jailhouse/blob/master/Documentation/sysfs-entries.txt
for what is reported as runtime information. Full configurations can't
be read back at this point. This might be reconsidered in the light of
[1], but I wouldn't plat for that yet.

  Ok, sure. I am looking into it. 

>
> 2. If Jailhouse is not enabled(again can we know this using some API)
> then, can libvirt enable/disable Jailhouse during the libvirt binding of
> the Jailhouse driver with a given set of Jailhouse cell configurations
> describing a complete partitioned system?

With the API above and a given configuration set, yes. The config set
would have to be provided to the libvirt driver in some to-be-defined
way (e.g. /etc/libvirt/jailhouse.conf -> /etc/libvirt/jailhouse/*.cell).

 Cool, got it. Thanks!

 

>
> 3. I was wondering, as you mentioned that libvirt driver should check
> for mismatch of the cell configuration with the root cell configuration,
> the question is, isn't that done by Jailhouse itself? If yes, then
> libvirt can just pass on the cell creation requests to Jailhouse and
> return the response to the user as it is, rather than driver doing any
> such mismatch check.

With matching I'm referring to a libvirt user request like "create a
domain with 2 CPUs", while there are no cells left that have more than
one CPU. Or "give the domain 1G RAM", and you need to find an available
cell with that much memory. Those are simple examples. A request that
states "connect the domain to the host network A" implies that a cell
has a shared-memory link to, say, the root cell that can be configured
to bridge this. But let's keep that for later and start as simple as
possible.

  Do I need to match the libvirt user-requested cell config with only root cells or with all cells present at that time?

I wanted to request you for a favor for the proposal as the deadline is approaching. Could I prepare a proposal for this project based on our discussion here and improve it later based on feedback comments after the deadline? I understand that I got late in starting on the project search and selection.

Thanks,

Prakhar

[Jan Kiszka]

On 30.03.20 10:02, PRAKHAR BANSAL wrote:
> Hi Jan,
>

With all non-root cells - the root cell will be occupied already (it
runs libvirt e.g.).

>
> I wanted to request you for a favor for the proposal as the deadline is
> approaching. Could I prepare a proposal for this project based on our
> discussion here and improve it later based on feedback comments after
> the deadline? I understand that I got late in starting on the project
> search and selection.

Sure, please go ahead.

Jan

>
> Thanks,
> Prakhar
>
>
> Jan
>
> [1]
> https://groups.google.com/d/msgid/jailhouse-dev/CADiTV-1QiRhSWZnw%2BkHhJMO-BoA4sAcOmTkQE7ZWbHkGh3Jexw%40mail.gmail.com
>
> >
> > -Prakhar
> >
> > On Thu, Mar 26, 2020 at 1:49 AM Jan Kiszka <jan.k...@web.de
> <mailto:jan.k...@web.de>

[PRAKHAR BANSAL]

Hi Jan,

Thanks for the confirmation to proceed on project proposal.

Also, I tried installing Jailhouse on my VM after enabling VT-x/EPT and IOMMU for my VM(Guest OS- Ubuntu 18.04) on VMware fusion hypervisor with MacOS on the host side.

However,  Jailhouse hardware check was failed because of missing Preemption timer and Virtualize APIC access, could you please suggest, if this is hardware limitation?  Is there any workaround here?

My laptop's processor is Intel quad-core i7.

Also, could you please suggest, if I can talk to you through an IRC or slack channel since it is a bit hard to communicate over email every time.

Thanks,

Prakhar

[Jan Kiszka]

On 31.03.20 04:43, PRAKHAR BANSAL wrote:
> Hi Jan,
>

> Thanks for the confirmation to proceed on project proposal.
>
> Also, I tried installing Jailhouse on my VM after enabling VT-x/EPT and
> IOMMU for my VM(Guest OS- Ubuntu 18.04) on VMware fusion hypervisor with
> MacOS on the host side.
> However,  Jailhouse hardware check was failed because of missing

> *Preemption timer and Virtualize APIC access*, could you please suggest,

> if this is hardware limitation?  Is there any workaround here?

You will need a hypervisor that supports both when nesting, but I have
no idea if there is one for the Mac. What is known to work is KVM on
Linux hosts.

> My laptop's processor is Intel quad-core i7.
>

> image.png
>
> Also, could you please suggest, if I can talk to you through an IRC or
> slack channel since it is a bit hard to communicate over email every time.

I'll be listening on #jailhouse, irc.freenode.net.

Jan

>
> Thanks,
> Prakhar
>
>
> On Mon, Mar 30, 2020 at 6:15 AM Jan Kiszka <jan.k...@web.de
> <mailto:jan.k...@web.de>> wrote:
>
> On 30.03.20 10:02, PRAKHAR BANSAL wrote:
> > Hi Jan,
> >
> > On Sat, Mar 28, 2020 at 4:12 AM Jan Kiszka <jan.k...@web.de
> <mailto:jan.k...@web.de>

[PRAKHAR BANSAL]

Hi Jan,

I have shared the draft proposal link with libvirt on GSoC's system.

Could you please check and provide your feedback, if possible.

Thanks,

Prakhar