Making Changes to Xen (qubes-vmm-xen) and Compiling

[Dylanger Daly]

Hey Guys,

I'm trying to add in a patch for Xen that allows me to pass-through my
NVIDIA MX150 to an appVM, I need to spoof the CPUID within Xen.

I've successfully setup Qubes Builder and built `vmm-xen` however if I
make changes in
`qubes-builder/chroot-dom0-fc29/home/user/rpmbuild/BUILD/xen-4.12.0-rc6`
the changes get restored when I run make.

I think this is due to the `qubes-vmm-xen` repo containing just .patch
files, but it's not very clear how these files are created or how to
make changes to Xen.

Any help would be appreciated, thank you!

[Frédéric Pierret]

Hi,

Indeed, it's overwritten. You need to change sources directly located in
: qubes-builder/qubes-src/vmm-xen/

Do your changes here and the 'make  vmm-xen' will take care of putting
your changes into the chroot.

Also, it seems that you are trying to build dom0 under fc29 which is for
release 4.1 (still work in progress).

Try to use r4.0 config file instead and not master except if you know
what you are doing :)

Best,

[Dylanger Daly]

Hmm, I can't seem to find the actual source, there are only .patch
files in qubes-src/vmm-xen, for example, hvmloader.c dosen't exist in vmm-xen.

Does it matter what AppVM I'm compiling in? Where can I configure 4.0?

Cheers!

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Apr 12, 2019 at 04:45:32AM -0700, Dylanger Daly wrote:
> On Friday, April 12, 2019 at 12:28:12 PM UTC+1, Frédéric Pierret wrote:
> > Hi,
> >
> > Indeed, it's overwritten. You need to change sources directly located in
> > : qubes-builder/qubes-src/vmm-xen/
> >
> > Do your changes here and the 'make  vmm-xen' will take care of putting
> > your changes into the chroot.
> >
> > Also, it seems that you are trying to build dom0 under fc29 which is for
> > release 4.1 (still work in progress).
> >
> > Try to use r4.0 config file instead and not master except if you know
> > what you are doing :)

setup script will ask you for the qubes version - choose 4.0.
Alternatively, if you use a config from example-configs/ directly,
choose the one with 4.0 in name.

In the end, you should get xen-4.8 branch in vmm-xen repository.

> Hmm, I can't seem to find the actual source, there are only .patch
> files in qubes-src/vmm-xen, for example, hvmloader.c dosen't exist in vmm-xen.

There is xen tarball with actual source code, but this not the place
you should change.
What you need to do, is to create a patch and add it to the list.
Since you already know how to change files in
qubes-builder/chroot-dom0-..., you can use that to create a patch.
You'll need "quilt" tool installed. Roughly do this:

1. Build vmm-xen unmodified
2. Go to
qubes-builder/chroot-dom0-fc25/home/user/rpmbuild/BUILD/xen-4.8.5
3. Execute "quilt new some-patch-name.patch" (you can use any name)
4. Identify what files you want to modify and mark them (before you
modify anything!) with quilt add
5. Do your modifications
6. Execute "quilt refresh" to create actual patch. You'll find it in
.patches dir there.

Then, copy resulting patch to qubes-src/vmm-xen and add it to the list
in xen.spec.in (Patch... lines, add it at the end, with a new number).

Alternatively, if you're comfortable with using git, it may be easier to
use it to create the patch instead. Simply clone upstream xen git
repository (https://xenbits.xen.org/git-http/xen.git), switch to
stable-4.8 branch and create a commit on top. Then export it with "git
format-patch -1"

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlywiaYACgkQ24/THMrX
1yyZyAf+ILCvqLddSz2EQoqHuXFtLRA913wakVD3Lqaqa5V2jkHJSypLlvUWmNyu
FH87o66KZjzWzz7K1Wqz24bx34sMDF1tCUX2QsuIZW1Me/LE67OgPb6MSP3oP/oZ
IaI1boZU3cG+88YLyB6cR1RzDwHzhj6XcUhvkmptFoDSBSwPeNbM/nEsUxmw/fmv
1LlMpcBGAS6V7yvcVXV3Ww8BzZwEJbMkBgAwIOkgF5OQ6C5uc22Tzoxmce3hGM9O
4INZ5by45jz2nvbMFiPgIARaxvTPbLscAZ47ppyqWAohMHpko3CFrKI34azwLVn3
srD0yLlrOV18sp6QCCd72E16xTrlgQ==
=0aKA
-----END PGP SIGNATURE-----

[Dylanger Daly]

That's fantastic, thank you very much for your help :)

[Dylanger Daly]

On Friday, April 12, 2019 at 1:50:54 PM UTC+1, Marek Marczykowski-Górecki wrote:

After adding an xl feature (spoof_xen) to be exact, does anyone know how to then enable the bool for an HVM appVM?

The feature is in the same position as pae, acpi and viridian, using `virsh edit <VM_NAME>` and adding `<spoof_xen/>` results in `error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng`

Then if I force: `unexpected feature 'spoof_xen'`, I would assume because the upper Qubes layer isn't aware the feature exists within Xen?

Cheers Guys! :)

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

> After adding an xl feature (spoof_xen) to be exact, does anyone know how to then enable the bool for an HVM appVM?
>
> The feature is in the same position as pae, acpi and viridian, using `virsh edit <VM_NAME>` and adding `<spoof_xen/>` results in `error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng`
>
> Then if I force: `unexpected feature 'spoof_xen'`, I would assume because the upper Qubes layer isn't aware the feature exists within Xen?

Yes, exactly. There is libvirt between those two...
Here you can see patches adding similar flag in libvirt:
https://github.com/QubesOS/qubes-core-libvirt/blob/master/0009-conf-add-xen-specific-feature-e820_host.patch
https://github.com/QubesOS/qubes-core-libvirt/blob/master/0010-libxl-make-use-of-e820_host-feature.patch

Since the first patch is already there, adding yet another feature is
much less code.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlyxI7YACgkQ24/THMrX
1yxqYwgAi8ihXxFGp8nc7DbBUjpMHkJhOqFadIZqU2f2BZWc+KsSQ/Gbeqe1XNFD
mT6oibw0MoxbRoBSH0V41UQ5Jld93SgzW9n0oiTNDv1eBHJHY4AMnYJ2yHaKdAJt
+sVB1d6VjvixN8FZQx+epnuCkZIzbyb5yZko3cJIFPB14qztaI6lAI1DC18YHIYZ
MePVLcPIp9vQWCXkwjAvBfRTAAoiznFZoYIbUkeT4ccwLYpQxibXsWtdDnWSlUNC
2wRXRVKIuKt6zoa2C2svVE/eKt48rZ9lSAGs4Ajz8cSQq/8f6vzYZJ6QlKUrGMq2
3/vfOZBS6fwyWyGWR7yjR86J543Olw==
=gOv8
-----END PGP SIGNATURE-----

[Dylanger Daly]

Thank you so much Marek! :D

[Dylanger Daly]

Anyone know how to install the changes without creating an ISO?

It looks like it's output `qubes-builder/qubes-src/core-libvirt/pkgs/dom0-fc25/x86_64` however when I install libvirt-3.3.0-7.fc25.x86_64.rpm I don't think it's actually installed my changes. `docs/schemas/domaincommon.rng` for example hasn't been updated.

Cheers!

[Dylanger Daly]

libvirt-libs is apart of qubes-core-dom0 protected packages, as well, hm

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

> Anyone know how to install the changes without creating an ISO?
>
> It looks like it's output `qubes-builder/qubes-src/core-libvirt/pkgs/dom0-fc25/x86_64` however when I install libvirt-3.3.0-7.fc25.x86_64.rpm I don't think it's actually installed my changes. `docs/schemas/domaincommon.rng` for example hasn't been updated.

You need to install other updates from that directory too. See list of
currently installed libvirt packages (rpm -q 'libvirt*' in dom0).
To ease the whole thing, you may want to enable
INCREMENT_DEVEL_VERSIONS=1 in builder.conf. This way, each build will
have increased revision number, which will allow you to simply use 'dnf
update path/to/packages/*.rpm' and it will pick the right ones.

If you trust your development VM, you can ease copying to dom0 by using
tar:

qvm-run -p devel 'tar c
qubes-builder/qubes-src/core-libvirt/pkgs/dom0-fc25'|tar xvk

("k" prevent tar to override already existing files)

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlyz14gACgkQ24/THMrX
1yy/jwf8DP6KCGpInaBIRrhdeNOf3eeiNJReHBGHxgS8iwwFrhrUw5hwSjUHOV7V
1pBmuQNrWc/BwQKkHZKKenuOzUlenJJuVuoFGVI0SiCA6CV9GpssCBer09EIiwn0
aTsfbJtxvxZ2oV14CpOmPUiLJN0qA2vE6Soy7KiB+Pk6Z8UO5gOh+AcpRk31ygC8
TSMSLWCxuD5mQGSqwobS8AfCqqZ7YYc0MjWM+idH2gE73Q7FgpS9447iZ8HY3WpC
kaeUzMTMWpkURuTpE+CzHc8j0f8XrsK8rMbKMferJWnBLsyBCsGV1ywM0ye0EcLH
1TqoyO8sze7EuRqgJaGMJrLhJuGEjg==
=ALpI
-----END PGP SIGNATURE-----

[Dylanger Daly]

That's perfect, thank you! I managed to install it successfully, however.

It appears I have two options.

A) Piggyback off of the <xen> Marek Patched
B) Grep all occurrences of `viridian` and attempt to add my `spoof_xen` option.

After disabling Testing I've been able to compile A, however it dosen't seem to actually flip the feature on.

Here's my patch: https://gist.github.com/dylangerdaly/74be3f316ce8f0ddfb27b0202aa5ec2d

The other thing is e820_host is a PV Feature, I'm trying to enable a HVM Feature, I'm not sure if the location of these patches matters.

Note, I'm disabling the Tests, I just want to PoC this before adding all of the testing, unless it's required?

[Dylanger Daly]

Here's a shot at B.

https://gist.github.com/dylangerdaly/daf5dcfb88a7b64c52850a8952d4ee46

I've basically just added my `spoof_xen` feature where `viridian` exists, it boots and I can add `<spoof_xen/>` to the xml of an appVM, but it dosn't _actually_ trigger it within Xen.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

> That's perfect, thank you! I managed to install it successfully, however.
>
> It appears I have two options.
>
> A) Piggyback off of the <xen> Marek Patched
> B) Grep all occurrences of `viridian` and attempt to add my `spoof_xen` option.
>
> After disabling Testing I've been able to compile A, however it dosen't seem to actually flip the feature on.
>
> Here's my patch: https://gist.github.com/dylangerdaly/74be3f316ce8f0ddfb27b0202aa5ec2d
>
> The other thing is e820_host is a PV Feature, I'm trying to enable a HVM Feature, I'm not sure if the location of these patches matters.

It shouldn't matter.

Your patch is mostly correct with one exception: missing part to
actually make use of the new option. See the other patch I've linked:
https://github.com/QubesOS/qubes-core-libvirt/blob/master/0010-libxl-make-use-of-e820_host-feature.patch

Note that "libxl" is the right driver, not "xenapi" or "xen".

> Note, I'm disabling the Tests, I just want to PoC this before adding all of the testing, unless it's required?

For PoC it's fine. To have it included in upstream libvirt, tests are
required.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAly0+sUACgkQ24/THMrX
1yxGIQf+K0Ws+70c7FUmvIBnWeQnh9UaWZzjotzjPGzrHcL5RyEhysgaSbGT9ZYT
pGqDmqH8AqEaF51HiryhPNskOxX6QEYXqs1rEp02ESkqaQxWP+X0FPc79ul9ikYs
Iq/zwSFvVDf+eydFNYfPHqSKjvnE7rc5AKheX8aSOMg2XNb3NDIvzziTWh1/vvmh
xbK88V+h2eR4sU8oFVb0Y1cRKCMu2UdX/LgS7Cloa95/+4AP0L8LhYurXCdXX8H6
o/RmVp0ze4QZrlDOmR08BwoWaZnuJUKna3neeOgE470tPM2H3rpDbEPUpjNyJBTI
54FnVNgXpguZJbClormq79FBaF72Hg==
=qG3v
-----END PGP SIGNATURE-----

[Dylanger Daly]

Cheers! That'd be why ;)

I've added the code to libxl_conf.c, however it complains
```
libxl/libxl_conf.c: In function 'libxlMakeDomBuildInfo':
libxl/libxl_conf.c:427:53: error: 'struct <anonymous>' has no member named 'spoof_xen'
libxl_defbool_set(&b_info->u.hvm.spoof_xen, true);
^
libxl/libxl_conf.c:430:53: error: 'struct <anonymous>' has no member named 'spoof_xen'
libxl_defbool_set(&b_info->u.hvm.spoof_xen, false);
```

spoof_xen hasn't been defined in the struct and I can't figure out where to define it.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Mon, Apr 15, 2019 at 03:54:42PM -0700, Dylanger Daly wrote:
> I've added the code to libxl_conf.c, however it complains
> ```
> libxl/libxl_conf.c: In function 'libxlMakeDomBuildInfo':
> libxl/libxl_conf.c:427:53: error: 'struct <anonymous>' has no member named 'spoof_xen'
> libxl_defbool_set(&b_info->u.hvm.spoof_xen, true);
> ^
> libxl/libxl_conf.c:430:53: error: 'struct <anonymous>' has no member named 'spoof_xen'
> libxl_defbool_set(&b_info->u.hvm.spoof_xen, false);
> ```
>
> spoof_xen hasn't been defined in the struct and I can't figure out where to define it.

How have you added it to xl then?

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAly1DnoACgkQ24/THMrX
1yyCKQf+KQ2WZ0B6x/2LgynOJ2CK3ELGoG3o9+94A0DXs6PJdIYsd16kG+C3W6Cc
mQHzI5udZuPhtc/l3jx6jZ9JPcuH50ZZFt2F2mi5cw/QLzeb8Q4RF3of7tbJrRSO
tUSGteYSZnCY+dYzKuppgKbCZ7JJHHPWkRSV+av3xyZvUe/p13EigzaS1UwCHOzm
S0s44PvDQj3AOcZZBR0Cm0KX3MESDcyiyI7nJHb8Z4sHlbkmxxyn2LpeiG19tM1b
KHwMBQcbjx24akMfYXGB0V3BprmAETc16AlAviZCE3W6eIHIhj0ah2zZgzeNPmeJ
7C9qWbP78SqYNGriVvTO3F6Oy7NbnQ==
=MVM3
-----END PGP SIGNATURE-----

[Qubes123]

I'm also trying to do secondary gpu pass-through, but this time with an AMD card. The problem in my case is that I need to pass also the ACPI VFCT table to the HVM guest, because otherwise the radeon/amdgpu KMS module cannot initialize card. The ACPI VFCT table contain the Atombios tables, so theoretically the vbios is not executed directly.
At fist I thought to implement this is a static way (not patching the ACPI table with actual BDF data) - similarly how the ACPI SLIC table can be passed through. As I have seen, I need to patch Xen (Qemu?) - to be able to reserve the ACPI memory space for the ca. 32k table and to be aware of the new table type in the ACPI headers. Also, I need to patch libvirt to handle the new type of ACPI table in the xl scripts.
My question: Is this the right way to do this? Are there any plans in Qubes R4.1 and Xen-4.12 to do this differently - or even is this requirement already identified in the current development?

[Dylanger Daly]

Ahh I think I might know what's going on, so I've added the feature to vmm-xen, I've just noticed when compiling libvirt it requires the `xen-devel` package, I would assume it's using headers/libs from that package, then complaining `spoof_xen` dosen't exit, I'll try to compile my specific xen-devel package with my feature in it, then install that into chroot.

[Dylanger Daly]

Actually, never mind with that! You need to `make vmm-xen` BEFORE you make libvirt.

It worked! It's now spoofing the CPUID for my HVM!

However, I'm getting
```
Debian GNU/Linux 9 debian-standalone hvc0

debian-standalone login: [ 126.526783] reboot: Power down
[ 0.772598] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
```

Looks to be related to VT-d potentially.

In any case, thank you for getting me to this point Marek!

Qubes123: Interesting, AMD don't store their vBIOS on an SPI Flash Chip?

[Dylanger Daly]

So as a note, the linux-stubdom crashes Xen if you spoof the CPUID. Using the old MiniOS, it 'kind of' works, I haven't been able to get Fedora Workstation or Silverblue (HVM) to boot.