Re: pre Sandy bridge IOMMU support (gm45)

[Thierry Laurion]

Hi devs!

XEN devs:
As per short discussion with ktemkin earlier in January in #xen:

"ktemkin Jan 10, 2016 16:21:50
This test patch did appear to make the system work, though: https://gist.github.com/ktemkin/0e81b93654ae800a5609

ktemkin Jan 10, 2016 16:24:55
Only real difference I see between that and the upstream behavior (besides limiting things to dom0 so things weren't accidentally passed through) is the call to disable_pmr on line 117 before aborting."

Makes total sense to my early understanding, since it seems that it is said that vt-d engine gets disabled, but disable_pmr(iommu) function is not called to enforce.

What do you think?

QUBES devs:
I'm still trying to understand how to apply this patch to qubes_builder to actually build a test iso or xen.gz image and report. All Qubes patches seem to be applied from git to local directory structure. Looking inside the code to understand how to generate the provided patch to git can apply it to local chrooted environment when building. Any documentation you could point me to would be greatly appreciated, as any feedback to actually fix the issue stopping this laptop from being a nearly perfect candidate for Qubes.

Thierry

Le sam. 23 janv. 2016 à 02:37, Thierry Laurion <thierry...@gmail.com> a écrit :

Hey devs,

Thinkpad x200 p8600 laptops have vt-d, vt-x and tpm. They also have intel integrated graphics 4 Series (gm45 chipset), supported through i915 driver.

In December, a fix got introduced to Xen 4.6 through iommu=no-igfx switch. Before that fix, it was impossible to boot xen without passing iommu=0.

With iommu=no-igfx passed on, Qubes boots xen, kernel, dom0 and domu until some graphic rendering is done from a domu to dom0 xserver.

I'm trying to push forward IOMMU support of gm45 chipset here. The problem is between i915 and xen iommu support for sure, but there is no crash or interesting debugging information given on a serial console.

Any dev help is welcome since that beast and t400 would be excellent Qubes candidates once that problem is fixed. I posted in December on the list just before Christmas but I guess the timing wasn't right;)

Thanks for your help.

Thierry

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Jan 24, 2016 at 06:21:05PM +0000, Thierry Laurion wrote:
> Hi devs!
>
> XEN devs:
> As per short discussion with ktemkin earlier in January in #xen:
>
> "ktemkin Jan 10, 2016 16:21:50
> This test patch did appear to make the system work, though:
> https://gist.github.com/ktemkin/0e81b93654ae800a5609
>
> ktemkin Jan 10, 2016 16:24:55
> Only real difference I see between that and the upstream behavior (besides
> limiting things to dom0 so things weren't accidentally passed through) is
> the call to disable_pmr on line 117 before aborting."
>
>
>
> Makes total sense to my early understanding, since it seems that it is said
> that vt-d engine gets disabled, but disable_pmr(iommu) function is not
> called to enforce.
>
> What do you think?
>
> QUBES devs:
> I'm still trying to understand how to apply this patch to qubes_builder to
> actually build a test iso or xen.gz image and report. All Qubes patches
> seem to be applied from git to local directory structure. Looking inside
> the code to understand how to generate the provided patch to git can apply
> it to local chrooted environment when building. Any documentation you could
> point me to would be greatly appreciated, as any feedback to actually fix
> the issue stopping this laptop from being a nearly perfect candidate for
> Qubes.

Actually for testing patched hypervisor, you can build xen the standard
way (http://wiki.xenproject.org/wiki/Compiling_Xen_From_Source). And
then copy just xen.gz. Qubes-specific patches are only for the
toolstack, not the hypervisor.

But if you want to build full xen package, simply place patches
somewhere in qubes-builder/qubes-src/vmm-xen (patches.misc subdir?) and
add them to series.conf. Then execute "make vmm-xen" from qubes-builder
directory.

>
> Thierry
>
> Le sam. 23 janv. 2016 à 02:37, Thierry Laurion <thierry...@gmail.com>
> a écrit :
>
> > Hey devs,
> >
> > Thinkpad x200 p8600 laptops have vt-d, vt-x and tpm. They also have intel
> > integrated graphics 4 Series (gm45 chipset), supported through i915 driver.
> >
> > In December, a fix got introduced to Xen 4.6 through iommu=no-igfx switch.
> > Before that fix, it was impossible to boot xen without passing iommu=0.
> >
> > With iommu=no-igfx passed on, Qubes boots xen, kernel, dom0 and domu until
> > some graphic rendering is done from a domu to dom0 xserver.
> >
> > I'm trying to push forward IOMMU support of gm45 chipset here. The problem
> > is between i915 and xen iommu support for sure, but there is no crash or
> > interesting debugging information given on a serial console.
> >
> > Any dev help is welcome since that beast and t400 would be excellent Qubes
> > candidates once that problem is fixed. I posted in December on the list
> > just before Christmas but I guess the timing wasn't right;)
> >
> > Thanks for your help.
> > Thierry
> >
>

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWpWIjAAoJENuP0xzK19csmBcH/jAkYioso8K0POq+hIPop9Ft
E9h0b964j/jaZsgqofmnZFj8ZA4zI/qr4mQEIuNdk+dUgN69awn/Ffa+/bxTtv0B
7AnCv65s+xMAOn8YHIc/pcwmL1/FymK1NAoVdk4wWXdWhxOW1PdGp+OCvFGFpOd1
L0rWwuY+EAV1UnUmd4OyPBLVh4f5fFG7B4tXnd1LaZ18noeSOaJpj5/o55zuwpgC
Fx3CtxtAlMLOpu7W1S/MzC73aOajKpFwoaS4RAMD8/Wby3nvtgcBJ6jmBmmSdn/J
9YUOxO9cflIKjKbqXmYZJFceK1CmGNYhYEjTI8m1K9e+ian3vWa3GOwEfBk1oIo=
=F+Eh
-----END PGP SIGNATURE-----

[Andrew Cooper]

On 24/01/16 18:21, Thierry Laurion wrote:

Hi devs!

XEN devs:
As per short discussion with ktemkin earlier in January in #xen:

"ktemkin Jan 10, 2016 16:21:50
This test patch did appear to make the system work, though: https://gist.github.com/ktemkin/0e81b93654ae800a5609

ktemkin Jan 10, 2016 16:24:55
Only real difference I see between that and the upstream behavior (besides limiting things to dom0 so things weren't accidentally passed through) is the call to disable_pmr on line 117 before aborting."

Makes total sense to my early understanding, since it seems that it is said that vt-d engine gets disabled, but disable_pmr(iommu) function is not called to enforce.

What do you think?

There is some confusion here.

"Unfortunately, quirks specific to the Clarkdale/Nehalem integrated graphics device (IGD) do not function correctly with Xen's VT-d implementation"

Either
1) There is a chipset errata which prevents it from functioning correctly, or
2) Xen's VT-d code doesn't program the chip correctly.

Which is it?

If it is 1), then there is a very good case for quirking the affected chipsets and unilaterally disabling them.  If it is 2), then the VT-d code should be corrected.

~Andrew

[Thierry Laurion]

I just tested freshly compiled xen.gz file produced from patched source, as recommended by ktempkin. (Previous post xen.diff attached file got applied to disable pmr).

Same behavior was observable with iommu=no-igfx: when net-vm tray icon gets rendered (corrupted graphics) and notification are draw on screen, system hang without logging any error. 

I will compile xen with debugging options.

If you guys have any insight or people I should talk to, please advise. It would be greatly appreciated. :)

Thierry

[Thierry Laurion]

 
Here is the output of xen (compiled with debug options in Config.mk and rules.mk as instucted here) debug trace when launched from grub2 with:

multiboot /xen-4.6.0-debug.gz placeholder console=none dom0_mem=min:1024M dom0_mem=max:4096M console_timestamps=datems loglvl=all guest_loglvl=all sync_console console_to_ring lapic=debug apic_verbosity=debug apic=debug iommu=no-igfx iommu=debug debug

module /vmlinuz-4.1.13-8.pvops.qubes.x86_64 placeholder root=/dev/mapper/qubes_dom0-root ro rd.lvm.lv=qubes_dom0/root vconsole.font=latarcyrheb-sun16 rd.lvm.lv=qubes_dom0/swap rd.luks.uuid=luks-blah

module /initramfs-4.1.13-8.pvops.qubes.x86_64.img

Any idea? hints? Tips?

[Thierry Laurion]

Hi all, this is debug trace I have with xen 4.6.0 debug build.

Attached files were generated with the single x200 laptop I have that has amt (The only one I have with a VPRO Centrino 2 Inside cpu sticker).

Those are the most complete debug traces I was able to generate.

4 use cases:
1-iommu=required (x200_xen_debug-iommu-required.txt)

2-normal boot (x200_xen_debug-normal.txt)

3-iommu=no-igfx (x200_xen_debug-iommu-no_igfx.txt)

4-iommu=0 (x200_xen_debug-no_iommu.txt)

4 different behaviors:
1-hang before xen is functional ("Couldn't enable Interrupt Remapping and iommu=required/force")

2-hang just after vgaarb and drm driver replacement.

3-hang at graphic rendering in dom0 from netvm domu and usbvm domu notifications when everything is ready to go.

4-functional but no netvm and usbvm device isolation

Any hint/tips/people to point me at?

Thierry

[Thierry Laurion]

The problem wasn't with xen iommu support but kms/drm and i915 driver.

Passing to the kernel i915.preliminary_hw_support=1 fixes it all :)

Thanks

Le sam. 20 févr. 2016 à 22:44, Thierry Laurion <thierry...@gmail.com> a écrit :

Le mar. 26 janv. 2016 à 05:52, Jan Beulich <JBeu...@suse.com> a écrit :

>>> On 25.01.16 at 22:49, <thierry...@gmail.com> wrote:
> The case is 1) disabling iommu for IGD, unilaterally since i915 + gm45
> doesn't play well together. Iommu is still desired to isolate usb and
> network devices, so we don't want to disable iommu completely. The side
> effect of this would be to have IGD only for dom0, which would also
> completely make sense in this use case.
>
> The point is the iommu=no-igfx doesn't fix the issue, since remapping seems
> to still happen for IGD. Does that make sense ?

It certainly may make sense, just that in what you have written so
far I don't think I've been able to spot any evidence thereof. Since,
as you say, nothing interesting gets logged by Xen, you must be
drawing this conclusion from something (or else you wouldn't say
"doesn't fix the issue").

Jan

Here is some interesting lines showing Xen failing without iommu=no-igfx:

--- /home/john/Downloads/amtterm/x200_xen_debug-normal-no_ts.txt
+++ /home/john/Downloads/amtterm/x200_xen_debug-iommu-no_igfx-no_ts.txt
@@ -339,23 +339,10 @@
(XEN) [VT-D]iommu.c:1465: d0:PCI: map 0000:00:1f.3
(XEN) [VT-D]iommu.c:1453: d0:PCIe: map 0000:03:00.0
(XEN) [VT-D]iommu.c:729: iommu_enable_translation: iommu->reg = ffff82c000205000
-(XEN) [VT-D]iommu.c:729: iommu_enable_translation: iommu->reg = ffff82c000203000
+(XEN) [VT-D]iommu.c:719: BIOS did not enable IGD for VT properly. Disabling IGD VT-d engine.
(XEN) [VT-D]iommu.c:729: iommu_enable_translation: iommu->reg = ffff82c000201000
(XEN) [VT-D]iommu.c:729: iommu_enable_translation: iommu->reg = ffff82c000207000
(XEN) Scrubbing Free RAM on 1 nodes using 2 CPUs
-(XEN) [VT-D]iommu.c:873: iommu_fault_status: Fault Overflow
-(XEN) [VT-D]iommu.c:875: iommu_fault_status: Primary Pending Fault
-(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr ffffff000, iommu reg = ffff82c000203000
-(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
-(XEN) print_vtd_entries: iommu ffff8301363fa7d0 dev 0000:00:02.0 gmfn ffffff
-(XEN) root_entry = ffff8301363f4000
-(XEN) root_entry[0] = 80fa001
-(XEN) context = ffff8300080fa000
-(XEN) context[10] = 1_8ae0001
-(XEN) l3 = ffff830008ae0000
-(XEN) l3_index = 3f
-(XEN) l3[3f] = 0
-(XEN) l3[3f] not present
(XEN) ....................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All

I restate my comprehension.

iommu=no-igfx needs to be passed to hypervisor for it to boot.

iommu=dom0-passthrough would also be needed for dom0 tobe able to unset iommu usage for itself?


For Dom0 to have access to device, I also understand that intel_iommu=igfx_off kernel option would need to be passed to i915 driver of dom0?

Doing so still fails without error. Any hint?

Doing so by providing dom0-pass

[Thierry Laurion]

Sorry for the precedent post that was written a bit too fast. Libreboot was flashed when I wrote it, which is the equivalent of a having vt-d deactivated (iommu=0). Thanks to a user that read this post and wrote to me personally so I could do my mea culpa. Sorry for the precedent misleading post.

Xen on a GM45 chipset and with IGD i915 driver is still getting the system hanged when vt-d is activated. I'm willing to borrow a machine to the Xen developer that could fix the iommu=no-igfx code for gm45 chipset to actually work.

A ticket is opened here with current states of thing: https://github.com/QubesOS/qubes-issues/issues/1594#issuecomment-209213917

Sorry about that.

Thierry