Qubes VT-d Questions (Rootkits and DMA, oh my)
364 views
Skip to first unread message
Andrew
Apr 8, 2015, 4:45:00 PM4/8/15
to qubes...@googlegroups.com
Hi,
Reading one of Joanna's old blog posts about ITL's "Ring -3 Rootkit"
(http://theinvisiblethings.blogspot.co.uk/2009/08/vegas-toys-part-i-ring-3-tools.html),
I came across this interesting paragraph:
An OS might attempt to protect itself from DMA accesses from the
rootkit in the
chipset by carefully setting VT-d protections. [...] it should be
possible to
modify Xen so that it set VT-d mappings in such a strict way, that
the AMT code
(and the AMT rootkit) could not access any useful information in any
of the
domains. This, in fact, would be a good idea anyway, as it would
also prevent
any sort of hardware-based backdoors (except for the backdoors in
the CPU).
Let's leave aside for the moment the fact that the AMT core could use
its DMA capabilities to change how Xen sets VT-d mappings--either
automatically or upon trigger by the integrated cellular connection,
before Xen even boots.
1) Does Qubes' Xen automatically set those VT-d mappings to restrict the
AMT core's DMA?
2) How do I check?
3) And if not, how can I tell Qubes to set those mappings on boot, like
'rd.qubes.hide_all_usb'? Related: is there a general way to tell Qubes
to bind a given device to pciback as early as possible?
On my system lspci shows "Intel Corporation 7 Series/C210 Series Chipset
Family MEI Controller #1 (rev 04)".
4) Is this the AMT core's bus presence?
5) If so, is it true that the AMT core can be isolated from the rest of
the system by restricting this device? After all, the "Ring -3 Rootkit"
slides show that the AMT Virtual CD-ROM is presented as a new PCI device
(!), and the AMT core appears to be integrated into the memory
controller itself.
FWIW I can assign this device to another domain without any problems,
but this happens well after boot, and after it's first bound to mei_me
(which has now been blacklisted). (related to question 3 above).
Sorry for the indigestible wall of text. Thanks for any answers.
Andrew
PS: * It's really insane that nearly all (?) Intel CPUs these days
apparently come with integrated 3G comms for OOB communication with the
AMT core for 'anti-theft'. Does anyone know if the antenna is
integrated on-die or in-package, or does it require that you have a 3G
WWAN card installed?
Reading one of Joanna's old blog posts about ITL's "Ring -3 Rootkit"
(http://theinvisiblethings.blogspot.co.uk/2009/08/vegas-toys-part-i-ring-3-tools.html),
I came across this interesting paragraph:
An OS might attempt to protect itself from DMA accesses from the
rootkit in the
chipset by carefully setting VT-d protections. [...] it should be
possible to
modify Xen so that it set VT-d mappings in such a strict way, that
the AMT code
(and the AMT rootkit) could not access any useful information in any
of the
domains. This, in fact, would be a good idea anyway, as it would
also prevent
any sort of hardware-based backdoors (except for the backdoors in
the CPU).
Let's leave aside for the moment the fact that the AMT core could use
its DMA capabilities to change how Xen sets VT-d mappings--either
automatically or upon trigger by the integrated cellular connection,
before Xen even boots.
1) Does Qubes' Xen automatically set those VT-d mappings to restrict the
AMT core's DMA?
2) How do I check?
3) And if not, how can I tell Qubes to set those mappings on boot, like
'rd.qubes.hide_all_usb'? Related: is there a general way to tell Qubes
to bind a given device to pciback as early as possible?
On my system lspci shows "Intel Corporation 7 Series/C210 Series Chipset
Family MEI Controller #1 (rev 04)".
4) Is this the AMT core's bus presence?
5) If so, is it true that the AMT core can be isolated from the rest of
the system by restricting this device? After all, the "Ring -3 Rootkit"
slides show that the AMT Virtual CD-ROM is presented as a new PCI device
(!), and the AMT core appears to be integrated into the memory
controller itself.
FWIW I can assign this device to another domain without any problems,
but this happens well after boot, and after it's first bound to mei_me
(which has now been blacklisted). (related to question 3 above).
Sorry for the indigestible wall of text. Thanks for any answers.
Andrew
PS: * It's really insane that nearly all (?) Intel CPUs these days
apparently come with integrated 3G comms for OOB communication with the
AMT core for 'anti-theft'. Does anyone know if the antenna is
integrated on-die or in-package, or does it require that you have a 3G
WWAN card installed?
WhonixQubes
Apr 8, 2015, 7:24:30 PM4/8/15
to kyb...@riseup.net, qubes...@googlegroups.com
On 2015-04-08 8:44 pm, Andrew wrote:
> PS: * It's really insane that nearly all (?) Intel CPUs these days
> apparently come with integrated 3G comms for OOB communication with the
> AMT core for 'anti-theft'. Does anyone know if the antenna is
> integrated on-die or in-package, or does it require that you have a 3G
> WWAN card installed?
This 2011 Intel doc says it requires an additional wireless card...
> PS: * It's really insane that nearly all (?) Intel CPUs these days
> apparently come with integrated 3G comms for OOB communication with the
> AMT core for 'anti-theft'. Does anyone know if the antenna is
> integrated on-die or in-package, or does it require that you have a 3G
> WWAN card installed?
http://www.intel.com/content/dam/doc/product-brief/mobile-computing-protect-laptops-and-data-with-intel-anti-theft-technology-brief.pdf
> Notification via an encrypted SMS text message
> over a 3g network. For this option, the laptop does
> not need to be connected to the Internet, but it
> must be within range of a 3G network. This feature
> works even if the OS is not running or has been re-
> installed, thanks to a hardware-to-hardware link
> between the 3G card and the Intel AT system.2
>
> ...
>
> New Intel AT features take advantage of 3g networks
> With Intel® Anti-Theft Technology (Intel® AT), IT administrators
> can now use encrypted SMS messages over a 3G network
> to send a poison pill, remotely unlock a recovered laptop
> quickly, or direct the system to send location information
> (GPS coordinates) back to the central server:
>
> Poison pill delivery via an encrypted SMS message over
> a 3g network. 3G connections can occur regardless of the
> state of the OS, via a direct hardware link between Intel AT
> and the 3G module.
>
> Remote unlock via an encrypted SMS message over
> a 3g network. This lets IT reactivate the laptop within
> minutes of recovering the PC.
>
> Location beaconing. Intel AT can now transmit latitude
> and longitude (using GPS coordinates) to the central server
> if the system is equipped with a supported 3G module.
>
> IT administrators can specify automated beaconing at
> regular intervals or location information on request when
> the laptop is marked as lost or stolen.
> To take advantage of 3G-based communication, the laptop
> does not need to be connected to the Internet, but it must
> be within range of a 3G network.2
>
> ...
>
> 2 This feature requires a laptop with Intel® Anti-Theft Technology
> (Intel® AT) 3.0, a 3G laptop modem that supports Intel AT 3.0
> functionality (for example, the Ericsson F5521gw), and OEM-enabled
> communication between the 3G modem and laptop.
AMT is quite concerning and Intel in general is too.
But if these processors were to be shown to have independent radio
capabilities, then I would trash all my Intel PCs immediately.
WhonixQubes
7v5w7go9ub0o
Apr 8, 2015, 7:54:14 PM4/8/15
to qubes...@googlegroups.com
On 04/08/15 19:23, WhonixQubes wrote:
> On 2015-04-08 8:44 pm, Andrew wrote:
>> PS: * It's really insane that nearly all (?) Intel CPUs these days
>> apparently come with integrated 3G comms for OOB communication with the
>> AMT core for 'anti-theft'. Does anyone know if the antenna is
>> integrated on-die or in-package, or does it require that you have a 3G
>> WWAN card installed?
>
>
> This 2011 Intel doc says it requires an additional wireless card...
>
> http://www.intel.com/content/dam/doc/product-brief/mobile-computing-protect-laptops-and-data-with-intel-anti-theft-technology-brief.pdf
>
>
>
>
> AMT is quite concerning and Intel in general is too.
>
> But if these processors were to be shown to have independent radio
> capabilities, then I would trash all my Intel PCs immediately.
>
>
> WhonixQubes
>
Ditto what he said.
> AMT is quite concerning and Intel in general is too.
>
> But if these processors were to be shown to have independent radio
> capabilities, then I would trash all my Intel PCs immediately.
>
>
> WhonixQubes
>
It is a configurable option in the Lenovo 440P BIOS, which also
indicates the need for a separate card.
I'd guess that some "helpful" OEMs might enable it, and then lock the
BIOS from disabling by the bad guy!?
Andrew
Apr 8, 2015, 10:50:05 PM4/8/15
to qubes...@googlegroups.com
7v5w7go9ub0o:
considering doing ;).
>
> Ditto what he said.
>
> It is a configurable option in the Lenovo 440P BIOS, which also
> indicates the need for a separate card.
>
> I'd guess that some "helpful" OEMs might enable it, and then lock the
> BIOS from disabling by the bad guy!?
>
>
What worried/worries me most is that even when you "disable" the
functionality in the BIOS (or "permanently disable"), the AMT core
*still* runs code. So I figured if the antenna and modem were
integrated in-package, the "disable" switch might not do anything to
prevent cellular comms, but merely disable the poison pill from working
unless you pay CompuTrace. And of course, NSA could still compromise
your system remotely whenever they want.
All the other questions still stand, though. I'd love to have an answer
for them :).
Andrew
>
>
> On 04/08/15 19:23, WhonixQubes wrote:
>> On 2015-04-08 8:44 pm, Andrew wrote:
>>> PS: * It's really insane that nearly all (?) Intel CPUs these days
>>> apparently come with integrated 3G comms for OOB communication with the
>>> AMT core for 'anti-theft'. Does anyone know if the antenna is
>>> integrated on-die or in-package, or does it require that you have a 3G
>>> WWAN card installed?
>>
>>
>> This 2011 Intel doc says it requires an additional wireless card...
>>
>> http://www.intel.com/content/dam/doc/product-brief/mobile-computing-protect-laptops-and-data-with-intel-anti-theft-technology-brief.pdf
>>
>>
>>
> <snip>
>>
>> AMT is quite concerning and Intel in general is too.
>>
>> But if these processors were to be shown to have independent radio
>> capabilities, then I would trash all my Intel PCs immediately.
>>
Thanks for the link. In my moment of panic, that's exactly what I was
>
> On 04/08/15 19:23, WhonixQubes wrote:
>> On 2015-04-08 8:44 pm, Andrew wrote:
>>> PS: * It's really insane that nearly all (?) Intel CPUs these days
>>> apparently come with integrated 3G comms for OOB communication with the
>>> AMT core for 'anti-theft'. Does anyone know if the antenna is
>>> integrated on-die or in-package, or does it require that you have a 3G
>>> WWAN card installed?
>>
>>
>> This 2011 Intel doc says it requires an additional wireless card...
>>
>> http://www.intel.com/content/dam/doc/product-brief/mobile-computing-protect-laptops-and-data-with-intel-anti-theft-technology-brief.pdf
>>
>>
>>
> <snip>
>>
>> AMT is quite concerning and Intel in general is too.
>>
>> But if these processors were to be shown to have independent radio
>> capabilities, then I would trash all my Intel PCs immediately.
>>
considering doing ;).
>
> Ditto what he said.
>
> It is a configurable option in the Lenovo 440P BIOS, which also
> indicates the need for a separate card.
>
> I'd guess that some "helpful" OEMs might enable it, and then lock the
> BIOS from disabling by the bad guy!?
>
>
functionality in the BIOS (or "permanently disable"), the AMT core
*still* runs code. So I figured if the antenna and modem were
integrated in-package, the "disable" switch might not do anything to
prevent cellular comms, but merely disable the poison pill from working
unless you pay CompuTrace. And of course, NSA could still compromise
your system remotely whenever they want.
All the other questions still stand, though. I'd love to have an answer
for them :).
Andrew
7v5w7go9ub0o
Apr 9, 2015, 9:29:06 AM4/9/15
to qubes...@googlegroups.com
Yep!
>
> All the other questions still stand, though. I'd love to have an answer
> for them :).
>
> Andrew
>
Andrew
Apr 9, 2015, 2:58:34 PM4/9/15
to qubes...@googlegroups.com
7v5w7go9ub0o:
I just saw that someone finally figured out the compression scheme for
the AMT modules that Igor Skochinsky didn't manage to decode:
http://io.smashthestack.org/me/.
Hopefully this means we'll see some disassemblies soon. Looks like
arc-gnu-tools are now free again since the Synopsys acquisition:
https://github.com/foss-for-synopsys-dwc-arc-processors.
Anyway, the page mentions this repo:
https://github.com/zamaudio/intelmetool. Judging from that code, it
looks like I can answer a few of my own questions:
> 1) Does Qubes' Xen automatically set those VT-d mappings to restrict
the AMT core's DMA?
No.
> 2) How do I check?
$ lspci -v | grep "MEI Controller" -A 6 | grep "Kernel driver"
Should at least show "pciback". But this only hides the device from Dom0.
I believe you also need to actually attach it to a DomU, and for a PV
DomU (as is the default Qubes AppVM), you might need to add iommu=pv to
/etc/default/grub "GRUB_CMDLINE_LINUX_DEFAULT".
You can check that a device has been passed through to a domain with
$ xl pci-list <domain>
(according to http://wiki.xen.org/wiki/VTd_HowTo).
I really don't know how to check that the VT-d mapping is *truly* set,
though. Maybe booting with iommu=verbose will tell?
> 3) And if not, how can I tell Qubes to set those mappings on boot,
like 'rd.qubes.hide_all_usb'? Related: is there a general way to tell
Qubes to bind a given device to pciback as early as possible?
Still don't know. This is also probably the only question that's
actually appropriate for this list ;).
> 4) Is this the AMT core's bus presence?
Still not sure. It's at least the presence for the host interface.
> 5) If so, is it true that the AMT core can be isolated from the rest
of the system by restricting this device?
Still have no idea!
Andrew
the AMT modules that Igor Skochinsky didn't manage to decode:
http://io.smashthestack.org/me/.
Hopefully this means we'll see some disassemblies soon. Looks like
arc-gnu-tools are now free again since the Synopsys acquisition:
https://github.com/foss-for-synopsys-dwc-arc-processors.
Anyway, the page mentions this repo:
https://github.com/zamaudio/intelmetool. Judging from that code, it
looks like I can answer a few of my own questions:
> 1) Does Qubes' Xen automatically set those VT-d mappings to restrict
the AMT core's DMA?
> 2) How do I check?
Should at least show "pciback". But this only hides the device from Dom0.
I believe you also need to actually attach it to a DomU, and for a PV
DomU (as is the default Qubes AppVM), you might need to add iommu=pv to
/etc/default/grub "GRUB_CMDLINE_LINUX_DEFAULT".
You can check that a device has been passed through to a domain with
$ xl pci-list <domain>
(according to http://wiki.xen.org/wiki/VTd_HowTo).
I really don't know how to check that the VT-d mapping is *truly* set,
though. Maybe booting with iommu=verbose will tell?
> 3) And if not, how can I tell Qubes to set those mappings on boot,
like 'rd.qubes.hide_all_usb'? Related: is there a general way to tell
Qubes to bind a given device to pciback as early as possible?
actually appropriate for this list ;).
> 4) Is this the AMT core's bus presence?
> 5) If so, is it true that the AMT core can be isolated from the rest
of the system by restricting this device?
Andrew
Radoslaw Szkodzinski
Apr 9, 2015, 4:36:44 PM4/9/15
to qubes...@googlegroups.com
On Thu, Apr 9, 2015 at 8:58 PM, Andrew <kyb...@riseup.net> wrote:
> 7v5w7go9ub0o:
>>
>>
>> On 04/08/15 22:48, Andrew wrote:
> I just saw that someone finally figured out the compression scheme for
> the AMT modules that Igor Skochinsky didn't manage to decode:
> http://io.smashthestack.org/me/.
>
> Hopefully this means we'll see some disassemblies soon. Looks like
> arc-gnu-tools are now free again since the Synopsys acquisition:
> https://github.com/foss-for-synopsys-dwc-arc-processors.
Disassembling the kernel won't do much. The
> 7v5w7go9ub0o:
>>
>>
>> On 04/08/15 22:48, Andrew wrote:
> I just saw that someone finally figured out the compression scheme for
> the AMT modules that Igor Skochinsky didn't manage to decode:
> http://io.smashthestack.org/me/.
>
> Hopefully this means we'll see some disassemblies soon. Looks like
> arc-gnu-tools are now free again since the Synopsys acquisition:
> https://github.com/foss-for-synopsys-dwc-arc-processors.
>
> Anyway, the page mentions this repo:
> https://github.com/zamaudio/intelmetool. Judging from that code, it
> looks like I can answer a few of my own questions:
>
>> 1) Does Qubes' Xen automatically set those VT-d mappings to restrict
> the AMT core's DMA?
>
> No.
>> 2) How do I check?
>
> $ lspci -v | grep "MEI Controller" -A 6 | grep "Kernel driver"
> Should at least show "pciback". But this only hides the device from Dom0.
>
> I believe you also need to actually attach it to a DomU, and for a PV
> DomU (as is the default Qubes AppVM), you might need to add iommu=pv to
> /etc/default/grub "GRUB_CMDLINE_LINUX_DEFAULT".
> You can check that a device has been passed through to a domain with
> $ xl pci-list <domain>
>
> (according to http://wiki.xen.org/wiki/VTd_HowTo).
>
> I really don't know how to check that the VT-d mapping is *truly* set,
> though. Maybe booting with iommu=verbose will tell?
>> 3) And if not, how can I tell Qubes to set those mappings on boot,
> like 'rd.qubes.hide_all_usb'? Related: is there a general way to tell
> Qubes to bind a given device to pciback as early as possible?
>
> Still don't know. This is also probably the only question that's
> actually appropriate for this list ;).
If you have the pciback module built into the kernel, it will hide
these devices before booting initramfs.
Earlier than that is not possible yet, could be turned into a Xen
kernel option to preset the mapping and not show the device directly
to the dom0. Nobody has written this kind of patch - could be useful
for PVH Dom0 with hardware redirected into PVH DomU. Ask xen-devel
perhaps?
>> 4) Is this the AMT core's bus presence?
>
> Still not sure. It's at least the presence for the host interface.
control), toggling antitheft, toggling remote network operation,
setting password locally, setting IDE mappings etc.
ME can additionally framegrab the iGPU (framebuffer in system memory),
provide PS/2 input devices then memory bounce data MMIO-like for
network firmware to read. The interface used to listen for the network
interfaces is not known to me. Rafał Wojtczuk knows much more than
that - and as part of ITL has managed to exploit the network interface
firmware to do generic DMA using this back in 2009.
How it is setup has not yet been reverse engineered, but it is in the
special firmware mode which is disabled after startup.
The framegrabbing and related networking parts actually use IOMMU - it
is pre-setup in DMAR by the EFI code. When IOMMU is enabled and AMT is
also enabled, the firmware uses a different DMAR table. There were
bugs related to that - garbage in the alternate tables.
>> 5) If so, is it true that the AMT core can be isolated from the rest
> of the system by restricting this device?
>
> Still have no idea!
base DMAR mapping set by firmware. Trying it will probably just fail
with an error.
--
Radosław Szkodziński
Andrew
Apr 9, 2015, 8:53:32 PM4/9/15
to qubes...@googlegroups.com
Radoslaw Szkodzinski:
which as far as I can tell is basically a nightmare ;).
If they are set by the EFI code, though, then it should be possible to
have the EFI firmware change them after loading the Management Engine
code, right? Or even just never set them in the first place.
I realize I may be completely wrong on this, and that this is anyway
probably more appropriate for the coreboot list.
>
>>> 2) How do I check?
>>
>> $ lspci -v | grep "MEI Controller" -A 6 | grep "Kernel driver"
>> Should at least show "pciback". But this only hides the device from Dom0.
>>
>> I believe you also need to actually attach it to a DomU, and for a PV
>> DomU (as is the default Qubes AppVM), you might need to add iommu=pv to
>> /etc/default/grub "GRUB_CMDLINE_LINUX_DEFAULT".
>
> Why PV? That makes the isolation completely worthless.
From what I understand, this isn't true. If it is, though, then doesn't
Qubes have a serious security issue by default, as the NetVM is PV?
Also, apparently, since Xen 4.0 it's not necessary to add "iommu=pv" to
enable VT-d for PV domains.
>
>> You can check that a device has been passed through to a domain with
>> $ xl pci-list <domain>
>>
>> (according to http://wiki.xen.org/wiki/VTd_HowTo).
>>
>> I really don't know how to check that the VT-d mapping is *truly* set,
>> though. Maybe booting with iommu=verbose will tell?
>
> Add "debug" option to Xen kernel line.
Thanks. Also iommu=verbose shows '[VT-D]iommu.c: ... unmap bdf = ...'
and '[VT-d]iommu.c: ... map bdf = ...' messages.
>
>>> 3) And if not, how can I tell Qubes to set those mappings on boot,
>> like 'rd.qubes.hide_all_usb'? Related: is there a general way to tell
>> Qubes to bind a given device to pciback as early as possible?
>>
>> Still don't know. This is also probably the only question that's
>> actually appropriate for this list ;).
>
> Linux option xen-pciback.hide=(00:01.0)(01:00.0) etc.
> If you have the pciback module built into the kernel, it will hide
> these devices before booting initramfs.
Unfortunately it looks like this is a module by default. But thanks for
the tip!
Marek: is there any reason to build this as a module? And in which
source can I find the implementation for rd.qubes.hide_all_usb?
>
> Earlier than that is not possible yet, could be turned into a Xen
> kernel option to preset the mapping and not show the device directly
> to the dom0. Nobody has written this kind of patch - could be useful
> for PVH Dom0 with hardware redirected into PVH DomU. Ask xen-devel
> perhaps?
Hmm, OK. I wonder now, though, what the benefit would be. The two
things that appear to matter are:
1) Preventing communication with the MEI interface, which could be
accomplished by building xen-pciback into the kernel, or just deleting
the mei_me.ko module (right?)
2) Actually isolating the AMT core from the rest of the system, to the
extent possible. This might be do-able by modifying the EFI BIOS. At
the least we can limit the MEI controller's access.
>>> 4) Is this the AMT core's bus presence?
>>
>> Still not sure. It's at least the presence for the host interface.
>
> This is just the MEI control, used for IPMI signalling (reboot
> control), toggling antitheft, toggling remote network operation,
> setting password locally, setting IDE mappings etc.
OK, so we definitely want this assigned to a quarantine domain.
> ME can additionally framegrab the iGPU (framebuffer in system memory),
> provide PS/2 input devices then memory bounce data MMIO-like for
> network firmware to read.
...Jesus Christ. Assigning the network card to a separate domain (i.e.
NetVM) should prevent its access to the NIC at least, right? Right??!
> The interface used to listen for the network
> interfaces is not known to me. Rafał Wojtczuk knows much more than
> that - and as part of ITL has managed to exploit the network interface
> firmware to do generic DMA using this back in 2009.
So the slides he made appear contradictory: at the same time, he says an
AMT rootkit can change the DMAR tables itself and therefore bypass VT-d,
but then says that assigning the ME PCI device to a driver domain
prevents access to Dom0 memory. How are both possible? And really:
what is the 'special interface' to the NIC? Do I need to remove my PCIe
NIC and use only a generic USB WiFi card?
> How it is setup has not yet been reverse engineered, but it is in the
> special firmware mode which is disabled after startup.
>
> The framegrabbing and related networking parts actually use IOMMU - it
> is pre-setup in DMAR by the EFI code. When IOMMU is enabled and AMT is
> also enabled, the firmware uses a different DMAR table. There were
> bugs related to that - garbage in the alternate tables.
I really don't know what this means :). Can you explain a bit more?
>>> 5) If so, is it true that the AMT core can be isolated from the rest
>> of the system by restricting this device?
>>
>> Still have no idea!
>
> You could try, but I think there is not really a way to countermand a
> base DMAR mapping set by firmware. Trying it will probably just fail
> with an error.
But is it possible to change the firmware itself and prevent the AMT
core from having access to any memory/MMIO devices other than its own range?
Thanks!
Andrew
> On Thu, Apr 9, 2015 at 8:58 PM, Andrew <kyb...@riseup.net> wrote:
>> 7v5w7go9ub0o:
>>>
>>>
>>> On 04/08/15 22:48, Andrew wrote:
>> I just saw that someone finally figured out the compression scheme for
>> the AMT modules that Igor Skochinsky didn't manage to decode:
>> http://io.smashthestack.org/me/.
>>
>> Hopefully this means we'll see some disassemblies soon. Looks like
>> arc-gnu-tools are now free again since the Synopsys acquisition:
>> https://github.com/foss-for-synopsys-dwc-arc-processors.
>
> Disassembling the kernel won't do much. The
>
>>
>> Anyway, the page mentions this repo:
>> https://github.com/zamaudio/intelmetool. Judging from that code, it
>> looks like I can answer a few of my own questions:
>>
>>> 1) Does Qubes' Xen automatically set those VT-d mappings to restrict
>> the AMT core's DMA?
>>
>> No.
>
> This might even be impossible, these mappings are set in EFI DMAR.
Disclaimer: I am woefully uninformed about the modern PC boot process,
>> 7v5w7go9ub0o:
>>>
>>>
>>> On 04/08/15 22:48, Andrew wrote:
>> I just saw that someone finally figured out the compression scheme for
>> the AMT modules that Igor Skochinsky didn't manage to decode:
>> http://io.smashthestack.org/me/.
>>
>> Hopefully this means we'll see some disassemblies soon. Looks like
>> arc-gnu-tools are now free again since the Synopsys acquisition:
>> https://github.com/foss-for-synopsys-dwc-arc-processors.
>
> Disassembling the kernel won't do much. The
>
>>
>> Anyway, the page mentions this repo:
>> https://github.com/zamaudio/intelmetool. Judging from that code, it
>> looks like I can answer a few of my own questions:
>>
>>> 1) Does Qubes' Xen automatically set those VT-d mappings to restrict
>> the AMT core's DMA?
>>
>> No.
>
> This might even be impossible, these mappings are set in EFI DMAR.
which as far as I can tell is basically a nightmare ;).
If they are set by the EFI code, though, then it should be possible to
have the EFI firmware change them after loading the Management Engine
code, right? Or even just never set them in the first place.
I realize I may be completely wrong on this, and that this is anyway
probably more appropriate for the coreboot list.
>
>>> 2) How do I check?
>>
>> $ lspci -v | grep "MEI Controller" -A 6 | grep "Kernel driver"
>> Should at least show "pciback". But this only hides the device from Dom0.
>>
>> I believe you also need to actually attach it to a DomU, and for a PV
>> DomU (as is the default Qubes AppVM), you might need to add iommu=pv to
>> /etc/default/grub "GRUB_CMDLINE_LINUX_DEFAULT".
>
> Why PV? That makes the isolation completely worthless.
Qubes have a serious security issue by default, as the NetVM is PV?
Also, apparently, since Xen 4.0 it's not necessary to add "iommu=pv" to
enable VT-d for PV domains.
>
>> You can check that a device has been passed through to a domain with
>> $ xl pci-list <domain>
>>
>> (according to http://wiki.xen.org/wiki/VTd_HowTo).
>>
>> I really don't know how to check that the VT-d mapping is *truly* set,
>> though. Maybe booting with iommu=verbose will tell?
>
> Add "debug" option to Xen kernel line.
and '[VT-d]iommu.c: ... map bdf = ...' messages.
>
>>> 3) And if not, how can I tell Qubes to set those mappings on boot,
>> like 'rd.qubes.hide_all_usb'? Related: is there a general way to tell
>> Qubes to bind a given device to pciback as early as possible?
>>
>> Still don't know. This is also probably the only question that's
>> actually appropriate for this list ;).
>
> Linux option xen-pciback.hide=(00:01.0)(01:00.0) etc.
> If you have the pciback module built into the kernel, it will hide
> these devices before booting initramfs.
the tip!
Marek: is there any reason to build this as a module? And in which
source can I find the implementation for rd.qubes.hide_all_usb?
>
> Earlier than that is not possible yet, could be turned into a Xen
> kernel option to preset the mapping and not show the device directly
> to the dom0. Nobody has written this kind of patch - could be useful
> for PVH Dom0 with hardware redirected into PVH DomU. Ask xen-devel
> perhaps?
things that appear to matter are:
1) Preventing communication with the MEI interface, which could be
accomplished by building xen-pciback into the kernel, or just deleting
the mei_me.ko module (right?)
2) Actually isolating the AMT core from the rest of the system, to the
extent possible. This might be do-able by modifying the EFI BIOS. At
the least we can limit the MEI controller's access.
>>> 4) Is this the AMT core's bus presence?
>>
>> Still not sure. It's at least the presence for the host interface.
>
> This is just the MEI control, used for IPMI signalling (reboot
> control), toggling antitheft, toggling remote network operation,
> setting password locally, setting IDE mappings etc.
> ME can additionally framegrab the iGPU (framebuffer in system memory),
> provide PS/2 input devices then memory bounce data MMIO-like for
> network firmware to read.
NetVM) should prevent its access to the NIC at least, right? Right??!
> The interface used to listen for the network
> interfaces is not known to me. Rafał Wojtczuk knows much more than
> that - and as part of ITL has managed to exploit the network interface
> firmware to do generic DMA using this back in 2009.
AMT rootkit can change the DMAR tables itself and therefore bypass VT-d,
but then says that assigning the ME PCI device to a driver domain
prevents access to Dom0 memory. How are both possible? And really:
what is the 'special interface' to the NIC? Do I need to remove my PCIe
NIC and use only a generic USB WiFi card?
> How it is setup has not yet been reverse engineered, but it is in the
> special firmware mode which is disabled after startup.
>
> The framegrabbing and related networking parts actually use IOMMU - it
> is pre-setup in DMAR by the EFI code. When IOMMU is enabled and AMT is
> also enabled, the firmware uses a different DMAR table. There were
> bugs related to that - garbage in the alternate tables.
>>> 5) If so, is it true that the AMT core can be isolated from the rest
>> of the system by restricting this device?
>>
>> Still have no idea!
>
> You could try, but I think there is not really a way to countermand a
> base DMAR mapping set by firmware. Trying it will probably just fail
> with an error.
core from having access to any memory/MMIO devices other than its own range?
Thanks!
Andrew
Marek Marczykowski-Górecki
Apr 9, 2015, 9:41:43 PM4/9/15
to Andrew, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Exactly, iommu=pv isn't currently needed to have VT-d for both HVM and
PV.
> >>> 3) And if not, how can I tell Qubes to set those mappings on boot,
> >> like 'rd.qubes.hide_all_usb'? Related: is there a general way to tell
> >> Qubes to bind a given device to pciback as early as possible?
> >>
> >> Still don't know. This is also probably the only question that's
> >> actually appropriate for this list ;).
> >
> > Linux option xen-pciback.hide=(00:01.0)(01:00.0) etc.
> > If you have the pciback module built into the kernel, it will hide
> > these devices before booting initramfs.
>
> Unfortunately it looks like this is a module by default. But thanks for
> the tip!
>
> Marek: is there any reason to build this as a module? And in which
> source can I find the implementation for rd.qubes.hide_all_usb?
This is because we want to build its arguments dynamically - enumerate
all network devices and assign all of them to pciback. This is done in
initramfs, so before mei.ko has a chance to load.
The code is in core-admin-linux repo in file
dracut/modules.d/90qubes-pciback/qubes-pciback.sh:
- ------
type getarg >/dev/null 2>&1 || . /lib/dracut-lib.sh
# Find all networking devices currenly installed...
HIDE_PCI=`lspci -mm -n | grep '^[^ ]* "02'|awk '{ ORS="";print "(" $1
")";}'`
if getargbool 0 rd.qubes.hide_all_usb; then
HIDE_PCI=$HIDE_PCI`lspci -mm -n | grep '^[^ ]* "0c03'|awk '{
ORS="";print "(" $1 ")";}'`
fi
HIDE_PCI=$HIDE_PCI`getarg rd.qubes.hide_pci | tr ',' '\n'|awk '{
ORS="";print "(" $1 ")";}'`
# ... and hide them so that Dom0 doesn't load drivers for them
modprobe pciback hide=$HIDE_PCI 2> /dev/null || modprobe xen-pciback
hide=$HIDE_PCI
- -----
As you can see, you can specify additional devices with
rd.qubes.hide_pci.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJVJypNAAoJENuP0xzK19csZfMH/At4VcFYCVjAkTF03G7bLubl
0EXart42CfSH1RF5bJJVRe89/DoYZWDvrwwlToHx9GV96XYIGzfg6/b86zgqoki8
177acOssWahI/c6UJKtdGvghx5zwsSKiR+5JTIBwBkdBjL46ktTrvp7MBBxush+E
JPO7xN3SuG9jvW4ePFJAw27a12D4WxzwkO7KR74NTm4FGA4aUWb1kqIJm7/zx8jF
OakNENS11TcXlWOKle3DfpkBTc0toIiTnxJy9QWsy+g82laInYLsFvdlguuckiGR
f4jV82sorPs0YkHnos76wwd9Ym/5j8BARRokY7hPwI7vbKIxkGbiHVDKg4MILOk=
=1Uqv
-----END PGP SIGNATURE-----
Hash: SHA1
PV.
> >>> 3) And if not, how can I tell Qubes to set those mappings on boot,
> >> like 'rd.qubes.hide_all_usb'? Related: is there a general way to tell
> >> Qubes to bind a given device to pciback as early as possible?
> >>
> >> Still don't know. This is also probably the only question that's
> >> actually appropriate for this list ;).
> >
> > Linux option xen-pciback.hide=(00:01.0)(01:00.0) etc.
> > If you have the pciback module built into the kernel, it will hide
> > these devices before booting initramfs.
>
> Unfortunately it looks like this is a module by default. But thanks for
> the tip!
>
> Marek: is there any reason to build this as a module? And in which
> source can I find the implementation for rd.qubes.hide_all_usb?
all network devices and assign all of them to pciback. This is done in
initramfs, so before mei.ko has a chance to load.
The code is in core-admin-linux repo in file
dracut/modules.d/90qubes-pciback/qubes-pciback.sh:
- ------
type getarg >/dev/null 2>&1 || . /lib/dracut-lib.sh
# Find all networking devices currenly installed...
HIDE_PCI=`lspci -mm -n | grep '^[^ ]* "02'|awk '{ ORS="";print "(" $1
")";}'`
if getargbool 0 rd.qubes.hide_all_usb; then
HIDE_PCI=$HIDE_PCI`lspci -mm -n | grep '^[^ ]* "0c03'|awk '{
ORS="";print "(" $1 ")";}'`
fi
HIDE_PCI=$HIDE_PCI`getarg rd.qubes.hide_pci | tr ',' '\n'|awk '{
ORS="";print "(" $1 ")";}'`
# ... and hide them so that Dom0 doesn't load drivers for them
modprobe pciback hide=$HIDE_PCI 2> /dev/null || modprobe xen-pciback
hide=$HIDE_PCI
- -----
As you can see, you can specify additional devices with
rd.qubes.hide_pci.
- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJVJypNAAoJENuP0xzK19csZfMH/At4VcFYCVjAkTF03G7bLubl
0EXart42CfSH1RF5bJJVRe89/DoYZWDvrwwlToHx9GV96XYIGzfg6/b86zgqoki8
177acOssWahI/c6UJKtdGvghx5zwsSKiR+5JTIBwBkdBjL46ktTrvp7MBBxush+E
JPO7xN3SuG9jvW4ePFJAw27a12D4WxzwkO7KR74NTm4FGA4aUWb1kqIJm7/zx8jF
OakNENS11TcXlWOKle3DfpkBTc0toIiTnxJy9QWsy+g82laInYLsFvdlguuckiGR
f4jV82sorPs0YkHnos76wwd9Ym/5j8BARRokY7hPwI7vbKIxkGbiHVDKg4MILOk=
=1Uqv
-----END PGP SIGNATURE-----
Radoslaw Szkodzinski
Apr 9, 2015, 9:58:44 PM4/9/15
to qubes...@googlegroups.com
On Fri, Apr 10, 2015 at 2:53 AM, Andrew <kyb...@riseup.net> wrote:
> Radoslaw Szkodzinski:
>> On Thu, Apr 9, 2015 at 8:58 PM, Andrew <kyb...@riseup.net> wrote:
>>> 7v5w7go9ub0o:
>>>>
>>>>
>>>> On 04/08/15 22:48, Andrew wrote:
>>> I just saw that someone finally figured out the compression scheme for
>>> the AMT modules that Igor Skochinsky didn't manage to decode:
>>> http://io.smashthestack.org/me/.
>>>
>>> Hopefully this means we'll see some disassemblies soon. Looks like
>>> arc-gnu-tools are now free again since the Synopsys acquisition:
>>> https://github.com/foss-for-synopsys-dwc-arc-processors.
>>
Disassembling the kernel won't do much. The important parts are in
> Radoslaw Szkodzinski:
>> On Thu, Apr 9, 2015 at 8:58 PM, Andrew <kyb...@riseup.net> wrote:
>>> 7v5w7go9ub0o:
>>>>
>>>>
>>>> On 04/08/15 22:48, Andrew wrote:
>>> I just saw that someone finally figured out the compression scheme for
>>> the AMT modules that Igor Skochinsky didn't manage to decode:
>>> http://io.smashthestack.org/me/.
>>>
>>> Hopefully this means we'll see some disassemblies soon. Looks like
>>> arc-gnu-tools are now free again since the Synopsys acquisition:
>>> https://github.com/foss-for-synopsys-dwc-arc-processors.
>>
complex interaction between EFI, NIC firmware and ME... plus the true
bugs might be in the ROM code.
>>>
>>> Anyway, the page mentions this repo:
>>> https://github.com/zamaudio/intelmetool. Judging from that code, it
>>> looks like I can answer a few of my own questions:
>>>
>>>> 1) Does Qubes' Xen automatically set those VT-d mappings to restrict
>>> the AMT core's DMA?
>>>
>>> No.
>>
>> This might even be impossible, these mappings are set in EFI DMAR.
>
> Disclaimer: I am woefully uninformed about the modern PC boot process,
> which as far as I can tell is basically a nightmare ;).
>
> If they are set by the EFI code, though, then it should be possible to
> have the EFI firmware change them after loading the Management Engine
> code, right? Or even just never set them in the first place.
way break AMT. That has partially happened in the past with broken
DMAR - hypervisor setting mappings such that iGPU to VM mapping
overrode the ME mapping.
In fact, those regions might even be set up as RMRR too nowadays,
which the hypervisor likely won't touch.
To decode these acronyms, read:
https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt
> I realize I may be completely wrong on this, and that this is anyway
> probably more appropriate for the coreboot list.
on an older chipset - probably at the cost of features.
That might be tricky with the newer chipsets...
>>>> 2) How do I check?
>>
>>> You can check that a device has been passed through to a domain with
>>> $ xl pci-list <domain>
>>>
>>> (according to http://wiki.xen.org/wiki/VTd_HowTo).
>>>
>>> I really don't know how to check that the VT-d mapping is *truly* set,
>>> though. Maybe booting with iommu=verbose will tell?
>>
>> Add "debug" option to Xen kernel line.
>
> Thanks. Also iommu=verbose shows '[VT-D]iommu.c: ... unmap bdf = ...'
> and '[VT-d]iommu.c: ... map bdf = ...' messages.
Those are only for the new mappings, you will need messages describing
>>> $ xl pci-list <domain>
>>>
>>> (according to http://wiki.xen.org/wiki/VTd_HowTo).
>>>
>>> I really don't know how to check that the VT-d mapping is *truly* set,
>>> though. Maybe booting with iommu=verbose will tell?
>>
>> Add "debug" option to Xen kernel line.
>
> Thanks. Also iommu=verbose shows '[VT-D]iommu.c: ... unmap bdf = ...'
> and '[VT-d]iommu.c: ... map bdf = ...' messages.
current DMAR and RMRR - this should be dumped by the Linux kernel on
dom0 boot and likewise by Xen in debug mode.
> Marek: is there any reason to build this as a module? And in which
> source can I find the implementation for rd.qubes.hide_all_usb?
Marek mentioned.
I think it was built as a module to simplify binding and unbinding
during initramfs part of the boot. Could be rewritten to use sysfs
instead.
>> Earlier than that is not possible yet, could be turned into a Xen
>> kernel option to preset the mapping and not show the device directly
>> to the dom0. Nobody has written this kind of patch - could be useful
>> for PVH Dom0 with hardware redirected into PVH DomU. Ask xen-devel
>> perhaps?
>
> Hmm, OK. I wonder now, though, what the benefit would be. The two
> things that appear to matter are:
> 1) Preventing communication with the MEI interface, which could be
> accomplished by building xen-pciback into the kernel, or just deleting
> the mei_me.ko module (right?)
> 2) Actually isolating the AMT core from the rest of the system, to the
> extent possible. This might be do-able by modifying the EFI BIOS. At
> the least we can limit the MEI controller's access.
See, some ME operations are done in a mode that is even higher in
security model than SMM (System Management Mode).
>>>> 4) Is this the AMT core's bus presence?
>>>
>>> Still not sure. It's at least the presence for the host interface.
>>
>> This is just the MEI control, used for IPMI signalling (reboot
>> control), toggling antitheft, toggling remote network operation,
>> setting password locally, setting IDE mappings etc.
>
> OK, so we definitely want this assigned to a quarantine domain.
- for reboots.
>> ME can additionally framegrab the iGPU (framebuffer in system memory),
>> provide PS/2 input devices then memory bounce data MMIO-like for
>> network firmware to read.
>
> ...Jesus Christ. Assigning the network card to a separate domain (i.e.
> NetVM) should prevent its access to the NIC at least, right? Right??!
then the NIC directly uses its own DMA engine to read this configured
memory area and turn it into packets.
(This including wifi and cellular modules.)
This is why AMT is not available in some devices and mainboards with
the cheap versions of network chips, without DMA engine.
>> The interface used to listen for the network
>> interfaces is not known to me. Rafał Wojtczuk knows much more than
>> that - and as part of ITL has managed to exploit the network interface
>> firmware to do generic DMA using this back in 2009.
>
> So the slides he made appear contradictory: at the same time, he says an
> AMT rootkit can change the DMAR tables itself and therefore bypass VT-d,
> but then says that assigning the ME PCI device to a driver domain
> prevents access to Dom0 memory.
mappings that have been set by the firmware - and remove all others
thanks to remapping.
This doesn't prevent bugs in ME code or network firmware from
installing a rootkit in this firmware, if that is possible; especially
one surviving only a warm reboot, which is enough to get full memory
access.
> How are both possible?
The confusion is caused due to talking about operations at different
times. Changing early DMAR is only possible around reboot, while
access to dom0 memory is possible right after running the exploit, if
mappings are not reconfigured. (such as by forwarding the device)
> And really:
> what is the 'special interface' to the NIC? Do I need to remove my PCIe
> NIC and use only a generic USB WiFi card?
with any other kind... well, maybe if said NIC supports DMA engine,
the firmware could configure it properly.
Alternatively, if an EFI driver is loaded it might work in some way
through an EFI service - never heard of such a way though, but it's
not impossible.
>> How it is setup has not yet been reverse engineered, but it is in the
>> special firmware mode which is disabled after startup.
>>
>> The framegrabbing and related networking parts actually use IOMMU - it
>> is pre-setup in DMAR by the EFI code. When IOMMU is enabled and AMT is
>> also enabled, the firmware uses a different DMAR table. There were
>> bugs related to that - garbage in the alternate tables.
>
> I really don't know what this means :). Can you explain a bit more?
Also this might help a bit:
http://firmware.intel.com/sites/default/files/resources/A_Tour_Beyond_BIOS_Using_Intel_VT-d_for_DMA_Protection.pdf
>>>> 5) If so, is it true that the AMT core can be isolated from the rest
>>> of the system by restricting this device?
>>>
>>> Still have no idea!
>>
>> You could try, but I think there is not really a way to countermand a
>> base DMAR mapping set by firmware. Trying it will probably just fail
>> with an error.
>
> But is it possible to change the firmware itself and prevent the AMT
> core from having access to any memory/MMIO devices other than its own range?
The situation with those is not yet well researched.
Adding a mapping for all unmapped memory (such as by moving the MEI
controller into a VM) will (partly?) disable the ability to read/write
dom0 memory (replacing it with chosen domU), but not the network
device, fake PS/2 devices or iGPU, which already have a mapping.
If you can modify DMAR setup code in firmware, you might be able to
disable those mappings. If they are not protected, this can also be
done by the hypervisor.
Heck, it might be enough to toggle AMT in the ME to not have them.
That said, ME <-> NIC interface is always on, as there is a way to
remotely setup and enable ME.
(Not sure if it is still possible with latest ME code versions, > 7.x.)
--
Radosław Szkodziński
Reply all
Reply to author
Forward
0 new messages