USB controller after resume

166 views
Skip to first unread message

Vít Šesták

unread,
May 9, 2015, 4:13:21 AM5/9/15
to qubes...@googlegroups.com
Hello,
I've some issue with an USB 3.0 controller “00:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04)”. I've attached it to one VM. But when I suspend and resume the laptop, the USB ports of this controller are no longer powered.

The lspci command in the VM shows the controller to be present.

The lsusb command in the VM freezes.

I have to reboot the VM in order to use the controller again.

Is there any fix/workaround?

Regards,
Vít Šesták 'v6ak'

Marek Marczykowski-Górecki

unread,
May 9, 2015, 6:05:11 AM5/9/15
to Vít Šesták, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Have you tried reloading its driver (xhci_hcd)? If that helps, add the
module to /rw/config/suspend-module-blacklist (in USB VM).
I have one system, where suspend does not work at all when this module
is loaded.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVTdvNAAoJENuP0xzK19csb+EH/AjrsfK1Ye4uf44xfrl+PVdi
rqkPXcHJBzyRcMwR+aD/EbLpUDRf81cA/Xh5RtQHYLXNV2YDDo9i5lmzdAqi8nIk
2OuyUlqw1im+MJTIx9Oj6jktpd8vf5ST7hBBSbQBbgX6X5uPzUrPxjfAgIMsue1p
PZEFDqDz08RxLBo6L6ilxEhZf7YA9axQjWvmCPdluBSJjcBaFWNK1v29725Z/bx3
ntRkRj35t6OgIFtcREIxQaEMmkss0Ifrma8KXdtlTrH5Z/y8Ed4scUbxaI9A2c2q
x1WJdv3gVu4eQ77N5T3GcekI6mIRd8oak7VBsZQ9L3aUuS2W5dtWAsc9PCfpVfE=
=hFDH
-----END PGP SIGNATURE-----

Vít Šesták

unread,
May 9, 2015, 7:53:09 AM5/9/15
to qubes...@googlegroups.com, groups-no-private-mail--con...@v6ak.com
I'lll try it. But I've to reboot the whole system first, because I have another issue now. The VM is somehow turned off and on at once :( I can't kill it (it is stopped), but I can't start it, because the PCI device it attached to a VM with the same name.

Regards,
Vít Šesták

Vít Šesták

unread,
May 10, 2015, 4:15:58 AM5/10/15
to qubes...@googlegroups.com
Well, I can't rmmod the module on Debian. Maybe I could do so if I had compiled a custom kernel with xhci_hcd as a module.

% sudo rmmod xhci_hcd
rmmod: ERROR: Module xhci_hcd is builtin.

Where does the Debian kernel come from? Since it is paravirtualized, we can't use standard Debian kernel. Is the kernel compiled by the Qubes team?

Regards,
Vít Šesták 'v6ak'

Marek Marczykowski-Górecki

unread,
May 10, 2015, 6:55:34 AM5/10/15
to Vít Šesták, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, May 10, 2015 at 01:15:58AM -0700, Vít Šesták wrote:
> Well, I can't rmmod the module on Debian. Maybe I could do so if I had
> compiled a custom kernel with xhci_hcd as a module.
>
> % sudo rmmod xhci_hcd
> > rmmod: ERROR: Module xhci_hcd is builtin.
> >
>
> Where does the Debian kernel come from? Since it is paravirtualized, we
> can't use standard Debian kernel. Is the kernel compiled by the Qubes team?

Yes, kernel is provided as dom0 package (kernel-qubes-vm).

If you can't unload the module, you can still try to detach it from the
device:
[root@usbvm ~]# lspci
00:00.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #2 (rev 04)
00:01.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #1 (rev 04)
[root@usbvm ~]# cd /sys/bus/pci
[root@usbvm pci]# ls -l devices/0000:00:00.0/driver
lrwxrwxrwx 1 root root 0 May 10 12:53 devices/0000:00:00.0/driver ->
../../../../bus/pci/drivers/ehci-pci
[root@usbvm pci]# echo 0000:00:00.0 > drivers/ehci-pci/unbind
[root@usbvm pci]# ls -l devices/0000:00:00.0/driver
ls: cannot access devices/0000:00:00.0/driver: No such file or directory
[root@usbvm pci]# echo 0000:00:00.0 > drivers/ehci-pci/bind
[root@usbvm pci]# ls -l devices/0000:00:00.0/driver
lrwxrwxrwx 1 root root 0 May 10 12:54 devices/0000:00:00.0/driver ->
../../../../bus/pci/drivers/ehci-pci

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVTzkcAAoJENuP0xzK19csbBsIAIfz7ihSkStaz3mi/KCvfY/Q
kGID+ueq96J/L8tMFge9BAekW9hPIsU5YNhAAzULaOUEVeMKn5i4hlzXLdAfljm9
t6i7798rkmkvDF/fJYwzeaF+wEH9aV0Wr54duslw+ginJmwSa09XFQeOfJ2Mz5g2
qTkACDQzMrpvNHxy1HT70YHgDpSIUC+If1r9+iogDGTQ+Vs3ohI2txUehvWEz8Fc
29E3X1UXUHw9W/Q/AkppacUC+lXCfLiK+rNXMq4h8rwQ+VlmeOdyl5elouCQ2eoR
aRuLVIVZu2lSKUckntBkYcxeY0/l7fwYohbCqE+sKFc3vetVJYpCHEd6Flj6rYk=
=Iwww
-----END PGP SIGNATURE-----

Vít Šesták

unread,
May 13, 2015, 4:45:20 PM5/13/15
to qubes...@googlegroups.com, groups-no-private-mail--con...@v6ak.com
Great, thank you Marek!

It seems that calling the following command before suspend:
% echo 0000:00:00.0  | sudo tee /sys/bus/pci/drivers/xhci_hcd/unbind
and calling the following command after resume:
% echo 0000:00:00.0  | sudo tee /sys/bus/pci/drivers/xhci_hcd/bind 
solves the issue.

When the laptop is suspended before unbind, it seems to be too late to unbind it.

The only question now is how to hook it. I am not sure if I can hook it from the guest VM. I probably can do it from Dom0, since there is /etc/pm/sleep.d, but I'd prefer to do this from DomU.

Regards,
Vít Šesták 'v6ak'

Marek Marczykowski-Górecki

unread,
May 13, 2015, 4:53:28 PM5/13/15
to Vít Šesták, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, May 13, 2015 at 01:45:20PM -0700, Vít Šesták wrote:
> Great, thank you Marek!
>
> It seems that calling the following command before suspend:
> % echo 0000:00:00.0 | sudo tee /sys/bus/pci/drivers/xhci_hcd/unbind
> and calling the following command after resume:
> % echo 0000:00:00.0 | sudo tee /sys/bus/pci/drivers/xhci_hcd/bind
> solves the issue.
>
> When the laptop is suspended before unbind, it seems to be too late to
> unbind it.
>
> The only question now is how to hook it. I am not sure if I can hook it
> from the guest VM. I probably can do it from Dom0, since there is
> /etc/pm/sleep.d, but I'd prefer to do this from DomU.

Qubes calls qubes.SuspendPre service (/etc/qubes-rpc/qubes.SuspendPre)
just before suspend (in all the VMs with any PCI device) and
qubes.SuspendPost after resume. Currently it doesn't support user
scripts, but it is trivial to add something there. Oops, this file isn't
marked as config file so will be overriden on update... I'll fix this.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVU7nAAAoJENuP0xzK19cszIoH/jTZE3k/LP/bozaqdXRF4bks
e+tTcoCpktmV6bn7kAccW5b0JLi7114dFItxhj9IZf9fXrVXHXZBGaNUw4OfE3yb
bHr3lZR08ZKx55ROSvJEdbcfynoHcMSLRJQcUTOduJanjAkmGZj9izE4wOBIXYo4
kd00CVBseO1vUisFNgXf7mJcSWUXijee33hEXaga5/70Gtht81SQK3h3HZDOGikz
klEMhWIzAdLfELM5Yq5cKG6BcUKKYFXeF+Fv4BthUNs5knOiiMuGkLbVTiQeN+y8
d6Ba37x+27LKj4vIbwcd6OVCDGLp+dwOewOS6LFkWq2j5mdr0HjrvmmMy/XeBYc=
=/asU
-----END PGP SIGNATURE-----

Vít Šesták

unread,
Jun 8, 2015, 2:50:42 AM6/8/15
to qubes...@googlegroups.com, groups-no-private-mail--con...@v6ak.com
I've found a way to overcome the limitations of no user scripts supported. I just added following two lines to /rw/config/rc.local of the one VM:

echo 'echo 0000:00:00.0 > /sys/bus/pci/drivers/xhci_hcd/bind' >> /etc/qubes-rpc/qubes.SuspendPost
echo 'echo 0000:00:00.0 > /sys/bus/pci/drivers/xhci_hcd/unbind' >> /etc/qubes-rpc/qubes.SuspendPre

I am not sure if it can be solved in a more generic way, but I am not sure how much generic is it desirable to be. That is, it probably does not make sense to unbind all PCI devices and I am not sure if all USB controllers do have the same issues.

But I have another idea. This seems to work without such hacks on many Linux distributions when running on bare metal. How do they handle such situations?
a) Such problems don't occur on bare metal, so we can't use their solution.
b) Such problems occur on bare metal, but they are solved by some sleep/suspend scripts.

The point is that in case of b), we could just call some suspend/resume scripts from the distribution. I've realized that such scripts are likely not called, as I get no notification from dbus. (As a result, my Pidgin autodisconnect script does not work.) In fact, some other potential problems could be solved this way.

Regards,
Vít Šesták 'v6ak'

Marek Marczykowski-Górecki

unread,
Jul 7, 2015, 7:05:09 PM7/7/15
to Vít Šesták, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sun, Jun 07, 2015 at 11:50:42PM -0700, Vít Šesták wrote:
> I've found a way to overcome the limitations of no user scripts supported.
> I just added following two lines to /rw/config/rc.local of the one VM:
>
> echo 'echo 0000:00:00.0 > /sys/bus/pci/drivers/xhci_hcd/bind' >>
> /etc/qubes-rpc/qubes.SuspendPost
> echo 'echo 0000:00:00.0 > /sys/bus/pci/drivers/xhci_hcd/unbind' >>
> /etc/qubes-rpc/qubes.SuspendPre
>
> I am not sure if it can be solved in a more generic way, but I am not sure
> how much generic is it desirable to be. That is, it probably does not make
> sense to unbind all PCI devices and I am not sure if all USB controllers do
> have the same issues.

BTW You can create /usr/local/etc/qubes-rpc/qubes.Suspend{Pre,Post} to
override default one in /etc/qubes-rpc. Those files are stored on /rw,
so can be customized per-VM (you'll need to create this directory).
Do not forget to call the original scripts from there.

> But I have another idea. This seems to work without such hacks on many
> Linux distributions when running on bare metal. How do they handle such
> situations?
> a) Such problems don't occur on bare metal, so we can't use their solution.
> b) Such problems occur on bare metal, but they are solved by some
> sleep/suspend scripts.
>
> The point is that in case of b), we could just call some suspend/resume
> scripts from the distribution. I've realized that such scripts are likely
> not called, as I get no notification from dbus. (As a result, my Pidgin
> autodisconnect script does not work.) In fact, some other potential
> problems could be solved this way.

I've just tried to convince "systemctl suspend" to work in Qubes VM.
While it is possible to do that (even when /sys/power/state isn't really
useful there), it doesn't do anything. For example NetworkManager does
not get notified about suspend - which is needed to turn down network
interfaces, and the reconfigure them after resume. Calling "pm-suspend"
in the VM also doesn't looks to do anything. Any other idea how to call
standard suspend scripts?

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVnFscAAoJENuP0xzK19cszIsIAJqUoBWcFiDwPOP239if5Nil
gXQlRXng6/No2F0GnwSCVnnjNaxbibsNsshKT9kwSW94OVNi/XPclnts1+qC8xQ0
kVrAPUDVF7K2+hjc4UJotGCV+5RSza0jODyokP8lyt2gX3tCm3D/ZULorXAzIbXo
kIPhCo2KtEjHgH1R4k8QUtJUbZ3MPi76fYJs9hMH0JgRS4Tk1Ds/sWAH6FNyCCaz
Z1omrx+RZtAGpZM/+4ZrCXHGFpvF2DuP6PNEkiUOvd1m2z2Awym/1y1NSNHU7kr2
FE1O0DKL+bxCw4JmbTvggotunyBmTvqdikTPZg+WdNTbUzyFCwR8H1mgHCA73JI=
=4Wbv
-----END PGP SIGNATURE-----

Vít Šesták

unread,
Sep 11, 2016, 2:17:54 PM9/11/16
to qubes-users, groups-no-private-mail--con...@v6ak.com
Qubes 3.2, essentially the same situation. Plus I am concerned also about dom0 USB, because I have external keyboard and touchpad.

Since there are similar issues in dom0, which is more close to bare metal than domUs, I suspect that those are issues would not be mitigated by calling standard suspend/resume scripts.

In dom0, I've created /usr/lib64/pm-utils/sleep.d/53usb file and registered it in /usr/lib/systemd/system/qubes-suspend.service . It seems to work for both USB2 and USB3. Maybe just the *_pci modules are needed to remove when sleeping.

In USBVM, I originally blacklisted on suspend just few modules, but it did not work with USB3. It works after extending the module list (i.e., /rw/config/suspend-module-blacklist):

xhci_pci
xhci_hcd
ehci_pci
ehci_hcd
vhci_hcd
usbip_core

It would be great to have USB working out-of-box. For reasons mentioned above, the blacklisting of modules might be way-to-go, despite I don't like it.

Unfortunately, I am not sure how much are my issues HW-specific. While I could create a pull request for those changes that fix it on my hardware, I am not sure if it does not break something it on other's HW.

Note that the module should be removed before suspend. When I skip this and reload it after resume, it sometimes works, but it also sometimes hangs, prevents sleeping or occasionally causes other issues like hanging with black screen.

Vít Šesták

unread,
Sep 11, 2016, 2:19:16 PM9/11/16
to qubes-users, groups-no-private-mail--con...@v6ak.com
On Sunday, September 11, 2016 at 8:17:54 PM UTC+2, Vít Šesták wrote:
> In dom0, I've created /usr/lib64/pm-utils/sleep.d/53usb file and registered it in /usr/lib/systemd/system/qubes-suspend.service

Oops, forgot to include link to the files: https://gist.github.com/v6ak/1a1f90f9f07f03d8c9c812dd9b817f0e

Marek Marczykowski-Górecki

unread,
Sep 11, 2016, 2:52:36 PM9/11/16
to Vít Šesták, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Sep 11, 2016 at 11:17:54AM -0700, Vít Šesták wrote:
> Qubes 3.2, essentially the same situation. Plus I am concerned also about dom0 USB, because I have external keyboard and touchpad.
>
> Since there are similar issues in dom0, which is more close to bare metal than domUs, I suspect that those are issues would not be mitigated by calling standard suspend/resume scripts.
>
> In dom0, I've created /usr/lib64/pm-utils/sleep.d/53usb file and registered it in /usr/lib/systemd/system/qubes-suspend.service . It seems to work for both USB2 and USB3. Maybe just the *_pci modules are needed to remove when sleeping.
>
> In USBVM, I originally blacklisted on suspend just few modules, but it did not work with USB3. It works after extending the module list (i.e., /rw/config/suspend-module-blacklist):
>
> xhci_pci
> xhci_hcd
> ehci_pci
> ehci_hcd
> vhci_hcd
> usbip_core
>
> It would be great to have USB working out-of-box. For reasons mentioned above, the blacklisting of modules might be way-to-go, despite I don't like it.

In fact those modules (besides USBIP) should be blacklisted by default.
Take a look at /etc/qubes-suspend-module-blacklist. Maybe it doesn't
work for some reason? Or the file wasn't updated (and the new on is in
.rpmnew)?

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJX1afuAAoJENuP0xzK19csgzQH/A3zRI58VJyEyy74HxvVK1Xp
UyRZ383GJPdd4qi/YrQcnU4wv1wLlp8G5O48ulcZpGgHfBRVgZJOMRXHqJfY/HrY
iFDtG7X40ilj0Gfkx3TYAfzbDXydcaAHGSLAK1HsboXTTbIuu5kZ2jtLuUjH7k0f
BlkrwtpkscpPDZbK6JttZ8KxeuE/Qy383KKzY0i/6g+hR3MTXLE9BBPG/TJ+w5Ka
qd8cwFjfIjTl1kgOrTWN27+eLd1ILSYaG8ZmaiNv0i9yTtjjs8wH807lXO1e3qiq
4wEqeJjTqjKjsmneFghaQ2rHGTb78QTGew6Rb7vXU/ZxvzCEerFGiGOxY7DhMQ4=
=PWro
-----END PGP SIGNATURE-----

Vít Šesták

unread,
Sep 11, 2016, 3:41:31 PM9/11/16
to qubes-users, groups-no-private-mail--con...@v6ak.com
On Sunday, September 11, 2016 at 8:52:36 PM UTC+2, Marek Marczykowski-Górecki wrote:
> In fact those modules (besides USBIP) should be blacklisted by default.
> Take a look at /etc/qubes-suspend-module-blacklist. Maybe it doesn't
> work for some reason? Or the file wasn't updated (and the new on is in
> .rpmnew)?

Well, there are just ehci_pci and xhci_pci modules, but not others. This by the way means that either my premise of *_pci modules was wrong (at least for DomU) or the file is somehow ignored.

No there is no *.dpkgnew (or how it is called) file for that. (This is based on debian-8 template, not of Fedora.)

Is there a similar attempt for Dom0 (except the my one)?

I connect keyboard through a different USB controller, which seems to be better than using the input proxy. (No added trust for the USBVM.) So, I think that using USB in dom0 still makes some sense.

Marek Marczykowski-Górecki

unread,
Sep 11, 2016, 3:54:42 PM9/11/16
to Vít Šesták, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Sep 11, 2016 at 12:41:31PM -0700, Vít Šesták wrote:
> On Sunday, September 11, 2016 at 8:52:36 PM UTC+2, Marek Marczykowski-Górecki wrote:
> > In fact those modules (besides USBIP) should be blacklisted by default.
> > Take a look at /etc/qubes-suspend-module-blacklist. Maybe it doesn't
> > work for some reason? Or the file wasn't updated (and the new on is in
> > .rpmnew)?
>
> Well, there are just ehci_pci and xhci_pci modules, but not others. This by the way means that either my premise of *_pci modules was wrong (at least for DomU) or the file is somehow ignored.

AFAIR only *_pci modules are talking to the hardware, so removing them
should be enough. Can you check if your approach still works with only
*_pci?

> No there is no *.dpkgnew (or how it is called) file for that. (This is based on debian-8 template, not of Fedora.)
>
> Is there a similar attempt for Dom0 (except the my one)?

No, that one is only for USB VM.

Take a look at manpage of systemd-sleep.service for dom0 place.

> I connect keyboard through a different USB controller, which seems to be better than using the input proxy. (No added trust for the USBVM.) So, I think that using USB in dom0 still makes some sense.

Using USB VM (a separate one) for keyboard still make some sense. It
will filter what device will have access to dom0. If you leave full USB
controller in dom0, any device plugged there will be handled by dom0.
This will include potentially malicious one plugged in when you're not
looking, or maybe even your keyboard with some "surprise". In case of
USB VM, the same will compromise only USB VM (instead of dom0), and it
will "only" allow to sniff/inject keystrokes. Then you can lower the
risk even further by regularly cleaning your USB VM for keyboard (remove
& create).
In Qubes 4.0 it will be probably possible to make such USB VM a
Disposable VM.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJX1bZ3AAoJENuP0xzK19cs7ksH/26ioT4mw4b0qRYADpy1+bO8
tPnfEVAKPRoVaNNdX4CrYWOEgyLdhfhkDymDBJBjenTgIPn9TUdzdz11DBEueqT5
IzTdaYnLJ8SfDIgm5kg/4Ps6l6cHJNekGMdwWUJNiP5XDfBLkDpMoHUVwcSfLBB4
n3DqFz/rxElGHW9GYw/8M7+eetOOgaX+SiTAVsRmY1Bn1KAvcuMfKASVDLTIn9Ej
KvYk5ty/6m5lW0wYm0dIM9vlWdXA6YhuI9MQnuIfIuvW+1/Io2kt2R9mt3V20/5h
k6vEmRzqrhBD6GsrbS/JhRUl17IigV3NykRfMQ8yNgRRRTIdISFHbAx2vrNBhjE=
=RwSQ
-----END PGP SIGNATURE-----

Vít Šesták

unread,
Sep 11, 2016, 4:12:13 PM9/11/16
to qubes-users, groups-no-private-mail--con...@v6ak.com
On Sunday, September 11, 2016 at 9:54:42 PM UTC+2, Marek Marczykowski-Górecki wrote:
> AFAIR only *_pci modules are talking to the hardware, so removing them
> should be enough. Can you check if your approach still works with only
> *_pci?

Got it, partially. The /usr/lib/qubes/prepare-suspend script ignores the /etc/qubes-suspend-module-blacklist if there is
/rw/config/suspend-module-blacklist.

However, after renaming the /rw/config/suspend-module-blacklist, rebooting USBVM and suspend-resume cycle, the USBVM does not see any USB device. (Currently, both controllers are assigned to USBVM and I use internal laptop keyboard instead.)

> Using USB VM (a separate one) for keyboard still make some sense. It
> will filter what device will have access to dom0. If you leave full USB
> controller in dom0, any device plugged there will be handled by dom0.
> This will include potentially malicious one plugged in when you're not
> looking, or maybe even your keyboard with some "surprise". In case of
> USB VM, the same will compromise only USB VM (instead of dom0), and it
> will "only" allow to sniff/inject keystrokes. Then you can lower the
> risk even further by regularly cleaning your USB VM for keyboard (remove
> & create).

In case of USBVM, it will allow to inject keystrokes, but it would not allow sniffing, would it?

(Anyway, my udev rules disconnects anything that tries to look like a keyboard.)

> In Qubes 4.0 it will be probably possible to make such USB VM a
> Disposable VM.

Cool!

Marek Marczykowski-Górecki

unread,
Sep 11, 2016, 4:15:52 PM9/11/16
to Vít Šesták, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Sep 11, 2016 at 01:12:12PM -0700, Vít Šesták wrote:
> On Sunday, September 11, 2016 at 9:54:42 PM UTC+2, Marek Marczykowski-Górecki wrote:
> > AFAIR only *_pci modules are talking to the hardware, so removing them
> > should be enough. Can you check if your approach still works with only
> > *_pci?
>
> Got it, partially. The /usr/lib/qubes/prepare-suspend script ignores the /etc/qubes-suspend-module-blacklist if there is
> /rw/config/suspend-module-blacklist.
>
> However, after renaming the /rw/config/suspend-module-blacklist, rebooting USBVM and suspend-resume cycle, the USBVM does not see any USB device. (Currently, both controllers are assigned to USBVM and I use internal laptop keyboard instead.)

Check USBVM kernel messages. Maybe driver didn't correctly reset the
device. Most likely full reboot will help.

> > Using USB VM (a separate one) for keyboard still make some sense. It
> > will filter what device will have access to dom0. If you leave full USB
> > controller in dom0, any device plugged there will be handled by dom0.
> > This will include potentially malicious one plugged in when you're not
> > looking, or maybe even your keyboard with some "surprise". In case of
> > USB VM, the same will compromise only USB VM (instead of dom0), and it
> > will "only" allow to sniff/inject keystrokes. Then you can lower the
> > risk even further by regularly cleaning your USB VM for keyboard (remove
> > & create).
>
> In case of USBVM, it will allow to inject keystrokes, but it would not allow sniffing, would it?

If your keyboard is on connected through that USB VM, it will allow
sniffing.

> (Anyway, my udev rules disconnects anything that tries to look like a keyboard.)
>
> > In Qubes 4.0 it will be probably possible to make such USB VM a
> > Disposable VM.
>
> Cool!
>


- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJX1bttAAoJENuP0xzK19cs+eAH/jtqNxuNGcZ900OXthTz2IB6
IOh2veGoUx2VbmlPOYCAoK8L3LCKwQqfNHDD4Ax+nzygIl+cvTW+5wZ9k2yp01HE
EO0XhXrJcJP7CelYxNZ7QZEtsO0//y/8IO9GS6JRNtfcfRaAaeaYkIQc3D4fgLxi
1lV5aFC/NSVVVWysn+t1QmYwQNwukvdHUeKa26S+I9g/hC2/t9jM79DBvRTxMBU9
pceb8PZSPrhijxZhamNvTk52P/3oPkWf5dAwGEVL/Fb1WFCPPxu+fcRG2+U9qAmi
29AMZrE6RppX7PpTnP4QN9OCY0cTFzlvEK0Q+516kR2iM5v/ZGxu9yT88agthrY=
=I0e4
-----END PGP SIGNATURE-----

Vít Šesták

unread,
Sep 11, 2016, 5:00:15 PM9/11/16
to qubes-users, groups-no-private-mail--con...@v6ak.com
On Sunday, September 11, 2016 at 10:15:52 PM UTC+2, Marek Marczykowski-Górecki wrote:
> > However, after renaming the /rw/config/suspend-module-blacklist, rebooting USBVM and suspend-resume cycle, the USBVM does not see any USB device. (Currently, both controllers are assigned to USBVM and I use internal laptop keyboard instead.)
>
> Check USBVM kernel messages. Maybe driver didn't correctly reset the
> device. Most likely full reboot will help.

Yes, USBVM reboot always helps there (unless it fails to start due to fragmented memory).

But this time, it was kind of easier. It seems that the modules just failed to be reinserted on resume for some unknown reason. After manual modprobe, it started working (at least lsusb can see a connected device) for both USB2 and USB3 controllers.

> If your keyboard is on connected through that USB VM, it will allow
> sniffing.

What adversary are you talking about?
* If USBVM is compromised, it can sniff the keystrokes. That's clear.
* If USBVM is OK, but there is a malicious USB device on the same USB controller (hub) as the keyboard, I assume it can sniff just commands sent to keyboards (e.g. NumLock/CapsLock/ScrollLock state, which is usually not much interesting), but not keys. (It can forge keystrokes, though.) Is this correct?

Regards,
Vít Šesták 'v6ak'

Marek Marczykowski-Górecki

unread,
Sep 11, 2016, 5:13:51 PM9/11/16
to Vít Šesták, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Sep 11, 2016 at 02:00:15PM -0700, Vít Šesták wrote:
> On Sunday, September 11, 2016 at 10:15:52 PM UTC+2, Marek Marczykowski-Górecki wrote:
> > > However, after renaming the /rw/config/suspend-module-blacklist, rebooting USBVM and suspend-resume cycle, the USBVM does not see any USB device. (Currently, both controllers are assigned to USBVM and I use internal laptop keyboard instead.)
> >
> > Check USBVM kernel messages. Maybe driver didn't correctly reset the
> > device. Most likely full reboot will help.
>
> Yes, USBVM reboot always helps there (unless it fails to start due to fragmented memory).
>
> But this time, it was kind of easier. It seems that the modules just failed to be reinserted on resume for some unknown reason. After manual modprobe, it started working (at least lsusb can see a connected device) for both USB2 and USB3 controllers.

Interesting. Any errors in journalctl?

> > If your keyboard is on connected through that USB VM, it will allow
> > sniffing.
>
> What adversary are you talking about?
> * If USBVM is compromised, it can sniff the keystrokes. That's clear.
> * If USBVM is OK, but there is a malicious USB device on the same USB controller (hub) as the keyboard, I assume it can sniff just commands sent to keyboards (e.g. NumLock/CapsLock/ScrollLock state, which is usually not much interesting), but not keys. (It can forge keystrokes, though.) Is this correct?

Yes, unless that malicious device successfully compromise your USBVM. And
given the number of available drivers, attack surface is huge.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJX1ckJAAoJENuP0xzK19cs01QH/1GfoopJ43xg+RDGEAHsV62N
6EjS5XzUsZll/ykSEo13kOgaUrF6ek/ZdxofncYlMojwHS3ZW1H+fwI10u5dOLnv
8P1Ax+T7qc6LC4IKmB6Hd4tG5kQ4Jm0gEuOre25kfIi8C3DaKVlN+DJ11MhpkINo
PcGjvZkATEsDHU7+a5UjNDNCcFjAAjNYd2pI524jXt2pxKzNuwoAi3sDUphcrWLb
TR8/7WlOgyYuNojYo8X4D3TRS/YDM68FWsyxt0SjH97fDyYYq649FI/NSJVy3sGP
EKnbfnxwO/Z7CmQgNtu2NcvhwuZZ14AXADdDDDO6LLMAm37f8dl7CtOSfsTWO/4=
=o8qi
-----END PGP SIGNATURE-----

Vít Šesták

unread,
Sep 11, 2016, 5:25:29 PM9/11/16
to qubes-users, groups-no-private-mail--con...@v6ak.com
On Sunday, September 11, 2016 at 11:13:51 PM UTC+2, Marek Marczykowski-Górecki wrote:
> Interesting. Any errors in journalctl?

Nothing interesting found there.

But... Today, I started having a similar issue with NetVM. It does not start NetrowkManager after resume. It looks like the script is just skipped.

Note that while USBVM is Debian 8, NetVM is Fedora 23.

> Yes, unless that malicious device successfully compromise your USBVM. And
> given the number of available drivers, attack surface is huge.

Thanks for clarification.

Hmm, maybe the attack surface could be dramatically reduced by a proper udev configuration. This would imply that even flash drives would be probably handled outside of USBVM, though.

Marek Marczykowski-Górecki

unread,
Sep 11, 2016, 7:21:39 PM9/11/16
to Vít Šesták, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Sep 11, 2016 at 02:25:29PM -0700, Vít Šesták wrote:
> On Sunday, September 11, 2016 at 11:13:51 PM UTC+2, Marek Marczykowski-Górecki wrote:
> > Interesting. Any errors in journalctl?
>
> Nothing interesting found there.
>
> But... Today, I started having a similar issue with NetVM. It does not start NetrowkManager after resume. It looks like the script is just skipped.

So, maybe something in dom0 logs? Especially for qubes-suspend.service?

> Note that while USBVM is Debian 8, NetVM is Fedora 23.
>
> > Yes, unless that malicious device successfully compromise your USBVM. And
> > given the number of available drivers, attack surface is huge.
>
> Thanks for clarification.
>
> Hmm, maybe the attack surface could be dramatically reduced by a proper udev configuration.

This is IMO a good idea. I'd start with setting
/sys/bus/usb/devices/usb*/authorized_default to 0. There was recently a
thread here about similar solution, check "epoxy on ram to prevent cold
boot attacks?". And this project:
https://dkopecek.github.io/usbguard/blog/2015/USBGuard-vs-UDev

> This would imply that even flash drives would be probably handled outside of USBVM, though.

We're talking here about having totally separate USB VM just for
keyboard. And a separate one for other devices (flash drives, camera
etc). Ideally we'd like to separate each device (or a class of devices)
to a separate VM, but in USB it isn't possible. We needs some trade-off
at controller level.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJX1eb9AAoJENuP0xzK19cs8awH/1eJIr8gQ6n9DBRDnIEgG5xU
tk33VGSEwspbWby7XkKswdRrN6PrxeBHJoChRnKngpL7YRGJXrN29trMjn1l150j
TKL52ZBh0yyLkcxQgYGj12+ErLe4JZCi4vO5gdLkVqIQ0K3HhTg1nYH95CGvUvr1
0gRgY8Zo93J/G1uQ1pkMf5t3k2dTBofnYzGaoWT/8Sjq4K8dfcidJXyQdoWaxFV3
9hJmDC5xOv6Jq1NZBSiTuwC0+vWSQrtpQT8mxxKnIMQ7zltrTAu7pIe4KwTgZX59
X1T1ceLKg8Erj3cfPAaRtwUChcHN97uovFgUyjtp+MneZ/JJSJ6/u1ZuL+fhuDM=
=kkXY
-----END PGP SIGNATURE-----

Vít Šesták

unread,
Sep 12, 2016, 2:04:45 AM9/12/16
to qubes-users
Thank you for pointing in the right direction. It was my fault in the script for dom0. It failed in some way, causing other scripts not being called. I am sorry for that. Now, I have a quick fix, but I should make it more robust.

On the keyboard: I had an idea about authenticated USB keyboard protocol. This would make inserting keystrokes impossible (either from usbvm or from other USB devices) and can also prevent some other modification attacks to some degree, but it can hardly prevent sniffing from USBVM, at least attacker can recognize timing of the keystrokes. But unfortunately:

* This would be implementable for me, because I can flash custom firmware to Ergodox. (Well, this introduces some other risks, but they are manageable in some way.) it might be implementable for some others (even via a proxy), but it is far from universal solution.
* lack of time for that 😔
Reply all
Reply to author
Forward
0 new messages