support for xen usb passthrough

[abb]

Hi,

Any plans to support USB passthrough http://wiki.xen.org/wiki/Xen_USB_Passthrough? This was partly discussed before https://groups.google.com/forum/?fromgroups=#!searchin/qubes-devel/usb/qubes-devel/jGA2rHlTr7s/OZby6IGUlKQJ, but that thread was about character devices specifically. As suggested in that post, attaching whole USB controller to a VM works, but it is rather inconvenient solution. Quite often I need to have a bunch of USB network adapters connected to different VMs, would be very nice to have proper USB devices passthrough.

Regards,
Alex

[Marek Marczykowski]

It is on our TODO list:
http://wiki.qubes-os.org/trac/ticket/531
But it isn't priority for now.

If you (or someone) are interested in contributing in this area, I can provide
some PoC scripts (as noted in task - only toolstack part require some work).

--
Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab

[Alexandre Bezroutchko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ok. I'll get back to you when I have some free time for this task.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQaYEsAAoJEJHFaQrJxOw3gKMIAJTnq1eDco4relX/DqMUAcyw
7n/1pQs5J2Fafz1qbBqtkWEJtgAs1m2rnYu4olnP4gtK0q8/nFjbB+eCzXbg3rSX
3IGWSleZa2wPMtzJX6t/rxdMi9XZW49iq0W5PTtMxxS2v/fRH/RcMcazW27oWoQb
j1+8yjyXCO1Vy+ckQzXRhw9b7+hh/dDWgFP58gGQYrD6E+KanejNUjY1KP5RNI8S
uKs2SsfCgC3fQ9qQTZApMXeTBCo8NPw69XMRT3VROSPcO6Zyw+bVcApii9e/e9SA
IFK8GdDICB+QgzITM/swd5DFaBfnf3X3hSZJw3x1eRikLhLGakeaRrC8zt+1+18=
=x12J
-----END PGP SIGNATURE-----

[abb]

I think I can give it a shot.

> 1. xen-usbfront and xen-usbback drivers (patch from xen-devel with some minor modification works ok). It is ready on my devel machine.

So you have this covered? Can you publish it? You have also mentioned you have PoC scripts, please include those too.

> 2. Add support for PV USB to libxl (or stay with standalone python scripts).

I would rather stay with standalone scripts for a start.

> 2a. This includes some script in backend VM (in some/most cases not dom0) that bind USB device to usbback driver - /usr/lib/qubes/unbind_pci_device.sh equivalent.

Should not be a problem. http://www.xen.org/files/xensummit_oracle09/PVUSB.pdf page 7 seems to explain how to bind/unbind devices via sysfs.

> 3. Add Qubes tool to manage USB devices (qvm-block equivalent).

I had a look at qvm-block. It looks like qvm-usb (or whatever the name will be) indeed will share a lot of code with it, in particular retrieving and storing pass-through config in xenstore. Should qvm-block (and qubesutils.block_*) be refactored to move generic code out of it?

There probably should be #4 as well: change qubes-manager GUI to support USB attach/detach.

[Marek Marczykowski]

On 03.10.2012 01:43, abb wrote:
> I think I can give it a shot.

:)

>> 1. xen-usbfront and xen-usbback drivers (patch from xen-devel with some
> minor modification works ok). It is ready on my devel machine.
>
> So you have this covered? Can you publish it?

It is included in my devel-3.4 kernel branch. Modules are named respectively
xen-usbback and xen-usbfront.

> You have also mentioned you
> have PoC scripts, please include those too.

Attached.

>> 2. Add support for PV USB to libxl (or stay with standalone python
> scripts).
>
> I would rather stay with standalone scripts for a start.
>
>> 2a. This includes some script in backend VM (in some/most cases not dom0)
> that bind USB device to usbback driver -
> /usr/lib/qubes/unbind_pci_device.sh equivalent.
>
> Should not be a problem.
> http://www.xen.org/files/xensummit_oracle09/PVUSB.pdf page 7 seems to
> explain how to bind/unbind devices via sysfs.

This is easy when backend is in dom0 (in which you call qvm-usb), but it's
getting more complicated in case of non-dom0 backend (eg UsbVM). I think it
can be done by some udev script or sth...

>> 3. Add Qubes tool to manage USB devices (qvm-block equivalent).
>
> I had a look at qvm-block. It looks like qvm-usb (or whatever the name will
> be) indeed will share a lot of code with it, in particular retrieving and
> storing pass-through config in xenstore. Should qvm-block (and
> qubesutils.block_*) be refactored to move generic code out of it?

This can be good idea. Some hints:
1. Block devices list from each VM is advertised by "qubes-block-devices"
xenstore dir, which is filled by udev script
(/etc/udev/rules.d/99-qubes_block.rules). Similar mechanism can be used for
USB devices.
2. You can start with with backend in dom0, but please, do not hardcode it
anywhere - it is much more secure to use UsbVM (which have USB controllers
attached as PCI devices).
3. qubesutils.block_* can be renamed to qubesutils.device_* with device type
as first parameter ("block" or "usb"). I think qvm-block tool should still
exists, but can be implemented as more generic qvm-device which choose device
type depending on argv[0] (qvm-block or qvm-usb) + symlinks.

> There probably should be #4 as well: change qubes-manager GUI to support
> USB attach/detach.

Indeed, but first of all there should be something which can be connected to
those actions :)

Let me know if you need anything else to know.

[Marek Marczykowski]

On 03.10.2012 09:49, Alexandre Bezroutchko wrote:
> Thanks, I'll have a look.
>
> Yes, dedicated domain for USB would be nice. The only problem is USB
> keyboard and mouse -- they will end up disconnected from dom0, as described
> in "usb security challenges" blog post of Joanna. I wonder if there is
> still a sane way to share them back with dom0.

This is problem only with USB keyboard and/or mouse (which isn't the case on
most laptops). In most cases they can be connected to USB controller which
stay in dom0.

[Joanna Rutkowska]

On 10/03/12 12:56, Marek Marczykowski wrote:
> On 03.10.2012 09:49, Alexandre Bezroutchko wrote:
>> > Thanks, I'll have a look.
>> >
>> > Yes, dedicated domain for USB would be nice. The only problem is USB
>> > keyboard and mouse -- they will end up disconnected from dom0, as described
>> > in "usb security challenges" blog post of Joanna. I wonder if there is
>> > still a sane way to share them back with dom0.
> This is problem only with USB keyboard and/or mouse (which isn't the case on
> most laptops). In most cases they can be connected to USB controller which
> stay in dom0.

... but in case you were unlucky and got a laptop that has
keyboard/touchpad implemented as USB devices (which is the case for
Apple laptops), then there is no "sane way to share them back with
dom0", i.e. there is no way to prevent the UsbVM from "owning" the
keyboard/mouse if it controls the underlying USB controller. The only
way, in that (unlikely) case, as Marek mentioned, would be to just leave
this particular USB controller to which keyboard and touchpad are
connected in Dom0, as it makes no sense to move it to any other domain.

joanna.

[Outback Dingo]

Ive got a USB video camera, would this possibly allow it to be used
with skype? without sharing the whole usb hub?

[Joanna Rutkowska]

Yeah, if you got a UsbVM with a working pvusb backend, then you could
just assign one of the USB devices, such as your camera, to some other
AppVM without moving the whole USB controller. Which is why the support
for pvusb is so desired...

joanna.

[Outback Dingo]

On Wed, Oct 3, 2012 at 12:40 PM, Joanna Rutkowska

Okay im a bit stumped, a USBVM ?? what create a VM, then add all USB
devices to it? Im confused.

>

[Joanna Rutkowska]

I'm a bit confused with your question and what specifically confuses
you? :) Can you rephrase?

joanna.

[Outback Dingo]

On Wed, Oct 3, 2012 at 7:54 PM, Joanna Rutkowska

You mentioned if i had a UsbVM with a working pvusb backend ?
I guess the right response is, no i dont, how do i create one? is there a doc?

> joanna.
>

[Marek Marczykowski]

UsbVM is VM with USB controllers assigned to it:
http://wiki.qubes-os.org/trac/wiki/AssigningDevices

The missing part is setting up pvusb backend there.

[Radoslaw Szkodzinski]

On Wed, Oct 3, 2012 at 12:56 PM, Marek Marczykowski
<marm...@invisiblethingslab.com> wrote:
> On 03.10.2012 09:49, Alexandre Bezroutchko wrote:
>> Thanks, I'll have a look.
>>
>> Yes, dedicated domain for USB would be nice. The only problem is USB
>> keyboard and mouse -- they will end up disconnected from dom0, as described
>> in "usb security challenges" blog post of Joanna. I wonder if there is
>> still a sane way to share them back with dom0.
>
> This is problem only with USB keyboard and/or mouse (which isn't the case on
> most laptops).

Unfortunately is the case on most desktops, such as mine. Also certain
laptops, as Joanna mentioned.
It also complicates my attempts at making a SoundVM. (I need to pass
an USB sound device or two.)

> In most cases they can be connected to USB controller which
> stay in dom0.

If you have more than one controller, maybe. Otherwise not.

Related fact: boot of Qubes takes pretty long, because it takes 10+
seconds to enable the USB keyboard, so I can enter the disk
password...

--
Radosław Szkodziński

[7v5w7go9ub0o]

On 10/03/12 12:10, Joanna Rutkowska wrote:

>
> ... but in case you were unlucky and got a laptop that has
> keyboard/touchpad implemented as USB devices (which is the case for
> Apple laptops), then there is no "sane way to share them back with
> dom0", i.e. there is no way to prevent the UsbVM from "owning" the
> keyboard/mouse if it controls the underlying USB controller. The
> only way, in that (unlikely) case, as Marek mentioned, would be to
> just leave this particular USB controller to which keyboard and
> touchpad are connected in Dom0, as it makes no sense to move it to
> any other domain.
>

Is this (keyboard/touchpad implemented as USB devices) something that
could/should be noted for units on the hardware page!? (please)

[Outback Dingo]

On Tue, Oct 2, 2012 at 8:51 PM, Marek Marczykowski
<marm...@invisiblethingslab.com> wrote:
> On 03.10.2012 01:43, abb wrote:
>> I think I can give it a shot.
>
> :)
>
>>> 1. xen-usbfront and xen-usbback drivers (patch from xen-devel with some
>> minor modification works ok). It is ready on my devel machine.
>>
>> So you have this covered? Can you publish it?
>
> It is included in my devel-3.4 kernel branch. Modules are named respectively
> xen-usbback and xen-usbfront.
>

Where is this devel branch and is it installable via yum in domO ? or
do i need to attempt the pain of getting a build environment

[Outback Dingo]

On Fri, Oct 5, 2012 at 11:43 PM, Outback Dingo <outbac...@gmail.com> wrote:
> On Tue, Oct 2, 2012 at 8:51 PM, Marek Marczykowski
> <marm...@invisiblethingslab.com> wrote:
>> On 03.10.2012 01:43, abb wrote:
>>> I think I can give it a shot.
>>
>> :)
>>
>>>> 1. xen-usbfront and xen-usbback drivers (patch from xen-devel with some
>>> minor modification works ok). It is ready on my devel machine.
>>>
>>> So you have this covered? Can you publish it?
>>
>> It is included in my devel-3.4 kernel branch. Modules are named respectively
>> xen-usbback and xen-usbfront.
>>
>
> Where is this devel branch and is it installable via yum in domO ? or
> do i need to attempt the pain of getting a build environment

Okay I got the kernel mentioned installed and when i rebooted the
netvm and firewall vm failed to start,
I started them manually, however, i failed to get networking in
personal vm. I terminaled into both
firewallvm and netvm, in netvm i could ping the outside, in firewallvm
there was some error message
about systemd qubes-firewall failed to start . i couldnt figure out
how to copy/paste it. not that i had
connectivity to email it but. Any ideas?

[Outback Dingo]

On Sat, Oct 6, 2012 at 12:23 AM, Outback Dingo <outbac...@gmail.com> wrote:
> On Fri, Oct 5, 2012 at 11:43 PM, Outback Dingo <outbac...@gmail.com> wrote:
>> On Tue, Oct 2, 2012 at 8:51 PM, Marek Marczykowski
>> <marm...@invisiblethingslab.com> wrote:
>>> On 03.10.2012 01:43, abb wrote:
>>>> I think I can give it a shot.
>>>
>>> :)
>>>
>>>>> 1. xen-usbfront and xen-usbback drivers (patch from xen-devel with some
>>>> minor modification works ok). It is ready on my devel machine.
>>>>
>>>> So you have this covered? Can you publish it?
>>>
>>> It is included in my devel-3.4 kernel branch. Modules are named respectively
>>> xen-usbback and xen-usbfront.
>>>
>>
>> Where is this devel branch and is it installable via yum in domO ? or
>> do i need to attempt the pain of getting a build environment
>
> Okay I got the kernel mentioned installed and when i rebooted the
> netvm and firewall vm failed to start,
> I started them manually, however, i failed to get networking in
> personal vm. I terminaled into both
> firewallvm and netvm, in netvm i could ping the outside, in firewallvm
> there was some error message
> about systemd qubes-firewall failed to start . i couldnt figure out
> how to copy/paste it. not that i had
> connectivity to email it but. Any ideas?

Okay, weird, i rebooted, and they came up. honestly, reading this readme
I dont get it......

xl list shows

personal id 3

I have a usb web cam in dom0 lsusb shows as
Bus 001 Device 003: ID 1bcf:2881 sunplus innovation

I just want to connect it to personal only for skype

[Marek Marczykowski]

On 06.10.2012 06:47, Outback Dingo wrote:
> On Sat, Oct 6, 2012 at 12:23 AM, Outback Dingo <outbac...@gmail.com> wrote:
>> On Fri, Oct 5, 2012 at 11:43 PM, Outback Dingo <outbac...@gmail.com> wrote:
>>> On Tue, Oct 2, 2012 at 8:51 PM, Marek Marczykowski
>>> <marm...@invisiblethingslab.com> wrote:
>>>> On 03.10.2012 01:43, abb wrote:
>>>>> I think I can give it a shot.
>>>>
>>>> :)
>>>>
>>>>>> 1. xen-usbfront and xen-usbback drivers (patch from xen-devel with some
>>>>> minor modification works ok). It is ready on my devel machine.
>>>>>
>>>>> So you have this covered? Can you publish it?
>>>>
>>>> It is included in my devel-3.4 kernel branch. Modules are named respectively
>>>> xen-usbback and xen-usbfront.
>>>>
>>>
>>> Where is this devel branch and is it installable via yum in domO ? or
>>> do i need to attempt the pain of getting a build environment
>>
>> Okay I got the kernel mentioned installed and when i rebooted the
>> netvm and firewall vm failed to start,
>> I started them manually, however, i failed to get networking in
>> personal vm. I terminaled into both
>> firewallvm and netvm, in netvm i could ping the outside, in firewallvm
>> there was some error message
>> about systemd qubes-firewall failed to start . i couldnt figure out
>> how to copy/paste it. not that i had
>> connectivity to email it but. Any ideas?

There should be some info in /var/log/messages (in firewallvm). Unfortunately
this file doesn't survive VM reboot...
In any case you can restart qubes-firewall service (service qubes-firewall
restart).

> Okay, weird, i rebooted, and they came up. honestly, reading this readme
> I dont get it......

>
> xl list shows
>
> personal id 3
>
> I have a usb web cam in dom0 lsusb shows as
> Bus 001 Device 003: ID 1bcf:2881 sunplus innovation
>
> I just want to connect it to personal only for skype

If you don't have any other USB device, which should stay in dom0 (connected
to the same USB bus), you can attach full USB controller to the VM:
1. Find out which PCI device it is: "ls -l /sys/bus/usb/devices/usb1" -
symlink target contains PCI BDF
2. Connect this device to your personal VM, as described in
http://wiki.qubes-os.org/trac/wiki/AssigningDevices

The above can be improved by PV USB, when it will be ready... But connecting
full USB controller can be more reliable (much more tested than PV USB).

[Outback Dingo]

On Sat, Oct 6, 2012 at 11:33 AM, Marek Marczykowski

I installed the PV USB kernel 3.4 and booted. What i dont get is the
initpv.py, usb-attach,py

i read the readme which is what i didnt understand quite well and only
need to attach 1-1.4 : ID 1bcf:2881 Integrated USB WebCam to dom ID 4
personal

[Marek Marczykowski]

They are in the same archive as the readme file...

[abb]

I can reproduce results described in Marek's readme.

There, you use 'xl list' to find ID of domU you want to attach the device to.
Then you make sure the modules are loaded in dom0 and domU.
Then you run initpv.py, which to my understanding initializes USB controller in domU and makes necessary arrangement for future attach/detach to work.
Then you run usb-list.py in dom0 (not lsusb) to learn the ID of your device.
Finally, you run usb-attach.py to actually attach the device.

The rest (qvm-run) you don't have to do, can as well do lsusb in the terminal of your domU.

So for me all this works ok. But if I try to go any further than what is described in readme, things are not that smooth anymore. If I attach USB stick, it turns out to be read-only and even then fsck hangs on it. Same goes for wireless adapter -- ifconfig up hangs.

[Outback Dingo]

On Sat, Oct 6, 2012 at 12:13 PM, Marek Marczykowski

Yes, I get that I see them, i read the readme.

Im doing

xl list - get personal dom ID 3

in dom0 i do modprobe xen-usbback
in personal i do modprobe xen-usbfront

dmesg in personal now shows a vusb-0 driver

i then in dom0 run ./initpv.py 0 3 0

dmesg in personal now shows a usb hub

i then run ./usb-attach.py 2-1.3 3 2
and

qvm-run personal --pass_io lsusb

it says

qvm-run: error: no such option: --pass_io

[abb]

--pass-io

[Outback Dingo]

On Sat, Oct 6, 2012 at 12:57 PM, abb <a...@gremwell.com> wrote:
> --pass-io
>

:P the doc says --pass_io .... anyway seems personalvm when i pass and
attach the external usb cam, it cant enumerate the device

and when i attach the internel usb web cam, keyboard and mouse and all
just locks up ......... so it seems this pvusb isnt a viable solution
either.

looks like im gunna be stuck on plain old kubuntu

> --
>
>

[Franz]

On Sat, Oct 6, 2012 at 2:15 PM, Outback Dingo <outbac...@gmail.com> wrote:

On Sat, Oct 6, 2012 at 12:57 PM, abb <a...@gremwell.com> wrote:
> --pass-io
>

:P the doc says --pass_io .... anyway seems personalvm when i pass and
attach the external usb cam, it cant enumerate the device

and when i attach the internel usb web cam, keyboard and mouse and all
just locks up ......... so it seems this pvusb isnt a viable solution
either.

looks like im gunna be stuck on plain old kubuntu

Perhaps, if you have an express card slot, it may be possible to buy an express card USB 3.0 card so then it may be possible to attach the whole new PCI device. I found a USB 3.0 (compatible with USB 2) that has 2 usb ports, so I may attach a webcam to one  usb port and a portable usb audio device (with audio in and out) to the other USB port.  Skype should work. Well I'm planning to do just that. Any problem with this arrangement?

Best
Franz

[Outback Dingo]

Yeah, If i pass the whole pci device, once i figured out the permissions
were wrong on the usb video0 device, skype sees the internal web cam fine.
The problem lies in skype requires a working sound system to use video.
And video is all I use when travelling, my wife and i use video only, not voice
chat in skype, we type messages. So skype requires a working sound system
and yeah if you strike gold let me know. Its feasible for me to do
similar. I was
testing the pv usb for my internal web cam, but as soon as i usb attach it, the
whole system freezes up. so pci pass thru will get video up, now just need a way
to resolve sound. Though amarok plays mp3s fine in the vm, skype doesnt like it.


> Best
> Franz
>
> --
>
>

[abb]

Hi Marek,

I have several questions.

I had a look at your PVUSB scripts. In usb-list.py you use xen vusb module to enumerate available USB devices. qvm-block works quite differently -- it relies on custom udev rules to populate 'qubes-block-devices/' directory in xenstore. As far as I can tell, vusb relies on sysfs, so it is limited to dom0. I suppose we still need 'qubes-usb-devices/' directory in xenstore if USB drivers are to run in a dedicated usbvm. I've made some rules and scripts to populate xenstore, see attachment.

There is a separate initpv.py script. I understand it configures both usbvm and target vm to interconnect usbfront and usbback. Is there a need for a standalone script? I suppose this operation (including allocation of devid) should happen automatically on usb-attach.

Do I understand correctly that qvm-usb tools is supposed to run in dom0? I mean even if USB drivers are in a separate domain.

What do I do with the scripts I put together -- just send to the maillist in a tarball?

Regards,
Alex

[Joanna Rutkowska]

On 10/07/12 14:44, abb wrote:
> Hi Marek,
>
> I have several questions.
>

/.../

> What do I do with the scripts I put together -- just send to the maillist
> in a tarball?

Ideally you can setup a git repo somewhere (e.g. at github), pull one of
our repos (the one where you apply changes), commit your changes, SIGN
THEM (git tag -s), and then send us a note about this with a link to
your repo (plus, if it's the first time, also send us your public key
you used for signing of your commits). We will then do git pull, review
and if all is ok, we will push it to our repos.

For people who make more contributions I would be happy to setup a git
account on our git.qubes-os.org server.

Thanks,
joanna.

[Marek Marczykowski]

On 07.10.2012 14:44, abb wrote:
> Hi Marek,
>
> I have several questions.
>
> I had a look at your PVUSB scripts. In usb-list.py you use xen vusb module
> to enumerate available USB devices. qvm-block works quite differently -- it
> relies on custom udev rules to populate 'qubes-block-devices/' directory in
> xenstore. As far as I can tell, vusb relies on sysfs, so it is limited to
> dom0. I suppose we still need 'qubes-usb-devices/' directory in xenstore if
> USB drivers are to run in a dedicated usbvm. I've made some rules and
> scripts to populate xenstore, see attachment.

Great! This is (almost) exactly what is needed here.
I think that subdirectory name should be after device connection (bus + device
number), not vendor+product numbers. First of all the bus+devid is needed to
attach the device to other VM (not the vendor+product). But also
vendor+product isn't guaranteed to be unique (what if I connect two identical
USB sticks?).

vendor:product can be placed as some xenstore entry, besides "desc".

> There is a separate initpv.py script. I understand it configures both usbvm
> and target vm to interconnect usbfront and usbback. Is there a need for a
> standalone script? I suppose this operation (including allocation of devid)
> should happen automatically on usb-attach.

Initpv should be called before attaching any USB device. For block devices it
is done by libxl at VM startup, but every VM uses block devices (at least its
system disk), which isn't the case for USB devices.

So usb-attach can first check if usbpv was initialised (by presence of
xenstore entries created by initpv) and if no - initialize it first.
Devid allocation also should be part of usb-attach. Something similar to
block_find_unused_frontend in qubesutils.py (actually somehow simpler).

> Do I understand correctly that qvm-usb tools is supposed to run in dom0? I
> mean even if USB drivers are in a separate domain.

Yes.

> What do I do with the scripts I put together -- just send to the maillist
> in a tarball?

As Joanna said - for bigger patches it will be better if you setup git
repository, clone our "core" repo and apply the changes there. If you don't
want to setup account anywhere on the web, you can also send patches using
standard way (git format-patch).
Are you familiar with git?

[abb]

I have github account, will try to use it for patches.

[abb]

Here is the procedure I've been following to clone Qubes core repo to Github. I'm posting it here in case someone else will find it useful.

=== setup ===

## configure your local git
# use your own name and email
git config --global user.name 'Alexandre Bezroutchko'
git config --global user.email 'a...@example.com'

## fetch code from Qubes repo
git clone git://git.qubes-os.org/mainstream/core.git qubes-core
cd qubes-core

## push the code to Github
# this assumes 'qubes-core' repo already exists on Github
# change 'grwl' to your github login
git remote add github g...@github.com:grwl/qubes-core.git
git push -u github master

## create a new branch for your development
# change pvusb to your own branch name
git checkout -b pvusb

=== working ===

# do-some-changes ...
# git add ...
# git commit ...

=== pushing ===

# this assumes GPG is configured
# change 0001 to your tag name
git tag -s 0001

# first time you have to use full form of push command, which sets Github repo as default upstream
git push --tags -u github pvusb
# after then it can be simply
git push ---tags

===

There seems to be no simple way to find URL of the created tag in Github. They only way is to scroll through the list of 500+ existing tags looking for one. For https://github.com/grwl/qubes-core/zipball/0001

Regards,
Alex

[abb]

On Sunday, October 7, 2012 3:06:02 PM UTC+2, Marek Marczykowski wrote:

On 07.10.2012 14:44, abb wrote:
> Hi Marek,
>
> I have several questions.
>
> I had a look at your PVUSB scripts. In usb-list.py you use xen vusb module
> to enumerate available USB devices. qvm-block works quite differently -- it
> relies on custom udev rules to populate 'qubes-block-devices/' directory in
> xenstore. As far as I can tell, vusb relies on sysfs, so it is limited to
> dom0. I suppose we still need 'qubes-usb-devices/' directory in xenstore if
> USB drivers are to run in a dedicated usbvm. I've made some rules and
> scripts to populate xenstore, see attachment.

Great! This is (almost) exactly what is needed here.
I think that subdirectory name should be after device connection (bus + device
number), not vendor+product numbers. First of all the bus+devid is needed to
attach the device to other VM (not the vendor+product). But also
vendor+product isn't guaranteed to be unique (what if I connect two identical
USB sticks?).

vendor:product can be placed as some xenstore entry, besides "desc".

Ok, it looks like this now:

 [abb@dom0 qubes-core]$ xenstore-ls -f qubes-usb-devices
qubes-usb-devices/2-2 = ""
qubes-usb-devices/2-2/desc = "045e:0745 Microsoft_Microsoft\xc2\xae_Nano_Transceiver_v1.0"
qubes-usb-devices/2-1 = ""
qubes-usb-devices/2-1/desc = "046a:0021 046a_0021"
qubes-usb-devices/7-5 = ""
qubes-usb-devices/7-5/desc = "0781:5530 SanDisk_Cruzer_1942531DB09038A6"
qubes-usb-devices/7-1 = ""
qubes-usb-devices/7-1/desc = "07d1:3c0a Ralink_11n_Adapter_1.0"

Is it ok to have vendor:product in the description or should I move it into a separate entry?

[abb@dom0 qubes-core]$ qvm-usb -l
dom0:7-5        0781:5530 SanDisk_Cruzer_1942531DB09038A6
dom0:2-2        045e:0745 Microsoft_Microsoft_Nano_Transceiver_v1.0
dom0:2-1        046a:0021 046a_0021
dom0:7-1        07d1:3c0a Ralink_11n_Adapter_1.0

Getting proper human-readable device names is not that straightforward. Your usb-list.py uses vusb xen python module (which falls back to lsusb in some cases), I will probably have to do the same to get the names right.

[Marek Marczykowski]

I think the better is to have it in separate key - it will be easier to make
some other interfaces to it (automatically translate them using vend/prod
list, automatic device attach etc).
This still can be presented by qvm-usb tool in addition to description.

> [abb@dom0 qubes-core]$ qvm-usb -l
> dom0:7-5 0781:5530 SanDisk_Cruzer_1942531DB09038A6
> dom0:2-2 045e:0745 Microsoft_Microsoft_Nano_Transceiver_v1.0
> dom0:2-1 046a:0021 046a_0021
> dom0:7-1 07d1:3c0a Ralink_11n_Adapter_1.0

Looks promising :)

> Getting proper human-readable device names is not that straightforward.
> Your usb-list.py uses vusb xen python module (which falls back to lsusb in
> some cases), I will probably have to do the same to get the names right.

I think the above is more than enough in initial version.

[abb]

I have a question: what exactly vusb_util.bind_usb_device() in usb-attach.py and  vusb_util.unbind_usb_device() in usb-detach.py are doing? Do they have to be executed in usbvm? These calls seems to require root privileges by the way, so I have to use sudo.

[Marek Marczykowski]

On 10.10.2012 00:00, abb wrote:
> I have a question: what exactly vusb_util.bind_usb_device() in
> usb-attach.py and vusb_util.unbind_usb_device() in usb-detach.py are
> doing?

They unbind device from original driver (eg usb-storage) and bind it to
xen-usbback.

> Do they have to be executed in usbvm? These calls seems to require
> root privileges by the way, so I have to use sudo.

Yes, they need to be called in usbvm as root. You can use vm.run("root:some
command") for this.

[Abel Luck]

Marek Marczykowski:

So.. given the work abb has done here, what's the status of a UsbVM with
pvusb delegating individual usb devices to other vms?

I definitely wouldn't mind building this functionality out and giving it
a test once the patches land.

~abel

[abb]

Abel,

I might have something usable by the end of the week. You are certainly welcome to test it :).

Regards,
Alex

[abb]

On Wednesday, October 10, 2012 12:33:39 AM UTC+2, Marek Marczykowski wrote:

On 10.10.2012 00:00, abb wrote:
> I have a question: what exactly vusb_util.bind_usb_device() in
> usb-attach.py and  vusb_util.unbind_usb_device() in usb-detach.py are
> doing?

They unbind device from original driver (eg usb-storage) and bind it to
xen-usbback.

> Do they have to be executed in usbvm? These calls seems to require
> root privileges by the way, so I have to use sudo.

Yes, they need to be called in usbvm as root. You can use vm.run("root:some
command") for this.

This vusb_util python module is only available in dom0, comes from dom0-cached package (). Now I am copying /usr/lib64/python2.6/site-packages/xen to /usr/lib from dom0 to /64/python2.7/site-packages/xen on usbvm. Is there a better way?

[Marek Marczykowski]

This is part of xen package (actually xen-devel subpackage). You can modify
xen.spec to include it in xen-qubes-vm-essentials. But copying this file to
core-vm package is also a good solution. Just make sure to not conflict with
xen-devel package (place it in different directory).

[abb]

Abel, I have implemented qvm-usb tool, it works for me. The code is on github [1], pvusb branch. There is a signed tag pvusb-20121013. There are proper packages yet, just get the code, copy it into dom0 and run the installer script, see readme [2]. Let me know how it goes.

[1] https://github.com/grwl/qubes-core/
[2] https://github.com/grwl/qubes-core/blob/pvusb/README.pvusb

On Wednesday, October 10, 2012 2:06:10 PM UTC+2, Abel Luck wrote:

[Abel Luck]

abb:

> Abel, I have implemented qvm-usb tool, it works for me. The code is on
> github [1], pvusb branch. There is a signed tag pvusb-20121013. There are
> proper packages yet, just get the code, copy it into dom0 and run the
> installer script, see readme [2]. Let me know how it goes.
>
> [1] https://github.com/grwl/qubes-core/
> [2] https://github.com/grwl/qubes-core/blob/pvusb/README.pvusb

The bottom of that README says it doesn't work so well, is that still
the case?

[Alexandre Bezroutchko]

Sent from my iPhone

On 13 Oct 2012, at 13:33, Abel Luck <ab...@guardianproject.info> wrote:

> abb:
>> Abel, I have implemented qvm-usb tool, it works for me. The code is on
>> github [1], pvusb branch. There is a signed tag pvusb-20121013. There are
>> proper packages yet, just get the code, copy it into dom0 and run the
>> installer script, see readme [2]. Let me know how it goes.
>>
>> [1] https://github.com/grwl/qubes-core/
>> [2] https://github.com/grwl/qubes-core/blob/pvusb/README.pvusb
>
> The bottom of that README says it doesn't work so well, is that still
> the case?

Well, as far as I am concerned qvm-usb tool itself works, there are either issues with xen pvusb subsystem or with my local setup. If Marek' scripts work for you, so will qvm-usb. Have you tried Marek's scripts?

[Abel Luck]

abb:

> Abel, I have implemented qvm-usb tool, it works for me. The code is on
> github [1], pvusb branch. There is a signed tag pvusb-20121013. There are
> proper packages yet, just get the code, copy it into dom0 and run the
> installer script, see readme [2]. Let me know how it goes.
>
> [1] https://github.com/grwl/qubes-core/
> [2] https://github.com/grwl/qubes-core/blob/pvusb/README.pvusb

Don't have time to check this out at the moment. Since it requires
letting you run code on my dom0, I'm going to have to give it a good
reading once I have time ;-)

Hopefully I will in the coming week. Or, hopefully some other brave soul
(Marek? *grin*) can review it and help package it up into an RPM.

I was surprised how easy making RPMs is. That should definitely be a
priority here.

~abel

[abb]

Abel, do Marek's scripts work for you?

[Marek Marczykowski]

On 13.10.2012 20:36, abb wrote:
> Abel, do Marek's scripts work for you?

I will test it tomorrow. Basically it looks good ("no exploits detected" ;) ).

[abb]

I've tried PVUSB on original FC17, still no luck (kernel 3.6.1-1 + pvusb patch from [1]). Same issues as with Qubes 1.0 with 3.4 kernel. Both with Marek's tools and from [2]. For the latter to work a bunch of syntax issues needs to be corrected and it also fails to do an equivalent of 'vusb-ctl.py bind' so this needs to be done manually for the device to appear in domU. I have not tried patched xm [3] yet.

I guess before continue spending time on qvm-usb, we need to have a stable pvusb kernel support. There are not that many success stories by the way, [2] is one of them and it is for kernel 2.6.

[1] http://wiki.xen.org/wiki/Xen_USB_Passthrough
[2] http://article.gmane.org/gmane.comp.emulators.xen.devel/70980
[3] http://old-list-archives.xen.org/archives/html/xen-devel/2009-12/msg00478.html

[Marek Marczykowski]

[Please, don't toppost]

On 14.10.2012 17:03, abb wrote:
> On Sunday, October 14, 2012 10:07:06 AM UTC+2, Marek Marczykowski wrote:
>>
>> On 13.10.2012 20:36, abb wrote:
>>> Abel, do Marek's scripts work for you?
>>
>> I will test it tomorrow. Basically it looks good ("no exploits detected"
>> ;) ).
> I've tried PVUSB on original FC17, still no luck (kernel 3.6.1-1 + pvusb
> patch from [1]). Same issues as with Qubes 1.0 with 3.4 kernel. Both with
> Marek's tools and from [2]. For the latter to work a bunch of syntax issues
> needs to be corrected and it also fails to do an equivalent of 'vusb-ctl.py
> bind' so this needs to be done manually for the device to appear in domU. I
> have not tried patched xm [3] yet.
>
> I guess before continue spending time on qvm-usb, we need to have a stable
> pvusb kernel support. There are not that many success stories by the way,
> [2] is one of them and it is for kernel 2.6.
>
> [1] http://wiki.xen.org/wiki/Xen_USB_Passthrough
> [2] http://article.gmane.org/gmane.comp.emulators.xen.devel/70980
> [3]
> http://old-list-archives.xen.org/archives/html/xen-devel/2009-12/msg00478.html
>

I've tried your changes, and qvm-usb tool is great work, but indeed pvusb
kernel driver isn't as stable as I though.

Some notes/observations:
0. I've tested only most simple scenarios: no dedicated UsbVM (usbback in
dom0), only one device at the same time. Kernel 3.4.12 (both dom0 and VM).
Used devices:
- usb stick (USB 2.0) - usb-storage driver
- bluetooth controller (USB 1.1) - btusb driver
- USB->serial converter (USB 1.1) - pl2303 driver
- Canon camera (USB 2.0) - generic usb driver + gphoto2
1. USB 1.1 devices don't work with USB 2.0 hub (the only created by usb_setup)
-> needed two USB hubs: one USB 2.0 and one USB 1.1 (and appropriate device
connection)
2. when changed default usb-ver to "1" (USB 1.1), xen-usbback crashes at
bluetooth controller attach (message 1 below)...
3. Sometimes qvm-usb -a failed to bind device at the first try. It success at
second try. Message 2 below.
4. Usb stick (USB 2.0) mount + read worked, but mkfs hanged, write to device
produced I/O errors...
5. USB->serial converter worked without any problem.
6. Camera (PTP mode) doesn't work - gphoto2 sends some data, but no answer
from device (not sure if it even get initial packet - no change on device
screen, but normally it blanks when communicating with computer)

Regarding the code technically:
1. Are you planning to integrate your files into rpm package (rpm_spec/*.spec)?
2. Regarding disconnecting USB device from VM, when device is physically
disconnected - there is some piece of code in block_remove for this - very
similar can be used for USB (it should clean one port instead of shutting down
whole hub).


So yes, you're right about stability of pvusb driver. When I've tried this
some time ago probably I had just more luck (and tried only usb sticks...).
Anyway most of above problems can be caused by one common bug (eg some ring
buffer overflow not handled properly), so I think it worth sending some
reports about the problems to driver author(s) + cc: xen-devel.



Listings:
1. kernel crash at bluetooth controller attach (backend):
[26852.334654] kernel BUG at
/home/user/qubes-src/kernel/kernel-3.4.12/linux-3.4.12/mm/slab.c:505!
[26852.334696] invalid opcode: 0000 [#1] SMP
[26852.334729] CPU 0
[26852.334743] Modules linked in: usb_storage uas xen_usbback dm_snapshot
binfmt_misc dm_mirror dm_region_hash dm_log xen_netback xen_gntdev xen_evtchn
snd_hda_codec_analog snd_hda_
intel xen_blkback snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm
thinkpad_acpi coretemp tpm_tis tpm nsc_ircc irda tpm_bios crc_ccitt thermal
battery ac btusb iTCO_wdt iTCO_v
endor_support snd_timer snd bluetooth i2c_i801 joydev soundcore rfkill pcspkr
snd_page_alloc mac_hid sg microcode ext4 jbd2 crc16 pcmcia sr_mod sd_mod cdrom
pata_acpi ata_generic at
a_piix ahci libahci yenta_socket pcmcia_rsrc pcmcia_core libata scsi_mod
ehci_hcd uhci_hcd usbcore intel_agp usb_common button video intel_gtt radeon
ttm drm_kms_helper drm agpgart
hwmon i2c_algo_bit i2c_core xen_pciback [last unloaded: scsi_wait_scan]
[26852.335440]
[26852.335454] Pid: 27665, comm: usbback.7.0 Not tainted
3.4.12-2.pvops.qubes.x86_64 #1 LENOVO 2007FVG/2007FVG
[26852.335514] RIP: e030:[<ffffffff811305dc>] [<ffffffff811305dc>]
kfree+0x1bc/0x200
[26852.335561] RSP: e02b:ffff88000f3dfda0 EFLAGS: 00010046
[26852.335591] RAX: ffffea0000088330 RBX: ffff8800026ea040 RCX: 0000000000000080
[26852.335625] RDX: 4000000000000000 RSI: 0000000000000200 RDI: ffff8800026ea040
[26852.335659] RBP: ffff88000f3dfdf0 R08: dead000000200200 R09: dead000000100100
[26852.335692] R10: 0000000019000000 R11: 0000000000000000 R12: ffff88003339c600
[26852.335726] R13: 0000000000000200 R14: ffff88002ee78640 R15: ffff88002ee78640
[26852.335764] FS: 00007f656622e700(0000) GS:ffff8800b5600000(0000)
knlGS:0000000000000000
[26852.335807] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[26852.335834] CR2: 00007fcabd1bd000 CR3: 000000000a4c3000 CR4: 0000000000002660
[26852.335868] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[26852.335902] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[26852.335937] Process usbback.7.0 (pid: 27665, threadinfo ffff88000f3de000,
task ffff88002ee78640)
[26852.335983] Stack:
[26852.335997] ffff880002f46800 0000000000000010 ffff8800026ea040
ffff880002f46ce8
[26852.336050] ffff88002ee78640 ffff88003339c600 ffff88003339c600
ffff880002f46ce8
[26852.336102] ffff88002ee78640 ffff88002ee78640 ffff88000f3dfe10
ffffffffa01e9abb
[26852.336154] Call Trace:
[26852.336184] [<ffffffffa01e9abb>] urb_destroy+0x1b/0x30 [usbcore]
[26852.336222] [<ffffffffa01e9af5>] usb_free_urb+0x25/0x30 [usbcore]
[26852.336259] [<ffffffffa00377a0>] usbbk_free_urbs+0x90/0x140 [xen_usbback]
[26852.338299] [<ffffffffa003909d>] xen_usbif_schedule+0x8d/0x20c [xen_usbback]
[26852.338299] [<ffffffff8106bfa0>] ? wake_up_bit+0x40/0x40
[26852.338299] [<ffffffffa0039010>] ? usbbk_start_submit_urb+0x160/0x160
[xen_usbback]
[26852.338299] [<ffffffff8106ba96>] kthread+0x96/0xa0
[26852.338299] [<ffffffff814703a4>] kernel_thread_helper+0x4/0x10
[26852.338299] [<ffffffff814675b8>] ? retint_restore_args+0x5/0x6
[26852.338299] [<ffffffff814703a0>] ? gs_change+0x13/0x13
[26852.338299] Code: 66 41 83 46 40 01 41 8b 14 24 4b 8d 34 38 4c 89 c7 29 c2
41 89 14 24 89 d2 48 c1 e2 03 e8 4d e5 0f 00 41 8b 04 24 e9 f9 fe ff ff <0f>
0b eb fe 4d 8d 44 24 18 89 c2 4c 89 ff 31 c9 89 45 c8 4c 89
[26852.338299] RIP [<ffffffff811305dc>] kfree+0x1bc/0x200
[26852.338299] RSP <ffff88000f3dfda0>
[26852.338299] ---[ end trace 076d5528b3b04718 ]---

============================
2. Failed first qvm-usb -a call:
[user@dom0 ~]$ qvm-usb -l
dom0:4-1 0a5c:2110 Broadcom_Corp_BCM2045B
dom0:5-5 0457:0150 0457_0150
dom0:4-2 0483:2016 STMicroelectronics_Biometric_Coprocessor
[user@dom0 ~]$ qvm-usb -a test2 dom0:5-5
Traceback (most recent call last):
File "/usr/lib/qubes/vusb-ctl.py", line 18, in <module>
vusb_util.bind_usb_device(device)
File "/usr/lib64/python2.6/site-packages/xen/util/vusb_util.py", line 325,
in bind_usb_device
raise UsbDeviceBindingError("can't bind intf (%s). " % intf)
xen.util.vusb_util.UsbDeviceBindingError: vusb: Failed to bind/unbind USB
device: can't bind intf (5-5:1.0).
[user@dom0 ~]$ qvm-usb -a test2 dom0:5-5
[user@dom0 ~]$

[Alexandre Bezroutchko]

Yes. I suppose I need to create new packages for install-pvusb-backend/frontend. What about dom0 -- do I integrate it into an existing RPM? Which one?

Any suggestions now to implement this?

> 2. Regarding disconnecting USB device from VM, when device is physically
> disconnected - there is some piece of code in block_remove for this - very
> similar can be used for USB (it should clean one port instead of shutting down
> whole hub).

In dom0, after each usbvm reboot:
  usbvm_xid=`xl list | awk '($1=="usbvm"){print $2}'`
  xenstore-write /local/domain/${usbvm_xid}/qubes-usb-devices ''
  xenstore-chmod /local/domain/${usbvm_xid}/qubes-usb-devices n0 b${usbvm_xid}

[Marek Marczykowski]

On 15.10.2012 09:37, Alexandre Bezroutchko wrote:
> On 10/15/2012 06:50 AM, Marek Marczykowski wrote:
>> Regarding the code technically:
>> 1. Are you planning to integrate your files into rpm package
> (rpm_spec/*.spec)?
> Yes. I suppose I need to create new packages for
> install-pvusb-backend/frontend. What about dom0 -- do I integrate it
> into an existing RPM? Which one?
>
> Any suggestions now to implement this?

I think all the files can be integrated into existing packages:
qubes-core-dom0 and qubes-core-vm.

>> 2. Regarding disconnecting USB device from VM, when device is physically
>> disconnected - there is some piece of code in block_remove for this - very
>> similar can be used for USB (it should clean one port instead of
> shutting down
>> whole hub).
>
> In dom0, after each usbvm reboot:
> usbvm_xid=`xl list | awk '($1=="usbvm"){print $2}'`
> xenstore-write /local/domain/${usbvm_xid}/qubes-usb-devices ''
> xenstore-chmod /local/domain/${usbvm_xid}/qubes-usb-devices n0
> b${usbvm_xid}

This can be added the same place where qubes-block-devices entry is created:
http://git.qubes-os.org/gitweb/?p=marmarek/core.git;a=blob;f=dom0/qvm-core/qubes.py;h=69f9e6f7269d51e33201fb4b9f622b7684116477;hb=HEAD#l863

[abb]

On Friday, October 12, 2012 9:35:16 PM UTC+2, Marek Marczykowski wrote:

On 12.10.2012 09:43, abb wrote:
>
>
> On Wednesday, October 10, 2012 12:33:39 AM UTC+2, Marek Marczykowski wrote:
>>
>> On 10.10.2012 00:00, abb wrote:
>>> I have a question: what exactly vusb_util.bind_usb_device() in
>>> usb-attach.py and  vusb_util.unbind_usb_device() in usb-detach.py are
>>> doing?
>>
>> They unbind device from original driver (eg usb-storage) and bind it to
>> xen-usbback.
>>
>>> Do they have to be executed in usbvm? These calls seems to require
>>> root privileges by the way, so I have to use sudo.
>>
>> Yes, they need to be called in usbvm as root. You can use
>> vm.run("root:some
>> command") for this.
>>
>
> This vusb_util python module is only available in dom0, comes from
> dom0-cached package (). Now I am copying
> /usr/lib64/python2.6/site-packages/xen to /usr/lib from dom0 to
> /64/python2.7/site-packages/xen on usbvm. Is there a better way?

This is part of xen package (actually xen-devel subpackage). You can modify
xen.spec to include it in xen-qubes-vm-essentials.

Can't find xen.spec in qubes-core nor in qubes-builder repos. What file you refer to exactly? Do you mean rpm_spec/core-vm.spec from qubes-core repo, like this:

diff --git a/rpm_spec/core-vm.spec b/rpm_spec/core-vm.spec
index f9ba774..7befdc0 100644
--- a/rpm_spec/core-vm.spec
+++ b/rpm_spec/core-vm.spec
@@ -40,6 +40,7 @@ Requires:   ethtool
 Requires:   tinyproxy
 Requires:   ntpdate
 Requires:   qubes-core-vm-kernel-placeholder
+Requires:   xen-devel
 Provides:   qubes-core-vm
 Obsoletes:  qubes-core-commonvm
 Obsoletes:  qubes-core-appvm

But copying this file to
core-vm package is also a good solution. Just make sure to not conflict with
xen-devel package (place it in different directory).

There are a lot of files to copy, I suppose the previous option is better.
 

[Marek Marczykowski]

xen.spec in xen repo.
http://git.qubes-os.org/gitweb/?p=marmarek/xen.git;a=blob;f=xen.spec;h=a202132d613185d6043894cb8f2e9fc983359410;hb=HEAD

[abb]

I've just realized I've been developing qvm-usb based on sources from joanna's repo. Do I finish as is or should I switch to mainstream/marmarek's tree?

How do I tag commit messages for this development? dom0/qvm-usb vm/qvm-usb usbvm/qvm-usb? What if certain commit affects two areas for example udev scripts get deployed to dom0 and usbvm.

[Marek Marczykowski]

On 20.10.2012 09:08, abb wrote:
> I've just realized I've been developing qvm-usb based on sources from
> joanna's repo. Do I finish as is or should I switch to
> mainstream/marmarek's tree?

Shouldn't matter. I don't expect any conflicts in merging this part of code
(new files and changes in qubesutils.py). qubes.py is quite different now, so
if you modify qubes.py, it's better to switch to my repo.

> How do I tag commit messages for this development? dom0/qvm-usb vm/qvm-usb
> usbvm/qvm-usb?

dom0/qvm-usb and vm/qvm-usb is ok.

What if certain commit affects two areas for example udev
> scripts get deployed to dom0 and usbvm.

dom0+vm/qvm-usb.

[Alexandre Bezroutchko]

On 20 Oct 2012, at 12:06, Marek Marczykowski <marm...@invisiblethingslab.com> wrote:

> On 20.10.2012 09:08, abb wrote:
>> I've just realized I've been developing qvm-usb based on sources from
>> joanna's repo. Do I finish as is or should I switch to
>> mainstream/marmarek's tree?
>
> Shouldn't matter. I don't expect any conflicts in merging this part of code
> (new files and changes in qubesutils.py). qubes.py is quite different now, so
> if you modify qubes.py, it's better to switch to my repo.

I think there is a hook to create/chmod usb-specific entries in xen store on domU creation.

[abb]

I'm building RPMs, using marmarek repo. Is it normal packages for fc13 (qubes-core-dom0 in particular) have fc17 in their names?

bash-4.2$ sudo find fc1? all-* -name \*.rpm
fc13/tmp/all-qubes-pkgs/rpm/xen-runtime-4.1.2-15.qubes.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/qubes-core-dom0-2.0.36-1.fc17.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/qubes-core-vm-kernel-placeholder-1.0-1.fc17.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/xen-hypervisor-4.1.2-15.qubes.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/xen-debuginfo-4.1.2-15.qubes.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/xen-4.1.2-15.qubes.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/qubes-core-vm-1.7.46-1.fc17.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/xen-libs-4.1.2-15.qubes.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/qubes-core-vm-libs-1.7.46-1.fc17.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/qubes-core-vm-sysvinit-1.7.46-1.fc17.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/xen-qubes-vm-essentials-4.1.2-15.qubes.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/xen-doc-4.1.2-15.qubes.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/xen-licenses-4.1.2-15.qubes.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/xen-devel-4.1.2-15.qubes.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/qubes-core-vm-systemd-1.7.46-1.fc17.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/xen-hvm-4.1.2gui2.0.9-15.qubes.x86_64.rpm
fc13/tmp/all-qubes-pkgs/rpm/qubes-core-vm-devel-1.7.46-1.fc17.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/xen-runtime-4.1.2-15.qubes.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/qubes-core-dom0-2.0.36-1.fc17.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/qubes-core-vm-kernel-placeholder-1.0-1.fc17.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/xen-hypervisor-4.1.2-15.qubes.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/xen-debuginfo-4.1.2-15.qubes.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/xen-4.1.2-15.qubes.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/qubes-core-vm-1.7.46-1.fc17.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/xen-libs-4.1.2-15.qubes.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/qubes-core-vm-libs-1.7.46-1.fc17.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/qubes-core-vm-sysvinit-1.7.46-1.fc17.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/xen-qubes-vm-essentials-4.1.2-15.qubes.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/xen-doc-4.1.2-15.qubes.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/xen-licenses-4.1.2-15.qubes.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/xen-devel-4.1.2-15.qubes.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/qubes-core-vm-systemd-1.7.46-1.fc17.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/xen-hvm-4.1.2gui2.0.9-15.qubes.x86_64.rpm
fc17/tmp/all-qubes-pkgs/rpm/qubes-core-vm-devel-1.7.46-1.fc17.x86_64.rpm
all-qubes-pkgs/rpm/xen-runtime-4.1.2-15.qubes.x86_64.rpm
all-qubes-pkgs/rpm/qubes-core-dom0-2.0.36-1.fc17.x86_64.rpm
all-qubes-pkgs/rpm/qubes-core-vm-kernel-placeholder-1.0-1.fc17.x86_64.rpm
all-qubes-pkgs/rpm/xen-hypervisor-4.1.2-15.qubes.x86_64.rpm
all-qubes-pkgs/rpm/xen-debuginfo-4.1.2-15.qubes.x86_64.rpm
all-qubes-pkgs/rpm/xen-4.1.2-15.qubes.x86_64.rpm
all-qubes-pkgs/rpm/qubes-core-vm-1.7.46-1.fc17.x86_64.rpm
all-qubes-pkgs/rpm/xen-libs-4.1.2-15.qubes.x86_64.rpm
all-qubes-pkgs/rpm/qubes-core-vm-libs-1.7.46-1.fc17.x86_64.rpm
all-qubes-pkgs/rpm/qubes-core-vm-sysvinit-1.7.46-1.fc17.x86_64.rpm
all-qubes-pkgs/rpm/xen-qubes-vm-essentials-4.1.2-15.qubes.x86_64.rpm
all-qubes-pkgs/rpm/xen-doc-4.1.2-15.qubes.x86_64.rpm
all-qubes-pkgs/rpm/xen-licenses-4.1.2-15.qubes.x86_64.rpm
all-qubes-pkgs/rpm/xen-devel-4.1.2-15.qubes.x86_64.rpm
all-qubes-pkgs/rpm/qubes-core-vm-systemd-1.7.46-1.fc17.x86_64.rpm
all-qubes-pkgs/rpm/xen-hvm-4.1.2gui2.0.9-15.qubes.x86_64.rpm
all-qubes-pkgs/rpm/qubes-core-vm-devel-1.7.46-1.fc17.x86_64.rpm

This was built in the following environment:

------------------------------------------------------------------------------
bash-4.2$ git remote -v
origin    git://git.qubes-os.org/marmarek/qubes-builder.git (fetch)
origin    git://git.qubes-os.org/marmarek/qubes-builder.git (push)

bash-4.2$ git status
# On branch master
nothing to commit (working directory clean)

bash-4.2$ diff -u builder.conf.default builder.conf
--- builder.conf.default    2012-10-20 22:29:03.368000172 +0200
+++ builder.conf    2012-10-20 22:29:28.316000173 +0200
@@ -48,3 +48,8 @@
 # GIT_BASEURL=/home/user/qubes-src
 # GIT_SUBDIR=
 # GIT_SUFFIX=
+
+GIT_SUBDIR="marmarek"
+BRANCH_kernel=devel-3.4
+BRANCH_dom0_updates=devel/xserver-1.12
+NO_SIGN=1

bash-4.2$ cd qubes-src/core/

bash-4.2$ git status
# On branch master
nothing to commit (working directory clean)
bash-4.2$ git remote -v
origin    git://git.qubes-os.org/marmarek/core.git (fetch)
origin    git://git.qubes-os.org/marmarek/core.git (push)

Built with: 'make get-sources xen core'
------------------------------------------------------------------------------

Just wondering, because in my current setup (1.0 release) qubes packages in dom0 are marked fc13.


Couple more question. Once the packages are built, what is the best way to apply them? Copy 'fc13/' packages to dom0 and 'fc17/' to the template vm and install with 'rpm -i'? Or do I need to create custom package repository and use standard

Is it ok to deploy xen and core packages alone, built as above, on top of 1.0 release?

Regards,
Alex

[Marek Marczykowski]

On 21.10.2012 14:18, abb wrote:
> On Saturday, October 20, 2012 12:06:27 PM UTC+2, Marek Marczykowski wrote:
>>
>> On 20.10.2012 09:08, abb wrote:
>>> I've just realized I've been developing qvm-usb based on sources from
>>> joanna's repo. Do I finish as is or should I switch to
>>> mainstream/marmarek's tree?
>>
>> Shouldn't matter. I don't expect any conflicts in merging this part of
>> code
>> (new files and changes in qubesutils.py). qubes.py is quite different now,
>> so
>> if you modify qubes.py, it's better to switch to my repo.
>>
>>> How do I tag commit messages for this development? dom0/qvm-usb
>> vm/qvm-usb
>>> usbvm/qvm-usb?
>>
>> dom0/qvm-usb and vm/qvm-usb is ok.
>>
>> What if certain commit affects two areas for example udev
>>> scripts get deployed to dom0 and usbvm.
>>
>> dom0+vm/qvm-usb.
>>
>>
>> --
>> Best Regards / Pozdrawiam,
>> Marek Marczykowski
>> Invisible Things Lab
>>
>>
> I'm building RPMs, using marmarek repo. Is it normal packages for fc13
> (qubes-core-dom0 in particular) have fc17 in their names?

qubes-core-* are built for both fc13 and fc17. qubes-core-dom0 built for fc17
isn't used, but it doesn't take much time to build, so it is simpler to build
all packages in all environments.

> bash-4.2$ sudo find fc1? all-* -name \*.rpm

Built packages are moved back to qubes-src/*/rpm/ dirs. So fc?? dirs isn't the
right place for searching resulting packages...

(...)

> Couple more question. Once the packages are built, what is the best way to
> apply them? Copy 'fc13/' packages to dom0 and 'fc17/' to the template vm
> and install with 'rpm -i'? Or do I need to create custom package repository
> and use standard

With exception of using "rpm" subdir in qubes-src/core instead of fc13, the
first option is ok. I use something like:
qvm-run -p devel "cd qubes-builder/qubes-src/core/rpm/x86_64; tar c
qubes-core-dom0-*fc13*rpm' | tar xv
rpm -Uhv qubes-core-dom0-*fc13*rpm

The same can be used for template - there instead of qvm-run + tar, you can
use simple qvm-copy-to-vm.

> Is it ok to deploy xen and core packages alone, built as above, on top of
> 1.0 release?

Yes.

[Marek Marczykowski]

Perhaps you need to add --force to install package with the same version
number as already installed one.

[abb]

I have made rpm packages. Repos git://github.com/grwl/qubes-core.git and git://github.com/grwl/qubes-xen.git. Signed tag pvusb-20121022, branch pvusb2 (now based on marmarek's repo).

I've tested it a bit, seems to work, at least in dom0 -> appvm (didn't deploy isolated usbvm this time). Anyway, there is no progress with the troubleshooting of kernel crashes etc, so usb pass-through subsystem remains unusable for all practical purposes. I've manage to reach the author of the module, Konrad, but so far there are no signs of him looking at the problem [1]. There is a bunch of minor issues with qvm-usb (see the end of README.pvusb), but I don't think it worth fixing them until there is some useable kernel support. I will be checking from time to time on pvusb, please let me know if you hear about any advances in this area.

[1] http://lists.xen.org/archives/html/xen-devel/2012-10/msg01645.html

[Marek Marczykowski]

Thanks! I've merged "core" changes into my repo. Also added some minor tweaks
(support subdevices like "1-1.3", one missing file in rpm package).
Regarding xen package, I've done it differently - included vusb_util in
xen-qubes-vm-essentials - to not pull full xen-devel package just for 2 files
(BTW I was wrong about xen-devel package, vusb_util is in main xen package...).

I've tested it somehow with dedicated usb vm and results are at least not
worse than with dom0 backend. USB stick is working in most cases, but other
devices have the same problems as in my previous tests (with dom0 backend).

> I've tested it a bit, seems to work, at least in dom0 -> appvm (didn't
> deploy isolated usbvm this time). Anyway, there is no progress with the
> troubleshooting of kernel crashes etc, so usb pass-through subsystem
> remains unusable for all practical purposes. I've manage to reach the
> author of the module, Konrad, but so far there are no signs of him looking
> at the problem [1]. There is a bunch of minor issues with qvm-usb (see the
> end of README.pvusb), but I don't think it worth fixing them until there is
> some useable kernel support. I will be checking from time to time on pvusb,
> please let me know if you hear about any advances in this area.
>
> [1] http://lists.xen.org/archives/html/xen-devel/2012-10/msg01645.html

IMHO one issue worth fixing now: support for USB 1.1 devices (this will
require setting up second pvusb connection). This looks to be very simple:
needs some info in qubes-usb-devices xenstore entry (like usb-vers with value
"1" or "2"), then additional parameter to usb_setup and
usb_find_unused_frontend (to search for specific usb version).
This will make testing much easier, because USB 1.1 devices are unusable with
USB 2.0 frontend.

[abb]

Oops. I suppose this is good example of why one should use RPMs during development ).
 

Regarding xen package, I've done it differently - included vusb_util in
xen-qubes-vm-essentials - to not pull full xen-devel package just for 2 files
(BTW I was wrong about xen-devel package, vusb_util is in main xen package...).  

I've tested it somehow with dedicated usb vm and results are at least not
worse than with dom0 backend. USB stick is working in most cases, but other
devices have the same problems as in my previous tests (with dom0 backend).

> I've tested it a bit, seems to work, at least in dom0 -> appvm (didn't
> deploy isolated usbvm this time). Anyway, there is no progress with the
> troubleshooting of kernel crashes etc, so usb pass-through subsystem
> remains unusable for all practical purposes. I've manage to reach the
> author of the module, Konrad, but so far there are no signs of him looking
> at the problem [1]. There is a bunch of minor issues with qvm-usb (see the
> end of README.pvusb), but I don't think it worth fixing them until there is
> some useable kernel support. I will be checking from time to time on pvusb,
> please let me know if you hear about any advances in this area.
>
> [1] http://lists.xen.org/archives/html/xen-devel/2012-10/msg01645.html

IMHO one issue worth fixing now: support for USB 1.1 devices (this will
require setting up second pvusb connection). This looks to be very simple:
needs some info in qubes-usb-devices xenstore entry (like usb-vers with value
"1" or "2"), then additional parameter to usb_setup and
usb_find_unused_frontend (to search for specific usb version).
This will make testing much easier, because USB 1.1 devices are unusable with
USB 2.0 frontend.

I've added USB version support to usb_find_unused_frontend() and usb_setup().
USB versions other than 1.0, 1.1, and 2.0 are silently ignored for now.
Code to detect which version required by particular device still to be added, now it is still hardcoded 2.0...

[abb]

Added proper support for USBv1 and fixed a bug in usb_find_unused_frontend(). Tag: pvusb-20121026.
 

[Marek Marczykowski]

Can you explain this [1] line?

if xs.read(xs_trans, "%s/backend-id" % fe_path) ==
str(backend_vm_xid):
(...)
if xs.read(xs_trans, "%s/backend-id" % fe_path) ==
str(backend_vm_xid):
last_frontend_dev = frontend_dev
continue

The second if (...) == str(backend_vm_xid) seems to be always true here, and
"continue" action not intended here (am I missing something?).

Also you should use "/sys/$DEVPATH/version" here [2] instead of open-coded
devices tree location location:
VERSION=`cat /sys/bus/usb/devices/${NAME}/version`

[1]
https://github.com/grwl/qubes-core/blob/pvusb2/dom0/qvm-core/qubesutils.py#L581
[2] https://github.com/grwl/qubes-core/blob/pvusb2/misc/usb_add_change#L25

PS Hmm.. perhaps I should use github code comment functionality?

[abb]

Thanks for the comments. I've fixed one (the last one), will fix remaining later -- it becomes difficult to refactor the code with vi and first attempt to use pycharm IDE has failed, not sure why.
 

[abb]

Konrad (technical contact for XEN PVUSB) says:
"""

I hadn't had a chance to look at it - and the only persons who would have
the expertise to do so are the authors of said patches. They seemed to
have moved on, so I think to answer your questions you would have to
wrap up your sleves and dig in the code yourself.

"""

Sounds like a plan :), I'll give it a try. By the way, Xen wiki [1] mentions USBIP [2] as an alternative to PVUSB.

[1] http://wiki.xen.org/xenwiki/XenUSBPassthrough
[2] http://usbip.sourceforge.net/

[Joanna Rutkowska]

On 11/02/12 19:15, abb wrote:
> Konrad (technical contact for XEN PVUSB) says:
> """
>
> I hadn't had a chance to look at it - and the only persons who would have
> the expertise to do so are the authors of said patches. They seemed to
> have moved on, so I think to answer your questions you would have to
> wrap up your sleves and dig in the code yourself.
>
> """
>
> Sounds like a plan :), I'll give it a try. By the way, Xen wiki [1]
> mentions USBIP [2] as an alternative to PVUSB.

Perhaps we could run the USBIP over qrexec/vchan? But does it really
work better than PVUSB backend?

joanna.

[Marek Marczykowski]

I've tried USBIP some time ago to connect some USB scanner over the network
and even device discovery didn't worked...

[abb]

Well, at least there is someone alive in usbip-devel maillist. I'll try troubleshooting pvusb a bit, perhaps, as Marek suggested earlier it is some stupid bug.
 

[Marek Marczykowski]

How about some fixes in your pvusb code for qubes-core (comments on github)?
This is the only thing that holds me from merging your code into master
branch. Most of qvm-usb code would be useful regardless of PVUSB
implementation (the current kernel modules or modified USBIP or anything else).

[Alexandre Bezroutchko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Done, except [1] (see comment). Unfortunately I can't test these changes
now, will do by Monday the latest.

[1]
https://github.com/grwl/qubes-core/commit/073ce8a6839fdc4541dbbba0bb4ac1fb01c123b4#commitcomment-2062215
[2] https://github.com/grwl/qubes-core/commits/pvusb2

>
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQmazMAAoJENsE8GcJgOzwtEAIAKJw4dovR4f2VlTJVhuWROS+
ZdC/6y6ZAInRpKjT2Mx4YzR1SUi3UiT62kwyqxYRurA99moXHprHSkS0m5kwBgO8
SQCFyKifGTlrTfc+d3tCNlRNY0bp4trbb3r2ZbeOj/36EpRsEKe1JmsvS1uYKXOr
8ZcAzN7Ri+SSSFLgNFzs4OnCCWTuNMZraa/6mhH3KvrTk5DBXUESQOe/UAos2IDU
YZbUOg0+UTZgNqeb9k9FOP3d5OvG3ARoLtVQaoIqVMWTahcZgxMjjcnxoDlmQzuN
lfA7+Y82uoe2GJY6fzVQ9ABNJCSVLdbPFs+8xeXhYRui0G4U2ZPp608wVQs/NSA=
=W3aS
-----END PGP SIGNATURE-----

[Marek Marczykowski]

I will test it, but please tag the sources.

> [1]
> https://github.com/grwl/qubes-core/commit/073ce8a6839fdc4541dbbba0bb4ac1fb01c123b4#commitcomment-2062215
> [2] https://github.com/grwl/qubes-core/commits/pvusb2

[abb]

I've tested it a little (only dom0 backend) and fixed couple more bugs. Both USB2.0 and USB1.1 devices seem to be handled right. Signed tag:  pvusb-20121108a

[Marek Marczykowski]

Thanks, merged. Also fixed one more bug regarding encoding device name.

[Alex Dubois]

Hi All,

I have a desktop with an intel motherboard with TPM and a CPU which support VT-d. This motherboard has an integrated Intel videochip.
I also have a NVidia card.
I would like to install Qubes and have the NVidia card dedicated to a Windows VM for casual games, the other video card would be for all the other VM.

I have 2 questions:

Would this setup be theoretically feasible?

Would it be as secure as having only an integrated video card?
This would save me having to build a dedicated Qubes hardware environment or to have to compromise on security with a dual boot system.

Practically, to set focus to the Windows VM (which would be a full screen attached to a second display) I would select it in the task bar, and to release focus, I would have a key combination to press.

Am I dreaming in aiming for such setup? Would the amount of patching be very substantial and beyond Python/C tweaking? I will not require copy/paste or file transfer, also tgat could be a plus...

Looking forward to your view
Alex

[Marek Marczykowski]

On 09.11.2012 00:32, Alex Dubois wrote:
> Hi All,
>
> I have a desktop with an intel motherboard with TPM and a CPU which support VT-d. This motherboard has an integrated Intel videochip.
> I also have a NVidia card.
> I would like to install Qubes and have the NVidia card dedicated to a Windows VM for casual games, the other video card would be for all the other VM.
>
> I have 2 questions:
>
> Would this setup be theoretically feasible?

For sure this will be difficult, but perhaps doable. There are (at least) two
problems:
1. General GPU passthrough problems. X86 architecture is quite complicated in
this area and GPU passthrough requires some additional operations than normal
PCI/PCI-E device. This is one of features of xen-4.2 (released in September) -
not sure how stable it is (for this particular GPU) and Qubes still uses
xen-4.1 - migration to xen-4.2 isn't trivial because major changes in some
APIs (especially libxl).

Not sure if all above still applies if your NVidia GPU will be "secondary"
video card for Windows VM (the "primary" one will be emulated by Qemu
device-model), but still I suspect some problems in this area.

Perhaps you will need some non standard config options in xen VM config file.
Some documentation is here [1], custom config file can be used by "qvm-start
--custom-config=path". You should base on autogenerated one (its location is
shown by qvm-prefs tool).

2. Input devices. Currently input devices are connected to dom0 and from there
proxied to appropriate VM by Qubes GUI (based on active window). It is limited
to windows known to dom0 window manager, so will not work for external display
connected directly to another VM...
This can be workarounded by using separate keyboard+mouse for that VM (or the
same, but physically switch it to other port each time). Both solution will
require USB ports connected to different USB controllers - connecting single
USB device to other VM isn't working well yet, only full PCI device is usable.

"Physical" switching can be done by some simple KVM switch.

> Would it be as secure as having only an integrated video card?

I think yes - assuming working VT-d. But be prepared to some stability issues...

> This would save me having to build a dedicated Qubes hardware environment or to have to compromise on security with a dual boot system.
>
> Practically, to set focus to the Windows VM (which would be a full screen attached to a second display) I would select it in the task bar, and to release focus, I would have a key combination to press.
>
> Am I dreaming in aiming for such setup? Would the amount of patching be very substantial and beyond Python/C tweaking? I will not require copy/paste or file transfer, also tgat could be a plus...
>
> Looking forward to your view
> Alex
>

[1] http://wiki.xen.org/wiki/Xen_Configuration_File_Options

[Alex Dubois]

Thanks Marek for this detailed answer.

I am going to install Qubes in dual boot and will try to explore from there first Windows on my integrated video card and will then explore what would be my ideal installation to gain 3d support.

Regards,
Alex

[Abel Luck]

abb:
> Hi,
>
> Any plans to support USB passthrough
> http://wiki.xen.org/wiki/Xen_USB_Passthrough? This was partly discussed
> before
> https://groups.google.com/forum/?fromgroups=#!searchin/qubes-devel/usb/qubes-devel/jGA2rHlTr7s/OZby6IGUlKQJ,
> but that thread was about character devices specifically. As suggested in
> that post, attaching whole USB controller to a VM works, but it is rather
> inconvenient solution. Quite often I need to have a bunch of USB network
> adapters connected to different VMs, would be very nice to have proper USB
> devices passthrough.
>
> Regards,
> Alex
>

So this thread is now a giant monster, and I'm having trouble following
the latest progress.

Alexandre and Marek, what's the current status of per-device usb
passthrough? Are UsbVMs a possibility yet?

If it is in a testable state, could we get a wiki page with instructions?

Thanks!
~abel

[Alexandre Bezroutchko]

I believe Marek has merged the userspace tool (qvm-usb) into his master
branch. You can have dedicated usbvm no problem, just deploy a fresh VM,
attach your USB controller there (qvm-pci -a ...), plug USB device, and
run 'qvm-usb -l' in dom0.

However, there is a bug in kernel which is not fixed yet. It is
discussed above, I'm afraid you have to read the thread if you need more
info. I'm following up on it, but unfortunately I don't have much free
time lately, so the progress is rather slow. This weekend I've sent a
mail to xen-devel with the description of the problem, but there was no
response yet. I suppose if the problem does not get resolved in couple
weeks we will try using usbip instead of pvusb. But I suspect
performance hit will be even worse than with pvusb, which is not nice.

[Joanna Rutkowska]

Hmm, but really, what's the fundamental difference between UsbIP project
vs. Xen PVUSB project? I really see no difference in writing a USB
backend for use with Xen backend-frontend scheme vs. writing the USB
backend for use with Over-IP backend-frontend... Is there any?
Essentially this should be the same code, no? And I understand that the
problem is with this backend code? Or is the problem related to
frontend-backend communication (which would be totally USB-unrelated)?

joanna.

[Alexandre Bezroutchko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I didn't look at usbip really. Yes, probably there is little difference between usbip and pvusb, except usbip probably using sockets and pvusb using xen rings. In fact performance can be the similar after usbip is ported to vchan or xen rings...

> And I understand that the
> problem is with this backend code? Or is the problem related to
> frontend-backend communication (which would be totally USB-unrelated)?

There might be more than one problem actually. The easiest to reproduce is kernel memory management issue which reliably causes oopses in kfree() USB Ethernet adapter is attached. This happens when the backend tries to free memory block used for USB event handling. I started debugging it, activated memory-related debugging options in the kernel and it started crashing much earlier, during bootstrap. Marek has suggested it is an issue for xen-devel, so I've posted there.

Another problem I saw is bus resets. When I try to pass-through USB stick, it gets attached but it takes a minute or so before it starts working in domU. During this time there are some bus timeout messages or something in dmesg. This could have something to do with missed events in frontend-backend communication. I didn't investigate it though, guess memory management issue needs to be sorted out first.

Alex

> joanna.
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQEcBAEBAgAGBQJQolxEAAoJENsE8GcJgOzwNWQIAIJbFCwMv6s7GYpR9cHMZpK1
TrYnLVN3KsHaYYiBKMv+CceslvW+9Duijg0KL/9ked/ADGhqiz/ww2qeZ8emOhjs
THPybuMc4WNF6W9ysCpY98ptF/lFys0LVCiziVoW9lC3AUeHm1EZL9MIm1W3N1Tq
IaNeBI+fatD88veEIT8aTNNa5p5AaiN/YxFHCjzDw8FSOSiGXiUi3uj7MiADJZ53
c4GqNjVU5ZzxZLEST86uq4CBNgDiahMXjVj4lUxjuErW6jg1XnK4FQq4J4T3PBVp
IFbB1DNM4npSn41Iz3fhf8Wiugnn7x82sE0YSqndModuWcJd49qPueJib/ocUaE=
=9s1f
-----END PGP SIGNATURE-----

[Abel Luck]

Alexandre Bezroutchko:

> On 11/13/2012 11:48 AM, Abel Luck wrote:
>> abb:
>>> Hi,
>>>
>>> Any plans to support USB passthrough
>>> http://wiki.xen.org/wiki/Xen_USB_Passthrough? This was partly discussed
>>> before
>>> https://groups.google.com/forum/?fromgroups=#!searchin/qubes-devel/usb/qubes-devel/jGA2rHlTr7s/OZby6IGUlKQJ,
>>> but that thread was about character devices specifically. As suggested in
>>> that post, attaching whole USB controller to a VM works, but it is rather
>>> inconvenient solution. Quite often I need to have a bunch of USB network
>>> adapters connected to different VMs, would be very nice to have proper USB
>>> devices passthrough.
>>>
>>> Regards,
>>> Alex
>>>
>> So this thread is now a giant monster, and I'm having trouble following
>> the latest progress.
>>
>> Alexandre and Marek, what's the current status of per-device usb
>> passthrough? Are UsbVMs a possibility yet?
> I believe Marek has merged the userspace tool (qvm-usb) into his master
> branch. You can have dedicated usbvm no problem, just deploy a fresh VM,
> attach your USB controller there (qvm-pci -a ...), plug USB device, and
> run 'qvm-usb -l' in dom0.

Which repos do I need to pull and rebuild? Are all the changes pushed
into rpms?

[abb]

There is one small patch I have not sent to Marek yet, it is tagged abb_e58b432 in git://github.com/grwl/qubes-core.git.

The rest is in RPMs, yes. I roughly follow this procedure to have pvusb on Qubes 1.0:

* rebuild kernel, core, and xen from marmarek's repo (devel-3.4 branch for kernel),
* make dedicated usbvm domain, passthrough USB controller PCI devices there
* in usbvm: "rpm -Uhv qubes-core-vm-2.1.1-1.fc17.x86_64.rpm xen-libs-2000:4.1.3-18.fc17.x86_64.rpm xen-qubes-vm-essentials-2000:4.1.3-18.fc17.x86_64.rpm"
* in appvm where you want to attach the device to: "rpm -Uhv qubes-core-vm-2.1.1-1.fc17.x86_64.rpm"
* in dom0 I think the following packages need to be updated, don't have exact list now.qubes-core-dom0 is actually important, the rest are dependencies
    kernel-qubes-vm-3.4.18-1.pvops.qubes.x86_64.rpm
    qubes-core-dom0-2.1.1-1.fc13.x86_64.rpm
    xen-4.1.3-18.fc13.x86_64.rpm
    xen-hvm-4.1.3gui2.0.9-18.fc13.x86_64.rpm
    xen-hypervisor-4.1.3-18.fc13.x86_64.rpm
    xen-libs-4.1.3-18.fc13.x86_64.rpm
    xen-licenses-4.1.3-18.fc13.x86_64.rpm
    xen-runtime-4.1.3-18.fc13.x86_64.rpm
* plug or replug some USB device -- it will be picked up by usbvm, and in dom0 it will appear in the output of qvm-usb

Note that installation of kernel-qubes-vm-3.4.18-1.pvops.qubes.x86_64.rpm in dom0 will make the patched kernel available to all other vms, if you don't want it change it in vm preferences and set some 3.2 kernel instead. I only run the new kernel in usbvm and work domains.

[Abel Luck]

abb:

Thanks! I've added these to a wiki page here:
https://wiki.qubes-os.org/trac/wiki/USBVM

I'll test it today and try to spruce it up as well as report any
problems I have

~abel

[Abel Luck]

abb:

> * rebuild kernel, core, and xen from marmarek's repo (devel-3.4 branch for
> kernel),

No branch is needed for dom0-updates?

I'm still configured to use devel/xserver-1.12 under marmarek's subdir.

[Alexandre Bezroutchko]

Right, I have the same.
 

--

[Marek Marczykowski]

devel/xserver-1.12 is already merged into master and will be included in 2.0
release.

[abb]

Hi,

Well, pvusb debugging is not going well. There seems to be some memory corruption. It causes oops in kfree() when freeing up URBs. Also some URBs reach the backend, get passed to the controller, but simply never get completed by USB subsystem, which is not supposed to happen either. I don't mind continue troubleshooting it, just for kicks, but I'm not sure it is the best idea really. I've spent quite a bit of time reading the code and plugging printk's all over the place, but I don't get a feeling I'm getting closer. Quality of pvusb code is not good, rather difficult to read (comparing to other drivers like xennet). I'm not certain I will be able to catch that memory corruption problem on time for R2. On another hand USBIP project seems to be alive and apparently its code is already in linux kernel tree (staging). Is there any good reason to continue with pvusb?

Cheers,
Alex

[Marek Marczykowski]

In this situation perhaps the better option is use of usbip. Most preferably
ported to use shared ring buffers... But for starting point just check if it
is working at all. USBIP modules are already included in qubes kernel,
userspace tools are in kernel sources.

[abb]

Crashes too, but differently, on deadlocks :). And there is quite a bit of debug info, at least there are some leads to follow. Build notes and kernel crash logs are at [1] in case anyone is curious. So far I've only tried it on kernel 3.4 with debugging options enabled and pvusb patches applied, probably this does not make much difference though. Will try without pvusb patches later on. Also, my test system is Qubes 1.0, only usbip rpms added.

[1] https://github.com/grwl/qubes-usbip/, signed tag abb_e4c28ebc

[abb]

On Thursday, November 22, 2012 12:06:36 AM UTC+1, abb wrote:

On Wednesday, November 21, 2012 11:51:45 AM UTC+1, Marek Marczykowski wrote:

On 21.11.2012 10:52, abb wrote:
> Hi,
>
> Well, pvusb debugging is not going well. There seems to be some memory
> corruption. It causes oops in kfree() when freeing up URBs. Also some URBs
> reach the backend, get passed to the controller, but simply never get
> completed by USB subsystem, which is not supposed to happen either. I don't
> mind continue troubleshooting it, just for kicks, but I'm not sure it is
> the best idea really. I've spent quite a bit of time reading the code and
> plugging printk's all over the place, but I don't get a feeling I'm getting
> closer. Quality of pvusb code is not good, rather difficult to read
> (comparing to other drivers like xennet). I'm not certain I will be able to
> catch that memory corruption problem on time for R2. On another hand USBIP
> project seems to be alive and apparently its code is already in linux
> kernel tree (staging). Is there any good reason to continue with pvusb?

In this situation perhaps the better option is use of usbip. Most preferably
ported to use shared ring buffers... But for starting point just check if it
is working at all. USBIP modules are already included in qubes kernel,
userspace tools are in kernel sources.

Crashes too,

Ah, forgot to mention -- before it crashes it actually manages to pass-through USB Ethernet device and get a DHCP lease on it.
 

[abb]

I've fixed a deadlock and a double free bug the backend in usbip kernel modules. It works better, but still glitches. If usbip backend is hosted on another physical server (fc17 with vanilla 3.4.18 kernel with usbip) it works the best (usb network adapter does works for regular browsing, but I really stress it with iperf, the frontend locks up. If I the backend is under qubes, I experienced some strange problems attaching usb stick, but usb ethernet adapter continues to work. The patches are in my repo g...@github.com:grwl/qubes-kernel.git, tag abb-20121125b.

Cheers,
Alex

[Marek Marczykowski]

Thanks for this work, I will look into it, but most likely next week... Can
you post here some instruction how to setup usbip frontend and backend?

[Alexandre Bezroutchko]

>> I've fixed a deadlock and a double free bug the backend in usbip kernel
>> modules. It works better, but still glitches. If usbip backend is hosted on
>> another physical server (fc17 with vanilla 3.4.18 kernel with usbip) it
>> works the best (usb network adapter does works for regular browsing, but I
>> really stress it with iperf, the frontend locks up. If I the backend is
>> under qubes, I experienced some strange problems attaching usb stick, but
>> usb ethernet adapter continues to work. The patches are in my repo
>> g...@github.com:grwl/qubes-kernel.git, tag abb-20121125b.
>
> Thanks for this work, I will look into it, but most likely next week... Can
> you post here some instruction how to setup usbip frontend and backend?

Nothing special for the kernel, just use patches from my repo:
 g...@github.com:grwl/qubes-kernel.git, tag abb-20121125b.
Two commits which fix the issues are:
 frontend deadlock: 79d91b069e1a3efce7881f9fdd9638d880187ea8
 backend doublefree: b7472a23ef91eb43645d57b0d5d19b995ff4c641, a97c88543762a64e3aa3ae53f69078386c7c767c

No changes to the userland tools since last week, see notes.txt for instructions how to prepare RPM. I'm not sure if it is the way to do it, I don't have much knowledge about RPMs. See git://github.com/grwl/qubes-usbip.git, tag abb_e4c28ebc

Cheers,
Alex

[abb]

Oops, something went wrong with formatting, sorry.

[abb]

On Tuesday, November 27, 2012 12:54:29 AM UTC+1, Marek Marczykowski wrote:

> I've fixed a deadlock and a double free bug the backend in usbip kernel
> modules. It works better, but still glitches. If usbip backend is hosted on
> another physical server (fc17 with vanilla 3.4.18 kernel with usbip) it
> works the best (usb network adapter does works for regular browsing, but I
> really stress it with iperf, the frontend locks up. If I the backend is
> under qubes, I experienced some strange problems attaching usb stick, but
> usb ethernet adapter continues to work. The patches are in my repo
> g...@github.com:grwl/qubes-kernel.git, tag abb-20121125b.

Thanks for this work, I will look into it, but most likely next week...

By the way I've done more testing with all debugging output removed now and it appears to be more usable. First iteration of iperf has completed fine, with 86MBps throughput over USB FastEthernet adapter. But then it stalled.

My fix for doublefree problem is not exactly correct and likely to lead to memory leak. The actual problem is USBIP module frees URB transfer buffers and USB core does the same afterwards. There is a flag controlling this behaviour (introduced in this patch [1] in kernel). Apparently this flag is controlled remotely by the frontend, it is mentioned in usbip_protocol.txt part of the protocol. This is odd, why it can possibly be up to the frontend to decide whether to free the transfer buffer or not. This transfer buffer is allocated at stub_rx.c:485, right after URB holding it is allocated with urb_alloc_urb() at lines 472/475. Eventually the transfer buffer is freed in stub_main.c:239 and stub_tx.c:31, right before URB is freed with usb_free_urb(). The latter internally triggers urb_destroy() which, if URB_FREE_BUFFER flag is set in URB, calls kfree() and actually causes all the troubles. I think proper solution would be to mask out URB_FREE_BUFFER from the received transfer flags. There is already some filtering for the flags stub_rx.c:437 but URB_FREE_BUFFER is explicitly permitted, which is odd.

By the way, it is possible that pvusb was plagued by the same issue -- at the first glance I don't find any attempt to filter value of transfer_flags, normally should happen somewhere around usbback.c:526, but it does not.

If I'm not mistaken the above problem only affects the backend, so frontend lockups something else probably. Maybe it is a side effect of my patch to the lockup problem :). To troubleshoot stalls I figure I can try to run debugger against the stalled kernel over a virtual serial link and dump stack trace on the stalled CPU, this might clairfy where the problem is.

Normally I will have more time over the next two weeks to work on this as well. But now I am seriously tempted to go back to pvusb and see if filtering out URB_FREE_BUFFER flag will have any effect there :). When/if you decide to start working on pvusb/usbip too, let me know, I'll share what I have.

[1] http://www.mail-archive.com/linux-u...@lists.sourceforge.net/msg54832.html