Slow performance of Docker containers in AppVMs

[Garrett Robinson]

I am working on transitioning my day-to-day software development work to
Qubes. The primary challenge that I face is widespread use of Vagrant
for provisioning development environments. I am aware of the challenges
and concerns around hacking Qubes to achieve nested virtualization, so I
am trying to avoid going down that road.

A potential alternative is to use Docker, because there are no inherent
issues with using Linux containers inside virtual machines. Vagrant
supports using Docker as a "provider," so this seems like a viable
solution that could allow me to use my existing Vagrant-based
development environments on Qubes with only minor modifications.

I set up a new TemplateVM (based on Fedora 24) and AppVM to experiment
with setting up a Vagrant+Docker-based development environment. It was
surprisingly easy; however, after some initial testing, I realized an
unfortunate truth: operations inside the Docker container are *very
slow* - so slow as to create an unacceptable level of overhead for
day-to-day development work.

The slowdown appears to be due to slow disk IO. Running htop shows that
processes in the container have status "D", meaning "disk sleep
(uninterruptible)", for long periods of time. I tried switching the
Docker storage backend to "overlay" instead of the Fedora default of
devicemapper with loopback devices, which is a well-known technique for
improving Docker performance on Fedora. This was an improvement, based
on my experience as well as some simple dd-based IO benchmarks, but
overall performance is still much too slow--borderline unusable.

My gut feeling is that this is due to the combination of Qubes' overlay
filesystem with Docker's overlay filesystem - I imagine that nesting COW
filesystems is naturally a prescription for degraded performance.
Unfortunately, I do not know enough about Qubes/Docker/filesystems to
know of the best way to test this hypothesis.
My questions for the list are:

1. Is anybody else successfully using Vagrant and/or Docker on Qubes? Do
you have any tips/tricks to share?
2. Does anybody know or have any alternative theories as to what might
be causing these performance problems? 3. Even better: does anybody have
good advice for how to experimentally analyze these problems? Good ideas
for how to analyze the source of the slowdown, ideas for benchmarks,
useful analysis tools or techniques, etc.?

I have some notes on my experience and a Docker environment for basic IO
benchmarking, which I am happy to share if anybody's interested.

Thanks,
Garrett

[Grzesiek Chodzicki]

Have You tried using StandaloneVMs to host Docker? StandaloneVMs don't use copy-on-write afaik so it should be at least a little faster.

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Try placing /var/lib/docker in /rw using bind-dirs[1]. Something like
this:

$ cat /rw/config/qubes-bind-dirs.d/docker.conf
binds+=( '/var/lib/docker' )

[1] https://www.qubes-os.org/doc/bind-dirs/

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJYhpRpAAoJENuP0xzK19csydsH+QETCL/mCE9gIRk3H7MghZVi
5Ry8ZIVznH+RM6eyF9o2zco4Q/tYdlrwbi2jyiZSEQa+kEFGAnzeS8G7403xE6ic
wp2EkKXX1n2L2zEFnfIOPdYyNV7Nd9JV7/hgGHKqa6Dv6yKeRIXpIllK978S96HV
bWW3MeWJVqKTZAK4ucUkFG3eOqgfUU83DL6/u5p9cfBkSOK91kp8xrT1QmD2JEba
anIF+C5/JTlqyH/W1aDz9u3m+JW+yDOIxAsYzwL7Xi5wUJ1yY3ZsiLzwJWWrlFTc
ScYogM/M/UhbMQAHynLXqihrUhclgZHgyz3+JnBQaDarTRlehTZSmg2dphANiok=
=nIBF
-----END PGP SIGNATURE-----

[Garrett Robinson]

On 01/23/2017 06:17 PM, Grzesiek Chodzicki wrote:

> Have You tried using StandaloneVMs to host Docker? StandaloneVMs don't use copy-on-write afaik so it should be at least a little faster.

Great suggestion! Unfortunately, I just tried using a StandaloneVM and
there was no noticeable performance improvement :(

[Garrett Robinson]

On 01/23/2017 06:40 PM, Marek Marczykowski-Górecki wrote:

Try placing /var/lib/docker in /rw using bind-dirs[1]. Something like
this:

    $ cat /rw/config/qubes-bind-dirs.d/docker.conf
    binds+=( '/var/lib/docker' )

[1] https://www.qubes-os.org/doc/bind-dirs/

Thanks for sharing bind-dirs, I hadn't come across that feature yet! However, do you think this would have different performance characteristics over just using a StandaloneVM (which I have tried, and which did not improve matters noticeably?)

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Mon, Jan 23, 2017 at 07:55:30PM -0500, Garrett Robinson wrote:
> On 01/23/2017 06:40 PM, Marek Marczykowski-Górecki wrote:
> > Try placing /var/lib/docker in /rw using bind-dirs[1]. Something like
> > this:
> >
> > $ cat /rw/config/qubes-bind-dirs.d/docker.conf
> > binds+=( '/var/lib/docker' )
> >
> > [1] https://www.qubes-os.org/doc/bind-dirs/
> >
>
> Thanks for sharing bind-dirs, I hadn't come across that feature yet!

> However, do you think this would have different /performance/

> characteristics over just using a StandaloneVM (which I have tried, and
> which did not improve matters noticeably?)

There is one device-mapper layer less - but in case of StandaloneVM it's
dm-linear (so 1:1 mapping), so not a big difference.

I'm not sure what performance do you expect, but here is some really
naive container startup benchmark (with the above bind-dir):

$ time sudo docker run -i ubuntu:16.04 true

real 0m2.206s
user 0m0.065s
sys 0m0.055s

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJYhqiCAAoJENuP0xzK19csuacH/RJVo8PkuB70G6f4GUh4sEKh
kVmZeTVJMMJhznfkRxxCgJ3o9YM8DwJs0hwXJb46MkJPlxULJ3SPGraqQHj9Ek11
1ywI4wtANkSNZgb7mH6cohR011rH0llSxhqbVcpqqIXUHqkiapCd50V9FGuhJUpz
j8D4gX/iBSFPkSfQytmDVFdMT1z+8A9LBqKscjjllBg7gBcKvG8wTP8bmD40I0HJ
B8yNdM8OXOG9CyefUqJitBnhua3JIclx7zZNiChlb3uouZok+VmpDA1fQT7ALmpf
gOKtTK0G8JMgf5ge7lIkQYY1YhR0CFZ10klkAPP/6ijttZf8PrzmySIov6s1tAk=
=Mptg
-----END PGP SIGNATURE-----

[Ángel]

Garrett Robinson wrote:
> I am working on transitioning my day-to-day software development work to
> Qubes. The primary challenge that I face is widespread use of Vagrant
> for provisioning development environments. I am aware of the challenges
> and concerns around hacking Qubes to achieve nested virtualization, so I
> am trying to avoid going down that road.

Supposedly, you could make vagrant use xen with
https://github.com/jonludlam/vagrant-xenserver at which point there
would be no double virtualization. I don't know how would it interact
with Qubes way, or if it would be breaking some security fence, but
seems worth investigating.


Best