Running containers with Docker + liblxc-virt instead of lxc-start

[Jérôme Petazzoni]

I did that earlier today, and here are my notes:

https://github.com/dotcloud/docker/wiki/libvirt-lxc

It's  very hackish at this point, it will be helpful after 1.0 if we want a plugin to use libvirt-lxc instead of lxc-start (could be useful because libvirt supposedly plays nicer with SELinux, while lxc-start is better integrated with AppArmor in Ubuntu).

[Michael Crosby]

Very nice Jerome.  I'm going to play with this a little.   I think we could write a nice Go wrapper for this lib....

[Josh Poimboeuf]

On Sun, Aug 25, 2013 at 07:42:56PM -0700, Michael Crosby wrote:

> On Thursday, August 22, 2013 3:42:12 PM UTC-7, J�r�me Petazzoni wrote:
> >
> > I did that earlier today, and here are my notes:
> > https://github.com/dotcloud/docker/wiki/libvirt-lxc
> >
> > It's very hackish at this point, it will be helpful after 1.0 if we want
> > a plugin to use libvirt-lxc instead of lxc-start (could be useful because
> > libvirt supposedly plays nicer with SELinux, while lxc-start is better
> > integrated with AppArmor in Ubuntu).

Hey! I've also been looking at integrating docker with libvirt-lxc. We
should probably all compare notes and coordinate our efforts (on IRC
maybe?)

I hacked up a template file for the libvirt XML config, and called
libvirt-lxc from docker. Here's my work so far (ugly hack alert):

https://github.com/jpoimboe/docker/commit/91b5b87ca3fc03150408e8a8f9f2385df9b56dfc

BTW, I was able to get the bind mount of /.dockerinit to work, so I'm
not sure why that didn't work for you J�r�me. I did my testing on
Fedora 19 (+ aufs kernel patch) if that makes a difference.


Josh

[Jérôme Petazzoni]

Nice!

Regarding networking, I don't know if there are plans in libvirt-lxc to allow passing down the network configuration to the container (or setting up the interfaces right away). The following approaches are possible:

(1) patching libvirt-lxc so that you can easily spec the IP address in the config (I think it's already possible, I'm just not familiar with the libvirt XMl format), then make sure that the libvirt-lxc driver can use that info to setup the veth interface correctly;

(2) passing all needed information to .dockerinit (instead of just the gateway as we do now);

(3) leaving the whole network setup to Docker (there is a simplified proof-of-concept implementation in dockerlite[1]), which is tempting but means that "native" management tools will see a "network-less" container, so it's probably sub-optimal in some scenarios.

[1] https://github.com/jpetazzo/dockerlite/blob/master/lib/dockerlite-runc.sh#L67 

On Wed, Aug 28, 2013 at 11:46 AM, Josh Poimboeuf <jpoi...@redhat.com> wrote:

On Sun, Aug 25, 2013 at 07:42:56PM -0700, Michael Crosby wrote:

> On Thursday, August 22, 2013 3:42:12 PM UTC-7, Jérôme Petazzoni wrote:
> >
> > I did that earlier today, and here are my notes:
> > https://github.com/dotcloud/docker/wiki/libvirt-lxc
> >
> > It's  very hackish at this point, it will be helpful after 1.0 if we want
> > a plugin to use libvirt-lxc instead of lxc-start (could be useful because
> > libvirt supposedly plays nicer with SELinux, while lxc-start is better
> > integrated with AppArmor in Ubuntu).

Hey!  I've also been looking at integrating docker with libvirt-lxc.  We
should probably all compare notes and coordinate our efforts (on IRC
maybe?)

I hacked up a template file for the libvirt XML config, and called
libvirt-lxc from docker.  Here's my work so far (ugly hack alert):

https://github.com/jpoimboe/docker/commit/91b5b87ca3fc03150408e8a8f9f2385df9b56dfc

BTW, I was able to get the bind mount of /.dockerinit to work, so I'm

not sure why that didn't work for you Jérôme.  I did my testing on

[Josh Poimboeuf]

Yeah, I think libvirt-lxc expects the container's init process to setup
the IP address. I don't know of any plans to change that and allow
setting up the IP address from libvirt-lxc, but I'll ask the libvirt
guys.

I think I prefer something like option 2, because it more closely
resembles how networks are setup in a non-container environment. Also
it gives the container more control over its setup. For example, the
container could in theory decide to run a DHCP client instead of a
static IP (assuming docker wanted to support that).


Josh

On Wed, Aug 28, 2013 at 12:21:23PM -0700, J�r�me Petazzoni wrote:
> Nice!
>
> Regarding networking, I don't know if there are plans in libvirt-lxc to
> allow passing down the network configuration to the container (or setting
> up the interfaces right away). The following approaches are possible:
>
> (1) patching libvirt-lxc so that you can easily spec the IP address in the
> config (I think it's already possible, I'm just not familiar with the
> libvirt XMl format), then make sure that the libvirt-lxc driver can use
> that info to setup the veth interface correctly;
>
> (2) passing all needed information to .dockerinit (instead of just the
> gateway as we do now);
>
> (3) leaving the whole network setup to Docker (there is a simplified
> proof-of-concept implementation in dockerlite[1]), which is tempting but
> means that "native" management tools will see a "network-less" container,
> so it's probably sub-optimal in some scenarios.
>
> [1]
> https://github.com/jpetazzo/dockerlite/blob/master/lib/dockerlite-runc.sh#L67
>
>
>
> On Wed, Aug 28, 2013 at 11:46 AM, Josh Poimboeuf <jpoi...@redhat.com>wrote:
>
> > On Sun, Aug 25, 2013 at 07:42:56PM -0700, Michael Crosby wrote:

> > > On Thursday, August 22, 2013 3:42:12 PM UTC-7, J�r�me Petazzoni wrote:
> > > >
> > > > I did that earlier today, and here are my notes:
> > > > https://github.com/dotcloud/docker/wiki/libvirt-lxc
> > > >
> > > > It's very hackish at this point, it will be helpful after 1.0 if we
> > want
> > > > a plugin to use libvirt-lxc instead of lxc-start (could be useful
> > because
> > > > libvirt supposedly plays nicer with SELinux, while lxc-start is better
> > > > integrated with AppArmor in Ubuntu).
> >
> > Hey! I've also been looking at integrating docker with libvirt-lxc. We
> > should probably all compare notes and coordinate our efforts (on IRC
> > maybe?)
> >
> > I hacked up a template file for the libvirt XML config, and called
> > libvirt-lxc from docker. Here's my work so far (ugly hack alert):
> >
> >
> > https://github.com/jpoimboe/docker/commit/91b5b87ca3fc03150408e8a8f9f2385df9b56dfc
> >
> > BTW, I was able to get the bind mount of /.dockerinit to work, so I'm

> > not sure why that didn't work for you J�r�me. I did my testing on

[Camilo Aguilar]

Why not using directly LXC bindings? https://github.com/caglar10ur/lxc

On Wednesday, August 28, 2013 4:02:04 PM UTC-4, Josh Poimboeuf wrote:

Yeah, I think libvirt-lxc expects the container's init process to setup
the IP address.  I don't know of any plans to change that and allow
setting up the IP address from libvirt-lxc, but I'll ask the libvirt
guys.

I think I prefer something like option 2, because it more closely
resembles how networks are setup in a non-container environment.  Also
it gives the container more control over its setup.  For example, the
container could in theory decide to run a DHCP client instead of a
static IP (assuming docker wanted to support that).


Josh

On Wed, Aug 28, 2013 at 12:21:23PM -0700, J�r�me Petazzoni wrote:
> Nice!
>
> Regarding networking, I don't know if there are plans in libvirt-lxc to
> allow passing down the network configuration to the container (or setting
> up the interfaces right away). The following approaches are possible:
>
> (1) patching libvirt-lxc so that you can easily spec the IP address in the
> config (I think it's already possible, I'm just not familiar with the
> libvirt XMl format), then make sure that the libvirt-lxc driver can use
> that info to setup the veth interface correctly;
>
> (2) passing all needed information to .dockerinit (instead of just the
> gateway as we do now);
>
> (3) leaving the whole network setup to Docker (there is a simplified
> proof-of-concept implementation in dockerlite[1]), which is tempting but
> means that "native" management tools will see a "network-less" container,
> so it's probably sub-optimal in some scenarios.
>
> [1]
> https://github.com/jpetazzo/dockerlite/blob/master/lib/dockerlite-runc.sh#L67
>
>
>
> On Wed, Aug 28, 2013 at 11:46 AM, Josh Poimboeuf <jpoi...@redhat.com>wrote:
>
> > On Sun, Aug 25, 2013 at 07:42:56PM -0700, Michael Crosby wrote:

> > > On Thursday, August 22, 2013 3:42:12 PM UTC-7, J�r�me Petazzoni wrote:
> > > >
> > > > I did that earlier today, and here are my notes:
> > > > https://github.com/dotcloud/docker/wiki/libvirt-lxc
> > > >
> > > > It's  very hackish at this point, it will be helpful after 1.0 if we
> > want
> > > > a plugin to use libvirt-lxc instead of lxc-start (could be useful
> > because
> > > > libvirt supposedly plays nicer with SELinux, while lxc-start is better
> > > > integrated with AppArmor in Ubuntu).
> >
> > Hey!  I've also been looking at integrating docker with libvirt-lxc.  We
> > should probably all compare notes and coordinate our efforts (on IRC
> > maybe?)
> >
> > I hacked up a template file for the libvirt XML config, and called
> > libvirt-lxc from docker.  Here's my work so far (ugly hack alert):
> >
> >
> > https://github.com/jpoimboe/docker/commit/91b5b87ca3fc03150408e8a8f9f2385df9b56dfc
> >
> > BTW, I was able to get the bind mount of /.dockerinit to work, so I'm

> > not sure why that didn't work for you J�r�me.  I did my testing on

[Jérôme Petazzoni]

Hi Camilo,

I wasn't aware of those go bindings ... Good to know :-)

In the long run, Docker will support plugins, so it should be easy to pick your backend of choice.

--

@jpetazzo

Latest blog post: http://blog.docker.io/2013/09/docker-joyent-openvpn-bliss/

[Camilo Aguilar]

I see, then it may make sense to use libvirt then, one good thing is that the docker community will join to the libvirt efforts too! Exciting times ahead! 

—
Sent from Mailbox for iPhone

[Jérôme Petazzoni]

Totally!

TBH, I was fairly neutral with both approaches, until someone pointed out that libvirt has a fairly badass networking model. A lot of things that are possible today with hacks like pipework, will very likely be possible tomorrow more cleanly with libvirt. Or at least, I hope so. :-)

[Camilo Aguilar]

Indeed, libvirt has a ton of work done already. It has been around for years and really smart people contributes to it. It will be a poor move not to use it, at least as a starting point.

—
Sent from Mailbox for iPhone

[Solomon Hykes]

Based on all this, my current thinking is that libvirt should be a first-class citizen, but probably not the hardcoded default (the way we're considering making device-mapper the harcoded default instead of aufs) since it carries rather large dependencies and relies on the libvirt daemon running on your machine.

So docker out of the box will rely on simple library bindings (instead of shelling out to lxc-start the way we do today), and offer a high-quality and well-maintained plugin to use libvirt. The main motivations for using libvirt would be 1) advanced customizations eg. exotic network topologies, custom security policies etc. and 2) you already have libvirt installed and want to integrate into it.

That's my current thinking anyway. I can always change my mind :)

[Filip Maj]

Hey everyone,

warning: noob here

I'm interested in kicking off docker containers with libvirt. I followed Jerome's wiki article to some success. I get some weird errors when attaching to the running docker container but that's another topic :)

My noob-ish question is: wouldn't we want to use libvirt to start the container in the first place? In the wiki article, we first start the container using docker, then attach to it after the fact.

Any info and patience with my questions would be greatly appreciated :)

Cheers,

Fil

[Jérôme Petazzoni]

Hi Filip,

Josh Poimboeuf is working on proper integration with libvirt (linking vs libvirt, instead of shelling out to virsh). The result should be pretty interesting.

lxc-start and libvirt both have pros and cons; lxc-start doesn't require another long-running daemon, but libvirt has more features.

Eventually we'll support both :-)

--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

@jpetazzo

Latest blog post: http://jpetazzo.github.io/2013/10/06/policy-rc-d-do-not-start-services-automatically/