Native integration of Open vSwitch in Docker.

[she...@nicira.com]

Hello All,
 Currently there is the instructions at [1] and the Pipework project at [2] that lets one add additional network interfaces inside an already created container and then attach it to an Open vSwitch bridge. One can also use the recently added 'ovs-docker' utility that comes with the Open vSwitch packages to add and remove interfaces from a container ([3]). The problem that I see with the current situation is that an application could already start and try to make a network connection before additional network interfaces are added. Isn't this a problem? (Maybe I am missing a popular workaround or a workflow.)

Would the Docker developers be open to adding a native integration support with Open vSwitch? I can create a GitHub issue and work on it if there is interest.
Right now, a container can be started with 'docker run' by providing it the --net option that lets one choose 'none', Linux bridge, host networking, or another container's network stack. We can extend it to also include --net=openvswitch. I can also see how it can be useful to provide options to let one create multiple network interfaces natively. The Docker daemon parsing the options, will create the veth pairs and add one end to the Open vSwitch bridge as a port. Docker can also modify the Open vSwitch database to specify the internal mac address, container uuid, interface uuid etc (similar to what Libvirt and Xenserver does currently with Open vSwitch. More information at [4]).

With the integration with Open vSwitch, one can set VLANs, create Tunnels, add OpenFlow rules etc for more network control locally or through cloud management platforms like OpenStack across multiple hosts.

[1] https://docs.docker.com/articles/networking/#how-docker-networks-a-container
[2] https://github.com/jpetazzo/pipework
[3] https://github.com/openvswitch/ovs/blob/master/INSTALL.Docker
[4] https://github.com/openvswitch/ovs/blob/master/IntegrationGuide

Thanks,
Guru

[Dave Tucker]

Hi Guru,

I'm already working on something...

So far I have got a rough draft of native OVS integration through libcontainer to create the necessary ports when a container is started.

This is in the form of a new network strategy called "ovs" which creates an ovs internal port and binds this to the netns.

Right now the I'm using ovs-vsctl through os.exec, but I'd rather use a native golang OVSDB client (which I've already started looking at too).

I haven't yet looked at what would be required to add a flag in docker daemon to use OVS networking by default...

Perhaps we can work on this together!

-- Dave

[Gurucharan Shetty]

Hello Dave,
It looks like you have already made some progress. Is there a github
issue/proposal that tracks your work? Or are you planning to present
your work once you finish it? Please point me to any discussion that
you have already had with the wider community . Couple of comments
inline.

>
> So far I have got a rough draft of native OVS integration through
> libcontainer to create the necessary ports when a container is started.
> This is in the form of a new network strategy called "ovs" which creates an
> ovs internal port and binds this to the netns.

What is your reason for using OVS internal port instead of say, using
a veth pair and then adding it as a port? I see how using the OVS
internal port makes it easy to administer. But it would also mean that
any OVS upgrade in the host machine (which includes replacing the
kernel module) would mean that the interfaces will disappear from
inside the container. So it will probably need a secondary effort of
moving them back.

> Right now the I'm using ovs-vsctl through os.exec, but I'd rather use a
> native golang OVSDB client (which I've already started looking at too).

Libvirt uses ovs-vsctl directly and AFAIK, it hasn't been a problem so
far. On the other hand Xenserver is more tightly integrated and uses
Python bindings, so I can see how it can be helpful to have new Golang
bindings in Open vSwitch, if you foresee a lot more things to do in
OVSDB.

>
> I haven't yet looked at what would be required to add a flag in docker
> daemon to use OVS networking by default...
>
> Perhaps we can work on this together!

I would be very happy to work together on this one.

> --
> You received this message because you are subscribed to the Google Groups
> "docker-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to docker-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Dave Tucker]

Adding libcontainer...

On 21 Oct 2014, at 20:39, Gurucharan Shetty wrote:

> Hello Dave,
> It looks like you have already made some progress. Is there a github
> issue/proposal that tracks your work?

Not yet..

> Or are you planning to present your work once you finish it?

Here's what I have so far. It's alpha quality at best and is still a
WIP.
https://github.com/dave-tucker/libcontainer/tree/openvswitch_support
https://github.com/dave-tucker/docker/tree/openvswitch_support

> Please point me to any discussion that
> you have already had with the wider community . Couple of comments
> inline.
>
>>
>> So far I have got a rough draft of native OVS integration through
>> libcontainer to create the necessary ports when a container is
>> started.
>> This is in the form of a new network strategy called "ovs" which
>> creates an
>> ovs internal port and binds this to the netns.
> What is your reason for using OVS internal port instead of say, using
> a veth pair and then adding it as a port? I see how using the OVS
> internal port makes it easy to administer. But it would also mean that
> any OVS upgrade in the host machine (which includes replacing the
> kernel module) would mean that the interfaces will disappear from
> inside the container. So it will probably need a secondary effort of
> moving them back.

See: http://www.opencloudblog.com/?p=96
There is a massive performance gain associated with using OVS Internal
Ports.
IIRC this the default in OpenStack these days.

I'm struggling to understand how updating the ovs userspace tools could
remove the port from within the container. Could you explain?

>> Right now the I'm using ovs-vsctl through os.exec, but I'd rather use
>> a
>> native golang OVSDB client (which I've already started looking at
>> too).
> Libvirt uses ovs-vsctl directly and AFAIK, it hasn't been a problem so
> far. On the other hand Xenserver is more tightly integrated and uses
> Python bindings, so I can see how it can be helpful to have new Golang
> bindings in Open vSwitch, if you foresee a lot more things to do in
> OVSDB.

We should have enough of OVSDB implemented to handle the heavy lifting
in libcontainer soon.
Watch this space!

[Gurucharan Shetty]

>
> Here's what I have so far. It's alpha quality at best and is still a WIP.
> https://github.com/dave-tucker/libcontainer/tree/openvswitch_support
> https://github.com/dave-tucker/docker/tree/openvswitch_support
>

Thanks for sharing.

>>
>> What is your reason for using OVS internal port instead of say, using
>> a veth pair and then adding it as a port? I see how using the OVS
>> internal port makes it easy to administer. But it would also mean that
>> any OVS upgrade in the host machine (which includes replacing the
>> kernel module) would mean that the interfaces will disappear from
>> inside the container. So it will probably need a secondary effort of
>> moving them back.
>
>
> See: http://www.opencloudblog.com/?p=96
> There is a massive performance gain associated with using OVS Internal
> Ports.
> IIRC this the default in OpenStack these days.
>
> I'm struggling to understand how updating the ovs userspace tools could
> remove the port from within the container. Could you explain?

I see where you are coming from. Typically, the upstream OVS
development includes changes to both OVS userspace and OVS kernel
module. There are new features added to OVS userspace which only work
with newer openvswitch kernel modules. There are also bug fixes added
to OVS kernel modules. So, a typical OVS upgrade changes userspace
utilities, daemons as well as kernel module and then to use the newer
kernel module, reloads the kernel module saving and restoring state
across the reload. When this is done, the internal interfaces get
deleted and re-added. This is the default behavior if anyone uses
debian packages coming from openvswitch.org. So using internal
interfaces would mean that you can only change OVS kernel module when
you do an entire kernel upgrade which is limiting in terms of
maintenance of servers. This hasn't been a problem with VMs because
VIFs are not OVS internal interfaces but rather a separate device.

>
>
> We should have enough of OVSDB implemented to handle the heavy lifting in
> libcontainer soon.

Have you thought about the strategy on how to manage a changed schema
that the Open vSwitch database managed by ovsdb-server is using?
Different OVS versions would have different schemas which could result
in particular versions of libcontainer not working with particular
versions of Open vSwitch, isn't it?. Using ovs-vsctl, if the use case
is simple, solves these issues. It also reduces the amount of code to
be maintained in libcontainer.

> Watch this space!
Okay. Does your work include plans to update the external_ids column
of OVSDB records? As in here:
https://github.com/openvswitch/ovs/blob/master/IntegrationGuide

Thanks,
Guru

[Dave Tucker]

Right. So I was speaking to Ben Pfaff about OVS deployment and the
recommendation he gave me was to use the linux kernel module over the
in-tree module in OVS. This is the approach that I took in packaging OVS
for docker.
https://github.com/dave-tucker/docker-ovs (i.e userspace only, kmod
provided by host).

I think I'd need to get some more info on how kernel upgrades affect
other parts of docker before I get concerned about usage of internal
ports.
E.g if the expectation is that all containers are ejected from a host
pre-upgrade this may be a non-issue

That said, we can easily support both. We should look at the existing
veth strategy in libcontainer and see if this can be re-used, or whether
we need to add an "ovs-veth" strategy.

>>
>>
>> We should have enough of OVSDB implemented to handle the heavy
>> lifting in
>> libcontainer soon.
> Have you thought about the strategy on how to manage a changed schema
> that the Open vSwitch database managed by ovsdb-server is using?
> Different OVS versions would have different schemas which could result
> in particular versions of libcontainer not working with particular
> versions of Open vSwitch, isn't it?. Using ovs-vsctl, if the use case
> is simple, solves these issues. It also reduces the amount of code to
> be maintained in libcontainer.

Yes. My team and I wrote a schema-independent OVSDB library for Java so
we're pretty comfortable with how to handle different versions of the
schema. Also I believe that the Open_vSwitch schema doesn't approve
backwards-incompatible changes. The only things we need to touch here
are the Bridge, Port and Interface tables and the necessary columns.

>> Watch this space!
> Okay. Does your work include plans to update the external_ids column
> of OVSDB records? As in here:
> https://github.com/openvswitch/ovs/blob/master/IntegrationGuide

Not initially.

Firstly I haven't checked the code to make sure there is enough context
about the container available to the method that will write these values
in to OVSDB.

Secondly, I'm not sure what to do re: "vm-id".
Yes we can re-use this for containers... but what about an environment
that has docker running on VMs instead of bare-metal, or a a mix of
docker and kvm or xen server?
Do we want to keep "vm-id" for VMs and "container-id" for containers to
make it easy to differntiate at the interface level?
If both "vm-id" and "container-id" are specified can we assume "vm-id"
is a reference to the parent-vm?
Should we just use "vm-id" and call it quits?

[Dave Tucker]

Premature send on my side so please ignore typos...
I'm speaking at the OVS Hackathon next month so perhaps we can spend
some time face to face on this then?

[Gurucharan Shetty]

>
> Right. So I was speaking to Ben Pfaff about OVS deployment and the
> recommendation he gave me was to use the linux kernel module over the
> in-tree module in OVS. This is the approach that I took in packaging OVS for
> docker.
> https://github.com/dave-tucker/docker-ovs (i.e userspace only, kmod provided
> by host).
>
> I think I'd need to get some more info on how kernel upgrades affect other
> parts of docker before I get concerned about usage of internal ports.
> E.g if the expectation is that all containers are ejected from a host
> pre-upgrade this may be a non-issue
>
> That said, we can easily support both. We should look at the existing veth
> strategy in libcontainer and see if this can be re-used, or whether we need
> to add an "ovs-veth" strategy.

Okay. I will be looking at the code too. I am hoping that we can come
up with something that is flexible.
(My experience over the last few years has been that the OVS kernel
module had to be upgraded every 3-4 months in large deployments,
simply because of rapid addition of features and performance
improvements.)

>
>
> Yes. My team and I wrote a schema-independent OVSDB library for Java so
> we're pretty comfortable with how to handle different versions of the
> schema. Also I believe that the Open_vSwitch schema doesn't approve
> backwards-incompatible changes. The only things we need to touch here are
> the Bridge, Port and Interface tables and the necessary columns.

Okay. ( A common case is that a new column gets added in a table, then
you change your bindings in libcontainer to reflect the new column,
but on a different machine, an older version of OVS would be running
and the new libcontainer should work with that.).

> Not initially.
>
> Firstly I haven't checked the code to make sure there is enough context
> about the container available to the method that will write these values in
> to OVSDB.
>
> Secondly, I'm not sure what to do re: "vm-id".
> Yes we can re-use this for containers... but what about an environment that
> has docker running on VMs instead of bare-metal, or a a mix of docker and
> kvm or xen server?
> Do we want to keep "vm-id" for VMs and "container-id" for containers to make
> it easy to differntiate at the interface level?
> If both "vm-id" and "container-id" are specified can we assume "vm-id" is a
> reference to the parent-vm?
> Should we just use "vm-id" and call it quits?

I would start with having the ability to set the columns, we can hash
out the key names when we get closer to acceptance.

[Madhu Venugopal]

Since every change to the schema comes with a version bump, this is handled by the
docker-ovs based CI systems. If you look at the history of the docker-ovs project, we introduced
for this exact purpose of testing an ovsdb client across various versions of the OVSDB schemas
and catching any incompatible usage at dev.
Also the use-case you mentioned will most likely impact the ovs-vsctl approach as well.
imho, having it managed using ovsdb (with a proper RFC) is far more solid and structured compared
to using an interface (ovs-vsctl) that doesn’t follow any particular specification.
As Dave suggested, we have a reasonable success with the Schema independent ovsdb library across
various versions of the Schemas and it pretty much scales to all ovsdb schemas (as defined by RFC 7047)
and not just to the Open_vSwtich schema.

>
>> Not initially.
>>
>> Firstly I haven't checked the code to make sure there is enough context
>> about the container available to the method that will write these values in
>> to OVSDB.
>>
>> Secondly, I'm not sure what to do re: "vm-id".
>> Yes we can re-use this for containers... but what about an environment that
>> has docker running on VMs instead of bare-metal, or a a mix of docker and
>> kvm or xen server?
>> Do we want to keep "vm-id" for VMs and "container-id" for containers to make
>> it easy to differntiate at the interface level?
>> If both "vm-id" and "container-id" are specified can we assume "vm-id" is a
>> reference to the parent-vm?
>> Should we just use "vm-id" and call it quits?
> I would start with having the ability to set the columns, we can hash
> out the key names when we get closer to acceptance.

Yes. Our work involves updating the external-ids with the relevant info.
and yes, lets work on the exact key-value pair details as we make progress.

[Gurucharan Shetty]

>
> Since every change to the schema comes with a version bump, this is handled by the
> docker-ovs based CI systems. If you look at the history of the docker-ovs project, we introduced
> for this exact purpose of testing an ovsdb client across various versions of the OVSDB schemas
> and catching any incompatible usage at dev.
> Also the use-case you mentioned will most likely impact the ovs-vsctl approach as well.

It shouldn't impact ovs-vsctl. ovs-vsctl is tied to the schema running
locally. OVS upgrades and downgrades automatically take care of schema
changes. Libvirt has been using ovs-vsctl to add VIFs to a OVS bridge
and set its external-ids for a few years now and we just had to make a
code addition once to Libvirt.

> imho, having it managed using ovsdb (with a proper RFC) is far more solid and structured compared
> to using an interface (ovs-vsctl) that doesn’t follow any particular specification.

IMHO, it really depends on the use case. If the use case is to just
add/delete ports and set external ids, then using ovs-vsctl means that
libcontainer needs a change in its code just once.

[Madhu Venugopal]

> On Oct 23, 2014, at 6:53 AM, Gurucharan Shetty <she...@nicira.com> wrote:
>
>>
>> Since every change to the schema comes with a version bump, this is handled by the
>> docker-ovs based CI systems. If you look at the history of the docker-ovs project, we introduced
>> for this exact purpose of testing an ovsdb client across various versions of the OVSDB schemas
>> and catching any incompatible usage at dev.
>> Also the use-case you mentioned will most likely impact the ovs-vsctl approach as well.
> It shouldn't impact ovs-vsctl. ovs-vsctl is tied to the schema running
> locally. OVS upgrades and downgrades automatically take care of schema
> changes. Libvirt has been using ovs-vsctl to add VIFs to a OVS bridge
> and set its external-ids for a few years now and we just had to make a
> code addition once to Libvirt.

I should have been more clearer. Yes the ovs-vsctl will not get affected because it is tied locally.
I meant the applications using the cli (to make use of the new Column).

>
>> imho, having it managed using ovsdb (with a proper RFC) is far more solid and structured compared
>> to using an interface (ovs-vsctl) that doesn’t follow any particular specification.
> IMHO, it really depends on the use case. If the use case is to just
> add/delete ports and set external ids, then using ovs-vsctl means that
> libcontainer needs a change in its code just once.

Yes. For simple scenarios, this will work. But, if we have to do more interesting stuffs,
even as simple as monitoring a bunch of table/column, the exec approach doesn’t scale too well.

[Gurucharan Shetty]

If I understand Madhu and Dave correctly, you are trying to have
ovs-docker as a separate repo/project and then provide the integration
for Open vSwitch through it. Is that the plan? I would actually prefer
that a straightforward integration exists natively in Docker code
base. Any extensions can be made through plugins.

[Madhu Venugopal]

> On Oct 27, 2014, at 11:30 AM, Gurucharan Shetty <she...@nicira.com> wrote:
>
> If I understand Madhu and Dave correctly, you are trying to have
> ovs-docker as a separate repo/project and then provide the integration
> for Open vSwitch through it. Is that the plan?

No. We are working on Native integration with Docker.
docker-ovs was a project that we worked on to provide user-space OVS in a container &
is not directly related to this discussion.

The discussion we were having was to use proper ovsdb natively instead of vsctl.

-Madhu

[Dave Tucker]

On 27 Oct 2014, at 18:30, Gurucharan Shetty wrote:

> If I understand Madhu and Dave correctly, you are trying to have
> ovs-docker as a separate repo/project and then provide the integration
> for Open vSwitch through it. Is that the plan?

No that's not the plan and is an incorrect assumption.
I linked you my PoC work in libcontainer and docker earlier in this
thread

docker-ovs is just OVS packaged to be deployed in a docker container.

> I would actually prefer
> that a straightforward integration exists natively in Docker code
> base. Any extensions can be made through plugins.

Precisely what we are proposing here.

-- Dave

[Gurucharan Shetty]

On Mon, Oct 27, 2014 at 11:39 AM, Dave Tucker <da...@dtucker.co.uk> wrote:
>
>
> On 27 Oct 2014, at 18:30, Gurucharan Shetty wrote:
>
>> If I understand Madhu and Dave correctly, you are trying to have
>> ovs-docker as a separate repo/project and then provide the integration
>> for Open vSwitch through it. Is that the plan?
>
>
> No that's not the plan and is an incorrect assumption.
> I linked you my PoC work in libcontainer and docker earlier in this thread
>
> docker-ovs is just OVS packaged to be deployed in a docker container.
>

Alright. I got a little confused with the continuous CI systems
through docker-ovs mentioned. Sorry about that.

[Dinesh Subhraveti]

This thread inspired me to create a tool called midware that addresses the issue of container applications racing with interfaces added by external tools.

https://github.com/Altiscale/midware

It lets you add additional network interfaces to the container *before* the application runs.  The interface could be assigned an IP address or it could just be left "raw" to be configured later (by init system, for example.)

Apart from consistent and general networking for containers, it provides a convenient place between Docker and the application to transparently perform any initialization or configuration before the application has a chance to run.

For example, instead of running:

docker run -d my-image

you'd run the following to give the container a preconfigured network interface bridged into br0 before the application has a chance to run.

midware -b br0 -i 10.10.10.2/24 -- -d my-image

Note that options to 'docker run' are simply passed down to midware.

The key idea is that it runs itself within the container and reaches out to the host through a reference to the root namespace device to setup host side config before running the target application.

It currently only supports Linux bridges as an example.  OVS bridges, macvlan interfaces etc. can be easily added.

All feedback and suggestions for improvements are welcome!

Thanks,
Dinesh

[Gurucharan Shetty]

On Tue, Oct 28, 2014 at 12:34 AM, Dinesh Subhraveti
<din...@altiscale.com> wrote:
> This thread inspired me to create a tool called midware that addresses the
> issue of container applications racing with interfaces added by external
> tools.
>
> https://github.com/Altiscale/midware

Thanks for sharing your code. The use of --entrypoint to load yourself
and be the first script to run inside a container is cool.

A few trivial comments on the script that I observed while running it
on Ubuntu 14.04:
* You may want to use /bin/bash instead of /bin/sh at the top of the
script as you seem to use BASH_SOURCE. Otherwise script fails.
* Ubuntu has /run/netns/root in the o/p of mount so your script fails.
* You need quotes around APP_ARGS
* Your script cannot handle arguments of the type passed to the
following docker run command.
docker run -d --net=none ubuntu:14.04 /bin/sh -c "while true; do echo
hello world; sleep 1; done"

I wonder whether sysadmins will be happy running their containers in
privileged mode. I wonder whether you can probably avoid running in
privileged mode if you create a dynamic script for the entrypoint that
simply waits for the existence of new network interfaces? I suppose
you need the privileged mode because you are trying to create network
interfaces from inside the container (I may have missed some other
reasons while reading the script.). You probably will get better
comments if you post your script to docker-dev separately instead of
this thread as many may not read the Open vSwitch specific agenda in
this thread. 'pipework --wait' tries to accomplish what you did but in
a different workflow.

[Vipin Jain]

Hi Dave,

I don't see any changes in:
https://github.com/dave-tucker/docker/tree/openvswitch_support

whereas following seem to do a dispatch of ovs-vsctl command with args built from the incoming function, which you said you'll replace with native interface I suppose.
https://github.com/dave-tucker/libcontainer/tree/openvswitch_support

Would it be possible to list out the overall flow of container and bridge creation with native OVS integration being talked about in this thread? Specifically the trigger of various events:
- Creation of bridge: perhaps when the docker daemon starts
- Creation of internal interfaces: perhaps just before the container creation in docker run
- If default policies/rules were needed to be installed, is it done outside the scope of docker/ovs integration? For example if I were to have some default rules installed before in the OVS before any container is spawned, I am thinking where is the window to do so and if it is outside docker (daemon or docker run), then it needs to be coordinated to be done after bridge creation and docker run. Because if default rules are not installed on the bridge (may be it is drop-all), technically the network isn't ready yet even though container interface may have been created.

One more follow-up on the native OVS integration: if the internal interfaces can be created using ovs-vsctl (i.e. from outside docker, either using a command or programmatic APIs), does it make sense to have the container orchestrator do the network creation (ovs and interface for the container) rather than this linked into the docker itself?

I appreciate the patience, if the questions sounds like obvious/basic to ovs experts.

Cheers,
-Vipin

[Gurucharan Shetty]

Recent changes to the upstream openvswitch project makes the 2 links
above invalid. I am updating here with valid links.
[3] https://github.com/openvswitch/ovs/blob/master/INSTALL.Docker.md
[4] https://github.com/openvswitch/ovs/blob/master/IntegrationGuide.md

Thanks!

[Dinesh Subhraveti]

Thanks for the comments and thanks for taking time to review the code as well.

It would be good to not require privilege but setting root net namespace within the container requires CAP_SYS_ADMIN.  An alternative would be to prepare the host side config while still running on the host and pass the other end of the veth pair into the container to wait on.  I was hoping to keep it such that the same utility can also be used within the container to create new interfaces.  But it is easy enough to check for access to root namespace and choose the right approach.

The objective is to provide a simple, transparent and consistent way to create containers with user-specified network interface, IP address and configuration.  I'll try a few more approaches and repost on docker-dev.

Thanks,

Dinesh

[Gurucharan Shetty]

>
> One more follow-up on the native OVS integration: if the internal interfaces
> can be created using ovs-vsctl (i.e. from outside docker, either using a
> command or programmatic APIs), does it make sense to have the container
> orchestrator do the network creation (ovs and interface for the container)
> rather than this linked into the docker itself?

I think what you suggest is a good alternative and has many advantages
over the current proposal. So in this case, one will create a network
interface asynchronously and then ask Docker client to add it as a
particular interface inside a container. This way, you get a chance to
pre-program your network and is good for short lived applications and
it is not Open vSwitch dependent. I am hoping that I am not missing
something obvious.

[Gurucharan Shetty]

I see that Madu and Dave have created good proposals for integration
of OVS with Docker.

I went ahead and created another proposal here based on Vipin's suggestion:
https://github.com/docker/docker/issues/8962

[Baohua Yang]

Hi dave

    Glad to hear that you've already worked on the topic.

    I am also looking solutions of it.

    I've many years of experience using and hacking ovs, and would like to contribute to this if possible.

[NITIN GOEL]

Hi ALL,

could you pls let me know from where i can get the latest source code for docker? 

thnaks in advance :)

[Satish Puranam]

Did you mean, https://github.com/docker/docker