Adding eth0 (control interface) to an OVS bridge

[Jacob Chappell]

Hello, everyone.

A quick and simple question this time. Does anyone know the procedure for adding the eth0 interface to an OVS bridge (and moving eth0's IP address over to that OVS bridge)? I have another node with a private link that I can use to SSH to the node, so it's okay if eth0 goes down temporarily.

End goal: All traffic that comes in on eth0 is first processed by OpenFlow rules before being sent to the kernel.

I've already tried the obvious procedure:
# 1. Create the bridge.
sudo ovs-vsctl add-br br-eth0
# 2. Remove the IP address from eth0.
sudo ip -4 addr del dev eth0
# 3. Add eth0 to the bridge.
sudo ovs-vsctl add-port br-eth0 eth0
# 4. Add eth0's old IP to the bridge.
sudo ip -4 addr add xyz/xyz broadcast xyz dev br-eth0
# 5. Restore "default" route.
sudo ip -4 route add <ARGS HERE>
# 6. Add the two routes for switching between the kernel and the interface.
sudo ovs-ofctl add-flow br-eth0 priority=1,in_port=LOCAL,actions=output:1
sudo ovs-ofctl add-flow br-eth0 priority=1,in_port=1,actions=LOCAL

Unfortunately, unlike any other interface, it seems that eth0 does not function correctly after being added to a bridge. One thing that happens immediately is that sudo whines with the following warning:
sudo: unable to resolve host ovs.jdc-test2.ch-geni-net.lan.sdn.uky.edu

Any thoughts on how to successfully accomplish this?

Thanks,
Jacob Chappell

[Nicholas Bastin]

On Thu, Aug 13, 2015 at 1:11 PM, Jacob Chappell <chapp...@gmail.com> wrote:
> A quick and simple question this time. Does anyone know the procedure for
> adding the eth0 interface to an OVS bridge (and moving eth0's IP address
> over to that OVS bridge)? I have another node with a private link that I can
> use to SSH to the node, so it's okay if eth0 goes down temporarily.

Why do you want to do this? In almost all circumstances this is a bad idea.

--
Nick

[Jacob Chappell]

Hi Nick,

We need the traffic on the data network to be able to get to the outside world, and vice versa. We want to filter that traffic with OpenFlow though.

If GENI allowed a public IP on interfaces other than eth0, we'd go that route. As far as I can tell though, this is our only choice.

Jacob Chappell

> --
> GENI Users is a community supported mailing list, so please help by responding to questions you know the answer to.
>
> If this is your first time posting a question to this list, please review http://groups.geni.net/geni/wiki/GENIExperimenter/CommunityMailingList
> ---
> You received this message because you are subscribed to a topic in the Google Groups "GENI Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/geni-users/bevwuf5so3g/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to geni-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ryan Izard]

Hi Jacob,

FWIW, I do the exact same thing with OVS on our control interfaces (outside the context of GENI though). If you'd like, I can dig out an example OVS configuration for you that you can compare against.

I allow Linux to handle some traffic on the OVS bridge on our control interface (e.g. me SSHing into the machine). I do this by sending the applicable traffic to/from the LOCAL port of OVS with flows. From the Linux side of things, the IP and route(s) are on the OVS bridge interface so Linux can pass traffic into and receive from the OVS bridge. We handle that traffic with OpenFlow though within the OVS.

Thanks,

Ryan 

Ryan Izard
Graduate Research/Teaching Assistant
306B Fluor Daniel Building
ECE Department, Clemson University
Clemson, SC 29634
riz...@g.clemson.edu

---------------------------------------------------

Big Switch Networks

ryan....@bigswitch.com

You received this message because you are subscribed to the Google Groups "GENI Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geni-users+...@googlegroups.com.

[Jacob Chappell]

Hi Ryan,

Good to see you over here :) I would very much appreciate your example config if it's not too much trouble.

Jacob Chappell

[Ryan Izard]

Sure, I've attached an example script I copied from our machine. br_campus is the OVS connected to the public interface (physically eth1). You can ignore any VLAN and tunnel stuff if it's not relevant to you. I removed our specific public IPs, since they are public after all :-)

Here is the route table and OVS config:

# route -n

Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

10.3.11.1       0.0.0.0         255.255.255.255 UH    0      0        0 br_tun

192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 br_tun

10.3.10.0       0.0.0.0         255.255.255.0   U     0      0        0 br_v760

10.40.84.0      0.0.0.0         255.255.255.0   U     0      0        0 br_v4084

10.40.85.0      0.0.0.0         255.255.255.0   U     0      0        0 br_v4085

<camp ntwk>     0.0.0.0         255.255.255.0   U     0      0        0 br_campus

10.40.83.0      0.0.0.0         255.255.255.0   U     0      0        0 br_v4083

10.43.0.0       0.0.0.0         255.255.0.0     U     0      0        0 br_v4087

0.0.0.0         <camp gw>       0.0.0.0         UG    0      0        0 br_campus

# ovs-vsctl show

157ebb0c-78e8-4271-b9cd-a5de20033574

    Bridge br_parent

        Controller "tcp:<controller ip:port>"

        fail_mode: standalone

        Port "eth0.4085"

            tag: 4085

            Interface "eth0.4085"

        Port "eth0.760"

            tag: 760

            Interface "eth0.760"

        Port "eth0.4084"

            tag: 4084

            Interface "eth0.4084"

        Port "br_v4087"

            tag: 4087

            Interface "br_v4087"

                type: internal

        Port "br_v760"

            tag: 760

            Interface "br_v760"

                type: internal

        Port "eth0.4083"

            tag: 4083

            Interface "eth0.4083"

        Port "br_v4083"

            tag: 4083

            Interface "br_v4083"

                type: internal

        Port br_parent

            Interface br_parent

                type: internal

        Port "br_v4084"

            tag: 4084

            Interface "br_v4084"

                type: internal

        Port "eth0.4087"

            tag: 4087

            Interface "eth0.4087"

        Port "br_v4085"

            tag: 4085

            Interface "br_v4085"

                type: internal

    Bridge br_tun

        Controller "tcp:<controller ip:port>"

        fail_mode: standalone

        Port "gre0"

            Interface "gre0"

                type: gre

                options: {remote_ip="<remote ip>"}

        Port br_tun

            Interface br_tun

                type: internal

    Bridge "br_campus"

        Controller "tcp:<controller ip:port>"

        fail_mode: standalone

        Port "br_campus"

            Interface "br_campus"

                type: internal

        Port "eth1"

            Interface "eth1"

Ryan Izard
Graduate Research/Teaching Assistant
306B Fluor Daniel Building
ECE Department, Clemson University
Clemson, SC 29634
riz...@g.clemson.edu

---------------------------------------------------

Big Switch Networks

ryan....@bigswitch.com

[Jacob Chappell]

Hi Ryan,

Thanks so much for the script. Aside from using the ip commands (instead of e.g. route, ifconfig, etc.), I can't find anything you've done that I haven't. I ignored all the VLAN/tunnel stuff as you said. Ultimately, that strips down the script to this:

$OVS_VSCTL add-br br_campus
$OVS_VSCTL set-fail-mode br_campus standalone
$OVS_VSCTL set bridge br_campus other-config:datapath-id=0000000000000101
$OVS_VSCTL set-controller br_campus tcp:$OVS_controllerIP
$OVS_VSCTL add-port br_campus eth1
ifconfig eth1 0.0.0.0 up
ifconfig br_campus <ip of br_campus>/24
echo "1" > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o br_campus -j SNAT --to <ip of br_campus>
route add default gw <gw of br_campus> dev br_campus

​The only thing in that list I hadn't done before was adding the NAT entry in the ip tables. I went ahead and did that, but to no avail. It behaves so bizarrely. If I ping the network from offsite (while running tcpdump), it takes about 10 seconds and then ICMP echo requests start coming in. Replies also get generated and sent out port 1 (which is attached to eth0, the control interface). What happens at that point, I don't know. The replies never make it back to the pinging host.

You said you were outside the context of GENI. Maybe GENI configures eth0 such that additional steps need to be taken?​

​Jacob Chappell​

[Ryan Izard]

Do you have persistent flows installed for your control traffic and ping? I'd try and insert those statically to make sure OVS is at least forwarding the packets in the right directions -- you can watch the packet/byte counts.

Ryan Izard
Graduate Research/Teaching Assistant
306B Fluor Daniel Building
ECE Department, Clemson University
Clemson, SC 29634
riz...@g.clemson.edu

---------------------------------------------------

Big Switch Networks

ryan....@bigswitch.com

[Jacob Chappell]

I went ahead and added these two flows:
 cookie=0x0, duration=94.223s, table=0, n_packets=35, n_bytes=3430, idle_age=0, priority=10,icmp,in_port=1 actions=LOCAL
 cookie=0x0, duration=72.625s, table=0, n_packets=35, n_bytes=3430, idle_age=0, priority=10,icmp,in_port=LOCAL actions=output:1

Pinging from off site reveals that both flows are being hit equally, and the counter goes up at the rate I would expect for ping (about 2 per second). The weird thing is, when I'm running tcpdump (filtering by icmp) on the interface as I'm pinging, both of the flows are being incremented, but tcpdump doesn't see any traffic until after about 10 seconds in.

I further went ahead and filtered by echo vs. reply (via icmp_type) also, to make sure the incoming echo packet wasn't just being duplicated:
 cookie=0x0, duration=29.054s, table=0, n_packets=18, n_bytes=1764, idle_age=0, priority=20,icmp,in_port=1,icmp_type=8 actions=LOCAL
 cookie=0x0, duration=25.516s, table=0, n_packets=18, n_bytes=1764, idle_age=0, priority=20,icmp,in_port=LOCAL,icmp_type=0 actions=output:1

The flows are still counting up equally, meaning a reply is being generated for every request.

Jacob Chappell

[Nicholas Bastin]

On Fri, Aug 14, 2015 at 2:45 AM, Jacob Chappell <chapp...@gmail.com> wrote:
> We need the traffic on the data network to be able to get to the outside world, and vice versa.
> We want to filter that traffic with OpenFlow though.

You need to PAT this traffic anyhow, though, at the very least. You
should be able to do your OF processing from the other OVS ports
before you then forward the packet to NORMAL or LOCAL which injects it
up the stack into the kernel for linux forwarding.

The testbed is not built to carry large amounts of traffic to the
internet from the data plane through the control interface (and in
fact that is a violation of most campuses' agreements with Internet2),
so in general you should keep this kind of traffic to as little as
possible. It also affects the ability of the testbed infrastructure
to distribute disk images on the local rack, as well as boot new
experiments, as all of these actions are carried out on the same
control network you are using to reach the internet.

> If GENI allowed a public IP on interfaces other than eth0, we'd go that route. As far as I can tell
> though, this is our only choice.

Just because you want traffic to go between your OVS bridge and eth0
doesn't mean this is your only choice - as above, you just need to
send it up to the local stack. Adding eth0 (an IP only interface) to
a layer 2 bridge (OVS) is only going to end up creating trouble. You
cannot have a public IP on an interface other than eth0, and this is
very important, because eth0 is on a *physically* separate network
from all the rest of your interfaces, and is on the only network that
has a path to the internet.

It is also important to note that eth0 is 100Mbits, and cannot support
jumbo frames, which means any bridge you attach it to will lose its'
ability to support jumbo frames, as well as any other feature that
doesn't work on 100Mbit networks.

--
Nick

[Jacob Chappell]

Hi Nick,

Thank you for your detailed comments. I'm going to go ahead and try a workaround with NAT to see if I can avoid messing with eth0 at all. I don't think the amount of traffic we want to send over the public network is extreme, but I will definitely keep in mind that Universities have limits (I didn't know about the whole Internet2 agreement).

Jacob Chappell