setting static IP on tap interface

[Robert Geiger]

Goal: want to set up external communication to/from containers so that they can communicate on a private network

Host Env: RHEL 6.4; qemu-kvm; one public interface, one private network interface (no DHCP)

I have configured my host to bring up a bridge interface tap, giving me a second interface:

eth1: flags=103<UP,BROADCAST,RUNNING>  mtu 1500

        inet  0.0.0.0  netmask 255.255.255.0  broadcast 0.0.0.255

        ether de:ad:be:ef:e6:ee

        RX packets 228  bytes 13848 (13.8 KiB)

        Rx errors  0  dropped 0

        TX packets 1  bytes 293 

        Tx errors  0  dropped 0 collisions 0

Used this in qemu command to get the tap configured:

-device virtio-net-pci,netdev=un0 -device virtio-net-pci,netdev=net0,mac=DE:AD:BE:EF:E6:EE -netdev tap,id=net0

I do not have DHCP on the private network, so OSv would not get an address via DHCP.

I am unable to assign this a static IP so that it can communicate ("ifconfig eth1 172.28.8.33 netmask 255.255.248.0: exception: osv_start_if failed")

Is there a way to assign a static address to an interface from within a container? (or is there an entirely different way this should be done?)

Thanks,

Bob

[Dor Laor]

On Wed, Jun 25, 2014 at 3:55 AM, Robert Geiger <rge...@gopivotal.com> wrote:

Goal: want to set up external communication to/from containers so that they can communicate on a private network

Host Env: RHEL 6.4; qemu-kvm; one public interface, one private network interface (no DHCP)

I have configured my host to bring up a bridge interface tap, giving me a second interface:

eth1: flags=103<UP,BROADCAST,RUNNING>  mtu 1500

        inet  0.0.0.0  netmask 255.255.255.0  broadcast 0.0.0.255

        ether de:ad:be:ef:e6:ee

        RX packets 228  bytes 13848 (13.8 KiB)

        Rx errors  0  dropped 0

        TX packets 1  bytes 293 

        Tx errors  0  dropped 0 collisions 0

Used this in qemu command to get the tap configured:

-device virtio-net-pci,netdev=un0 -device virtio-net-pci,netdev=net0,mac=DE:AD:BE:EF:E6:EE -netdev tap,id=net0

I do not have DHCP on the private network, so OSv would not get an address via DHCP.

I am unable to assign this a static IP so that it can communicate ("ifconfig eth1 172.28.8.33 netmask 255.255.248.0: exception: osv_start_if failed")

When I executed qemu using the standard way (using scripts/run.py -nv) it booted OSv w/ DHCP but afterwards

I managed to override the IP settings and configure a static IP address. It did return an error message but 

the configuration did work (tested by pinging to that subnet). However, apparently we have bugs there and also missing

some functionality to set a static IP. Our idea to do that is to use cloudInit which will pull a config file that will contain the static IP setup. Amnon (CCed) already sent the cloudInit series (generic, w/o the static ip setup). We should add such api (Amnon please open a matching github issue(s))

[/]% ifconfig 

ifconfig argc=1 argv[0]=/tools/ifconfig.so

lo0: flags=37777700111<UP,LOOPBACK,RUNNING,MULTICAST,PPROMISC,MONITOR,STATICARP>  mtu 16384

        inet  127.0.0.1  netmask 255.0.0.0  broadcast 

        RX packets 0  bytes 0 

        Rx errors  0  dropped 0

        TX packets 0  bytes 0 

        Tx errors  0  dropped 0 collisions 0

eth0: flags=103<UP,BROADCAST,RUNNING>  mtu 1500

        inet  192.168.122.89  netmask 255.255.255.0  broadcast 192.168.122.255

        ether 52:54:0:12:34:56

        RX packets 9  bytes 1262 (1.3 KiB)

        Rx errors  0  dropped 0

        TX packets 3  bytes 640 

        Tx errors  0  dropped 0 collisions 0

[/]% ifconfig eth0 3.3.3.3 255.255.255.0

ifconfig eth0 3.3.3.3 255.255.255.0: failed to set ip

[/]% ifconfig 

ifconfig argc=1 argv[0]=/tools/ifconfig.so

lo0: flags=37777700111<UP,LOOPBACK,RUNNING,MULTICAST,PPROMISC,MONITOR,STATICARP>  mtu 16384

        inet  127.0.0.1  netmask 255.0.0.0  broadcast 

        RX packets 0  bytes 0 

        Rx errors  0  dropped 0

        TX packets 0  bytes 0 

        Tx errors  0  dropped 0 collisions 0

eth0: flags=103<UP,BROADCAST,RUNNING>  mtu 1500

        inet  3.3.3.3  netmask 255.255.255.0  broadcast 3.3.3.255

        ether 52:54:0:12:34:56

        RX packets 10  bytes 1332 (1.3 KiB)

        Rx errors  0  dropped 0

        TX packets 4  bytes 682 

        Tx errors  0  dropped 0 collisions 0

Is there a way to assign a static address to an interface from within a container? (or is there an entirely different way this should be done?)

Thanks,

Bob

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Geiger]

Thanks; yes, I can set the eth0 address as well, but cannot set the eth1 bridged interface that is on the private network. I need the eth0 iface for it to come up with DHCP and give me shell, but want to set up the private eth1 interface on the tap to allow external network access. Will study run.sh more and see if I can figure it out.

Bob

--

Robert Geiger

Architect, Pivotal HD

408-833-4527

[Robert Geiger]

OK, got it:

ifconfig eth1 172.28.8.33 netmask 255.255.248.0   fails

ifconfig eth1 172.28.8.30 255.255.248.0   works, so I had the incorrect command format. Still need to make it reachable, but can set the address now. 

Thanks,

Bob

[Dor Laor]

On Wed, Jun 25, 2014 at 10:27 PM, Robert Geiger <rge...@gopivotal.com> wrote:

OK, got it:

ifconfig eth1 172.28.8.33 netmask 255.255.248.0   fails

ifconfig eth1 172.28.8.30 255.255.248.0   works, so I had the incorrect command format. Still need to make it reachable, but can set the address now. 

Can you please specify how you have two eth's ? Did you spawn qemu w/ two NICs?

When I manually set an ip/netmask addr on eth0 inside OSv and I configure another ip with the

same subnet mask on the host bridge interface virbr0 I can ping each of them successfully.

[Robert Geiger]

We (Roman S. and myself) had trouble using run.sh on our RHEL boxes. I think I'm going to revisit that; the command used was:

qemu-system-x86_64  -m 1024 -smp 2 -device virtio-blk-pci,id=blk0,bootindex=0,drive=hd0 -drive file=//root/.capstan/instances/qemu/cloudius-osv/disk.qcow2,if=none,id=hd0,aio=native,cache=none  -chardev stdio,mux=on,id=stdio,signal=off -device isa-serial,chardev=stdio -netdev user,id=un0,net=192.168.122.0/24,host=192.168.122.1 -device virtio-net-pci,netdev=un0 -device virtio-net-pci,netdev=net0,mac=DE:AD:BE:EF:E6:EE -netdev tap,id=net0 -chardev socket,id=charmonitor,path=/root/.capstan/instances/qemu/cloudius-osv/osv.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -enable-kvm -cpu host,+x2apic

With the tap being set-up by creating a bridge and tunnel as in qemu instructions as:

#!/bin/sh

set -x

switch=br100

if [ -n "$1" ];then

        /usr/bin/sudo /usr/sbin/tunctl -u `whoami` -t $1

        /usr/bin/sudo /sbin/ip link set $1 up

        sleep 0.5s

        /usr/bin/sudo /usr/sbin/brctl addif $switch $1

        exit 0

else

        echo "Error: no interface specified"

        exit 1

fi

Bob

[Dor Laor]

On Thu, Jun 26, 2014 at 12:51 AM, Robert Geiger <rge...@gopivotal.com> wrote:

We (Roman S. and myself) had trouble using run.sh on our RHEL boxes. I think I'm going to revisit that; the command used was:

qemu-system-x86_64  -m 1024 -smp 2 -device virtio-blk-pci,id=blk0,bootindex=0,drive=hd0 -drive file=//root/.capstan/instances/qemu/cloudius-osv/disk.qcow2,if=none,id=hd0,aio=native,cache=none  -chardev stdio,mux=on,id=stdio,signal=off -device isa-serial,chardev=stdio -netdev user,id=un0,net=192.168.122.0/24,host=192.168.122.1 -device virtio-net-pci,netdev=un0 -device virtio-net-pci,netdev=net0,mac=DE:AD:BE:EF:E6:EE -netdev tap,id=net0 -chardev socket,id=charmonitor,path=/root/.capstan/instances/qemu/cloudius-osv/osv.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -enable-kvm -cpu host,+x2apic

Why do you need two NICs?

[Robert Geiger]

Without the user NIC defined I don't get console up; possibly due to no DHCP being found on the net interface so hanging in set-up. If I set the arguments and qemu-ifup.sh exactly as in run.sh same behavior- I can get eth1 and assign an address to it, but cannot communicate with it.

Bob

[Dor Laor]

On Thu, Jun 26, 2014 at 11:59 PM, Robert Geiger <rge...@gopivotal.com> wrote:

Without the user NIC defined I don't get console up; possibly due to no DHCP being found on the net interface so hanging in set-up. If I set the arguments and qemu-ifup.sh exactly as in run.sh same behavior- I can get eth1 and assign an address to it, but cannot communicate with it.

Got it. Good idea to hack around it. Once we complete the static ip support it

won't be needed. I just booted w/ dhcp and change the address of the very same interface but

if it works for you, no problems

[Pekka Enberg]

On Thu, Jun 26, 2014 at 11:59 PM, Robert Geiger <rge...@gopivotal.com> wrote:
> Without the user NIC defined I don't get console up; possibly due to no DHCP
> being found on the net interface so hanging in set-up. If I set the
> arguments and qemu-ifup.sh exactly as in run.sh same behavior- I can get
> eth1 and assign an address to it, but cannot communicate with it.

Bob, were you able to resolve this? Vlad, Asias, is there anything in
virtio-net that assumes just one NIC?

- Pekka

[Robert Geiger]

We have not resolved completely yet; we have tried two things in our environment so far:

- bridge/tap method: we are able to bring up one or two interfaces and see the DHCP requests, but no replies are being recorded and the server running on the interface is not logging any incoming DHCP requests.

- Macvtap method: DHCP server on the interface gets and responds to the request to assign an address, reply shows up when we sniff the tap interface, OSv does not seem to pick up the reply.

It is possible the problems are related to the drivers for the 10Gb NIC's we are using, though unlikely. Also it does not appear to be an issue with the # of interfaces as OSv seems to handle 2 interfaces correctly.

Bob

[Roman Shaposhnik]

On Tue, Jul 8, 2014 at 9:49 AM, Robert Geiger <rge...@gopivotal.com> wrote:
> - Macvtap method: DHCP server on the interface gets and responds to the
> request to assign an address, reply shows up when we sniff the tap
> interface, OSv does not seem to pick up the reply.

Actually, on that note: I'd be really grateful if anybody on this list can give
us any hints or advice. I gotta be honest with you though: at this point it
has absolutely *nothing* to do with OSv (which means we're essentially
hitting you for free Linux kernel networking advice ;-)).

Like Bob said, we're trying the macvtap/macvlan route and there's how
it breaks for us (I'll use macvlan in this example, but macvtap breaks
exactly the same way):
[host a] # ip link add link eth2 macvlan2 type macvlan mode bridge
[host a] # dhclient -v -4 macvlan2
DHCPDISCOVER on macvlan2 to 255.255.255.255 port 67 interval 3
(xid=0x2a943d62)
DHCPDISCOVER on macvlan2 to 255.255.255.255 port 67 interval 3
(xid=0x2a943d62)
....
No DHCPOFFERS received.
No working leases in persistent database - sleeping.

Meanwhile on host b where we run our DHCP server I see this:
Jul 8 10:07:52 sjc-w31 dhcpd: DHCPDISCOVER from f6:23:f4:2f:1e:61 via eth2
Jul 8 10:07:53 sjc-w31 dhcpd: DHCPOFFER on 172.28.8.100 to
f6:23:f4:2f:1e:61 via eth2
Jul 8 10:08:00 sjc-w31 dhcpd: DHCPDISCOVER from f6:23:f4:2f:1e:61 via eth2
Jul 8 10:08:00 sjc-w31 dhcpd: DHCPOFFER on 172.28.8.100 to
f6:23:f4:2f:1e:61 via eth2

Which means that packets are actually traveling up and down the
network, but for some reason something somewhere doesn't
quite work.

iptables/ebtable are emtpy.

Here's how ip link on host a looks like:

4: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq
state UP qlen 1000
link/ether 00:05:33:48:92:32 brd ff:ff:ff:ff:ff:ff
32: macvlan2@eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
qdisc noqueue state UNKNOWN
link/ether f6:23:f4:2f:1e:61 brd ff:ff:ff:ff:ff:ff

Thanks,
Roman.

[Asias He]

Are you trying this on VirtualBox with bridge mode? It does not work
for me neither.

I switch to VMware Fusion with bridge mode. I tried on CentOS 6. It works fine.

iptables -F
ifconfig eth1 down
ip link add link eth1 v1 type macvlan mode bridge
ip link set eth1 up
ip link set v1 up
ifconfig v1 192.168.6.10

ip link add link eth1 v2 type macvlan mode bridge
ip link set v2 up
ifconfig v2 192.168.6.20


ip link add link eth1 v3 type macvtap mode bridge
ip link set v3 up
ifconfig v3 192.168.6.30

I can communicate v1 v2 v3 on another physical machine as well as on
the host where I ran the VM.



--
Asias

[Asias He]

BTW, Try this with kvm guest + macvtap:

# ip link add link eth0 name v0 type macvtap mode bridge
# ip link set v0 address da:4e:17:88:42:b1 up
# idx=`ip link show v0 | grep mtu| awk -F":" '{print $1}'`
# kvm -net nic,macaddr=da:4e:17:88:42:b1 -net tap,fd=3 -hda
linux.img 3<>/dev/tap${idx}

Note, you are not supposed to communicate from host to guest using the
underneath physical device eth0. If you want guest/host communication
you can create a macvlan device and set the ip as above.




--
Asias