Strange networking issue...

[Timothy Scoppetta]

Hey Folks,

I;m have a strange networking issue that I was hoping you guys could help with. Or maybe point me in the right direction. I have a very simple bridge set up on my vmhosts:

~# brctl show br0

bridge name bridge id STP enabled interfaces

br0 8000.bc305bedf538 yes eth0

eth1

eth2

Nothing fancy going on there, no NAT configured, etc. However I'm noticing that on some hosts (specifically windows hosts) incoming traffic is having it's source IP address rewritten to be that of the vmhost as opposed to the original sender of the packet. I can't imagine WHY it would be doing this. I have no NAT configured or anything like that, just a very basic bridge. However, I came across it while configuring zabbix and got the following error:

260:20120823:142808.938 Listener error: Connection from [128.32.224.132] rejected. Allowed server is [imon-0.EECS.Berkeley.EDU,imon-1.EECS.Berkeley.EDU,imon-2.EECS.Berkeley.EDU].

This (the error message, not the issue) makes perfect sense because that's the IP address of the vmhost itself. I went ahead and captured some traffic and it appears that's exactly whats happening. I'm having trouble tacking down WHY that ip address is getting changed at all. I know this is a bit vague but... I'm not quite sure where else to look at this point.

Any ideas?

Tim

--

Timothy Scoppetta
UC Berkeley

E: t...@eecs.berkeley.edu
P: 845-459-3002

[Guido Trotter]

Hi Timothy,

Could you please show us a brctl show on the node on which the Windows System is up, and perhaps an ip link list, and an iptables -t nat -L just to be sure?

Also can you show a gnt-instance info, and the setting for proxy arp in /proc/sys/net/... ?

Thanks,

Guido

[Timothy Scoppetta]

So, while gathering up this information I realized something. Apparently the windows host using tap0? I'm not entirely sure why (as the device is listed as br0 in gnt-instance info) but if I take the tap interface out of the bridge things come tumbling down. is this the expected behavior for a windows host? I've never seen my linux instances behave like this.

All info requested below, let me know if I missed anything or if anything looks out of place...

vmhost-0:~# brctl show

bridge name bridge id STP enabled interfaces

br0 8000.b221a3821727 yes eth0

eth1

eth2

eth3

tap0

tap1

vmhost-0:~# ip link list

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP mode DEFAULT qlen 1000

    link/ether bc:30:5b:ee:06:30 brd ff:ff:ff:ff:ff:ff

3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br0 state DOWN mode DEFAULT qlen 1000

    link/ether bc:30:5b:ee:06:31 brd ff:ff:ff:ff:ff:ff

4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br0 state DOWN mode DEFAULT qlen 1000

    link/ether bc:30:5b:ee:06:32 brd ff:ff:ff:ff:ff:ff

5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP mode DEFAULT qlen 1000

    link/ether bc:30:5b:ee:06:33 brd ff:ff:ff:ff:ff:ff

6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000

    link/ether a0:36:9f:07:12:c4 brd ff:ff:ff:ff:ff:ff

7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000

    link/ether a0:36:9f:07:12:c6 brd ff:ff:ff:ff:ff:ff

8: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT 

    link/ether bc:30:5b:ee:06:30 brd ff:ff:ff:ff:ff:ff

10: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 500

    link/ether b2:21:a3:82:17:27 brd ff:ff:ff:ff:ff:ff

11: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 500

    link/ether ca:ef:89:13:00:da brd ff:ff:ff:ff:ff:ff

vmhost-0:~# iptables -t nat -L

Chain PREROUTING (policy ACCEPT)

target     prot opt source               destination         

Chain INPUT (policy ACCEPT)

target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)

target     prot opt source               destination         

MASQUERADE  all  --  anywhere             anywhere    

vmhost-1:~# gnt-instance info dc-dev

Instance name: dc-dev.eecs.berkeley.edu

UUID: 0033c0ca-5eb9-40f2-b9f9-e828e678bdd9

Serial number: 10

Creation time: 2012-08-15 16:19:20

Modification time: 2012-08-20 15:14:27

State: configured to be up, actual state is up

  Nodes:

    - primary: vmhost-0.eecs.berkeley.edu

      group: default (UUID 6e1b2abd-69c2-4b31-b2df-f96aa5122d76)

    - secondaries: vmhost-1.eecs.berkeley.edu (group default, group UUID 6e1b2abd-69c2-4b31-b2df-f96aa5122d76)

  Operating system: image+default

  Allocated network port: 11028

  Hypervisor: kvm

    - console connection: vnc to 127.0.0.1:11028 (node vmhost-0.eecs.berkeley.edu) (display 5128)

    - acpi: default (True)

    - boot_order: default (disk)

    - cdrom2_image_path: default ()

    - cdrom_disk_type: default ()

    - cdrom_image_path: default ()

    - cpu_mask: default (all)

    - disk_cache: default (default)

    - disk_type: default (paravirtual)

    - floppy_image_path: default ()

    - initrd_path: default ()

    - kernel_args: default (ro)

    - kernel_path: default ()

    - keymap: default ()

    - kvm_flag: default ()

    - mem_path: default ()

    - migration_downtime: default (30)

    - nic_type: default (paravirtual)

    - reboot_behavior: default (reboot)

    - root_path: default (/dev/vda1)

    - security_domain: default ()

    - security_model: default (none)

    - serial_console: default (True)

    - spice_bind: default ()

    - spice_image_compression: default ()

    - spice_ip_version: default (0)

    - spice_jpeg_wan_compression: default ()

    - spice_password_file: default ()

    - spice_playback_compression: default (True)

    - spice_streaming_video: default ()

    - spice_tls_ciphers: default (HIGH:-DES:-3DES:-EXPORT:-ADH)

    - spice_use_tls: default (False)

    - spice_use_vdagent: default (True)

    - spice_zlib_glz_wan_compression: default ()

    - usb_mouse: default ()

    - use_chroot: default (False)

    - use_localtime: default (False)

    - vhost_net: default (False)

    - vnc_bind_address: 127.0.0.1

    - vnc_password_file: default ()

    - vnc_tls: default (False)

    - vnc_x509_path: default ()

    - vnc_x509_verify: default (False)

  Hardware:

    - always_failover: default (False)

    - auto_balance: default (True)

    - maxmem: 1024

    - memory: default (1024)

    - minmem: 1024

    - spindle_use: default (1)

    - vcpus: default (1)

    - NICs:

      - nic/0: MAC: AA:00:00:71:FF:D2, IP: None, mode: bridged, link: br0

  Disk template: drbd

  Disks:

    - disk/0: drbd8, size 20.0G

      access mode:  rw

      nodeA:        vmhost-0.eecs.berkeley.edu, minor=2

      nodeB:        vmhost-1.eecs.berkeley.edu, minor=2

      port:         11030

      auth key:     947f50988193e7217ef1f4f630c6454ddc1ab392

      on primary:   /dev/drbd2 (147:2) in sync, status ok

      on secondary: /dev/drbd2 (147:2) in sync, status ok

      child devices:

        - child 0: lvm, size 20.0G

          logical_id:   ganeti/b6cdae80-3134-4ecd-a426-a9519816edf4.disk0_data

          on primary:   /dev/ganeti/b6cdae80-3134-4ecd-a426-a9519816edf4.disk0_data (254:4)

          on secondary: /dev/ganeti/b6cdae80-3134-4ecd-a426-a9519816edf4.disk0_data (254:4)

        - child 1: lvm, size 128M

          logical_id:   ganeti/b6cdae80-3134-4ecd-a426-a9519816edf4.disk0_meta

          on primary:   /dev/ganeti/b6cdae80-3134-4ecd-a426-a9519816edf4.disk0_meta (254:5)

          on secondary: /dev/ganeti/b6cdae80-3134-4ecd-a426-a9519816edf4.disk0_meta (254:5)

    - disk/1: drbd8, size 20.0G

      access mode:  rw

      nodeA:        vmhost-0.eecs.berkeley.edu, minor=3

      nodeB:        vmhost-1.eecs.berkeley.edu, minor=3

      port:         11031

      auth key:     843408ec249db76dd6b5cae72ca13dd258c06c61

      on primary:   /dev/drbd3 (147:3) in sync, status ok

      on secondary: /dev/drbd3 (147:3) in sync, status ok

      child devices:

        - child 0: lvm, size 20.0G

          logical_id:   ganeti/a3b0afde-c9bd-4dda-bdf0-b6ea818df5fb.disk1_data

          on primary:   /dev/ganeti/a3b0afde-c9bd-4dda-bdf0-b6ea818df5fb.disk1_data (254:6)

          on secondary: /dev/ganeti/a3b0afde-c9bd-4dda-bdf0-b6ea818df5fb.disk1_data (254:6)

        - child 1: lvm, size 128M

          logical_id:   ganeti/a3b0afde-c9bd-4dda-bdf0-b6ea818df5fb.disk1_meta

          on primary:   /dev/ganeti/a3b0afde-c9bd-4dda-bdf0-b6ea818df5fb.disk1_meta (254:7)

          on secondary: /dev/ganeti/a3b0afde-c9bd-4dda-bdf0-b6ea818df5fb.disk1_meta (254:7)

Thanks!

Tim

[Timothy Scoppetta]

Oh, ip link list was slightly out of date as I removed tap0 and tap1 and then put them back in when I realized my windows host had stopped networking:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP mode DEFAULT qlen 1000

    link/ether bc:30:5b:ee:06:30 brd ff:ff:ff:ff:ff:ff

3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br0 state DOWN mode DEFAULT qlen 1000

    link/ether bc:30:5b:ee:06:31 brd ff:ff:ff:ff:ff:ff

4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br0 state DOWN mode DEFAULT qlen 1000

    link/ether bc:30:5b:ee:06:32 brd ff:ff:ff:ff:ff:ff

5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP mode DEFAULT qlen 1000

    link/ether bc:30:5b:ee:06:33 brd ff:ff:ff:ff:ff:ff

6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000

    link/ether a0:36:9f:07:12:c4 brd ff:ff:ff:ff:ff:ff

7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000

    link/ether a0:36:9f:07:12:c6 brd ff:ff:ff:ff:ff:ff

8: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT 

    link/ether b2:21:a3:82:17:27 brd ff:ff:ff:ff:ff:ff

10: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN mode DEFAULT qlen 500

    link/ether b2:21:a3:82:17:27 brd ff:ff:ff:ff:ff:ff

11: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN mode DEFAULT qlen 500

    link/ether ca:ef:89:13:00:da brd ff:ff:ff:ff:ff:ff

[Guido Trotter]

Hi Timothy,

It's normal for a kvm guests to use a tap interface : br0 is just the bridge to connect it to: so yes, tap0 must be enslaved to br0, usually by the kvm ifup script provided by ganeti : did you unenslave it or override the script?

Also as you see your postrouting table is masquerading all routed (l3) traffic by default, so you are natting.

So the behaviour you see is completely expected, for how your system is configured.

Thanks,

Guido

[Timothy Scoppetta]

I noticed the postrouting rule and removed it and everything seems to be better now. Oddly it was only that one vmhost (out of seven) that was set up like that. Someone's been monkeying with my cluster again.

Thanks for the help :)

-Tim