Communication Issue
55 views
Skip to first unread message
hll.harr...@gmail.com
Jun 6, 2016, 4:01:17 PM6/6/16
to GENI Users, Harrison Lee Ledford, Xenia Mountrouidou
GENI Portal Account, Jacks, COSC460-Simple, Kentucky ProtoGENI
I am doing research, but I keep running into an issue. I have a topology that consists of two nodes and two switches. Each node has two paths; one path to each switch. The switches connect the two switch but are not connected to one another. I have attached a picture of my topology for a clearer idea.
My problem is when trying to send traffic. When I run an iperf server on node bob and bind it to bob's eth1 (10.10.4.1), and then run node alice as an iperf client to 10.10.4.1, the traffic doesn't pass through ovs1 which connects with 10.10.4.1 and node alice. The traffic only goes through ovs2 even though 10.10.4.1 is connected to ovs1.
By running an iperf server on node bob's other interface, eth2 (10.10.1.1), and then running node alice as an iperf client to 10.10.1.1, the traffic passes through ovs2 which connects with 10.10.1.1 and node alice.
Why am I not able send traffic through one switch (ovs2) but not the other switch (ovs1)? What do you think the problem could be?
Thank you,
Harrison
Nicholas Bastin
Jun 6, 2016, 4:19:53 PM6/6/16
to geni-...@googlegroups.com, Harrison Lee Ledford, Xenia Mountrouidou
On Mon, Jun 6, 2016 at 4:01 PM, <hll.harr...@gmail.com> wrote:
I am doing research, but I keep running into an issue. I have a topology that consists of two nodes and two switches. Each node has two paths; one path to each switch. The switches connect the two switch but are not connected to one another. I have attached a picture of my topology for a clearer idea.
There's no topology attached, but I think I understand what you have configured.
My problem is when trying to send traffic. When I run an iperf server on node bob and bind it to bob's eth1 (10.10.4.1), and then run node alice as an iperf client to 10.10.4.1, the traffic doesn't pass through ovs1 which connects with 10.10.4.1 and node alice. The traffic only goes through ovs2 even though 10.10.4.1 is connected to ovs1.By running an iperf server on node bob's other interface, eth2 (10.10.1.1), and then running node alice as an iperf client to 10.10.1.1, the traffic passes through ovs2 which connects with 10.10.1.1 and node alice.
It seems that you are experiencing an issue called "arp flux". Each host needs to make an arp request for the interface you want to reach, and thus broadcasts (on all interfaces) that request. So, if Alice wants to reach 10.10.4.1 that node sends out an arp request on both interfaces, and Bob receives this request twice. The *first* interface Bob receives the ARP request on is the one that Bob will reply on, even if it's not the "right" interface (because Bob knows that Bob has this IP address somewhere, so why not respond). Depending on host parameters it might also respond on the second interface it receives the request on, and Alice will handle the replies based on the receiving order (and either overwriting or not based on various settings). Either way you can get into a scenario where Alice believes that 10.10.4.1 is reachable over the "wrong" path (because it is).
If this is the extent of your problem, you can likely just set arp_filter in your linux VMs:
sysctl -w net.ipv4.conf.all.arp_filter=1
And they will stop behaving in this manner. However, it's also possible that your subnet masks are wrong, and thus all of your interfaces are in the same subnet, in which case arp filter won't help. What is the subnet mask of your interfaces? Is it 255.255.255.0, or is it 255.255.0.0 (or larger)? If it is 255.255.0.0 (or 255.0.0.0, etc.), then just setting the arp_filter won't solve your problem, and you will need to fix your subnet masks as well.
--
Nick
hll.harr...@gmail.com
Jun 7, 2016, 9:28:44 AM6/7/16
to GENI Users, ledf...@email.wofford.edu, mountr...@wofford.edu
Thank you for your response, Nick. I tried your recommended solution but sending traffic from any interface only seems to pass through one switch.
When I ping from alice's eth1 (10.10.3.1) to bob's eth1 (10.10.4.1), the request passes through the correct switch (ovs1) and bob's eth1 does receive the request. However, when I ping from alice's eth2 (10.10.2.2) to bob's eth2 (10.10.1.1), the request passes through the wrong switch (ovs1) instead of the right switch (ovs2). Consequently, bob's eth2 doesn't receive the request but bob's eth1 receives the request.
The ping commands I am using for the alice node are:
- ping -c 1 -I 10.10.3.1 10.10.4.1
- This should be going through ovs1 and it does
- ping -c 1 -I 10.10.2.2 10.10.1.1
- This should be going through ovs2 but it goes through ovs2
To further troubleshoot, I tried pinging from the bob node using these:
- ping -c 1 -I 10.10.4.1 10.10.3.1
- This should be going through ovs1 but it goes through ovs2
- ping -c 1 -I 10.10.1.1 10.10.2.2
- This should be going through ovs2 and it does
All the interfaces of my nodes have the mask of 255.255.255.0.
Any further help would be appreciated!
Thank you for the previous help,
~Harrison
hll.harr...@gmail.com
Jun 7, 2016, 2:38:49 PM6/7/16
to GENI Users, ledf...@email.wofford.edu, mountr...@wofford.edu
Sorry I made a typing mistake on the ping commands I am using for the alice node in my previous reply. I meant to type this:
The ping commands I am using for the alice node are:
- ping -c 1 -I 10.10.3.1 10.10.4.1
- This should be going through ovs1 and it does
- ping -c 1 -I 10.10.2.2 10.10.1.1
- This should be going through ovs2 but it goes through ovs1
On Monday, June 6, 2016 at 4:19:53 PM UTC-4, nick.bastin wrote:
pjayant
Jul 19, 2016, 2:10:44 PM7/19/16
to GENI Users, ledf...@email.wofford.edu, mountr...@wofford.edu
Hi Harrison,
This is a follow up to find out if your issue has been solved so that appropriate action can be taken.
hll.harr...@gmail.com
Jul 20, 2016, 9:09:42 AM7/20/16
to GENI Users, ledf...@email.wofford.edu, mountr...@wofford.edu
Unfortunately my issue has not been resolved. Just to update the details, I have a GENI portal account, I used Jacks to reserve my topology, my slices name is COSC460-Simple, and I am using Kentucky ProtoGENI as the aggregate.
My issue is that communication (ICMP requests) between my nodes can't go between both switches but only seem to go through one switch.
If I run ping 10.10.1.1 -I 10.10.2.2 from node-1 I would of thought the requests from 10.10.2.2 and replies from 10.10.1.1 would travel through node-2. However, this is not the case. The requests travel through node-3 and reach node-0 at 10.10.3.1, the replies travel through node-2 and reach node-0 at 10.10.2.2. I have attached the tcpdumps of this scenario. They are titled "ping 10.10.1.1" followed by the node's name.
If I run ping 10.10.3.1 -I 10.10.4.2 from node-1 I would of thought the requests from 10.10.4.2 and replies from 10.10.3.1 would travel through node-3. However, this is not the case. The requests travel through node-3 and reach node-0 at 10.10.1.1, the replies travel through node-2 and reach node-1 at 10.10.2.2. I have attached the tcpdumps of this scenario. They are titled "ping 10.10.3.1" followed by the node's name.
Why do all the requests travel through one switch, and all the replies travel through the other. Is it possible to have requests travel through both switches? Ideally I mean if I ping from 10.10.2.2 to 10.10.1.1 is it possible to have the ICMP requests travel through node-2 and if I ping from 10.10.4.2 to 10.10.3.1 is it possible to have the ICMP requests travel through node-3?
I've tried the solution posted above but to no success. Any help would be much appreciated. Thank you.
Harrison
hll.harr...@gmail.com
Jul 21, 2016, 1:56:28 PM7/21/16
to GENI Users, ledf...@email.wofford.edu, mountr...@wofford.edu
Has anyone found a possible solution for this? Any help is appreciated.
Nicholas Bastin
Jul 21, 2016, 4:21:33 PM7/21/16
to geni-...@googlegroups.com, Harrison Lee Ledford, Xenia Mountrouidou
On Wed, Jul 20, 2016 at 9:09 AM, <hll.harr...@gmail.com> wrote:
My issue is that communication (ICMP requests) between my nodes can't go between both switches but only seem to go through one switch.
First, your "switch" nodes don't appear to be switches, they appear to be routers. Otherwise your host nodes would be on the same IP subnet, but instead each link has a different subnet.
If I run ping 10.10.1.1 -I 10.10.2.2 from node-1 I would of thought the requests from 10.10.2.2 and replies from 10.10.1.1 would travel through node-2.
These decisions are being made using the ARP and routing tables on your nodes.
However, this is not the case. The requests travel through node-3 and reach node-0 at 10.10.3.1, the replies travel through node-2 and reach node-0 at 10.10.2.2. I have attached the tcpdumps of this scenario. They are titled "ping 10.10.1.1" followed by the node's name.
This is not unusual. In linux, by default, despite you configuring IP addresses on interfaces, they actually all belong to the same kernel stack. For the most part userspace applications (like ping) cannot choose which interface they want to send a packet on, they simply send that packet to the kernel and the kernel chooses for them. If the kernel learned the destination MAC on more than one interface, it is free to pick any interface to send the packet on (and it is rarely the one you expect it to be).
Why do all the requests travel through one switch, and all the replies travel through the other.
Again, these are routed packets, not switched ones. All packets are traveling through a single router either because it was "first" when it forwarded broadcast packets, and thus the interface facing that device has a higher priority in the routing or arp tables when it comes to choosing two paths which are equal cost, or because the AM only inserted one route into the table (I'm not sure which is the case here, either are equally valid).
Is it possible to have requests travel through both switches? Ideally I mean if I ping from 10.10.2.2 to 10.10.1.1 is it possible to have the ICMP requests travel through node-2 and if I ping from 10.10.4.2 to 10.10.3.1 is it possible to have the ICMP requests travel through node-3?
In your current configuration you would have to alter your routing tables to make one path preferred over the other.
Also if you want to truly make the intermediate nodes switches, you need to change all of your IP addressing and forwarding schemes.
--
Nick
hll.harr...@gmail.com
Jul 22, 2016, 10:32:18 AM7/22/16
to GENI Users, ledf...@email.wofford.edu, mountr...@wofford.edu
Thank you for the reply and the help, Nick!
I have edited my topology:
I edited the routing tables and ARP tables for each node:
node-0:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.0.1 0.0.0.0 UG 0 0 0 eth0
10.10.1.4 10.10.1.1 255.255.255.255 UGH 0 0 0 eth2
10.10.1.5 10.10.1.8 255.255.255.255 UGH 0 0 0 eth1
172.16.0.0 0.0.0.0 255.240.0.0 U 0 0 0 eth0
Address HWtype HWaddress Flags Mask Iface
10.10.1.5 ether 02:4e:6e:fa:6a:d6 CM eth2
172.16.0.1 ether fe:ff:ff:ff:ff:ff C eth0
10.10.1.4 ether 02:1b:18:d4:b9:02 CM eth2
node-1:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.0.1 0.0.0.0 UG 0 0 0 eth0
10.10.1.1 10.10.1.4 255.255.255.255 UGH 0 0 0 eth1
10.10.1.8 10.10.1.5 255.255.255.255 UGH 0 0 0 eth2
172.16.0.0 0.0.0.0 255.240.0.0 U 0 0 0 eth0
Address HWtype HWaddress Flags Mask Iface
10.10.1.8 ether 02:df:f6:f2:f5:61 CM eth1
10.10.1.1 ether 02:58:af:54:4c:91 CM eth1
172.16.0.1 ether fe:ff:ff:ff:ff:ff C eth0
node-2:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.0.1 0.0.0.0 UG 0 0 0 eth0
10.10.1.1 10.10.1.2 255.255.255.255 UGH 0 0 0 eth1
10.10.1.4 10.10.1.3 255.255.255.255 UGH 0 0 0 eth2
172.16.0.0 0.0.0.0 255.240.0.0 U 0 0 0 eth0
Address HWtype HWaddress Flags Mask Iface
172.16.0.3 ether fe:ff:ff:ff:ff:ff C eth0
10.10.1.4 ether 02:1b:18:d4:b9:02 CM eth2
172.17.253.254 ether fe:ff:ff:ff:ff:ff C eth0
10.10.1.1 ether 02:58:af:54:4c:91 C eth1
172.16.0.1 ether fe:ff:ff:ff:ff:ff C eth0
node-3:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.0.1 0.0.0.0 UG 0 0 0 eth0
10.10.1.5 10.10.1.6 255.255.255.255 UGH 0 0 0 eth1
10.10.1.8 10.10.1.7 255.255.255.255 UGH 0 0 0 eth2
172.16.0.0 0.0.0.0 255.240.0.0 U 0 0 0 eth0
Address HWtype HWaddress Flags Mask Iface
172.16.0.3 ether fe:ff:ff:ff:ff:ff C eth0
10.10.1.5 ether 02:4e:6e:fa:6a:d6 CM eth2
172.17.253.254 ether fe:ff:ff:ff:ff:ff C eth0
10.10.1.8 ether 02:df:f6:f2:f5:61 CM eth2
172.16.0.1 ether fe:ff:ff:ff:ff:ff C eth0
After setting up the routing tables and ARP tables as above, when I ping 10.10.1.4 from node-0 (ping 10.10.1.4) I see ICMP requests leave from 10.10.1.1 travel to node-2 (10.10.1.2) but then node-2 doesn't do anything with the ICMP requests. Node-2 doesn't pass them from 10.10.1.2 to 10.10.1.3 to 10.10.1.4.
The same thing happens when I ping 10.10.1.5 from node-0 (ping 10.10.1.5). I see ICMP requests leave from 10.10.1.8 travel to node-3 (10.10.1.7) but then node-3 doesn't do anything with the ICMP request. Node-3 doesn't pass them from 10.10.1.7 to 10.10.1.6 to 10.10.1.5.
Do the above route and ARP tables look correct? Am I missing something else?
Thank you for the help!
Harrison
hll.harr...@gmail.com
Jul 22, 2016, 11:27:37 AM7/22/16
to GENI Users, ledf...@email.wofford.edu, mountr...@wofford.edu
I noticed that the entries for my ARP tables had the wrong interface attached to them. I have update the ARP tables so the right address, hardware address and interface correspond.
node-0:
Address HWtype HWaddress Flags Mask Iface
10.10.1.5 ether 02:4e:6e:fa:6a:d6 CM eth1
172.16.0.1 ether fe:ff:ff:ff:ff:ff C eth0
10.10.1.4 ether 02:1b:18:d4:b9:02 CM eth2
node-1:
Address HWtype HWaddress Flags Mask Iface
10.10.1.1 ether 02:58:af:54:4c:91 CM eth1
172.16.0.1 ether fe:ff:ff:ff:ff:ff C eth0
10.10.1.8 ether 02:df:f6:f2:f5:61 CM eth2
node-2:
Address HWtype HWaddress Flags Mask Iface
172.16.0.3 ether fe:ff:ff:ff:ff:ff C eth0
10.10.1.4 ether 02:1b:18:d4:b9:02 CM eth2
172.17.253.254 ether fe:ff:ff:ff:ff:ff C eth0
10.10.1.1 ether 02:58:af:54:4c:91 C eth1
172.16.0.1 ether fe:ff:ff:ff:ff:ff C eth0
node-3:
Address HWtype HWaddress Flags Mask Iface
10.10.1.5 ether 02:4e:6e:fa:6a:d6 CM eth1
172.16.0.3 ether fe:ff:ff:ff:ff:ff C eth0
172.17.253.254 ether fe:ff:ff:ff:ff:ff C eth0
10.10.1.8 ether 02:df:f6:f2:f5:61 CM eth2
172.16.0.1 ether fe:ff:ff:ff:ff:ff C eth0
Even after correcting the ARP tables of the nodes so they correspond with the correct interfaces I am still having the same problems. When I ping 10.10.1.4 from node-0 (ping 10.10.1.4) I see ICMP requests leave from 10.10.1.1 travel to node-2 (10.10.1.2) but then node-2 doesn't do anything with the ICMP requests. Node-2 doesn't pass them from 10.10.1.2 to 10.10.1.3 to 10.10.1.4.
The same thing happens when I ping 10.10.1.5 from node-0 (ping 10.10.1.5). I see ICMP requests leave from 10.10.1.8 travel to node-3 (10.10.1.7) but then node-3 doesn't do anything with the ICMP request. Node-3 doesn't pass them from 10.10.1.7 to 10.10.1.6 to 10.10.1.5.
Nicholas Bastin
Jul 22, 2016, 1:16:06 PM7/22/16
to geni-...@googlegroups.com, Harrison Lee Ledford, Xenia Mountrouidou
On Fri, Jul 22, 2016 at 11:27 AM, <hll.harr...@gmail.com> wrote:
I noticed that the entries for my ARP tables had the wrong interface attached to them. I have update the ARP tables so the right address, hardware address and interface correspond.
Your intermediate nodes still aren't going to act as switches - they're just hosts. You will need to actually run switch software on them and attach the interfaces to that software if you want them to act as switches.
--
Nick
Reply all
Reply to author
Forward
0 new messages