Quagga is unable to ping upstream router

291 views
Skip to first unread message

Teja Bachu

unread,
Feb 24, 2017, 9:06:39 AM2/24/17
to CORD Discuss
Hi,

I have CORD 2.0 multi node setup.

I have Quagga running in a host outside the CORD POD.
I've configured static route in zebrad.conf as below

!
hostname cord-quagga
password cord
!
ip forwarding
ip route 0.0.0.0/0 10.15.134.253
!
fpm connection ip 10.15.0.13 port 2620
!
end



I am trying to ping upstream router from Quagga.
Quagga is able to resolve arp, however ping failed.


I see the below flow is being hit for the above ping request 

Criteria: IN_PORT:6, ETH_SRC:02:42:0A:0F:86:46, ETH_TYPE:arp, ARP_SPA:10.15.134.70
Treatment Instructions: OUTPUT:CONTROLLER, GROUP:0xffe002f


However, the group 0xffe002f shows zero packets.



I see ONOS is showing correct routes and next-hops as below

onos> next-hops
ip=10.15.134.253, mac=E8:F7:24:BC:43:D2, loc=of:0000480fcfb07084/47, numRoutes=1
onos> routes
Table: ipv4
   Network            Next Hop        Source
   0.0.0.0/0          10.15.134.253   FPM
   Total: 1

Table: ipv6
   Network            Next Hop        Source
   Total: 0



How to fix this ?



network-cfg-quickstart.json

Jonathan Hart

unread,
Feb 24, 2017, 11:56:28 AM2/24/17
to Teja Bachu, CORD Discuss
Hi,

Not all switches support group counters, so depending on what switch you are using you may or may not see the counters go up. You could check if any of the other group counters are increasing.

Is the MAC address you configured in your upstream interface config in ONOS the same MAC address as the Quagga host's NIC?  The other thing I noticed is that in your segmentrouting app config the vrouterId is not correct - this should be the device ID of the device where the upstream is connected.

Thanks,

Jono

--
You received this message because you are subscribed to the Google Groups "CORD Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss+unsubscribe@opencord.org.
To post to this group, send email to cord-d...@opencord.org.
Visit this group at https://groups.google.com/a/opencord.org/group/cord-discuss/.
To view this discussion on the web visit https://groups.google.com/a/opencord.org/d/msgid/cord-discuss/CAGpRgg2qe0hxMPOH0APHvZ-A6BfP1FS_QQskg1%3DGpW%3D9KMDdrQ%40mail.gmail.com.
For more options, visit https://groups.google.com/a/opencord.org/d/optout.

Teja Bachu

unread,
Feb 25, 2017, 7:12:23 AM2/25/17
to Jonathan Hart, CORD Discuss
Hi Jono,

Thanks a lot.

I am using HP Altoline 6920 (Leaf) and 6940 (Spine) switches.

For both the leaf and spine switches, the group counters in ONOS are showing zero packets.
So, as you mentioned, these switches may not support group counters.


I changed the vrouterId as you suggested.
Now, Quagga is able to ping upstream router.


I launched exampleservice. However, the exampleservice VM is not able to reach internet.

I noticed the below:
 1. exampleservice VM is able to resolve MAC of it's default gateway 192.168.2.1, but not able to ping
  
 2. I did tcpdump at multiple points and noticed the traffic is leaving the compute node hosting the exampleservice VM.
     However, I am not sure if the traffic is reaching the Leaf switch ( the counters are showing zero packets as they may not be supported).
     So, can you please let me know if there are other ways to find out if the traffic is reaching Leaf switch.

 3. I see few messages in ONOS log like "Unsupported action extension; defaulting to native OF"
     I am not sure if this can cause any issues 



Can you please check the attached config files and logs and let me know if I am missing any configuration.


Regards,
Tejaswi





onos_cord_netcfg.txt
cord-services.yaml
onos_fabric_logs.txt
onos_fabric_netcfg.txt

Charles Chan

unread,
Feb 26, 2017, 4:26:10 AM2/26/17
to Teja Bachu, Jonathan Hart, CORD Discuss
Hi,

I leave some comments inline.

Charles

On Sat, Feb 25, 2017 at 4:12 AM Teja Bachu <teja....@gmail.com> wrote:
Hi Jono,

Thanks a lot.

I am using HP Altoline 6920 (Leaf) and 6940 (Spine) switches.

For both the leaf and spine switches, the group counters in ONOS are showing zero packets.
So, as you mentioned, these switches may not support group counters.


I changed the vrouterId as you suggested.
Now, Quagga is able to ping upstream router.


I launched exampleservice. However, the exampleservice VM is not able to reach internet.

I noticed the below:
 1. exampleservice VM is able to resolve MAC of it's default gateway 192.168.2.1, but not able to ping
  
 2. I did tcpdump at multiple points and noticed the traffic is leaving the compute node hosting the exampleservice VM.
     However, I am not sure if the traffic is reaching the Leaf switch ( the counters are showing zero packets as they may not be supported).
     So, can you please let me know if there are other ways to find out if the traffic is reaching Leaf switch.

Can you do a tcpdump on the physical interface? Do you see the packet? Does the packet have correct destination MAC (= leaf switch MAC)?

The port counter should still work on these switches, although it cannot give you detail information as a flow counters.
 
 3. I see few messages in ONOS log like "Unsupported action extension; defaulting to native OF"
     I am not sure if this can cause any issues 

This is expected since OFDPA3 does not support set_vlan_vid extension. It should be safe to ignore this message.


Jono

To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss...@opencord.org.

--
You received this message because you are subscribed to the Google Groups "CORD Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss...@opencord.org.

To post to this group, send email to cord-d...@opencord.org.
Visit this group at https://groups.google.com/a/opencord.org/group/cord-discuss/.
--
Charles Chan
Member of Technical Staff, ON.Lab

Teja Bachu

unread,
Feb 26, 2017, 11:23:56 PM2/26/17
to Charles Chan, Jonathan Hart, CORD Discuss
Hi Charles,

If I ping google IP from exampleservice VM, I see the packets on the physical interface and tcpdump on the physical interface shows as below:

04:18:58.386510 fa:16:3e:00:4d:d2 (oui Unknown) > a4:23:05:34:56:78 (oui Unknown), ethertype IPv4 (0x0800), length 98: 192.168.2.2 > dfw25s08-in-f142.1e100.net: ICMP echo request, id 2583, seq 67, length 64


fa:16:3e:00:4d:d2 - eth0's MAC in exampleservice VM
 a4:23:05:34:56:78 - vRouterMacs in segment routing section in network-cfg-quickstart.json = gateway_mac in cord-services.yaml


Regards,
Tejaswi








Charles


Jono

To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss+unsubscribe@opencord.org.

--
You received this message because you are subscribed to the Google Groups "CORD Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss+unsubscribe@opencord.org.

To post to this group, send email to cord-d...@opencord.org.
Visit this group at https://groups.google.com/a/opencord.org/group/cord-discuss/.

Charles Chan

unread,
Feb 27, 2017, 5:57:41 PM2/27/17
to Teja Bachu, Jonathan Hart, CORD Discuss
Hi Teja,

The packet dump seems correct. The onos-fabric configuration also seems correct - you have vRouterMac and vRouterId configured.

Have you received a route from upstream yet?
More specifically, if you run "routes" in onos-fabric CLI, do you see the prefix that contains dfw25s08-in-f142.1e100.net ?

Have the next hop of that route been resolved?
More specifically, if you run "next-hops" in onos-fabric CLI, do you see the nexthop (upstream router) ?

Charles

Charles


Jono

To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss...@opencord.org.

--
You received this message because you are subscribed to the Google Groups "CORD Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss...@opencord.org.

To post to this group, send email to cord-d...@opencord.org.
Visit this group at https://groups.google.com/a/opencord.org/group/cord-discuss/.
--
Charles Chan
Member of Technical Staff, ON.Lab

Teja Bachu

unread,
Feb 27, 2017, 6:34:17 PM2/27/17
to Charles Chan, Jonathan Hart, CORD Discuss
Hi Charles,

I see the below outputs in ONOS fabric. I've set static route in Quagga to upstream router.

onos> next-hops
ip=10.15.134.253, mac=E8:F7:24:BC:43:D2, loc=of:0000480fcfb07084/47, numRoutes=1

onos> routes
Table: ipv4
   Network            Next Hop        Source
   0.0.0.0/0          10.15.134.253   FPM
   Total: 1

Table: ipv6
   Network            Next Hop        Source
   Total: 0


Regards,
Tejaswi



Charles


Jono

To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss+unsubscribe@opencord.org.

--
You received this message because you are subscribed to the Google Groups "CORD Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss+unsubscribe@opencord.org.

To post to this group, send email to cord-d...@opencord.org.
Visit this group at https://groups.google.com/a/opencord.org/group/cord-discuss/.
--
Charles Chan
Member of Technical Staff, ON.Lab

Jonathan Hart

unread,
Feb 28, 2017, 12:39:26 PM2/28/17
to Teja Bachu, Charles Chan, CORD Discuss
Hi,

All the configuration seems correct. Are you able to tell if the packets are leaving the fabric towards the upstream router at all?

Thanks,

Charles


Jono

To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss...@opencord.org.

--
You received this message because you are subscribed to the Google Groups "CORD Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss...@opencord.org.

To post to this group, send email to cord-d...@opencord.org.
Visit this group at https://groups.google.com/a/opencord.org/group/cord-discuss/.
--
Charles Chan
Member of Technical Staff, ON.Lab
--
Charles Chan
Member of Technical Staff, ON.Lab

--
You received this message because you are subscribed to the Google Groups "CORD Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss...@opencord.org.

To post to this group, send email to cord-d...@opencord.org.
Visit this group at https://groups.google.com/a/opencord.org/group/cord-discuss/.

Teja Bachu

unread,
Feb 28, 2017, 7:12:10 PM2/28/17
to Jonathan Hart, Charles Chan, CORD Discuss
Hi Jono,

The packets are not reaching the upstream router.

I am not sure if the packets are reaching the leaf switch as the counters in ONOS are showing zero packets.

I tried to find if there are any packet dumping tools in whitebox switches. However, I did not find any commands.

Can you please let me know if there are any commands to dump packets in whitebox switches?


Regards,
Tejaswi




Charles


Jono

To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss+unsubscribe@opencord.org.

--
You received this message because you are subscribed to the Google Groups "CORD Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss+unsubscribe@opencord.org.

To post to this group, send email to cord-d...@opencord.org.
Visit this group at https://groups.google.com/a/opencord.org/group/cord-discuss/.
--
Charles Chan
Member of Technical Staff, ON.Lab
--
Charles Chan
Member of Technical Staff, ON.Lab

--
You received this message because you are subscribed to the Google Groups "CORD Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss+unsubscribe@opencord.org.

To post to this group, send email to cord-d...@opencord.org.
Visit this group at https://groups.google.com/a/opencord.org/group/cord-discuss/.

Charles Chan

unread,
Feb 28, 2017, 10:07:19 PM2/28/17
to Teja Bachu, Jonathan Hart, CORD Discuss
We also found it very difficult to determine whether the packet hit the switch or not.

The port counter works. What we usually do is sending a large amount of ping packets (ping -i 0.2 <ip>) and observe the port counter.
It should at least tell you where the packet get stuck. (where you see a large RxDrop counter)

It might also be helpful if you can send the flow/group table dump to us.

Cheers,
Charles

Charles


Jono

To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss...@opencord.org.

--
You received this message because you are subscribed to the Google Groups "CORD Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss...@opencord.org.

To post to this group, send email to cord-d...@opencord.org.
Visit this group at https://groups.google.com/a/opencord.org/group/cord-discuss/.
--
Charles Chan
Member of Technical Staff, ON.Lab
--
Charles Chan
Member of Technical Staff, ON.Lab

--
You received this message because you are subscribed to the Google Groups "CORD Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss...@opencord.org.

To post to this group, send email to cord-d...@opencord.org.
Visit this group at https://groups.google.com/a/opencord.org/group/cord-discuss/.

Teja Bachu

unread,
Feb 28, 2017, 11:31:28 PM2/28/17
to Charles Chan, Jonathan Hart, CORD Discuss
Hi Charles,

Please check the attached configs and dumps.


Regards,
Tejaswi

Charles


Jono

To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss+unsubscribe@opencord.org.

--
You received this message because you are subscribed to the Google Groups "CORD Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss+unsubscribe@opencord.org.

To post to this group, send email to cord-d...@opencord.org.
Visit this group at https://groups.google.com/a/opencord.org/group/cord-discuss/.
--
Charles Chan
Member of Technical Staff, ON.Lab
--
Charles Chan
Member of Technical Staff, ON.Lab

--
You received this message because you are subscribed to the Google Groups "CORD Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss+unsubscribe@opencord.org.

To post to this group, send email to cord-d...@opencord.org.
Visit this group at https://groups.google.com/a/opencord.org/group/cord-discuss/.
--
Charles Chan
Member of Technical Staff, ON.Lab

--
You received this message because you are subscribed to the Google Groups "CORD Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cord-discuss+unsubscribe@opencord.org.

To post to this group, send email to cord-d...@opencord.org.
Visit this group at https://groups.google.com/a/opencord.org/group/cord-discuss/.
onos_cord_dump.txt
onos_cord_flows.txt
onos_cord_netcfg.txt
onos_fabric_dump.txt
onos_fabric_flows.txt
onos_fabric_groups.txt
onos_fabric_netcfg.txt
onos_fabric_ports.txt

Nandan Bharadwaj

unread,
Feb 28, 2017, 11:58:09 PM2/28/17
to Teja Bachu, Charles Chan, Jonathan Hart, CORD Discuss
Hi Tejaswi,

Do update the zebra.conf file with the interface info as below,

interface eth1 #eth1 is the quagga interface on the Quagga host
  link-detect
  ip address 192.168.2.2/24

Hope this works.

And I don't see the Quagga info in the ports configuration of the fabric, do add that as well.

Regards,
Nandan

Teja Bachu

unread,
Mar 1, 2017, 12:17:06 AM3/1/17
to Nandan Bharadwaj, Charles Chan, Jonathan Hart, CORD Discuss
Hi Nandan,


I am running Quagga as a Docker container in a host outside the POD.
The Quagga container has 2 NICs - eth0 (10.15.0.70) and eth1 (10.15.134.70).

eth0's IP is used to connect to ONOS fabric
eth1's IP is used to connect to upstream router



I have a port as below in ONOS fabric netcfg for upstream router.
The "ips" and "mac" fields are Quagga eth1's IP and MAC.

    "of:0000480fcfb07084/47" : {
      "interfaces" : [ {
        "ips" : [ "10.15.134.70/24" ],
        "name" : "uplink",
        "mac" : "02:42:0a:0f:86:46"
      } ]

So, can you please let me know if any changes are required in zebrad.conf or any other config file.

It will also help me if you can share your configurations (ONOS fabric netcfg, ONOS cord netcfg, zebrad.conf).


Regards,
Tejaswi













Nandan Bharadwaj

unread,
Mar 1, 2017, 1:02:30 AM3/1/17
to Teja Bachu, Charles Chan, Jonathan Hart, CORD Discuss
Hi Tejaswi

Do add the zebra.conf file with the upstream interface info as below,

interface eth1
  link-detect
  ip address 10.15.134.70/24

Restart the zebra daemon. 

If following above steps doesn't solve the problem, please post the below logs.

1. routing table of exampleservice VM
2. sudo tcpdump -i <fabric_interface_compute> -ne 

Regards,
Nandan

Nandan Bharadwaj

unread,
Mar 1, 2017, 2:03:25 AM3/1/17
to Teja Bachu, Charles Chan, Jonathan Hart, CORD Discuss
Hi Tejaswi,

This usually is how the packets flow.

Flows:
Missing tableId=10 flow rule.
id=4c0000fa788de8, state=ADDED, bytes=0, packets=0, duration=63051, priority=32768, tableId=20, appId=org.onosproject.segmentrouting, payLoad=null, selector=[ETH_DST:A4:23:05:34:56:78, ETH_TYPE:ipv4], treatment=DefaultTrafficTreatment{immediate=[], deferred=[], transition=TABLE:30, meter=None, cleared=false, metadata=null}
id=4c0000faad1003, state=ADDED, bytes=0, packets=0, duration=21093, priority=10, tableId=30, appId=org.onosproject.segmentrouting, payLoad=null, selector=[ETH_TYPE:ipv4], treatment=DefaultTrafficTreatment{immediate=[], deferred=[GROUP:0x70000450], transition=TABLE:60, meter=None, cleared=false, metadata=null}

Groups:
id=0x70000450, bucket=1, bytes=0, packets=0, actions=[GROUP:0x9200044f]
id=0x9200044f, bucket=1, bytes=0, packets=0, actions=[MPLS_PUSH:mpls_unicast, MPLS_LABEL:101, MPLS_BOS:true, TTL_OUT, GROUP:0x9000044e]
id=0x9000044e, bucket=1, bytes=0, packets=0, actions=[ETH_DST:48:0F:CF:AE:69:50, ETH_SRC:48:0F:CF:B0:10:28, VLAN_ID:4094, GROUP:0xffe0031]
id=0xffe0031, bucket=1, bytes=0, packets=0, actions=[VLAN_POP:unknown, OUTPUT:49]

Request you to update the exampleservice VM information in the hosts field of the fabric configuration so that a rule gets created in the tableId=10 .

Later coming to the groups, below should have been the actions criteria,

id=0x9000044e, bucket=1, bytes=0, packets=0, actions=[ETH_DST:<E8:F7:24:BC:43:D2 MAC_of_next_hop_router>, ETH_SRC:<Quagga_upstream_interface_MAC>, VLAN_ID:4094, GROUP:0xffe0031]

id=0xffe0031, bucket=1, bytes=0, packets=0, actions=[VLAN_POP:unknown, OUTPUT:<Port_of_next_hop_router>]

Looking at the above, I see issues in the fabric switch configuration. Please update the fabric configuration with proper ports and hosts info.

Regards,
Nandan


On Wed, Mar 1, 2017 at 11:49 AM, Teja Bachu <teja....@gmail.com> wrote:
Hi Nandan,

I made the changes you suggested. However, the exampleservice VM is unable to reach internet.
Please check the attached files.

Regards,
Tejaswi

On Wed, Mar 1, 2017 at 11:41 AM, Nandan Bharadwaj <nandan07...@gmail.com> wrote:
Updates inline.

On Wed, Mar 1, 2017 at 11:32 AM, Nandan Bharadwaj <nandan07...@gmail.com> wrote:
Hi Tejaswi

Do add the zebra.conf file with the upstream interface info as below,

interface eth1
  link-detect
  ip address 10.15.134.70/24

Restart the zebra daemon. 

If following above steps doesn't solve the problem, please post the below logs.

1. routing table of exampleservice VM
2. sudo tcpdump -i <fabric_interface_sugary_visitor> -ne 

Regards,
Nandan

Teja Bachu

unread,
Mar 1, 2017, 4:04:00 AM3/1/17
to Nandan Bharadwaj, Charles Chan, Jonathan Hart, CORD Discuss
Hi Nandan,

I see the below 2 flows are added after adding exampleservice VM to hosts section of fabric conf.

id=4c00007fca0e5f, state=ADDED, bytes=0, packets=0, duration=490, priority=64010, tableId=30, appId=org.onosproject.segmentrouting, payLoad=null, selector=[ETH_TYPE:ipv4, IPV4_DST:192.168.2.2/32], treatment=DefaultTrafficTreatment{immediate=[], deferred=[GROUP:0x20000471], transition=TABLE:60, meter=None, cleared=false, metadata=null}

id=4c00008b19ffc6, state=ADDED, bytes=0, packets=0, duration=490, priority=100, tableId=50, appId=org.onosproject.segmentrouting, payLoad=null, selector=[ETH_DST:FA:16:3E:0D:7C:D1, EXTENSION:of:0000480fcfb07084/OfdpaMatchVlanVid{vlanId=4093}], treatment=DefaultTrafficTreatment{immediate=[], deferred=[GROUP:0xffd0005], transition=TABLE:60, meter=None, cleared=false, metadata=null}


Regards,
Tejaswi


Teja Bachu

unread,
Mar 1, 2017, 11:36:52 PM3/1/17
to Nandan Bharadwaj, Jonathan Hart, Charles Chan, CORD Discuss
Hi Nandan/Jono/Charles,


I've added exampleservice VM to hosts section. However, I still do not get internet access in VM.

Please check the attached configs and dumps and let me know if anything else is missing.




Regards,
Tejaswi





On Wed, Mar 1, 2017 at 3:44 PM, Nandan Bharadwaj <nandan07...@gmail.com> wrote:
Hi Tejaswi,

Just try reactivating "org.onosproject.segmentrouting" app from onos-fabric as below,

onos> app deactivate org.onosproject.segmentrouting
onos> app activate org.onosproject.segmentrouting

This the only option I can think of now, looking at all the above info.

Ragards,
Nandan
onos_fabric_dump.txt
network-cfg-quickstart.json
onos_fabric_flows.txt
onos_fabric_netcfg.txt

Charles Chan

unread,
Mar 2, 2017, 9:22:08 PM3/2/17
to Teja Bachu, Nandan Bharadwaj, Jonathan Hart, CORD Discuss
There is only one routing table entry in of:7084 pushed by vRouter, which matches the DST_IP 10.15.12.0/24.
In your previous mail, you mentioned that the default route (0.0.0.0/0) is listed in routes command and the next-hop is resolved. You should see a route on of:7084 that matches 0.0.0.0/0, instead of 10.15.12.0/24. I am not sure where this inconsistency comes from. Maybe you change something between these two results?

I haven't identified the root cause. But the missing routing table entry is probably the reason why none of your VM can reach the upstream router.

Charles

Teja Bachu

unread,
Mar 2, 2017, 9:44:48 PM3/2/17
to Charles Chan, Nandan Bharadwaj, Jonathan Hart, CORD Discuss
Hi Charles,

Yes, I've changed 0.0.0.0/0 to 10.15.12.0/24 to check if ONOS is adding a new flow.

I've changed it back to 0.0.0.0/0. However, I do not see any entry for 0.0.0.0/0



Please check the attached flows.

Regards,
Tejaswi
onos_fabric_flows.txt
Reply all
Reply to author
Forward
0 new messages