Re: [seba-dev] ONOS org.onosproject.onos-apps-segmentrouting-app crash on aarch64

[Charles Chan]

The exception is unlikely related to different CPU architecture. It is mainly about the distributed store time out when XConnectManager tries to write to the store.

Have you tried ONOS 1.13.9? This patch may solve the problem, but it is not part of 1.13.5.

Thanks,

Charles Chan, Ph.D.

Member of Technical Staff, Open Networking Foundation

On Thu, Sep 19, 2019 at 2:48 AM <cipria...@gmail.com> wrote:

Hello,

First of all I apologize if this is not the appropriate forum, I assume there should be a mailing list for ONOS, but I can't figure out which is it.

In the context of Akrino IEC, I've been investigating a problem on a SEBA-in-a-Box installation, at least sort of, because I'm deploying SEBA using the cord-platform -> seba -> att-workflow set of charts and then install PONSim. All of these use manually built Docker images that work on aarch64!

Right now I can do the authentication step, but the DHCP fails, although I can see the DHCP replies coming from the BNG in the mininet POD. Which then revealed that ONOS is not catching them.

There are a lot of details in an Akraino Jira card:

https://jira.akraino.org/browse/IEC-22

Right now the most obvious problem is that Xconnet Manager crashes during what looks to be the operation of adding the corresponding flows in ONOS. Not all the flows are lost, by comparing against a working setup on x86 there were suppose to be 6 new flows added, but only 4 got there.

The problem is reproducible with the same outcome every time. Here is the relevant bactrace in the ONOS log:

2019-09-13 15:09:55,807 | INFO  | AAA-radius-0     | AaaManager                       | 185 - org.opencord.aaa - 1.8.0 | Auth event APPROVED for of:0000aabbccddeeff/128

2019-09-13 15:10:23,278 | INFO  | qtp300471503-38  | Olt                              | 187 - org.opencord.olt-app - 2.1.0 | Programming vlans for subscriber: [id:PSMO12345678,cTag:111,sTag:222,nasPortId:,uplinkPort:-1,slot:-1,hardwareIdentifier:null,ipaddress:null,nasId:null,circuitId:,remoteId:]

2019-09-13 15:10:23,287 | INFO  | ispatch-default0 | AccessDeviceKafkaIntegration     | 189 - org.opencord.kafka - 1.0.0 | Got AccessDeviceEvent: SUBSCRIBER_REGISTERED

2019-09-13 15:10:23,350 | INFO  | ce-operations-29 | Olt                              | 187 - org.opencord.olt-app - 2.1.0 | DHCP v4 filter for device of:0000aabbccddeeff on port 128 installed.

2019-09-13 15:10:23,350 | WARN  | tive-installer-3 | OltPipeline                      | 161 - org.onosproject.onos-drivers-default - 1.13.5 |

Only the following are Supported in OLT for filter ->

ETH TYPE : EAPOL, LLDP and IPV4

IPV4 TYPE: IGMP and UDP (for DHCP)

2019-09-13 15:10:23,351 | WARN  | tive-installer-3 | InOrderFlowObjectiveManager      | 130 - org.onosproject.onos-core-net - 1.13.5 | Flow objective onError DefaultFilteringObjective{id=1084928967, type=PERMIT, op=ADD, priority=10000, key=IN_PORT:128, conditions=[ETH_TYPE:ipv6, IP_PROTO:17, UDP_SRC:547, UDP_DST:546], meta=DefaultTrafficTreatment{immediate=[OUTPUT:CONTROLLER], deferred=[], transition=None, meter=[], cleared=false, StatTrigger=null, metadata=null}, appId=DefaultApplicationId{id=176, name=org.opencord.olt}, permanent=true, timeout=0}. Reason = UNSUPPORTED

2019-09-13 15:10:23,351 | INFO  | tive-installer-3 | Olt                              | 187 - org.opencord.olt-app - 2.1.0 | DHCP v6 filter for device of:0000aabbccddeeff on port 128 failed installation because UNSUPPORTED

2019-09-13 15:10:25,097 | INFO  | qtp300471503-38  | XconnectManager                  | 177 - org.onosproject.onos-apps-segmentrouting-app - 1.13.5 | Adding or updating xconnect. deviceId=of:0000000000000001, vlanId=222, ports=[1, 2]

2019-09-13 15:10:40,118 | ERROR | nt-partition-1-0 | ThreadPoolContext                | 90 - io.atomix - 2.0.23 | An uncaught exception occurred

java.lang.IllegalStateException: org.onosproject.store.service.ConsistentMapException$Timeout: onos-sr-xconnect-next

        at org.onosproject.store.primitives.impl.MeteredAsyncConsistentMap$InternalMeteredMapEventListener.event(MeteredAsyncConsistentMap.java:310)

        at org.onosproject.store.primitives.impl.CachingAsyncConsistentMap.lambda$null$1(CachingAsyncConsistentMap.java:93)

        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399)[95:com.google.guava:22.0.0]

        at org.onosproject.store.primitives.impl.CachingAsyncConsistentMap.lambda$null$2(CachingAsyncConsistentMap.java:93)

        at java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1597)[:1.8.0_201]

        at org.onosproject.store.primitives.impl.CachingAsyncConsistentMap.lambda$new$3(CachingAsyncConsistentMap.java:93)

        at org.onosproject.store.primitives.impl.TranscodingAsyncConsistentMap$InternalBackingMapEventListener.event(TranscodingAsyncConsistentMap.java:366)

        at org.onosproject.store.primitives.impl.TranscodingAsyncConsistentMap$InternalBackingMapEventListener.event(TranscodingAsyncConsistentMap.java:366)

        at org.onosproject.store.primitives.resources.impl.AtomixConsistentMap.lambda$null$1(AtomixConsistentMap.java:128)[133:org.onosproject.onos-core-primitives:1.13.5]

        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399)[95:com.google.guava:22.0.0]

        at org.onosproject.store.primitives.resources.impl.AtomixConsistentMap.lambda$null$2(AtomixConsistentMap.java:128)[133:org.onosproject.onos-core-primitives:1.13.5]

        at java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1597)[:1.8.0_201]

        at org.onosproject.store.primitives.resources.impl.AtomixConsistentMap.lambda$handleEvent$3(AtomixConsistentMap.java:128)[133:org.onosproject.onos-core-primitives:1.13.5]

        at java.util.ArrayList.forEach(ArrayList.java:1257)[:1.8.0_201]

        at org.onosproject.store.primitives.resources.impl.AtomixConsistentMap.handleEvent(AtomixConsistentMap.java:127)[133:org.onosproject.onos-core-primitives:1.13.5]

        at io.atomix.protocols.raft.proxy.impl.DelegatingRaftProxy.lambda$addEventListener$4(DelegatingRaftProxy.java:122)

        at io.atomix.protocols.raft.proxy.impl.BlockingAwareRaftProxyClient.lambda$null$2(BlockingAwareRaftProxyClient.java:67)

        at io.atomix.utils.concurrent.ThreadPoolContext.lambda$new$0(ThreadPoolContext.java:81)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_201]

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_201]

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)[:1.8.0_201]

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)[:1.8.0_201]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_201]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_201]

        at java.lang.Thread.run(Thread.java:748)[:1.8.0_201]

Caused by: org.onosproject.store.service.ConsistentMapException$Timeout: onos-sr-xconnect-next

        at org.onosproject.store.primitives.DefaultConsistentMap.complete(DefaultConsistentMap.java:258)

        at org.onosproject.store.primitives.DefaultConsistentMap.containsKey(DefaultConsistentMap.java:77)

        at org.onosproject.segmentrouting.xconnect.impl.XconnectManager.populateNext(XconnectManager.java:485)

        at org.onosproject.segmentrouting.xconnect.impl.XconnectManager.populateXConnect(XconnectManager.java:455)

        at org.onosproject.segmentrouting.xconnect.impl.XconnectManager.access$300(XconnectManager.java:98)

        at org.onosproject.segmentrouting.xconnect.impl.XconnectManager$XconnectMapListener.event(XconnectManager.java:297)

        at org.onosproject.store.primitives.impl.MeteredAsyncConsistentMap$InternalMeteredMapEventListener.event(MeteredAsyncConsistentMap.java:306)

        ... 24 more

At this point I can't match the trace against Xconnect version 1.13.5, I have no idea how the app is built and installed in the Docker image.

What information I received so far, the aarch64 Docker image has been built using this Dockerfile:

https://github.com/cachengo/cachengo-onos/commit/6af2d66935f7b450965ef07040da4c2721637938

The cord-platform chart and other related charts can be found here:

https://github.com/iecedge/seba_charts

PONSim charts can be found for now at:

https://github.com/iecedge/helm-charts

Any help is greatly appreciated as this points to a serious problem on aarch64 which needs to be fixed.

Thanks and regards,

Ciprian

--
You received this message because you are subscribed to the Google Groups "SEBA Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seba-dev+u...@opennetworking.org.
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/seba-dev/6e617200-c56f-450b-abfc-c8c1a904b6fc%40opennetworking.org.

[Zack Williams]

Hi Ciprian,

Did upgrading to 1.13.9 help resolve this issue?

Thanks,
Zack

> On Sep 20, 2019, at 4:03 AM, cipria...@gmail.com wrote:
>
> Hi Charles,
>
> Thank you for the suggestion. I will try to test with version 1.13.9 as indicated.
> I assume the segmentrouting app is built together with ONOS, so I will need to take the newer ONOS altogether. I will need some time to built a new image for aarch64 and get back to you as soon as possible.
>
> Thank and regards,
> /Ciprian

> To unsubscribe from this group and stop receiving emails from it, send an email to seba...@opennetworking.org.

> To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/seba-dev/6e617200-c56f-450b-abfc-c8c1a904b6fc%40opennetworking.org.

---
Zack Williams
z...@opennetworking.org

[Saurav Das]

Hi Tina,

SEBA meetings have been temporarily suspended as the community focuses on VOLTHA and meets frequently in the VOLTHA calls or the various brigade calls.

We will resume the SEBA calls at a future date, with advance notice.

Sorry for the confusion

Thanks

Saurav

On Tue, Oct 15, 2019 at 9:04 AM Tina Tsou <tina...@gmail.com> wrote:

Dear all,

Is there SEBA meeting today?

I’m waiting at the GoToMeeting bridge.

On Mon, Oct 14, 2019 at 5:32 AM <cipria...@gmail.com> wrote:

Hello guys,

I'm back from vacation now. There was no update on this thread since two weeks ago, but my colleagues have managed to build an ONOS 1.13.9 Docker image which unfortunately yields the same result.

I need to look into the details, but I think the flows are again not programmed properly, so at this point we are still kind of blocked on identifying this issue.

BR,

Ciprian

On Tuesday, October 1, 2019 at 1:44:46 AM UTC+3, cipria...@gmail.com wrote:

Hello Zack,

Unfortunately the timing was bad for me to start this thread in the first place, I went into a 3 weeks vacation starting last week.

Fortunately this work has been taken over by other colleagues and I think they made a preliminary build which might be tested this week.

We will update as soon as possible, for now things are going a bit slow.

Thanks and regards,

/Ciprian

To unsubscribe from this group and stop receiving emails from it, send an email to seba-dev+u...@opennetworking.org.
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/seba-dev/ec95904f-9f60-4b68-95bb-86b0e409c1e3%40opennetworking.org.

--

Thank you, Tina

--
You received this message because you are subscribed to the Google Groups "SEBA Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to seba-dev+u...@opennetworking.org.
To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/seba-dev/CAMS-Z8A%3DZV5svErmdobjYaUXVibSD4iukYi--jiMpA%3DW3LNcEQ%40mail.gmail.com.

[Saurav Das]

Sure no problem. First question -- Is this a SEBA 1.0 pod which used ONOS 1.13.5 and related apps, or is this a SEBA 2.0 pod which used ONOS 1.13.9 and related apps.

You cannot use 1.13.9 and still use the older apps for 1.13.5.

Regarding your problem with DHCP

do you have this flow?

https://guide.opencord.org/cord-6.1/profiles/seba/troubleshoot/no-dhcp.html#agg-switch-dhcp-flows

Thanks

Saurav

On Tue, Oct 15, 2019 at 12:00 PM Tina Tsou <tina...@gmail.com> wrote:

Dear Saurav,

Thanks for the update.

We only need someone to answer Ciprian’s questions from Akraino SEBA blueprint’s upstream requests.

Would you kindly answer his question he posted and reposted at seba mailing list? It has been a month.

--

Thank you, Tina

[Saurav Das]

But these missing flows and groups have to do with the vlan-xconnect. 

They should not have any impact on dhcp.

You said dhcp was not working -- is that working now?

On Wed, Oct 16, 2019 at 9:11 AM Ciprian Barbu <cipria...@gmail.com> wrote:

Hi,

Sorry for the late reply, I've used my personal email for this thread, my bad.

This is SEBA 1.0 with the related apps. I too realized that using ONOS 1.13.9 might not be a good idea, after analyzing the flows and logs and noticing things are not quite in order.

The flow you pointed is not the issue, I did some extensive work capturing logs and flows and comparing them to a working (x86) setup. From this work, I ended up with the following set of flows and flow groups that are missing on the aggregation switch (Openvswitch in this case). In fact, I'm showing the output from ovs-ofctl:

Missing flows:
=====================================
 cookie=0xac00004df3cb3e, duration=593.794s, table=10, n_packets=0, n_bytes=0, send_flow_rem in_port=1,dl_vlan=222 actions=goto_table:20
 cookie=0xac00003d987c88, duration=593.794s, table=10, n_packets=0, n_bytes=0, send_flow_rem in_port=2,dl_vlan=222 actions=goto_table:20
 cookie=0xac000074fa5059, duration=593.042s, table=50, n_packets=0, n_bytes=0, send_flow_rem priority=1000,dl_vlan=222 actions=write_actions(group:1088290816),goto_table:60
 cookie=0xac000062d0f17c, duration=593.779s, table=60, n_packets=0, n_bytes=0, send_flow_rem priority=60000 actions=drop
 cookie=0xb20000420c9478, duration=593.794s, table=63, n_packets=0, n_bytes=0, send_flow_rem priority=40000,in_port=2,dl_vlan=222 actions=CONTROLLER:65535
 cookie=0xb2000001f9e66b, duration=593.794s, table=63, n_packets=0, n_bytes=0, send_flow_rem priority=40000,in_port=1,dl_vlan=222 actions=CONTROLLER:65535

Missing groups:
=====================================
 group_id=14548993,type=indirect,bucket=actions=output:1
 group_id=14548994,type=indirect,bucket=actions=output:2
 group_id=1088290816,type=all,bucket=actions=group:14548993,bucket=actions=group:14548994

So as you can see I'm missing flows that handle the double tagged packets sent from the BNG (simulated by mininet app), to the aggregation switch.

But I think I might have another option, which is rebuilding ONOS 1.13.5 together with the aforementioned patch (by Charles Chan. Hopefully there won't be other dependencies.

I will update you as soon as I have news.

BR,

/Ciprian

On Wed, Oct 16, 2019, 16:50 Tina Tsou, <tina...@gmail.com> wrote:

Dear Ciprian,

Would you please answer it?

--

Thank you, Tina

[Saurav Das]

The vlan-tagged replies from the DHCP server for DHCP packets should not go through the AGG switch dataplane. 

They should be trapped to the controller from the AGG switch if there is a flow like this in table 60

the second pic here: https://guide.opencord.org/profiles/seba/troubleshoot/no-dhcp.html

requires this flow

ADDED, bytes=0, packets=0, table=60, priority=61000, selector=[IN_PORT:32, ETH_TYPE:ipv4, IP_PROTO:17, UDP_SRC:67, UDP_DST:68], treatment=[immediate=[OUTPUT:CONTROLLER], clearDeferred]

It is possible the SiaB in SEBA 1.0 did not work like this, and instead depended on a vlan-xconnect (which is not working for you)

But my question is - do you see such a flow?

Thanks
Saurav

On Wed, Oct 16, 2019 at 10:12 AM Ciprian Barbu <cipria...@gmail.com> wrote:

They do have an impact, because the DHCP replies from the simulated BNG are tagged. They never reach the agg switch. But the requests from the RG pod do reach the BNG.

[Charles Chan]

 Below is what I get in latest SiaB.

onos> flows -s any of:0000000000000001 60
deviceId=of:0000000000000001, flowRuleCount=13
    ADDED, bytes=1072, packets=3, table=60, priority=61000, selector=[IN_PORT:1, ETH_TYPE:ipv4, IP_PROTO:17, UDP_SRC:67], treatment=[clearDeferred, transition=TABLE:63]
    ADDED, bytes=850, packets=12, table=60, priority=60000, selector=[VLAN_VID:222], treatment=[immediate=[NOACTION]]
    ADDED, bytes=0, packets=0, table=60, priority=40000, selector=[ETH_TYPE:eapol], treatment=[clearDeferred, transition=TABLE:63]
    ADDED, bytes=0, packets=0, table=60, priority=40000, selector=[ETH_TYPE:lldp], treatment=[clearDeferred, transition=TABLE:63]
    ADDED, bytes=9460, packets=110, table=60, priority=40000, selector=[ETH_TYPE:bddp], treatment=[clearDeferred, transition=TABLE:63]
    ADDED, bytes=0, packets=0, table=60, priority=40000, selector=[ETH_TYPE:ipv4, IPV4_DST:192.168.0.201/32], treatment=[clearDeferred, transition=TABLE:63]
    ADDED, bytes=0, packets=0, table=60, priority=40000, selector=[ETH_TYPE:ipv6, IPV6_DST:fe80::200:2ff:fe01:601/128], treatment=[clearDeferred, transition=TABLE:63]
    ADDED, bytes=0, packets=0, table=60, priority=30000, selector=[ETH_TYPE:arp], treatment=[transition=TABLE:63]
    ADDED, bytes=0, packets=0, table=60, priority=30000, selector=[ETH_TYPE:ipv6, IP_PROTO:58, ICMPV6_TYPE:134], treatment=[transition=TABLE:63]
    ADDED, bytes=0, packets=0, table=60, priority=30000, selector=[ETH_TYPE:ipv6, IP_PROTO:58, ICMPV6_TYPE:135], treatment=[transition=TABLE:63]
    ADDED, bytes=0, packets=0, table=60, priority=30000, selector=[ETH_TYPE:ipv6, IP_PROTO:58, ICMPV6_TYPE:136], treatment=[transition=TABLE:63]
    ADDED, bytes=1542, packets=21, table=60, priority=30000, selector=[ETH_TYPE:ipv6, IP_PROTO:58, ICMPV6_TYPE:133], treatment=[transition=TABLE:63]
    ADDED, bytes=4294, packets=62, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]

I can reproduce the same issue if I manually remove xconnect. DHCP no longer works in that case.
I think the problem is in table 63, the packet-in table we implement in OVS to preserve original VLAN of the packet.

onos> flows -s any of:0000000000000001 63
deviceId=of:0000000000000001, flowRuleCount=4
    ADDED, bytes=19436, packets=226, table=63, priority=65535, selector=[ETH_TYPE:bddp], treatment=[immediate=[OUTPUT:CONTROLLER]]
    ADDED, bytes=0, packets=0, table=63, priority=65535, selector=[ETH_TYPE:lldp], treatment=[immediate=[OUTPUT:CONTROLLER]]
    ADDED, bytes=0, packets=0, table=63, priority=40000, selector=[IN_PORT:2, VLAN_VID:4090], treatment=[immediate=[OUTPUT:CONTROLLER]]
    ADDED, bytes=0, packets=0, table=63, priority=40000, selector=[IN_PORT:1, VLAN_VID:4090], treatment=[immediate=[OUTPUT:CONTROLLER]]

There should be a flow matching on VLAN=222, IN_PORT=1 and send to CONTROLLER

I guess the problem didn't surface with xconnect installed since the packet will go through ASG data plane and being packet_in at OLT.

I will follow up with a proposed solution soon.

Thanks,

Charles Chan, Ph.D.

Member of Technical Staff, Open Networking Foundation

On Wed, Oct 16, 2019 at 10:55 AM Ciprian Barbu <cipria...@gmail.com> wrote:

I need to double check, but I'm quite certain I have this flow, I'm no longer at the office right now.

You're right that the packets need to be trapped by the switch and sent to the controller, I didn't express myself correctly earlier.

I tried to use ovs-appctl to understand how these packets should travel through the flow tables, but I couldn't reach a conclusion.

But I can tell you for sure that the DHCP requests arrive to the dnsmasq process, double tagged with tags 222 and 111 (these are hardcoded by the ponsim charts). Then the dnsmasq replies with DHCP offer, and the packets are tagged back and sent on the same interfaces where the requests originated from.

At this point, ovs receives them, but they are never sent to the controller. They also do not reach the dhcpl2relay app in ONOS. The DHCP requests however can be seen in the dhcpl2relay trace in ONOS. So you see, the problem must be the missing flows and flow groups, I'm just confused how it is supposed to work.

To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/seba-dev/CAHpdbYFdi34Nv_5QbwUf%3D8kwFx-3VOrhCC3RC5rOmkOGMLj6SQ%40mail.gmail.com.

[Charles Chan]

To clarify, there will be such table 63 flow when xconnect is installed.
In that case, the DHCP packet will first hit table 60

ADDED, bytes=1072, packets=3, table=60, priority=61000, selector=[IN_PORT:1, ETH_TYPE:ipv4, IP_PROTO:17, UDP_SRC:67], treatment=[clearDeferred, transition=TABLE:63]

and then table 63 
ADDED, bytes=0, packets=0, table=63, priority=40000, selector=[IN_PORT:1, VLAN_VID:222], treatment=[immediate=[OUTPUT:CONTROLLER]]

[Saurav Das]

Thanks Charles, in the meantime, sounds like Ciprian was right - his problem will be solved if the xconnect works, the AGG switch forwards the dhcp packets to the OLT, and the OLT traps the packets to the controller.

The xconnect needs to work anyway for all other dataplane traffic to work. 

But the problem he is seeing with the xconnect is not one we have encountered with SiaB before - nevertheless lets go with the approach Ciprian is trying now to rebuild 1.13.5 with your patch and try again

[Charles Chan]

Here's my proposed fix: https://gerrit.onosproject.org/#/c/22723/

Thanks,
Charles Chan, Ph.D.

Member of Technical Staff, Open Networking Foundation

[Saurav Das]

Nice work Ciprian. Thanks Charles

On Fri, Oct 18, 2019 at 9:15 AM <cipria...@gmail.com> wrote:

Hi Charles,

I finally managed to build ONOS 1.13.5 with the patch you first indicated. I looked for dependencies with git-cherry, I noticed the change was actually cherry-picking another commit, but in the end I did git cherry-pick 6c48dba766f0e79a65098b94bc87f5f4d29dc3ff.

As I expected, this indeed did the trick for me, now I have all the flows for the ovs switch and DHCP step completed successfully. I also pinged the BNG, which in SiaB with PONSim has a fixed value of 172.18.0.10. It works fine.

I did notice a weird behavior, sometimes I would get a bunch of missing flows from table 0 (handling lldp, eapol bddp and ipv6), and also weird some of them seemed to appear in table 60. But this doesn't seem to be related to our issue and I still can't tell when it happens.

So I think you were right on point indicating this fix to me.

About your last change, I assume it applies to latest (master) ONOS, I don't expect it will land in SEBA 1.0 as well, but if it does, let me know, so I will build another ONOS image for aarch64.

I think we can consider this fixed, thanks for all the help!

BR,

/Ciprian

To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/seba-dev/ec95904f-9f60-4b68-95bb-86b0e409c1e3%40opennetworking.org.

--

Thank you, Tina

--
You received this message because you are subscribed to the Google Groups "SEBA Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to seba...@opennetworking.org.

To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/seba-dev/CAMS-Z8A%3DZV5svErmdobjYaUXVibSD4iukYi--jiMpA%3DW3LNcEQ%40mail.gmail.com.

--

Thank you, Tina

--

Thank you, Tina

--
You received this message because you are subscribed to the Google Groups "SEBA Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to seba...@opennetworking.org.

To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/seba-dev/CAHpdbYFdi34Nv_5QbwUf%3D8kwFx-3VOrhCC3RC5rOmkOGMLj6SQ%40mail.gmail.com.

[Charles Chan]

Ciprian,

Glad that you figure it out.

The second patch (https://gerrit.onosproject.org/#/c/22723/) fixes the issue that packet-in doesn't happen correctly on OVS under certain circumstances.
In real OF-DPA hardware, the DHCP flow in ACL [1] should send the packet to controller, even when the cross connect [2] is not setup. That didn't happen in OVS without the patch.
I just discussed with Saurav. We both agree that this is a critical issue and we decided to back port this to all ONOS LTS versions as well as 1.13.
Saurav mentioned that there is currently no plan to cut another release for SEBA 1.0.
Having said that, I would still encourage you to cherry-pick the second patch and try it out.

Please find the rest of my reply inline.

[1] ADDED, bytes=1072, packets=3, table=60, priority=61000, selector=[IN_PORT:1, ETH_TYPE:ipv4, IP_PROTO:17, UDP_SRC:67], treatment=[clearDeferred, transition=TABLE:63]
[2] More accurately, the filtering objective generated by cross connect

Thanks,
Charles Chan, Ph.D.

Member of Technical Staff, Open Networking Foundation

On Fri, Oct 18, 2019 at 9:15 AM <cipria...@gmail.com> wrote:

Hi Charles,

I finally managed to build ONOS 1.13.5 with the patch you first indicated. I looked for dependencies with git-cherry, I noticed the change was actually cherry-picking another commit, but in the end I did git cherry-pick 6c48dba766f0e79a65098b94bc87f5f4d29dc3ff.

As I expected, this indeed did the trick for me, now I have all the flows for the ovs switch and DHCP step completed successfully. I also pinged the BNG, which in SiaB with PONSim has a fixed value of 172.18.0.10. It works fine.

I did notice a weird behavior, sometimes I would get a bunch of missing flows from table 0 (handling lldp, eapol bddp and ipv6), and also weird some of them seemed to appear in table 60. But this doesn't seem to be related to our issue and I still can't tell when it happens.

[Charles] Table 0 flow typically indicates that the network config is not correctly loaded when the device (OVS) first connecting to ONOS.
This can be resolved by disconnecting and reconnecting OVS to ONOS.

 

So I think you were right on point indicating this fix to me.

About your last change, I assume it applies to latest (master) ONOS, I don't expect it will land in SEBA 1.0 as well, but if it does, let me know, so I will build another ONOS image for aarch64.

I think we can consider this fixed, thanks for all the help!

BR,

/Ciprian

On Friday, October 18, 2019 at 6:17:03 AM UTC+3, Charles Chan wrote:

To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/seba-dev/ec95904f-9f60-4b68-95bb-86b0e409c1e3%40opennetworking.org.

--

Thank you, Tina

--
You received this message because you are subscribed to the Google Groups "SEBA Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to seba...@opennetworking.org.

To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/seba-dev/CAMS-Z8A%3DZV5svErmdobjYaUXVibSD4iukYi--jiMpA%3DW3LNcEQ%40mail.gmail.com.

--

Thank you, Tina

--

Thank you, Tina

--
You received this message because you are subscribed to the Google Groups "SEBA Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to seba...@opennetworking.org.

To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/seba-dev/CAHpdbYFdi34Nv_5QbwUf%3D8kwFx-3VOrhCC3RC5rOmkOGMLj6SQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "SEBA Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seba-dev+u...@opennetworking.org.

To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/seba-dev/98249a8a-bc5d-42eb-ae4e-eb8a0c44b59d%40opennetworking.org.

[Charles Chan]

Hi Ciprian,

Thanks for testing it. I will merge this shortly.

Charles

On Fri, Nov 1, 2019 at 07:38 <cipria...@gmail.com> wrote:

Hi Charles,

Just FYI, since I'm not a contributor on opennetworking, I backported your second patch to 1.13.5 and verified it on my setup, it's ok, no problems encountered.

I cherry-picked this to our iecedge/onos repo here:

https://github.com/iecedge/onos/commit/fb493468359ded392006e24d978d3d8ca58efab6

To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/seba-dev/98249a8a-bc5d-42eb-ae4e-eb8a0c44b59d%40opennetworking.org.

--
You received this message because you are subscribed to the Google Groups "SEBA Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seba-dev+u...@opennetworking.org.

To view this discussion on the web visit https://groups.google.com/a/opennetworking.org/d/msgid/seba-dev/c5943d00-3806-4dc0-8385-a0d522b2ab42%40opennetworking.org.

--

Sent from mobile. Please excuse briefness and typo.