Problems when using application framework

Charles Min-Cheng Chan

unread,

Apr 22, 2015, 10:13:21 AM4/22/15

to onos...@onosproject.org

Hi,

I encountered several problems when using the application framework.

When uninstalling fwd app (either through REST or CLI), some warnings are observed in log.

Please refer to ONOS-1736 for detail.

In addition, after uninstall (deactivate) and reinstall (activate), two flow entries installed by fwd app disappear. The pings fail as a result.

I am wondering if this problem is a result of ONOS-1736 and the fwd app is not uninstalled and reinstalled properly.

Thanks in advance if someone can help investigating this problem.

Best,

Charles

--

Charles Min-Cheng Chan 詹珉誠

Ph.D. Candidate, National Chiao Tung University

mcchan.io

Thomas Vachuska

unread,

Apr 22, 2015, 11:43:19 AM4/22/15

to Charles Min-Cheng Chan, onos...@onosproject.org

Hello Charles,

The warnings you are seeing actually come from the component configuration subsystem and they are caused by the simultaneous withdrawal of configuration properties as the application is deactivated across the cluster. They are purely cosmetic and have no effect on the application uninstall.

Currently, the behaviour of the reactive forwarding app is to remove all flows as part of its deactivation so what you are seeing is an intended behaviour. Perhaps, if persistence of flows is desired, we could make this configurable.

Thomas

--
You received this message because you are subscribed to the Google Groups "ONOS Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+u...@onosproject.org.
To post to this group, send email to onos...@onosproject.org.
Visit this group at http://groups.google.com/a/onosproject.org/group/onos-dev/.
To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/CAJJLwsNAtcfqfwxXDS4RKPmsFV-xj0i%3DTaL08JRcCA2wMiJ-%3DQ%40mail.gmail.com.

Charles Min-Cheng Chan

unread,

Apr 22, 2015, 11:50:59 AM4/22/15

to Thomas Vachuska, onos...@onosproject.org

Thomas,

I totally agree with "the behaviour of the reactive forwarding app is to remove all flows as part of its deactivation."

However, these flows should be back after reinstallation/reactivation of fwd app.

In my case those flows are not back even after I reinstall and reactivate the fwd app.

Thanks,

Charles

--

Charles Min-Cheng Chan 詹珉誠

Ph.D. Candidate, National Chiao Tung University

mcchan.io

Hari Krishna

unread,

Apr 22, 2015, 12:00:04 PM4/22/15

to Charles Min-Cheng Chan, Thomas Vachuska, onos...@onosproject.org

Hi Charles,

I have verified this couple times yesterday and it should be working as part of ONOS-1647 (see the verify logs).

For exceptions and other errors we have couple bugs (ONOS-1733 and ONOS-1731) see if any of them match.

-Hari

To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/CAJJLwsPr0WEVNeFDPQ-mVaEY-Qz7yxgKjzLxXCxzoVM57-v%3DXA%40mail.gmail.com.

Thomas Vachuska

unread,

Apr 22, 2015, 12:12:15 PM4/22/15

to Hari Krishna, Charles Min-Cheng Chan, onos...@onosproject.org

Hari, I think Charles is encountering the opposite issue.

Charles, I made a fix for the log.warn bit. However, I was not able to replicate the behaviour where reactive forwarding fails to respond to IP or ARP traffic. I am able to repeatedly activate/deactivate and even uninstall/install the org.onosproject.fwd app and pings stop/start working as expected. When using this command, the pings don’t even flinch.

onos-app $OCI reinstall! org.onosproject.fwd ~/onos/apps/fwd/target/onos-app-fwd-1.2.0-SNAPSHOT.oar

Thomas

Charles Min-Cheng Chan

unread,

Apr 22, 2015, 12:13:22 PM4/22/15

to Hari Krishna, Thomas Vachuska, onos...@onosproject.org

Hari,

Thanks for the information.

ONOS-1733 includes the warning message I reported in ONOS-1736.

Sorry I didn't search carefully enough and created a duplicated issue.

The situation in ONOS-1647 is a little bit different with my case.

Here is my steps to reproduce the problem. Hope that helps. (I am reproducing it with commit d17abc2)

onos> apps -s

// the fwd app is activated

$ sudo mn --topo=tree,2 --controller=remote,ip=192.168.136.1

mininet> pingall

// ping OK

onos> app deactivate org.onosproject.fwd

// warnings in ONOS-1736 appears

onos> apps -s

// the fwd app is installed but deactivated

onos> flows

// flows installed by fwd disappear

onos> app activate org.onosproject.fwd

onos> apps -s

// the fwd app is activated

onos> flows

// flows installed by fwd still disappear

mininet> pingall

// ping fails

Thanks,

Charles

--

Charles Min-Cheng Chan 詹珉誠

Ph.D. Candidate, National Chiao Tung University

mcchan.io

Charles Min-Cheng Chan

unread,

Apr 22, 2015, 1:20:36 PM4/22/15

to Thomas Vachuska, Hari Krishna, onos...@onosproject.org

Thomas,

Thanks for fixing the warning message.

However, the reinstall! also does not work properly in my case.

At the end of this mail you can find the log during reinstallation.

There is one warning. The rest of them seem fine.

I fetched the latest master, and run onos-build, cell, onos-package, onos-install -nf, /opt/onos/apache-karaf-3.0.3/bin/karaf clean.

Am I missing anything?

Thanks,

Charles

--

Charles Min-Cheng Chan 詹珉誠

Ph.D. Candidate, National Chiao Tung University

mcchan.io

Charles Min-Cheng Chan

unread,

Apr 22, 2015, 2:10:25 PM4/22/15

to Hari Krishna, Thomas Vachuska, onos...@onosproject.org

Hari,

Hmm...That's weird.

In my case, the flows are removed after deactivation, but not added back after re-activation.

Charles

onos> flows | grep fwd

id=6000041598028, state=ADDED, bytes=0, packets=0, duration=31, priority=5, tableId=0 appId=org.onosproject.fwd

id=60000415996ae, state=ADDED, bytes=0, packets=0, duration=31, priority=5, tableId=0 appId=org.onosproject.fwd

id=600004159f487, state=ADDED, bytes=0, packets=0, duration=31, priority=5, tableId=0 appId=org.onosproject.fwd

id=60000415a0b0d, state=ADDED, bytes=0, packets=0, duration=31, priority=5, tableId=0 appId=org.onosproject.fwd

id=60000415a68e6, state=ADDED, bytes=0, packets=0, duration=31, priority=5, tableId=0 appId=org.onosproject.fwd

id=60000415a7f6c, state=ADDED, bytes=0, packets=0, duration=31, priority=5, tableId=0 appId=org.onosproject.fwd

onos> summary

node=192.168.136.1, version=1.2.0.SNAPSHOT

nodes=1, devices=3, links=4, hosts=0, SCC(s)=1, flows=15, intents=0

onos> app deactivate org.onosproject.fwd

onos> apps -s | grep fwd

6 org.onosproject.fwd 1.2.0.SNAPSHOT Reactive forwarding application using flow subsystem

onos> flows | grep fwd

onos> summary

node=192.168.136.1, version=1.2.0.SNAPSHOT

nodes=1, devices=3, links=4, hosts=0, SCC(s)=1, flows=9, intents=0

onos> app activate org.onosproject.fwd

onos> apps -s | grep fwd

* 6 org.onosproject.fwd 1.2.0.SNAPSHOT Reactive forwarding application using flow subsystem

onos> flows | grep fwd

onos> summary

node=192.168.136.1, version=1.2.0.SNAPSHOT

nodes=1, devices=3, links=4, hosts=0, SCC(s)=1, flows=9, intents=0

--

Charles Min-Cheng Chan 詹珉誠

Ph.D. Candidate, National Chiao Tung University

mcchan.io

On Thu, Apr 23, 2015 at 1:49 AM, Hari Krishna <ha...@onlab.us> wrote:

Hi Charles and Thomas,

I am able to reproduce the issue. Yes, this is the opposite of issue fixed by Thomas.
If ONOS boot up with fwd enabled I see the flows are added, but when I try to deactivate
now I do not see flows deleted.

-Hari
****************************************
Welcome to Open Network Operating System (ONOS)!
____ _ ______ ____
/ __ \/ |/ / __ \/ __/
/ /_/ / / /_/ /\ \
\____/_/|_/\____/___/

Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or type 'system:shutdown' or 'logout' to shutdown ONOS.

onos>
onos>
onos>
onos> nodes
Command not found: nodes
onos> nodes
Command not found: nodes
onos> nodes
id=10.128.40.41, address=10.128.40.41:9876, state=ACTIVE, updated=8m ago *
id=10.128.40.42, address=10.128.40.42:9876, state=ACTIVE, updated=8m ago
id=10.128.40.43, address=10.128.40.43:9876, state=ACTIVE, updated=8m ago
id=10.128.40.44, address=10.128.40.44:9876, state=ACTIVE, updated=8m ago
id=10.128.40.45, address=10.128.40.45:9876, state=ACTIVE, updated=8m ago
onos>
onos> summary
node=10.128.40.41, version=1.2.0.admin~2015/04/21@13:12
nodes=5, devices=2, links=2, hosts=0, SCC(s)=1, flows=10, intents=0
onos>
onos> apps | grep fwd
* id=16, name=org.onosproject.fwd, version=1.2.0, origin=ON.Lab, description=Reactive forwarding application using flow subsystem, features=[onos-app-fwd], featuresRepo=mvn:org.onosproject/onos-app-fwd/1.2.0-SNAPSHOT/xml/features, permissions=[]
onos> app deactivate org.onosproject.
org.onosproject.fwd org.onosproject.metrics org.onosproject.mobility
org.onosproject.openflow org.onosproject.proxyarp
onos> app deactivate org.onosproject.fwd
onos> apps | grep fwd
id=16, name=org.onosproject.fwd, version=1.2.0, origin=ON.Lab, description=Reactive forwarding application using flow subsystem, features=[onos-app-fwd], featuresRepo=mvn:org.onosproject/onos-app-fwd/1.2.0-SNAPSHOT/xml/features, permissions=[]
onos> summary
node=10.128.40.41, version=1.2.0.admin~2015/04/21@13:12
nodes=5, devices=2, links=2, hosts=0, SCC(s)=1, flows=10, intents=0
onos>
onos> summary
node=10.128.40.41, version=1.2.0.admin~2015/04/21@13:12
nodes=5, devices=2, links=2, hosts=0, SCC(s)=1, flows=10, intents=0
onos>

Thomas Vachuska

unread,

Apr 22, 2015, 2:16:38 PM4/22/15

to Charles Min-Cheng Chan, Hari Krishna, onos...@onosproject.org

Just to be sure… is this what happens?

fwd app installed and pings work
fwd app uninstalled and pings stop
fwd app installed and pings no longer work

Thomas

Hari Krishna

unread,

Apr 22, 2015, 1:49:57 PM4/22/15

to Thomas Vachuska, Charles Min-Cheng Chan, onos...@onosproject.org

Thomas Vachuska

unread,

Apr 22, 2015, 2:46:18 PM4/22/15

to Hari Krishna, Charles Min-Cheng Chan, onos...@onosproject.org

Hari,

I think what you are seeing is different from what Charles is seeing.

What you are seeing is ‘packet request’ flows, which are added by the fwd app stick around after that app is deactivated. This is on purpose. These are different from the flows that are added as part of reactive forwarding of IP & ARP traffic between hosts. The latter should be removed, but the former should stay. This has been done deliberately to keep the packet request mechanism simple at this time. We may add “reference counting” to packet requests at a later time, but for now packet requests, once issued, cannot be revoked unless one uses remove flow specifically - which is not recommended.

What Charles is seeing, if I understand correctly, is failure for reactive forwarding to install reactive forwarding flows.

Thomas

Charles Min-Cheng Chan

unread,

Apr 22, 2015, 2:47:03 PM4/22/15

to Thomas Vachuska, Hari Krishna, onos...@onosproject.org

Thomas,

In my case, yes.

The good news is I found the root cause.

When fwd first activated, we request IPv4 and ARP packets in ReactiveForwarding.java line 165 and 168 respectively.

When fwd is deactivated, all rules including the previous two is removed by ReactiveForwarding.java line 183.

However, we did NOT unregister the packet request with PacketStore.

When fwd is activated again, it will try to request packet

At this moment, the PacketStore believes that the same requests already exist so it returns a false in DistributedPacketStore line 194.

Therefore the request is not pushed to the devices.

Thanks,

Charles

--

Charles Min-Cheng Chan 詹珉誠

Ph.D. Candidate, National Chiao Tung University

mcchan.io

Charles Min-Cheng Chan

unread,

Apr 22, 2015, 3:03:05 PM4/22/15

to Thomas Vachuska, Hari Krishna, onos...@onosproject.org

Thomas,

What Charles is seeing, if I understand correctly, is failure for reactive forwarding to install reactive forwarding flows.

Not exactly. I am also seeing the failure of the packet_in request.

Something like this is missing:

id=60000c543ad37, state=ADDED, bytes=0, packets=0, duration=0, priority=5, tableId=0 appId=org.onosproject.fwd
selector=[ETH_TYPE{ethType=800}]
treatment=DefaultTrafficTreatment{immediate=[OUTPUT{port=CONTROLLER}], deferred=[], transition=None, cleared=false}

Of course, as a result, there are no forwarding rules since the IPv4 and ARP packets don't even trigger packet in.

A possible cause is mentioned in my previous mail.

Thomas Vachuska

unread,

Apr 22, 2015, 3:13:31 PM4/22/15

to Charles Min-Cheng Chan, Hari Krishna, onos...@onosproject.org

Ah… packet requests flows should not be attributed to the app. That’s where the defect lies. Working on it.

Thomas

Reply all

Reply to author

Forward