FW: [onos-dev] Inconsistent node in DCS on master cluster node when configuration data node added/updated on leader cluster node.

[Ashish Mittal]

Forwarded to DCS Brigade

 

Regards

Ashish Mittal

 

From: suhail khan [mailto:suhailk...@gmail.com]
Sent: Thursday, August 30, 2018 06:05 PM
To: ONOS Developers <onos...@onosproject.org>
Subject: Re: [onos-dev] Inconsistent node in DCS on master cluster node when configuration data node added/updated on leader cluster node.

 

 

Hi community,

 

Kindly, help us. We are stuck due to issue mentioned in  trailing mail 😞.

 

regards,

Suhail

 

On Wed, Aug 29, 2018 at 12:04 PM suhail khan <suhailk...@gmail.com> wrote:

Hi all,

 

We have created an app using DCS to push configuration on netconf devices with ONOS-1.12.0.

As per app design, leader cluster node create/update/remove configuration node in DCS. After listening the events from DCS, master node push configuration on device using netconf protocol. Master cluster node need to read the node from DCS, after full configuration is successfully added/updated in DCS by leader.

 

Issues we faced with this design.

 

DCS start sending DynamicConfigEvent for every node updation/addition in DCS in place of single event after configuration is fully added/updated by leader in DCS.

 

To resolve this limitation, we tried multiple solution and faced following new issue.

1.  Accumulator with large idle time and small batch life time to trigger processing in accumulator at regular time interval. Here we had used isRelavent() method to filter out unrelated events.
   
   >> Issue faced: Many times, when master read the configuration node, it was not complete. Seems the event we relied on came before full configuration was added/updated. To resolve this we increased the batch life time but still less than idle time. Frequency of appearance of issue reduced but still sometime we faced same issue.
2.  Accumulator with large batch life time and small idle time. Removed our implementation of isRelevant() and filtered out non relevant events while processing. With small idle time, we had the assumption that when configuration is fully updated/added by leader, accumulator will become idle. At this time processing of batch will be triggered.
   
   >> This resolved the issue quite efficiently with single threaded implementation. But as app was taking huge time when multiple requests were submitted simultaneously. We moved to multi-threaded implementation. Here we again faced new issue with DCS. When multiple thread starts updating/adding in DCS on behalf of multiple request. As accumulator continuously receive event for long duration, idle time reach after long period. This result into expiration of multiple timers in our app. As time of accumulator process trigger is not fixed. We cannot configure other app timer as per accumulator trigger timer.
3. Removed accumulator, maintained consistent map with ResourceId as key, and operation as value. Leader after add/update in DCS, add/update in consistent map. Master node stop listening to DCS event, in place start listening to map event. Here our assumption was, createNode()/updateNode() api provided by DynamicConfigService is a blocking call, means main thread on leader will wait until configuration node is correctly updated/added in DCS and properly synced on all cluster node. After listening to events from consistent map, app use resourceId of map event to read from DCS.
   
   >>  It seems, our assumption is wrong. Although issue became rare, but still we have observed, sometime when reading DCS node on master configuration is not full.

 

Community, kindly suggest how can we detect on master node when configuration is completely added/updated in DCS by leader in multi-threaded environment.

 

Regards,

Suhail

--
You received this message because you are subscribed to the Google Groups "ONOS Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+u...@onosproject.org.
To post to this group, send email to onos...@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.
To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/CAM%2BXdaYKKWg0yKYVX%2BP6S8JFVJFFMmPsU2Obu0P0MHUUZF%2Br%3Dw%40mail.gmail.com.

=====================================================
Please refer to http://www.aricent.com/email-disclaimer
for important disclosures regarding this electronic communication.
=====================================================