Clustering, Mastership not changed if node goes down

rohit sharan

unread,

Nov 5, 2017, 1:01:53 AM11/5/17

to ONOS Developers

HI,

we are facing strange issue in cluster.

we have 3 node cluster, if we make one cluster down repatedly. one node goes down,

but devices under that node still intact with node. In "nodes" command, it shows node is inactive, but in masters command

it is showing some devices are intact with it. Observe one more thing, if before 10 devices are under that node before going INACTIVE,

then after making node down, some devices only moved to other controller node, some remain intact with INACTIVE node.

we are using onos version 1.9.0.

ONOS logs below:-

BEFORE Making node down:-

onos> nodes

id=192.168.0.10, address=192.168.0.10:9876, state=READY, version=1.9.0, updated=2m37s ago

id=192.168.0.11, address=192.168.0.11:9876, state=READY, version=1.9.0, updated=4m5s ago

id=192.168.0.12, address=192.168.0.12:9876, state=READY, version=1.9.0, updated=6m9s ago *

onos> masters

192.168.0.10: 12 devices

of:c0a800a000000000

of:a0a800e000000002

of:45a800e000000000

of:c230a80000000002

of:250a800e00000001

of:1242c0a800000000

of:087a800e00000001

of:1309a80000000000

of:320a800e00000001

of:2209801200000000

of:009087e000000001

of:a0a800e000000001

192.168.0.11: 0 devices

192.168.0.12: 0 devices

AFTER Making one node down

onos> nodes

id=192.168.0.10, address=192.168.0.10:9876, state=INACTIVE, version=1.9.0, updated=1m5s ago

id=192.168.0.11, address=192.168.0.11:9876, state=READY, version=1.9.0, updated=5m11s ago

id=192.168.0.12, address=192.168.0.12:9876, state=READY, version=1.9.0, updated=7m15s ago *

onos> masters

192.168.0.10: 5 devices

of:1309a80000000000

of:320a800e00000001

of:2209801200000000

of:009087e000000001

of:a0a800e000000001

192.168.0.11: 0 devices

192.168.0.12: 7 devices

of:c0a800a000000000

of:a0a800e000000002

of:45a800e000000000

of:c230a80000000002

of:250a800e00000001

of:1242c0a800000000

of:087a800e00000001

More the devices, more frequent this issue came. If we try make nodes down sequentially one by another (node which hase devices), issue generate in some iterations.

and this situation is not recoverable, if we make node again up, still devices stuck with same node,

Tried with balance-masters, but same issue.

we have observe same issue in 1.11.0 also.

Regards

Rohit

Thomas Vachuska

unread,

Nov 5, 2017, 12:19:14 PM11/5/17

to rohit sharan, ONOS Developers

This is odd indeed. We will have to investigate why this is happening as this is something we’re testing for and therefore should be seeing as well in our HA tests.

Thomas

--
You received this message because you are subscribed to the Google Groups "ONOS Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+u...@onosproject.org.
To post to this group, send email to onos...@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.
To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/6d0e35c7-5eba-4bad-a09c-ac925bcfc3c6%40onosproject.org.

Jordan Halterman

unread,

Nov 5, 2017, 1:45:43 PM11/5/17

to Thomas Vachuska, rohit sharan, ONOS Developers

I haven't been able to reproduce this yet today over dozens of restarts... Are you only killing one node at a time? Are you killing only the same node over and over? It sounds like one of the Raft partitions is not detecting the failure and reassigning the masters.

Also, looking at the mastership code, I think there's a problem with balancing masters when devices are stuck on an unavailable node right now. It looks like we're only balancing masters for active nodes and are effectively ignoring the inactive node, so using balance-masters has no effect in this case. I'm not sure there's a good reason to ignore devices that are assigned to inactive nodes.

Thomas

To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+unsubscribe@onosproject.org.

To post to this group, send email to onos...@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.
To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/6d0e35c7-5eba-4bad-a09c-ac925bcfc3c6%40onosproject.org.

--
You received this message because you are subscribed to the Google Groups "ONOS Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+unsubscribe@onosproject.org.

To post to this group, send email to onos...@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.

To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/35E25982-3CC5-4FD5-9D87-30857A65BEC2%40opennetworking.org.

rohit sharan

unread,

Nov 6, 2017, 1:51:55 AM11/6/17

to ONOS Developers

Hi,
We are killing node which has devices. Not same node again. When Node goes to inactive, all devices shifted to other node.
We have noted also, when devices shifting happens, all devices under an inactive node shifted to only single READY node. Devices not balanced automatically.
And in our test we have not used balance-master. So all devices exist under mastership of single node only.

Regards
Rohit

Thomas Vachuska

unread,

Nov 6, 2017, 1:01:08 PM11/6/17

to Jordan Halterman, rohit sharan, ONOS Developers

Even though this is really a secondary problem to what Rohit is seeing, we should definitely fix that. There is not reason to let an inactive node to hold onto mastership of devices during a balancing scenario. Looking at it now.

Thomas

Thomas Vachuska

unread,

Nov 6, 2017, 1:04:23 PM11/6/17

to rohit sharan, ONOS Developers

As things are now, this is not a defect, but rather what should be expected as controller node load balancing is not at all engaged during failover scenario; only a mastership election is run in order to fail over as quickly as possible. Thus what you are seeing, where all devices are adopted by another controller node is normal. We can certainly discuss on how to improve this.

Thomas

Thomas Vachuska

unread,

Nov 6, 2017, 2:08:27 PM11/6/17

to Vivek Soni, ONOS Developers, Jordan Halterman

Yes, I understand that this is the primary problem and that it should not happen. Inactive controller nodes should not be left in charge of mastership over any devices whatsoever. At most what should happen is that a device will be left with no master and that is only in cases where the device has no connection to any of the remaining active controller nodes.

From what Rohit described, there are to unexpected behaviours. 1) nodes retain mastership of devices even when inactive and 2) after node failure, mastership of devices is assigned without regard to balancing the mastership load. The fact that manual rebalancing does not serve as a work-around for the first issue is a secondary issue as our mastership load-balancing assumes that only active nodes have mastership and therefore does not attempt to include any “orphaned devices”; devices, which for some reason failed to be re-assigned a new master when their old one failed.

Clearly we should fix #1 and potentially also the secondary issue of the load-balancing algorithm so that it considers possibility of such orphaned devices. With respect to #2, that is currently designed to operate as observed, but we can certainly discuss whether to change this.

Thomas

On Nov 6, 2017, at 10:56, Vivek Soni <tellv...@gmail.com> wrote:

Hi Thomas
The issue is not about mastership load balancing. The original issue what rohit mentioned is that sometimes we see that a node retains the mastership even after it is inactive.

Can you suggest any pointers where to look for to find the root cause.

Thanks

--
You received this message because you are subscribed to the Google Groups "ONOS Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+unsubscribe@onosproject.org.
To post to this group, send email to onos...@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.

To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/A90C0ABB-41F3-4A70-B722-B2450459FDC9%40opennetworking.org.

Thomas Vachuska

unread,

Nov 6, 2017, 2:27:02 PM11/6/17

to rohit sharan, ONOS Developers

Rohit,

Can you describe your setup in more detail so that we can duplicate the issue in our environment? Without that, it will be difficult to troubleshoot it.

What environment are you using to run the cluster? (VMs, containers, bare-metal? How is the controller inter-networking setup?)

What network devices are you using? (HW, mining with ovs or some other soft-switch?)

Thomas

On Nov 5, 2017, at 22:51, rohit sharan <rohit...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "ONOS Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+u...@onosproject.org.

To post to this group, send email to onos...@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.

To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/c092c568-789f-4bb0-926a-4f82c1e7eca0%40onosproject.org.

rohit sharan

unread,

Nov 6, 2017, 3:04:38 PM11/6/17

to ONOS Developers

Hello Thomas ,
Test environment is as follows:
4 compute nodes running software ovs switch with two bridges each. i.e. equal to 8 devices
4 pica8 openflow switches
One cisco hw switch

We are running 3 onos controller nodes in cluster.
First all devices Are in mastership of one node.
Make that node down
Check masters. devices should move to other node.
Make that node up again.
As soon as, nodes command shows ready of all nodes, make other node down which have all devices. And check masters again.
We need to do some iterations of above procedure.

Thanks
Rohit

Jordan Halterman

unread,

Nov 7, 2017, 3:25:59 AM11/7/17

to rohit sharan, ONOS Developers

FYI the reason mastership appears to be assigned to one node is just because it's a queue. The longest running node will typically be assigned mastership for all orphaned devices. But that can certainly be changed. In fact, recent changes to partitioning make this a lot more feasible.

Anyways, I went back with the additional info about how the issue has been produced and I ran tests killing nodes in the same pattern. I haven't yet been able to reproduce this particular issue after numerous runs, but it did produce one interesting but seemingly unrelated problem that I'm looking into.

> --
> You received this message because you are subscribed to the Google Groups "ONOS Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+u...@onosproject.org.
> To post to this group, send email to onos...@onosproject.org.
> Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.

> To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/dbfa55fe-bb04-4a4d-b3ac-71405a3f49a7%40onosproject.org.

Thomas Vachuska

unread,

Nov 7, 2017, 2:50:36 PM11/7/17

to rohit sharan, ONOS Developers

Thanks for the info Rohit.

In the mean-time, here’s a fix for the secondary issue. This fix could at least provide a work-around to allow explicit load-balancing to re-distribute any devices which may be stuck with mastership on an inactive node. Clearly, we’ll try to get at the bottom of the primary issue on a separate timeline.

https://gerrit.onosproject.org/#/c/15969/

Thomas

--
You received this message because you are subscribed to the Google Groups "ONOS Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+u...@onosproject.org.
To post to this group, send email to onos...@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.

To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/dbfa55fe-bb04-4a4d-b3ac-71405a3f49a7%40onosproject.org.

Kishore Darapu

unread,

Nov 13, 2017, 4:49:33 AM11/13/17

to Thomas Vachuska, rohit sharan, ONOS Developers

Thanks Thomas,

Even we faced similar issue, I thought the below fix would be helpful for Rohit's issue. Could you please review and provide your comments?

https://gerrit.onosproject.org/#/c/16011/

Regards,

Kishore

Regards,

Kishore

On Wed, Nov 8, 2017 at 1:20 AM, Thomas Vachuska <t...@opennetworking.org> wrote:

Thanks for the info Rohit.

In the mean-time, here’s a fix for the secondary issue. This fix could at least provide a work-around to allow explicit load-balancing to re-distribute any devices which may be stuck with mastership on an inactive node. Clearly, we’ll try to get at the bottom of the primary issue on a separate timeline.

https://gerrit.onosproject.org/#/c/15969/

Thomas

On Nov 6, 2017, at 12:04, rohit sharan <rohit...@gmail.com> wrote:

Hello Thomas ,
Test environment is as follows:
4 compute nodes running software ovs switch with two bridges each. i.e. equal to 8 devices
4 pica8 openflow switches
One cisco hw switch

We are running 3 onos controller nodes in cluster.
First all devices Are in mastership of one node.
Make that node down
Check masters. devices should move to other node.
Make that node up again.
As soon as, nodes command shows ready of all nodes, make other node down which have all devices. And check masters again.
We need to do some iterations of above procedure.

Thanks
Rohit

--
You received this message because you are subscribed to the Google Groups "ONOS Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+unsubscribe@onosproject.org.

To post to this group, send email to onos...@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.
To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/dbfa55fe-bb04-4a4d-b3ac-71405a3f49a7%40onosproject.org.

--
You received this message because you are subscribed to the Google Groups "ONOS Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+unsubscribe@onosproject.org.

To post to this group, send email to onos...@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.

To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/924525F4-2BCD-41B8-ABD2-6DA983CDB804%40opennetworking.org.

Jordan Halterman

unread,

Nov 13, 2017, 2:14:31 PM11/13/17

to Kishore Darapu, Thomas Vachuska, rohit sharan, ONOS Developers

I'm concerned this change is just a bandaid that's going to hide an underlying problem. If mastership is not being changed on failures, there are really two possible reasons why:

• A Raft partition lost availability and thus can't commit any state changes

• A Raft session is not being expired

Mastership changes occur on Raft session expirations, so if mastership is not being reassigned on a failure, that implies Raft sessions are not being expired. That would have implications far beyond mastership elections.

IMO we need to get TRACE logs from a cluster that is experiencing this problem to fix it. I haven't been able to reproduce it, but I'll take another shot at it today.

To unsubscribe from this group and stop receiving emails from it, send an email to onos-dev+u...@onosproject.org.

To post to this group, send email to onos...@onosproject.org.
Visit this group at https://groups.google.com/a/onosproject.org/group/onos-dev/.

To view this discussion on the web visit https://groups.google.com/a/onosproject.org/d/msgid/onos-dev/CAPO-qz1RL1ebFW09eEizGbBPb%2BH1qq6OJHzZzQi6HLZOunjkVQ%40mail.gmail.com.

Subhasis Mohapatra

unread,

Nov 14, 2017, 4:23:57 AM11/14/17

to ONOS Developers

Hi,

I faced similar issue.Please find the logs below.

In my test exception is coming while sending heartbeat to peer in DistributedClusterStore.

Please check if it is helpful.

2017-11-14 01:59:11,859 | DEBUG | 9876-partition-2 | LeaderAppender | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent AppendRequest[term=20, leader=1802383516, logIndex=1232091, logTerm=20, entries=[0], commitIndex=1232091, globalIndex=1231989] to /107.110.6.198:9876

2017-11-14 01:59:11,868 | DEBUG | 9876-partition-2 | LeaderAppender | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent AppendRequest[term=20, leader=1802383516, logIndex=1232091, logTerm=20, entries=[1], commitIndex=1232091, globalIndex=1231989] to /107.110.6.198:9876

2017-11-14 01:59:11,870 | DEBUG | 9876-partition-2 | LeaderState | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent KeepAliveResponse[status=OK, error=null, leader=/107.110.6.181:9876, members=[/107.110.6.184:9876, /107.110.6.181:9876, /107.110.6.198:9876]]

2017-11-14 01:59:11,951 | DEBUG | 9876-partition-3 | LeaderAppender | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent AppendRequest[term=7, leader=1802383516, logIndex=32436874, logTerm=7, entries=[0], commitIndex=32436874, globalIndex=32432427] to /107.110.6.198:9876

2017-11-14 01:59:12,030 | DEBUG | 9876-partition-1 | FollowerState | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Appended KeepAliveEntry[index=1230269, term=5, session=303880, commandSequence=2792, eventIndex=1230258, timestamp=1510642750657] to log at index 1230269

2017-11-14 01:59:12,031 | DEBUG | 9876-partition-1 | FollowerState | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent KeepAliveResponse[status=OK, error=null, leader=/107.110.6.198:9876, members=[/107.110.6.184:9876, /107.110.6.181:9876, /107.110.6.198:9876]]

2017-11-14 01:59:12,109 | DEBUG | 9876-partition-2 | LeaderAppender | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent AppendRequest[term=20, leader=1802383516, logIndex=1232092, logTerm=20, entries=[0], commitIndex=1232092, globalIndex=1231989] to /107.110.6.198:9876

2017-11-14 01:59:12,201 | DEBUG | 9876-partition-3 | LeaderAppender | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent AppendRequest[term=7, leader=1802383516, logIndex=32436874, logTerm=7, entries=[0], commitIndex=32436874, globalIndex=32432427] to /107.110.6.198:9876

2017-11-14 01:59:12,328 | DEBUG | 9876-partition-3 | LeaderAppender | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent AppendRequest[term=7, leader=1802383516, logIndex=32436874, logTerm=7, entries=[1], commitIndex=32436874, globalIndex=32432427] to /107.110.6.198:9876

2017-11-14 01:59:12,331 | DEBUG | 9876-partition-3 | LeaderState | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent KeepAliveResponse[status=OK, error=null, leader=/107.110.6.181:9876, members=[/107.110.6.184:9876, /107.110.6.181:9876, /107.110.6.198:9876]]

2017-11-14 01:59:12,345 | DEBUG | 9876-partition-1 | FollowerState | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Appended KeepAliveEntry[index=1230270, term=5, session=306694, commandSequence=2485, eventIndex=1230258, timestamp=1510642750972] to log at index 1230270

2017-11-14 01:59:12,360 | DEBUG | 9876-partition-2 | LeaderAppender | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent AppendRequest[term=20, leader=1802383516, logIndex=1232092, logTerm=20, entries=[0], commitIndex=1232092, globalIndex=1231989] to /107.110.6.198:9876

2017-11-14 01:59:12,434 | DEBUG | 9876-partition-3 | LeaderAppender | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent AppendRequest[term=7, leader=1802383516, logIndex=32436875, logTerm=7, entries=[1], commitIndex=32436875, globalIndex=32432427] to /107.110.6.198:9876

2017-11-14 01:59:12,437 | DEBUG | 9876-partition-3 | LeaderState | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent KeepAliveResponse[status=OK, error=null, leader=/107.110.6.181:9876, members=[/107.110.6.184:9876, /107.110.6.181:9876, /107.110.6.198:9876]]

2017-11-14 01:59:12,451 | DEBUG | 9876-partition-3 | LeaderAppender | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent AppendRequest[term=7, leader=1802383516, logIndex=32436876, logTerm=7, entries=[0], commitIndex=32436876, globalIndex=32432427] to /107.110.6.198:9876

2017-11-14 01:59:12,609 | DEBUG | 9876-partition-2 | LeaderAppender | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent AppendRequest[term=20, leader=1802383516, logIndex=1232092, logTerm=20, entries=[0], commitIndex=1232092, globalIndex=1231989] to /107.110.6.198:9876

2017-11-14 01:59:12,701 | DEBUG | 9876-partition-3 | LeaderAppender | 90 - io.atomix.all - 1.0.3 | /107.110.6.181:9876 - Sent AppendRequest[term=7, leader=1802383516, logIndex=32436876, logTerm=7, entries=[0], commitIndex=32436876, globalIndex=32432427] to /107.110.6.198:9876

2017-11-14 01:59:12,840 | TRACE | sagingEvt-nioC-2 | DistributedClusterStore | 144 - org.onosproject.onos-core-dist - 1.9.0 | Sending heartbeat to Endpoint{ip=107.110.6.184, port=9876} failed

io.netty.channel.ConnectTimeoutException: connection timed out: /107.110.6.184:9876

at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267)[98:io.netty.transport:4.1.5.Final]

at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)[96:io.netty.common:4.1.5.Final]

at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)[96:io.netty.common:4.1.5.Final]

at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)[96:io.netty.common:4.1.5.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418)[96:io.netty.common:4.1.5.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:440)[98:io.netty.transport:4.1.5.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)[96:io.netty.common:4.1.5.Final]

at java.lang.Thread.run(Thread.java:745)[:1.8.0_65]

2017-11-14 01:59:12,840 | DEBUG | heartbeat-sender | NettyMessagingManager | 144 - org.onosproject.onos-core-dist - 1.9.0 | Closing connection to Endpoint{ip=107.110.6.184, port=9876}