Vertx-Hazelcast SEVERE error can leave node in state with event bus timeouts or No Handlers errors

[Gordon Roylance]

The following exception occurred when two nodes shutdown. It left the remaining node that got this error in a bad state where the event bus was getting timeouts. After more testing I have found that the loss of the partition data lead to other failures where the event bus lost it's map of registered handlers. Is there a way to make the vertx class: HazelcastClusterManager more robust to retry removing the failed nodes registered handlers. Also, is there a way to cause each node to re-register their handlers without restarting all of the nodes, ideas?

WARNING: [172.17.0.8]:5701 [canopy] [3.5.2] This node does not have a connection to Member [172.17.0.20]:5701
 Dec 11, 2015 7:37:13 PM io.vertx.spi.cluster.hazelcast.HazelcastClusterManager
 SEVERE: Failed to handle memberRemoved
 com.hazelcast.spi.exception.WrongTargetException: WrongTarget! this:Address[172.17.0.16]:5701, target:Address[172.17.0.22]:5701, partitionId: 0, replicaIndex: 0, operation: com.hazelcast.map.impl.operation.QueryPartitionOperation, service: hz:impl:mapService
 	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:246)
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:127)
...

 	at ------ End remote and begin local stack-trace ------.(Unknown Source)
 	at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:384)
	at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:334)
	at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
 	at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
	at com.hazelcast.map.impl.BasicMapContextQuerySupport.addResultsOfPredicate(BasicMapContextQuerySupport.java:350)
	at com.hazelcast.map.impl.BasicMapContextQuerySupport.query(BasicMapContextQuerySupport.java:262)
	at com.hazelcast.map.impl.proxy.MapProxySupport.query(MapProxySupport.java:1157)
 	at com.hazelcast.map.impl.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:575)
	at com.hazelcast.map.impl.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:560)
 	at io.vertx.core.impl.HAManager.nodeLeft(HAManager.java:278)
 	at io.vertx.core.impl.HAManager.access$100(HAManager.java:96)
	at io.vertx.core.impl.HAManager$1.nodeLeft(HAManager.java:144)
	at io.vertx.spi.cluster.hazelcast.HazelcastClusterManager.memberRemoved(HazelcastClusterManager.java:232)
	at com.hazelcast.cluster.impl.ClusterServiceImpl.dispatchEvent(ClusterServiceImpl.java:1422)
...

[Clement Escoffier]

Hi,

Could you provide a reproducer ?

Clement

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/36655668-2e32-455d-b35d-41f281381ef0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Fox]

https://github.com/hazelcast/hazelcast/issues/3395

It might be worth asking on the Hazelcast group if there is any resolution.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/1977D14F-BE43-4627-8644-DDD3A2D58BD2%40gmail.com.

[Gordon Roylance]

A simple reproducer is to create 3 services. The first service registers a handler and periodically displays the cluster maps __vertx.haInfo and __vertx.subs details. The second service registers a handler and sets a timer to System.exit(1) in 30 secs. The third service registers a handler and exits in 60 seconds. You may have to adjust the timing but I was able to see hazelcast repeat errors and/or lose cluster map data related to the event bus. 

Hazelcast appears to be getting in a bad state when either a migration is occurring, or it has detected that a node is down and is in the process of recovering and then another hazelcast node goes down. Hazelcast setting changes that have improved hazelcast's ability to work better in a double failure scenario.

        <property name="hazelcast.max.no.heartbeat.seconds">default 500, set it to 5 seconds</property>

        <property name="hazelcast.migration.min.delay.on.member.removed.seconds">default 5, but for testing purposes set it to 60 seconds</property>

Another issue is making sure that there are sufficient hazelcast backups of both __vertx.haInfo and __vertx.subs cluster maps. By default there is only 1 backup for __vertx.subs in the default xml. But with the way hazelcast works with partitions and with a double failure scenario, data loss can occur if there is only 1 backup. Particularly for the registered handlers this can be troubling, even with graceful exiting nodes.

I have also tested rapid scaling and descaling and have thrown in periodic sig-term and sig-kill events. Data loss is unfortunately happening sometimes, even with the improvements mentioned above. This is what lead to me asking if there is a way to get the vertx-hazelcast cluster manager to be self-healing. Perhaps on a memberRemoved event the node checks to see if it's own verticle and handler data is still properly in the maps. I am sure old stale data is a concern too. Do you think this can be added? When the cluster gets in this bad state it is very detrimental to the event bus messaging of the whole cluster. Not to mention the difficulty of monitoring and knowing how to recover in production. Incidentally, I am working on building a small periodic process that checks the node's map when a node is removed and forcing vertx to close when missing data occurs, since our services are auto-restarted when the process dies. Not ideal, but better than an unhealthy event bus.

[Tim Fox]

Hi Gordon,

I think the behaviour you have seen with hazelcast is to be expected.

Hazelcast maps work by partitioning the data across different nodes of the cluster. On top of that you can also configure it to backup the data to N other nodes. This number is configurable in the cluster config and the default value in the Vert.x cluster config is 1 (as you point out).

What this means is there is possibility of data loss if more than N nodes are shutdown or crash at the same time - after all those N nodes you shutdown might be the only ones that contain a specific piece of data. This isn't a bug in Hazelcast or Vert.x it's just the way HZ works - I also think you would find the same behaviour in most other similar clustered map or other similar technologies.

In order to allow more nodes to fail/shutdown simultaneously you can increase N in your config. The best reliability you can get is where N = the number of nodes in the cluster (i.e. data is replicated on to every node) Clearly there is a trade off between reliability and performance/memory overhead here.

I'm not an expert on hazelcast but I'd definitely recommend posting on their google group, they may have some more advanced hints and tips that allow you to configure replication/backups to suit your setup.

HTH

-- You received this message because you are subscribed to the Google Groups "vert.x" group. To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com. Visit this group at https://groups.google.com/group/vertx. To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/6cf50507-63a5-4dab-b5eb-8e9c0d81000c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.

[Gordon Roylance]

Thanks Tim, I agree that it's an expected behavior. I didn't mean to imply that the hazelcast map backup configuration was a vertx issue, we have been looking to improve the various issues in our specific type of environment. We have been learning and adjusting, over the last few weeks, our hazelcast configuration, our deployment process and our deployment tools. We have reached out to hazelcast support, there response took a few days (just one of our various hazelcast issues we have seen) regarding the exception above: 

"It appears to be caused by Vert.x calling back into Hazelcast on the Hazelcast event thread. When a node leaves the cluster, Vert.x is notified and on the same thread queries the Hazelcast IMap it uses to keep track of its cluster. This likely prevents the event from propagating properly, or some deadlock situation occurs where the reties are exhausted.

Hazelcast encourages offloading event processing to a separate thread, to improve isolation. This is a qualified guess, since it would take some engineering effort to really dig into this."

I am not sure if this is helpful to you or not, but we haven't seen the exception for some time now, due to some adjustments we have made. I added the ability for our verticles to auto-restart if they notice that the registered handler count drops for a specified period of time. This effectively self heals our cluster. We are anxious to try ZooKeeeper cluster manager once it's available, we think it will be better suited for our environment. We use Rancher to deploy and manager our cluster. Issues we are currently looking into in our environment:

1) During a verticle upgrade, we deploy a new docker container, allow it to run for 20 secs and then take down the old verticle docker container. Once in a while we will still see a handler or two disappear from the map. In this scenario, we signal the container to shutdown gracefully, we have ensured that the map is backed up on more than one other node. We think that there may be a race condition for allowing hazelcast enough time shutdown properly.

2) We have seen issues with hazelcast members dropping off when a particular host is temporarily strained by heavy use. Configuration changes can help in this scenario, but it's difficult to balance the problem where a node has failed and we want to have the member removed quickly (trying to avoid as many issues with the event bus as possible), and that the system will recover and we just need to give it more time.

I am just sharing these in case others have seen similar issues and want to share what they did. Also, it has been interesting to consider how the vertx eventbus could be even more robust in some of these scenarios.

[Guillermo Fritz]

Hi guys,

I think that I'm having a similar issue like you described in this post. I'm using vertx 3.2.1 & hazelcast 3.6.1 (the latest) because I needed the IMap's EntryEvictedListener. 

As far I play with it it works fine. For example I have two instances of a webapp using vertx, and other two using hazelcast. My intention is to share the same HZ cluster (same HZ group & config xml) between the 4 members because I wanted to share a ITopic without adding vertx dependencies (for now) to the only-hz projects.

So, when I restart some of the non-vertx instances the vertx-EventBus starts to give timeouts. Then I saw (with log trace level of HZ classes) this message in the non-vertx instance log: TRACE 25503 [hz.simHZ.partition-operation.thread-3] -> (StandardLoggerFactory.java:49) Config: No configuration found for __vertx.subs, using default config!

If I stop the non-vertx apps or change it's hz-group name it works ok... any ideas or workaround for sharing HZ with instances that not have vertx?

Thank you!

[Clement Escoffier]

Hello,

Sharing a HZ cluster with non-vert.x systems is not recommended as it may breaks HA. 

If you want to go down this road, be aware that the vert.x configuration needs to be included in the HZ configuration. Snippet is provided here: https://github.com/vert-x3/vertx-hazelcast/blob/master/src/main/asciidoc/java/index.adoc#using-an-existing-hazelcast-cluster.

Clement

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/beda91e2-0efb-4c2d-a271-6af8ed553f38%40googlegroups.com.