Issues with vertx 3.7, hazelcast, and rolling restarts in openshift

[annette....@gmail.com]

Since updating to vertx 3.7, we have been experiencing rolling restart issues with hazelcast clustering. 

I'm aware that Hazelcast has released fixes for similar issues in their 3.12 release

Is there any plan to move vertx-hazelcast to Hazelcast 3.12?

Since vertx-hazelcast uses hazelcast 3.10.5, we are using hazelcast-kubernetes 1.2.2

Here is one example of the errors we're seeing continuously :

2019-04-24 15:15:05,763 [vertx-blocked-thread-checker] WARN  io.vertx.core.impl.BlockedThreadChecker$1 run:55 - Thread Thread[vert.x-worker-thread-9,5,main] has been blocked for 1570096 ms, time limit is 60000 ms io.vertx.core.VertxException: Thread blocked

at jdk.internal.misc.Unsafe.park(Native Method) ~[?:?]

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:323) ~[?:?]

at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:160) ~[api-gateway-1.0-SNAPSHOT-fat.jar:1.0-SNAPSHOT]

at com.hazelcast.multimap.impl.MultiMapProxySupport.invoke(MultiMapProxySupport.java:266) ~[api-gateway-1.0-SNAPSHOT-fat.jar:1.0-SNAPSHOT]

at com.hazelcast.multimap.impl.MultiMapProxySupport.putInternal(MultiMapProxySupport.java:69) ~[api-gateway-1.0-SNAPSHOT-fat.jar:1.0-SNAPSHOT]

at com.hazelcast.multimap.impl.ObjectMultiMapProxy.put(ObjectMultiMapProxy.java:110) ~[api-gateway-1.0-SNAPSHOT-fat.jar:1.0-SNAPSHOT]

at io.vertx.spi.cluster.hazelcast.impl.HazelcastAsyncMultiMap.lambda$add$1(HazelcastAsyncMultiMap.java:89) ~[api-gateway-1.0-SNAPSHOT-fat.jar:1.0-SNAPSHOT]

at io.vertx.spi.cluster.hazelcast.impl.HazelcastAsyncMultiMap$$Lambda$296/0x00000008405d2040.handle(Unknown Source) ~[?:?]

at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(Unknown Source) ~[api-gateway-1.0-SNAPSHOT-fat.jar:1.0-SNAPSHOT]

at io.vertx.core.impl.ContextImpl$$Lambda$114/0x00000008401a3040.run(Unknown Source) ~[?:?]

at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[api-gateway-1.0-SNAPSHOT-fat.jar:1.0-SNAPSHOT]

at io.vertx.core.impl.TaskQueue$$Lambda$98/0x00000008401a7840.run(Unknown Source) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]

at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[api-gateway-1.0-SNAPSHOT-fat.jar:1.0-SNAPSHOT]

at java.lang.Thread.run(Thread.java:834) ~[?:?]

Has anyone else experienced similar issues?

[Darren Jones]

+1

[Ronoel Júnior]

I upgraded my project to use Hazelcast 3.12 since the release of vert.x 3.7 and I have no issue since then.

To upgrade in grade I just use the following code on the build.gradle file:

allprojects {

    ext {
        vertxVersion = '3.7.0'
        hazelcastKubernetesVersion = '1.4'
    }

    configurations.all {

        resolutionStrategy.eachDependency {
            details ->
                if (details.requested.name == 'hazelcast') {
                    details.useVersion '3.12'
                    details.because 'Hazelcast Update'
                }
        }
    }
}

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/ac5f94c9-3b49-489c-820d-c8e5d2de9ae9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Ronoel Júnior

[Thomas SEGISMONT]

+1, you can upgrade to a later Hazelcast version if the Kubernetes plugin does not work well for you.

Note that if you use HZ 3.11+, you must upgrade to the HZ Kubernetes plugin 1.3+ (see https://github.com/hazelcast/hazelcast-kubernetes#requirements-and-recommendations)

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CALdDpGo_HE3deC7SJ1ucZNp%3DnUFmTdA%2BSJG0uYvwPnosWSkidQ%40mail.gmail.com.