Couldn't discover Hazelcast members using Kubernetes API

Shikhar Srivastava

unread,

Apr 7, 2022, 1:07:27 AM4/7/22

to Hazelcast

Hello

I have implemented Hazelcast configuration in my API and have deployed it on kubernetes but the pods are unable to discover each other and i get the following logs :

022-04-06 09:50:55.351 INFO 1 --- [ main] com.hazelcast.system.logo : [10.191.144.45]:5701 [dev] [5.0-BETA-1]
+ + o o o o---o o----o o o---o o o----o o--o--o
+ + + + | | / \ / | | / / \ | |
+ + + + + o----o o o o o----o | o o o o----o |
+ + + + | | / \ / | | \ / \ | |
+ + o o o o o---o o----o o----o o---o o o o----o o
2022-04-06 09:50:55.351 INFO 1 --- [ main] com.hazelcast.system : [10.191.144.45]:5701 [dev] [5.0-BETA-1] Copyright (c) 2008-2021, Hazelcast, Inc. All Rights Reserved.
2022-04-06 09:50:55.351 INFO 1 --- [ main] com.hazelcast.system : [10.191.144.45]:5701 [dev] [5.0-BETA-1] Hazelcast Platform 5.0-BETA-1 (20210713 - e8e010a) starting at [10.191.144.45]:5701
2022-04-06 09:50:55.352 INFO 1 --- [ main] com.hazelcast.system : [10.191.144.45]:5701 [dev] [5.0-BETA-1] Cluster name: dev
2022-04-06 09:50:55.352 INFO 1 --- [ main] com.hazelcast.system : [10.191.144.45]:5701 [dev] [5.0-BETA-1] Jet is disabled. Enable it by setting the "hz.jet.enabled" property to true.
2022-04-06 09:50:56.478 INFO 1 --- [ main] c.h.s.d.integration.DiscoveryService : [10.191.144.45]:5701 [dev] [5.0-BETA-1] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: hazelcast-service, service-port: 5701, service-label: null, service-label-value: true, namespace: superapp-dev, pod-label: null, pod-label-value: null, resolve-not-ready-addresses: true, use-node-name-as-external-address: false, kubernetes-api-retries: 3, kubernetes-master: https://kubernetes.default.svc}
2022-04-06 09:50:56.534 INFO 1 --- [ main] c.h.s.d.integration.DiscoveryService : [10.191.144.45]:5701 [dev] [5.0-BETA-1] Kubernetes Discovery activated with mode: KUBERNETES_API
2022-04-06 09:50:56.659 INFO 1 --- [ main] com.hazelcast.instance.impl.Node : [10.191.144.45]:5701 [dev] [5.0-BETA-1] Using Discovery SPI
2022-04-06 09:50:56.663 WARN 1 --- [ main] com.hazelcast.cp.CPSubsystem : [10.191.144.45]:5701 [dev] [5.0-BETA-1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2022-04-06 09:50:57.457 INFO 1 --- [ main] c.h.internal.diagnostics.Diagnostics : [10.191.144.45]:5701 [dev] [5.0-BETA-1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2022-04-06 09:50:57.464 INFO 1 --- [ main] com.hazelcast.core.LifecycleService : [10.191.144.45]:5701 [dev] [5.0-BETA-1] [10.191.144.45]:5701 is STARTING
2022-04-06 09:50:58.144 WARN 1 --- [ main] com.hazelcast.kubernetes.RetryUtils : Couldn't discover Hazelcast members using Kubernetes API, [1] retrying in 1 seconds...
2022-04-06 09:50:59.659 WARN 1 --- [ main] com.hazelcast.kubernetes.RetryUtils : Couldn't discover Hazelcast members using Kubernetes API, [2] retrying in 2 seconds...
2022-04-06 09:51:01.918 WARN 1 --- [ main] com.hazelcast.kubernetes.RetryUtils : Couldn't discover Hazelcast members using Kubernetes API, [3] retrying

After a few more logs I am getting the error

com.hazelcast.kubernetes.RestClientException: Failure in executing REST call
at com.hazelcast.kubernetes.RestClient.call(RestClient.java:115) ~[hazelcast-all-5.0-BETA-1.jar!/:5.0-BETA-1]
at com.hazelcast.kubernetes.RestClient.get(RestClient.java:80) ~[hazelcast-all-5.0-BETA-1.jar!/:5.0-BETA-1]
at com.hazelcast.kubernetes.KubernetesClient$1.call(KubernetesClient.java:508) ~[hazelcast-all-5.0-BETA-1.jar!/:5.0-BETA-1]
at com.hazelcast.kubernetes.KubernetesClient$1.call(KubernetesClient.java:502) ~[hazelcast-all-5.0-BETA-1.jar!/:5.0-BETA-1]
at com.hazelcast.kubernetes.RetryUtils.retry(RetryUtils.java:52) ~[hazelcast-all-5.0-BETA-1.jar!/:5.0-BETA-1]
at com.hazelcast.kubernetes.KubernetesClient.callGet(KubernetesClient.java:502) ~[hazelcast-all-5.0-BETA-1.jar!/:5.0-BETA-1]
at com.hazelcast.kubernetes.KubernetesClient.endpointsByName(KubernetesClient.java:116) ~[hazelcast-all-5.0-BETA-1.jar!/:5.0-BETA-1]
at com.hazelcast.kubernetes.KubernetesApiEndpointResolver.resolve(KubernetesApiEndpointResolver.java:62) ~[hazelcast-all-5.0-BETA-1.jar!/:5.0-BETA-1]
at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.discoverNodes(HazelcastKubernetesDiscoveryStrategy.java:135) ~[hazelcast-all-5.0-BETA-1.jar!/:5

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [com/bfl/config/HazelcastConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:657) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:485) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1341) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1181) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) ~[spring-beans-5.2.

Caused by: java.lang.IllegalStateException: Node failed to start!
at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:126) ~[hazelcast-all-5.0-BETA-1.jar!/:5.0-BETA-1]
at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:211) ~[hazelcast-all-5.0-BETA-1.jar!/:5.0-BETA-1]
at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:190) ~[hazelcast-all-5.0-BETA-1.jar!/:5.0-BETA-1]
at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:128) ~[hazelcast-all-5.0-BETA-1.jar!/:5.0-BETA-1]
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:61) ~[hazelcast-all-5.0-BETA-1.jar!/:5.0-BETA-1]
at com.bfl.config.HazelcastConfig.hazelcastInstance(HazelcastConfig.java:51) ~[classes!/:na]
at com.bfl.config.HazelcastConfig$$EnhancerBySpringCGLIB$$f84cd122.CGLIB$hazelcastInstance$1(<generated>) ~[classes!/:na]
at com.bfl.config.HazelcastConfig$$EnhancerBySpringCGLIB$$f84cd122$$FastClassBySpringCGLIB$$de516997.invoke(<generated>) ~[classes!/:na]
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at com.bfl.config.HazelcastConfig$$EnhancerBySpringCGLIB$$f84cd122.hazelcastInstance(<generated>) ~[classes!/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:567) ~[na:na]

I have tried to do all the configuration correctly but still I am facing this error!

If anyone could please help me in the issue then it would be great as I am stuck on this since more than a month.

Thanks & Regards

Shikhar

Neil Stevenson

unread,

Apr 7, 2022, 3:15:10 AM4/7/22

to Hazelcast

It may be your dependencies

You are using "5.0-BETA-1" and "hazelcast-all"

Can you try with com.hazelcast:hazelcast:5.1.1

This message

"2022-04-06 09:50:59.659 WARN 1 --- [ main] com.hazelcast.kubernetes.RetryUtils : Couldn't discover Hazelcast members using Kubernetes API, [2] retrying in 2 seconds..."

is ok, the first to start won't find others.

But it won't cause the REST error.

If upgrading doesn't fix, can you post the full logs please ?

Neil

Shikhar Srivastava

unread,

Apr 7, 2022, 5:45:42 AM4/7/22

to Hazelcast

I have tried using all the versions of the dependencies of hazelcast and hazelcast-all but I end up having the same errors. also if i remove the kubernetes configuration then it works by auto detecting multicast but it only creates 1 member even after running 2 pods.

Can you tell me the version of the dependencies which would actually work!

My use-case is to implement hazelcast in a kubernetes environment where for e.g. if I have 2 pods then I should get 2 members in the logs. Any help would be appreciated. Thanks a lot.

Shikhar Srivastava

unread,

Apr 7, 2022, 6:09:52 AM4/7/22

to Hazelcast

After using the com.hazelcast:hazelcast:5.1.1 dependency I am getting the following logs

- Change member config using Java API: config.setIntegrityCheckerEnabled(true);
- Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
- Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
- Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2022-04-07 09:57:15.962 INFO 1 --- [ main] com.hazelcast.system : [10.191.144.42]:5701 [dev] [5.1.1] The Jet engine is disabled.
To enable the Jet engine on the members, do one of the following:
- Change member config using Java API: config.getJetConfig().setEnabled(true)
- Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
- Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
- Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2022-04-07 09:57:16.876 INFO 1 --- [ main] c.h.s.d.integration.DiscoveryService : [10.191.144.42]:5701 [dev] [5.1.1] Kubernetes Discovery properties: { service-dns: null, service-dns-timeout: 5, service-name: hazelcast-service, service-port: 5701, service-label: null, service-label-value: true, namespace: superapp-dev, pod-label: null, pod-label-value: null, resolve-not-ready-addresses: true, expose-externally-mode: AUTO, use-node-name-as-external-address: false, service-per-pod-label: null, service-per-pod-label-value: null, kubernetes-api-retries: 3, kubernetes-master: https://kubernetes.default.svc}
2022-04-07 09:57:17.453 WARN 1 --- [ main] com.hazelcast.spi.utils.RetryUtils : Couldn't connect to the service, [1] retrying in 1 seconds...
2022-04-07 09:57:18.971 WARN 1 --- [ main] com.hazelcast.spi.utils.RetryUtils : Couldn't connect to the service, [2] retrying in 2 seconds...
2022-04-07 09:57:21.236 WARN 1 --- [ main] com.hazelcast.spi.utils.RetryUtils : Couldn't connect to the service, [3] retrying in 3 seconds...
2022-04-07 09:57:24.624 INFO 1 --- [ main] c.h.s.d.integration.DiscoveryService : [10.191.144.42]:5701 [dev] [5.1.1] Kubernetes Discovery activated with mode: KUBERNETES_API
2022-04-07 09:57:24.627 INFO 1 --- [ main] com.hazelcast.system.security : [10.191.144.42]:5701 [dev] [5.1.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security or use -Dhazelcast.security.recommendations system property to see 🔒 security recommendations and the status of current config.
2022-04-07 09:57:24.774 INFO 1 --- [ main] com.hazelcast.instance.impl.Node : [10.191.144.42]:5701 [dev] [5.1.1] Using Discovery SPI
2022-04-07 09:57:24.779 WARN 1 --- [ main] com.hazelcast.cp.CPSubsystem : [10.191.144.42]:5701 [dev] [5.1.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2022-04-07 09:57:25.444 INFO 1 --- [ main] c.h.internal.diagnostics.Diagnostics : [10.191.144.42]:5701 [dev] [5.1.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2022-04-07 09:57:25.450 INFO 1 --- [ main] com.hazelcast.core.LifecycleService : [10.191.144.42]:5701 [dev] [5.1.1] [10.191.144.42]:5701 is STARTING
2022-04-07 09:57:25.487 WARN 1 --- [ main] com.hazelcast.spi.utils.RetryUtils : Couldn't connect to the service, [1] retrying in 1 seconds...
2022-04-07 09:57:25.534 INFO 1 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection : [10.191.144.42]:5701 [dev] [5.1.1] Connection[id=1, /10.191.144.42:5701->/10.191.144.182:38523, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2022-04-07 09:57:25.534 INFO 1 --- [.IO.thread-in-2] c.h.i.server.tcp.TcpServerConnection : [10.191.144.42]:5701 [dev] [5.1.1] Connection[id=3, /10.191.144.42:5701->/10.191.144.182:46742, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2022-04-07 09:57:25.534 INFO 1 --- [.IO.thread-in-1] c.h.i.server.tcp.TcpServerConnection : [10.191.144.42]:5701 [dev] [5.1.1] Connection[id=2, /10.191.144.42:5701->/10.191.144.57:42403, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2022-04-07 09:57:26.590 INFO 1 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection : [10.191.144.42]:5701 [dev] [5.1.1] Connection[id=4, /10.191.144.42:5701->/10.191.144.4:55479, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2022-04-07 09:57:27.000 WARN 1 --- [ main] com.hazelcast.spi.utils.RetryUtils : Couldn't connect to the service, [2] retrying in 2 seconds...
2022-04-07 09:57:35.193 INFO 1 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection : [10.191.144.42]:5701 [dev] [5.1.1] Connection[id=13, /10.191.144.42:5701->/10.191.144.182:59463, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2022-04-07 09:57:36.431 WARN 1 --- [ main] com.hazelcast.spi.utils.RetryUtils : Couldn't connect to the service, [3] retrying in 3 seconds...
2022-04-07 09:57:37.407 INFO 1 --- [.IO.thread-in-1] c.h.i.server.tcp.TcpServerConnection : [10.191.144.42]:5701 [dev] [5.1.1] Connection[id=14, /10.191.144.42:5701->/10.191.145.96:34946, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2022-04-07 09:57:39.814 WARN 1 --- [ main] c.h.s.d.integration.DiscoveryService : [10.191.144.42]:5701 [dev] [5.1.1] Cannot fetch name of the node, NODE_AWARE feature is disabled
2022-04-07 09:57:39.825 WARN 1 --- [ main] com.hazelcast.spi.utils.RetryUtils : Couldn't connect to the service, [1] retrying in 1 seconds...
2022-04-07 09:57:40.141 INFO 1 --- [.IO.thread-in-2] c.h.i.server.tcp.TcpServerConnection : [10.191.144.42]:5701 [dev] [5.1.1] Connection[id=15, /10.191.144.42:5701->/10.191.144.4:4541, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2022-04-07 09:57:40.500 INFO 1 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection : [10.191.144.42]:5701 [dev] [5.1.1] Connection[id=16, /10.191.144.42:5701->/10.191.144.57:32216, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2022-04-07 09:57:41.334 WARN 1 --- [ main] com.hazelcast.spi.utils.RetryUtils : Couldn't connect to the service, [2] retrying in 2 seconds...
2022-04-07 09:57:43.419 INFO 1 --- [.IO.thread-in-1] c.h.i.server.tcp.TcpServerConnection : [10.191.144.42]:5701 [dev] [5.1.1] Connection[id=17, /10.191.144.42:5701->/10.191.145.96:48084, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2022-04-07 09:57:46.991 ERROR 1 --- [ main] c.h.i.cluster.impl.DiscoveryJoiner : [10.191.144.42]:5701 [dev] [5.1.1] Failure in executing REST call

com.hazelcast.spi.exception.RestClientException: Failure in executing REST call
at com.hazelcast.spi.utils.RestClient.call(RestClient.java:163) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.spi.utils.RestClient.lambda$callWithRetries$0(RestClient.java:130) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.spi.utils.RetryUtils.retry(RetryUtils.java:65) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.spi.utils.RetryUtils.retry(RetryUtils.java:51) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.spi.utils.RestClient.callWithRetries(RestClient.java:130) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.spi.utils.RestClient.get(RestClient.java:122) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.kubernetes.KubernetesClient.lambda$callGet$0(KubernetesClient.java:441) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.spi.utils.RetryUtils.retry(RetryUtils.java:65) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.kubernetes.KubernetesClient.callGet(KubernetesClient.java:438) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.kubernetes.KubernetesClient.endpointsByName(KubernetesClient.java:159) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.kubernetes.KubernetesApiEndpointResolver.resolve(KubernetesApiEndpointResolver.java:62) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy.discoverNodes(HazelcastKubernetesDiscoveryStrategy.java:136) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:72) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddresses(DiscoveryJoiner.java:72) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddressesForInitialJoin(DiscoveryJoiner.java:61) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.internal.cluster.impl.TcpIpJoiner.joinViaPossibleMembers(TcpIpJoiner.java:141) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.internal.cluster.impl.TcpIpJoiner.doJoin(TcpIpJoiner.java:100) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.internal.cluster.impl.AbstractJoiner.join(AbstractJoiner.java:139) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.instance.impl.Node.join(Node.java:826) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.instance.impl.Node.start(Node.java:484) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:124) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:217) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:196) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:134) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:61) ~[hazelcast-5.1.1.jar!/:5.1.1]

at com.bfl.config.HazelcastConfig.hazelcastInstance(HazelcastConfig.java:51) ~[classes!/:na]

at com.bfl.config.HazelcastConfig$$EnhancerBySpringCGLIB$$cddbe84f.CGLIB$hazelcastInstance$0(<generated>) ~[classes!/:na]
at com.bfl.config.HazelcastConfig$$EnhancerBySpringCGLIB$$cddbe84f$$FastClassBySpringCGLIB$$101b278d.invoke(<generated>) ~[classes!/:na]

at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]

at com.bfl.config.HazelcastConfig$$EnhancerBySpringCGLIB$$cddbe84f.hazelcastInstance(<generated>) ~[classes!/:na]

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:567) ~[na:na]

at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:652) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]

at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:485) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1341) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1181) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:311) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1109) ~[spring-context-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869) ~[spring-context-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) ~[spring-context-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143) ~[spring-boot-2.3.11.RELEASE.jar!/:2.3.11.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:755) ~[spring-boot-2.3.11.RELEASE.jar!/:2.3.11.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) ~[spring-boot-2.3.11.RELEASE.jar!/:2.3.11.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:402) ~[spring-boot-2.3.11.RELEASE.jar!/:2.3.11.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312) ~[spring-boot-2.3.11.RELEASE.jar!/:2.3.11.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1247) ~[spring-boot-2.3.11.RELEASE.jar!/:2.3.11.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1236) ~[spring-boot-2.3.11.RELEASE.jar!/:2.3.11.RELEASE]
at com.bfl.IAMApplication.main(IAMApplication.java:17) ~[classes!/:na]

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:567) ~[na:na]

at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[app.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[app.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[app.jar:na]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[app.jar:na]
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1715) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1514) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:451) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:422) ~[na:na]
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:574) ~[na:na]
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1653) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1577) ~[na:na]
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) ~[na:na]
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308) ~[na:na]
at com.hazelcast.spi.utils.RestClient.checkResponseCode(RestClient.java:173) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.spi.utils.RestClient.call(RestClient.java:160) ~[hazelcast-5.1.1.jar!/:5.1.1]
... 66 common frames omitted
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:483) ~[na:na]
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472) ~[na:na]
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160) ~[na:na]
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506) ~[na:na]
... 77 common frames omitted

2022-04-07 09:57:46.993 ERROR 1 --- [ main] com.hazelcast.instance.impl.Node : [10.191.144.42]:5701 [dev] [5.1.1] Could not join cluster. Shutting down now!
2022-04-07 09:57:46.993 INFO 1 --- [ main] com.hazelcast.core.LifecycleService : [10.191.144.42]:5701 [dev] [5.1.1] [10.191.144.42]:5701 is SHUTTING_DOWN
2022-04-07 09:57:46.996 WARN 1 --- [ main] com.hazelcast.instance.impl.Node : [10.191.144.42]:5701 [dev] [5.1.1] Terminating forcefully...
2022-04-07 09:57:46.996 INFO 1 --- [ main] com.hazelcast.instance.impl.Node : [10.191.144.42]:5701 [dev] [5.1.1] Shutting down connection manager...
2022-04-07 09:57:46.998 INFO 1 --- [ main] c.h.i.server.tcp.TcpServerConnection : [10.191.144.42]:5701 [dev] [5.1.1] Connection[id=32, /10.191.144.42:5701->/10.191.144.35:8694, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: TcpServer is stopping
2022-04-07 09:57:47.169 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [com/bfl/config/HazelcastConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:657) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:485) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1341) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1181) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:311) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1109) ~[spring-context-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869) ~[spring-context-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) ~[spring-context-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143) ~[spring-boot-2.3.11.RELEASE.jar!/:2.3.11.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:755) ~[spring-boot-2.3.11.RELEASE.jar!/:2.3.11.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) ~[spring-boot-2.3.11.RELEASE.jar!/:2.3.11.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:402) ~[spring-boot-2.3.11.RELEASE.jar!/:2.3.11.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312) ~[spring-boot-2.3.11.RELEASE.jar!/:2.3.11.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1247) ~[spring-boot-2.3.11.RELEASE.jar!/:2.3.11.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1236) ~[spring-boot-2.3.11.RELEASE.jar!/:2.3.11.RELEASE]
at com.bfl.IAMApplication.main(IAMApplication.java:17) ~[classes!/:na]

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:567) ~[na:na]

at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[app.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[app.jar:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[app.jar:na]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[app.jar:na]
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:652) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
... 30 common frames omitted

Caused by: java.lang.IllegalStateException: Node failed to start!

at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:126) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:217) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:196) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:134) ~[hazelcast-5.1.1.jar!/:5.1.1]
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:61) ~[hazelcast-5.1.1.jar!/:5.1.1]

at com.bfl.config.HazelcastConfig.hazelcastInstance(HazelcastConfig.java:51) ~[classes!/:na]

at com.bfl.config.HazelcastConfig$$EnhancerBySpringCGLIB$$cddbe84f.CGLIB$hazelcastInstance$0(<generated>) ~[classes!/:na]
at com.bfl.config.HazelcastConfig$$EnhancerBySpringCGLIB$$cddbe84f$$FastClassBySpringCGLIB$$101b278d.invoke(<generated>) ~[classes!/:na]

at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]

at com.bfl.config.HazelcastConfig$$EnhancerBySpringCGLIB$$cddbe84f.hazelcastInstance(<generated>) ~[classes!/:na]

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:567) ~[na:na]

at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
... 31 common frames omitted

My HazelcastConfig file is:

@Configuration
public class HazelcastConfig {
/**
* Hazelcast instance hazelcast instance.
*
* @return the hazelcast instance
*/
private static final Logger log = LogManager.getLogger(HazelcastConfig.class);

@Bean
public HazelcastInstance hazelcastInstance() {
Config config = new Config();
MapConfig mapConfig = new MapConfig();
mapConfig.setName("authtoken");
mapConfig.setTimeToLiveSeconds(120);
config.setInstanceName("token_hazelcast");
config.addMapConfig(mapConfig);
String env = System.getenv().get("environment");
if ("k8s".equalsIgnoreCase(env)) {
log.debug("Kubernetes environment discovered");
String serviceName = System.getenv().get("service-name");
String servicePort = System.getenv().get("service-port");
JoinConfig joinConfig = config.getNetworkConfig().getJoin();
joinConfig.getMulticastConfig().setEnabled(false);
joinConfig.getTcpIpConfig().setEnabled(false);
joinConfig
.getKubernetesConfig()
.setEnabled(true)
.setProperty("service-name", serviceName)
.setProperty("namespace", "superapp-dev")
.setProperty("service-port", servicePort);
} else {
log.debug("Kubernetes environment not DETECTED");
}

log.debug("Creating NEW Hazelcast Instance");
return Hazelcast.newHazelcastInstance(config);
}

My deployment.yaml file is:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: hazelcast-role
rules:
- apiGroups:
- ""
resources:
- endpoints
- pods
- nodes
- services
verbs:
- get
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: hazelcast-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: hazelcast-role
subjects:
- kind: ServiceAccount
name: hazelcast-service-account
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: hazelcast-service-account
---

apiVersion: apps/v1
kind: Deployment
metadata:
name: hoz-jav-iam-cache
namespace: superapp-dev
spec:
progressDeadlineSeconds: 600
replicas: 2
revisionHistoryLimit: 1
selector:
matchLabels:
app: hoz-jav-iam-cache
template:
metadata:
labels:
app: hoz-jav-iam-cache
spec:
serviceAccountName: hazelcast-service-account
containers:
- name: hoz-jav-iam-cache
image: bflsuperappacr.azurecr.io/hoz-jav-iam-cache:#{Build.BuildNumber}#
imagePullPolicy: Always
env:
- name: environment
value: "k8s"
- name: service-name
value: "hazelcast-service"
- name: service-port
value: "5701"
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP

ports:
- name: "http"
containerPort: 8080
protocol: TCP
- name: "hz"
containerPort: 5701
protocol: TCP
resources:
requests:
cpu: 1000m
memory: 2Gi
limits:
cpu: 1000m
memory: 2Gi
restartPolicy: Always
imagePullSecrets:
- name: apisecret

---
apiVersion: v1
kind: Service
metadata:
name: iam-cache-service
namespace: superapp-dev
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: 'true'
spec:
ports:
- protocol: TCP
port: 8080
targetPort: 8080
selector:
app: hoz-jav-iam-cache
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
name: hazelcast-service
namespace: superapp-dev
spec:
type: NodePort
selector:
app: hoz-jav-iam-cache
ports:
- name: "http"
port: 8080
targetPort: 8080
- name: "hz"
port: 5701
targetPort: 5701

My build.gradle is:

dependencies {
implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
implementation 'org.springframework.boot:spring-boot-starter-web'
implementation 'com.google.code.gson:gson:2.8.6'
// implementation 'com.azure.spring:azure-spring-boot-starter-cosmos'
runtimeOnly 'mysql:mysql-connector-java'
compileOnly 'org.projectlombok:lombok'
developmentOnly 'org.springframework.boot:spring-boot-devtools'
annotationProcessor 'org.projectlombok:lombok'
implementation 'com.hazelcast:hazelcast:5.1.1'
implementation 'org.springframework.boot:spring-boot-starter-cache'
implementation group: 'com.azure', name: 'azure-cosmos', version: '4.11.0'
}

This is everything I have. I really need to get my project running and a fast solution to the problem would mean a lot to me.

Thanks

Shikhar

Shikhar Srivastava

unread,

Apr 8, 2022, 2:53:05 AM4/8/22

to Hazelcast

Any Help? Neil?

Bence Eros

unread,

Apr 8, 2022, 4:33:58 AM4/8/22

to haze...@googlegroups.com

Hello,

the REST call made by HZ to the /api/v1/namespaces/{namespace}/endpoints/{endpointName} endpoint of k8s fails.

What kubernetes implementation do you use?

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/a0661d39-9ff6-4853-830d-da7c07a4052dn%40googlegroups.com.

--

Bence Erős

dev@management-center

er...@hazelcast.com

2 W 5th Ave, Ste 300 | San Mateo, CA 94402 | USA

+1 (650) 521-5453 | hazelcast.com

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast

Neil Stevenson

unread,

Apr 8, 2022, 4:36:06 AM4/8/22

to haze...@googlegroups.com

Can you try with Kubernetes DNS instead as a temporary step ?

Set "service-dns" to "hazelcast-service.default.svc.cluster.local" in Hazelcast config

if your service name in Kubernetes is "hazelcast-service"

Neil

--

You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/a0661d39-9ff6-4853-830d-da7c07a4052dn%40googlegroups.com.

Shikhar Srivastava

unread,

Apr 11, 2022, 3:36:44 AM4/11/22

to Hazelcast

Hi Neil even after trying Kubernetes DNS I am only getting 1 member created even though 2 replicas are running.

2022-04-11 07:31:42.254 INFO 1 --- [ main] com.hazelcast.system.logo : [10.191.144.41]:5701 [dev] [5.1.1]

+ + o o o o---o o----o o o---o o o----o o--o--o
+ + + + | | / \ / | | / / \ | |
+ + + + + o----o o o o o----o | o o o o----o |
+ + + + | | / \ / | | \ / \ | |
+ + o o o o o---o o----o o----o o---o o o o----o o

2022-04-11 07:31:42.254 INFO 1 --- [ main] com.hazelcast.system : [10.191.144.41]:5701 [dev] [5.1.1] Copyright (c) 2008-2022, Hazelcast, Inc. All Rights Reserved.
2022-04-11 07:31:42.254 INFO 1 --- [ main] com.hazelcast.system : [10.191.144.41]:5701 [dev] [5.1.1] Hazelcast Platform 5.1.1 (20220317 - 5b5fa10) starting at [10.191.144.41]:5701
2022-04-11 07:31:42.254 INFO 1 --- [ main] com.hazelcast.system : [10.191.144.41]:5701 [dev] [5.1.1] Cluster name: dev
2022-04-11 07:31:42.254 INFO 1 --- [ main] com.hazelcast.system : [10.191.144.41]:5701 [dev] [5.1.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.
To enable integrity checker do one of the following:

- Change member config using Java API: config.setIntegrityCheckerEnabled(true);
- Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
- Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
- Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)

2022-04-11 07:31:42.258 INFO 1 --- [ main] com.hazelcast.system : [10.191.144.41]:5701 [dev] [5.1.1] The Jet engine is disabled.

To enable the Jet engine on the members, do one of the following:
- Change member config using Java API: config.getJetConfig().setEnabled(true)
- Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
- Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
- Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)

2022-04-11 07:31:43.449 INFO 1 --- [ main] c.h.s.d.integration.DiscoveryService : [10.191.144.41]:5701 [dev] [5.1.1] Kubernetes Discovery properties: { service-dns: hazelcast-service.superapp-dev.svc.cluster.local, service-dns-timeout: 5, service-name: null, service-port: 5701, service-label: null, service-label-value: true, namespace: superapp-dev, pod-label: null, pod-label-value: null, resolve-not-ready-addresses: true, expose-externally-mode: AUTO, use-node-name-as-external-address: false, service-per-pod-label: null, service-per-pod-label-value: null, kubernetes-api-retries: 3, kubernetes-master: https://kubernetes.default.svc}
2022-04-11 07:31:43.968 WARN 1 --- [ main] com.hazelcast.spi.utils.RetryUtils : Couldn't connect to the service, [1] retrying in 1 seconds...
2022-04-11 07:31:45.485 WARN 1 --- [ main] com.hazelcast.spi.utils.RetryUtils : Couldn't connect to the service, [2] retrying in 2 seconds...
2022-04-11 07:31:47.749 WARN 1 --- [ main] com.hazelcast.spi.utils.RetryUtils : Couldn't connect to the service, [3] retrying in 3 seconds...
2022-04-11 07:31:51.141 INFO 1 --- [ main] c.h.s.d.integration.DiscoveryService : [10.191.144.41]:5701 [dev] [5.1.1] Kubernetes Discovery activated with mode: DNS_LOOKUP
2022-04-11 07:31:51.143 INFO 1 --- [ main] com.hazelcast.system.security : [10.191.144.41]:5701 [dev] [5.1.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security or use -Dhazelcast.security.recommendations system property to see 🔒 security recommendations and the status of current config.
2022-04-11 07:31:51.340 INFO 1 --- [ main] com.hazelcast.instance.impl.Node : [10.191.144.41]:5701 [dev] [5.1.1] Using Discovery SPI
2022-04-11 07:31:51.345 WARN 1 --- [ main] com.hazelcast.cp.CPSubsystem : [10.191.144.41]:5701 [dev] [5.1.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2022-04-11 07:31:52.065 INFO 1 --- [ main] c.h.internal.diagnostics.Diagnostics : [10.191.144.41]:5701 [dev] [5.1.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2022-04-11 07:31:52.071 INFO 1 --- [ main] com.hazelcast.core.LifecycleService : [10.191.144.41]:5701 [dev] [5.1.1] [10.191.144.41]:5701 is STARTING
2022-04-11 07:31:52.179 INFO 1 --- [.IO.thread-in-1] c.h.i.server.tcp.TcpServerConnection : [10.191.144.41]:5701 [dev] [5.1.1] Connection[id=2, /10.191.144.41:5701->/10.191.144.35:8357, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2022-04-11 07:31:52.231 INFO 1 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection : [10.191.144.41]:5701 [dev] [5.1.1] Connection[id=1, /10.191.144.41:5701->/10.191.144.4:42523, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2022-04-11 07:31:52.231 INFO 1 --- [.IO.thread-in-2] c.h.i.server.tcp.TcpServerConnection : [10.191.144.41]:5701 [dev] [5.1.1] Connection[id=3, /10.191.144.41:5701->/10.191.144.4:10774, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2022-04-11 07:31:52.232 INFO 1 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection : [10.191.144.41]:5701 [dev] [5.1.1] Connection[id=4, /10.191.144.41:5701->/10.191.144.182:27711, qualifier=null, endpoint=null, remoteUuid=null, alive=false, connectionType=NONE, planeIndex=-1] closed. Reason: Connection closed by the other side
2022-04-11 07:31:52.235 INFO 1 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection : [10.191.144.41]:5701 [dev] [5.1.1] Initialized new cluster connection between /10.191.144.41:5701 and /10.191.144.35:20888
2022-04-11 07:31:52.237 INFO 1 --- [.IO.thread-in-1] c.h.i.server.tcp.TcpServerConnection : [10.191.144.41]:5701 [dev] [5.1.1] Initialized new cluster connection between /10.191.144.41:51393 and /10.191.154.50:5701
2022-04-11 07:31:52.243 INFO 1 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection : [10.191.144.41]:5701 [dev] [5.1.1] Connection[id=12, /10.191.144.41:5701->/10.191.144.35:20888, qualifier=null, endpoint=[10.191.144.41]:5701, remoteUuid=2f92af6e-1efd-4807-a037-7b9ce8b63e3a, alive=false, connectionType=MEMBER, planeIndex=0] closed. Reason: Connecting to self!
2022-04-11 07:31:52.245 INFO 1 --- [.IO.thread-in-1] c.h.i.server.tcp.TcpServerConnection : [10.191.144.41]:5701 [dev] [5.1.1] Connection[id=13, /10.191.144.41:51393->/10.191.154.50:5701, qualifier=null, endpoint=[10.191.144.41]:5701, remoteUuid=2f92af6e-1efd-4807-a037-7b9ce8b63e3a, alive=false, connectionType=MEMBER, planeIndex=0] closed. Reason: Connecting to self!
2022-04-11 07:31:52.334 WARN 1 --- [Server.thread-1] c.h.i.s.tcp.TcpServerConnectionManager : [10.191.144.41]:5701 [dev] [5.1.1] Packet send task is rejected. Target is this node! Target[uuid=2f92af6e-1efd-4807-a037-7b9ce8b63e3a, address=[10.191.154.50]:5701]
2022-04-11 07:31:53.236 INFO 1 --- [ main] c.h.internal.cluster.ClusterService : [10.191.144.41]:5701 [dev] [5.1.1]

Members {size:1, ver:1} [
Member [10.191.144.41]:5701 - 2f92af6e-1efd-4807-a037-7b9ce8b63e3a this
]

Why am i not able to see multiple members even though multiple pods/replicas are running?

Neil Stevenson

unread,

Apr 11, 2022, 9:10:03 AM4/11/22

to haze...@googlegroups.com

If you do `kubectl describe service hazelcast-service --namespace=superapp-dev` what does it show for endpoints ?

These lines

2022-04-11 07:31:52.235 INFO 1 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection : [10.191.144.41]:5701 [dev] [5.1.1] Initialized new cluster connection between /10.191.144.41:5701 and /10.191.144.35:20888

2022-04-11 07:31:52.243 INFO 1 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection : [10.191.144.41]:5701 [dev] [5.1.1] Connection[id=12, /10.191.144.41:5701->/10.191.144.35:20888, qualifier=null, endpoint=[10.191.144.41]:5701, remoteUuid=2f92af6e-1efd-4807-a037-7b9ce8b63e3a, alive=false, connectionType=MEMBER, planeIndex=0] closed. Reason: Connecting to self!

appear to show the 2nd pod trying to connect to the 1st but it failing/being rejected

What does the logs on the other pod show ?

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/e44c37fd-0ce7-4494-8ed8-62b69ae3a1f2n%40googlegroups.com.

Shikhar Srivastava

unread,

Apr 18, 2022, 3:54:32 AM4/18/22

to Hazelcast

Thankyou for the suggestions Neil the service-dns method worked and now I can discover all my pods.

But, a new problem arose and now once my data is stored in the cache it is not getting evicted. TTL is not working. How to resolve that issue?

I am always getting data from the cache even after the expiration of TTL.

Neil Stevenson

unread,

Apr 19, 2022, 7:27:01 AM4/19/22

to Hazelcast

If it's a different problem can you raise a different question for it please -- this helps others who might hit the same issue and google for a solution

It's probably a configuration mismatch, so be sure to confirm what you have configured

pratik parshetti

unread,

Dec 15, 2022, 4:51:50 AM12/15/22

to Hazelcast

Hi Neil,

I'm facing same error and here are the endpoints,

172.18.116.10:8080

Can you please help.

Reply all

Reply to author

Forward