Issues connecting to kubernetes hazelcast cluster from client code

[Nicolas Parfait]

Hi, I'm unable to connect to a hazelcast cluster running on kubernetes from my SpringBoot app.

I'm able to deploy a multi node hazelcast server cluster within my k8s cluster.

From k8s if i use the following config (kubernetes enabled)

I see the following error (log file attached)

2018-11-25 12:04:54.392 TRACE 7 --- [           main] c.h.i.metrics.impl.MetricsRegistryImpl   : hz.client_0 [development] [3.11] Registered probeInstance executionService.userExecutorQueueSize
2018-11-25 12:04:54.411  WARN 7 --- [           main] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcastInstance' defined in class path resource [digital/test/bootstrap/configuration/HazelcastConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'create' threw exception; nested exception is com.hazelcast.config.InvalidConfigurationException: Invalid configuration
2018-11-25 12:04:54.412  INFO 7 --- [           main] uration$$EnhancerBySpringCGLIB$$1bb041db : Shutdown hook called

I've got a headless service defined (ClusterIP, no ip address). I've tried using service-name/namespace, same error.

If i put the pod IP address in cluster-member its able to connect.

Any help would be appreciated.

hazelcast-client.xml

<hazelcast-client xmlns="http://www.hazelcast.com/schema/client-config"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.hazelcast.com/schema/client-config hazelcast-client-config-3.11.xsd">
  <group>
      <name>development</name>
      <password/>
  </group>
  <properties>
      <property name="hazelcast.client.statistics.enabled">true</property>
      <!--property name="hazelcast.discovery.enabled">true</property-->
  </properties>
  <network>
    <!--cluster-members>
        <address>172.17.0.7:5701</address>
    </cluster-members-->
    <kubernetes enabled="true">
        <service-dns>hazelcast.default.svc.cluster.local</service-dns>
        <!--namespace>default</namespace>
        <service-name>hazelcast</service-name-->
    </kubernetes>
  </network>
</hazelcast-client>

Config

OSX / minikube / istio

hazelcast 3.11

hazelcast-kubernetes 1.3.1

[rafal....@gmail.com]

Hi Nicolas,

I think your issue is related to some problems with Java 9+ and Hazelcast metrics.

This part of the log: "module jdk.management does not "opens com.sun.management.internal" to unnamed module" looks similar to this GH Issue. You may consider reopening it or creating a new one.

As of your Kubernetes configuration, it doesn't make sense to specify both service-name and service-dns, since one is related to Kubernete API Discovery and the other to the DNS Lookup Discovery. Read more at hazelcast-kubernetes. 

You can see examples of Spring Boot + Hazelcast on the k8 environment here:

- Embedded Hazelcast Code Sample

- Complete Example of Spring Boot + Hazelcast

Still I believe it's not the configuration which is the issue here.

Cheers,

Rafał

[Josef Cacek]

Hi,

the exceptions in the log don't cause the problem here. They are just
printed (on TRACE level) when an OS metric can't be registered for
some reason.
This behavior on Java 9 and newer can be fixed by opening packages as
described in our documentation:
https://docs.hazelcast.org/docs/3.11/manual/html-single/index.html#running-in-modular-java

The real problem seems to be the client configuration. Maybe the
classpath is not correct and it's missing Kubernetes discovery
strategy factory.
Can you check if hazelcast-kubernetes jar is really on the classpath?

Best regards,

-- Josef

> --
> You received this message because you are subscribed to the Google Groups "Hazelcast" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
> To post to this group, send email to haze...@googlegroups.com.
> Visit this group at https://groups.google.com/group/hazelcast.
> To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/bd9a4a44-609e-49b8-b28b-de13b2f8004e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Josef Cacek]

One more note:
There is a bug in the hazelcast-kubernetes 1.3.1 Maven dependencies.
It wrongly contains a transitive dependency on hazelcast 3.11-BETA-1.
It can be fixed by explicitly excluding the transitive dependency.

<dependency>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast-kubernetes</artifactId>
<version>1.3.1</version>
<exclusions>
<exclusion>
<groupId>com.hazelcast</groupId>
<artifactId>hazelcast</artifactId>
</exclusion>
</exclusions>
</dependency>

This problem is already fixed in the master branch:
https://github.com/hazelcast/hazelcast-kubernetes/pull/116

Regards,
-- Josef

[Nicolas Parfait]

Thanks all, this is working now. I need to post a separate issue of istio and mTLS not working when I get back around to looking at it (have it disabled for now)