Kubernetes Hazelcast DNS not found

[Simon LEDUNOIS]

Hello,

I'm experiencing issues with hazelcast and kubernetes DNS discovery policy. Actually, I need help. After several research on the web and many tries, I don't find the solution.

I have an openshift "cluster" with a Hazelcast Vert.x application configured to use Kubernetes and DNS discovery strategy but something goes wrong on start. I have multiple logs like : 

May 21, 2021 12:29:34 PM com.hazelcast.spi.discovery.integration.DiscoveryService
WARNING: [172.17.0.10]:5701 [dev] [4.0.2] DNS lookup for serviceDns 'service-hazelcast-cluster.default.svc.cluster.local' failed: name not found

Scaling my service to 2 pods produce the same and creates 2 single hazelcast clusters.

Here is :

1. the full logs : https://gist.github.com/SLedunois/4d627a40838dfc271ea83539aed430ec
2. A part of my OpenShift configuration: https://gist.github.com/SLedunois/bad5ad2045efab7db18a772eea4dac5b
3. My cluster.xml: https://gist.github.com/SLedunois/acca0b099727d1865c2a29abe6fff1cf

Thanks for your time,

SLedunois

[Thomas SEGISMONT]

I can't see anything wrong in the service/deployment config or cluster config (regarding discovery).

Have you checked with oc or the console that the headless service has been created?

If it's been created, can you check under which name it is accessible? Perhaps it is just "service-hazelcast-cluster"...

Note that in your cluster.xml file you need to add configuration for the distributed maps required by Vert.x

(see an example in https://vertx.io/docs/vertx-hazelcast/java/#_using_an_existing_hazelcast_cluster)

It is not your problem right now but it could be later after the DNS discovery issue is resolved.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/d56bf27d-e2ca-4923-9b6a-cc8b4a816cc7n%40googlegroups.com.

[Simon LEDUNOIS]

Thanks for you reply.

Indeed, my service is up:

oc get services

NAME                        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE

service-hazelcast-cluster   ClusterIP   None         <none>        5701/TCP   1m

[Simon LEDUNOIS]

Another thing: the deployment config map the network with an internal service => service-hazelcast-cluster

[Thomas SEGISMONT]

Can you run this command on the pod:

nslookup service-hazelcast-cluster

If you get a result then update the cluster.xml file: in the  "service-dns" config property set "service-hazelcast-cluster"

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/f7816700-2353-4eb5-92f1-8944ea9f09d8n%40googlegroups.com.

[Simon LEDUNOIS]

Ok, I solved it with you last answer !  

nslookup was not found in the pod so I apply a new conf : https://gist.github.com/SLedunois/fe34bbf39be73cb6d7fade2e491178df and run nslookup into the new pod.

/ $ nslookup service-hazelcast-cluster

Server:         172.30.0.2

Address:        172.30.0.2#53

Name:   service-hazelcast-cluster.vertx-modular-platform.svc.cluster.local

Address: 172.17.0.11

Replaced the service-dns property with service-hazelcast-cluster.vertx-modular-platform.svc.cluster.local and scaling my service add a new member to the cluster : 

Members {size:2, ver:2} [
    Member [172.17.0.14]:5701 - 9dd64ef8-31d3-4a94-bdfd-538e47bebf11 this
    Member [172.17.0.13]:5701 - 661fdff0-6194-459f-916f-01e915d65400
]

Thanks for your help !