Python Hazelcast Client timeout

115 views
Skip to first unread message

Dũng Lê

unread,
Oct 22, 2021, 12:17:51 AM10/22/21
to Hazelcast

I'm using the open-source 5.0 Hazelcast cluster which was deployed on k8s. However, I have been met connection issues from client to server with smart client strategy.

I used python code example as above:

Capture.PNG

Issue details:

Capture.PNG

Thanks all,

Dũng


Yuce Tekol

unread,
Oct 22, 2021, 3:07:09 AM10/22/21
to Hazelcast
Hi,

You should enable External Smart Client Discovery by passing use_public_ip=True to the client constructor:

client  = HazelcastClient(
...
use_public_ip=True
...
)

That should solve the issue, but let me know if it still persists.

Yuce

Yuce Tekol

unread,
Oct 22, 2021, 3:22:22 AM10/22/21
to Hazelcast

Dũng Lê

unread,
Oct 22, 2021, 4:24:05 AM10/22/21
to Hazelcast
Hi Yuce ...

When I added use_public_ip=True parameter, I still saw that issue. However, I checked the number of keys in management center, it was enough.

Can you suggest for me how to solve that persistently "Connection timed out" issue?

Thanks

Dũng



Yuce Tekol

unread,
Oct 22, 2021, 7:21:40 AM10/22/21
to Hazelcast
Hi Dũng,

You don't need to specify addresses for all members, a single one is adequate. Python client can discover other members.

It would be great if you could paste the latest configuration you've tried just to be sure there's no configuration problem (such as ssl_password is unnecessary in the configuration you've pasted).

Could you also elaborate your Kubernetes setup a bit so we can provide a more helpful solution?

Thanks,

Yüce
Message has been deleted

Dũng Lê

unread,
Oct 26, 2021, 12:21:05 AM10/26/21
to Hazelcast
Hi  Yuce Tekol,

I'm using Client.py file to connect and write data into Hazelcast.

K8s Configuration:
- Deploy: Helm chart open source https://github.com/hazelcast/charts
- Hazelcast-client and hazelcast-config: using default configuration.
- Services: expose each pods are nodeport type.

If you need any additional information about Hazelcast configuration, pls let me know.

Thanks,
Dũng
client.py

Muhammet Yazici

unread,
Oct 26, 2021, 8:41:21 AM10/26/21
to Hazelcast
Hi Dũng,

I set up the Hazelcast cluster for external smart client connection using the guide https://guides.hazelcast.org/kubernetes-external-client/#_smart_client. I used the `Helm and Metacontroller` tab for cluster creation. I changed the cluster_members in your client code to IP address of the discovery service Load Balancer. I ran the client without any connection timeout messages.

You can try to create the cluster using the guide. There are a few things you might consider to change if you want to use NodePort instead of LoadBalancer. Assuming you cloned the repository for the guide.
  • You should change `LoadBalancer` to `NodePort` in `hooks/sync-service-per-pod.jsonnet` if you want to use NodePort for service-per-pode.
  • If you want the discovery service to be `NodePort`, you should change `LoadBalancer` to `NodePort` in the following command.
---
helm install hz-hazelcast \
   --set service.type=LoadBalancer \
  --set annotations.service-per-pod-label=statefulset.kubernetes.io/pod-name,annotations.service-per-pod-ports="5701:5701" \
  hazelcast/hazelcast
---

Please tell me if this set up works or not.

Best,

Muhammet
This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast

Dũng Lê

unread,
Oct 31, 2021, 6:49:14 AM10/31/21
to Hazelcast

Hi Muhammet Yazici,

I have been tried many times but it's still the same issue. The details configuration which I deployed.

- Hazelcast cluster on k8s:
Members {size:3, ver:3} [
Member [10.44.182.18]:5701 - e4d93737-ca19-4242-a023-f04d8d086214
Member [10.44.182.19]:5701 - 35d56cbb-a432-489f-a46f-5278b72cab1d
Member [10.44.182.3]:5701 - 7936ade1-5c54-4cce-b81f-e907321c4899 this
]
- NodePort service per pod.

ihazelcast             NodePort       10.45.39.147    <none>        5701:32645/TCP                 90m
ihazelcast-0           NodePort       10.45.63.91     <none>        5701:32114/TCP                 91m
ihazelcast-1           NodePort       10.45.99.80     <none>        5701:30161/TCP                 91m
ihazelcast-2           NodePort       10.45.114.141   <none>        5701:30660/TCP                 91m

- Pods IP:
NAME                     READY   STATUS    RESTARTS   AGE   IP             NODE      NOMINATED NODE   READINESS GATES
ihazelcast-0             1/1     Running   0          79m   10.44.182.18   worker3   <none>           <none>
ihazelcast-1             1/1     Running   0          78m   10.44.182.19   worker3   <none>           <none>
ihazelcast-2             1/1     Running   0          77m   10.44.182.3    worker3   <none>           <none>
ihazelcast-mancenter-0   0/1     Pending   0          79m   <none>         <none>    <none>           <none>
 ihazelcast-1  and ihazelcast-2
- Describe svc which contain only endpoint pods (similar ihazelcast-1 and ihazelcast-2 )
Name:                     ihazelcast-0
Namespace:                ihazelcast
Labels:                   <none>
Annotations:              field.cattle.io/publicEndpoints:
                            [{"addresses":["worker1"],"port":32114,"protocol":"TCP","serviceName":"ihazelcast:ihazelcast-0","allNodes":true}]
Selector:                 role=hazelcast,statefulset.kubernetes.io/pod-name=ihazelcast-0
Type:                     NodePort
IP:                       10.45.63.91
Port:                     ihazelcast-0  5701/TCP
TargetPort:               5701/TCP
NodePort:                 ihazelcast-0  32114/TCP
Endpoints:                10.44.182.18:5701
- Use default Hazelcast.xml and Hazelcast-client.xml configurations.

Please can you send your manifest and step by step which was deployed with loadbalancer or nodeport successfully?  
If you need any additional information, pls let me know.
Thanks 
Dũng
nodeport.png
hazelcast-cluster.PNG
pods.PNG
describe-svc.PNG

Muhammet Yazici

unread,
Nov 2, 2021, 5:31:41 AM11/2/21
to Hazelcast
Hi Dũng,

I prepared a zip file containing all the instructions for enabling exposeExternally using Helm Charts. You can follow the `instructions.sh` file to deploy the Hazelcast cluster and run the Python client you provided.

I also want to mention about Hazelcast Platform Operator that we are currently developing. You can enable exposeExternally feature easily using the operator. You can read about how to use the operator in the documentation. It may seem like you can only use the operator if you have a license key. However we intend to release OS version of the operator soon. Until that time there is a workaround you can use. At license key secret creation process you can create the secret with arbitrary license key as follows

kubectl create secret generic hazelcast-license-key --from-literal=license-key=arbitrary-key

Also to use the open source image you need to change the `repository` field in Hazelcast Custom Resource to 'docker.io/hazelcast/hazelcast' instead of 'docker.io/hazelcast/hazelcast-enterprise'

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast-sample
spec:
  ...
  repository: 'docker.io/hazelcast/hazelcast'
  ...

Please tell me if one of the two set-ups solve your problem.

Best,
Muhammet
python-smart-kubernetes.zip

Dũng Lê

unread,
Nov 2, 2021, 6:36:52 AM11/2/21
to Hazelcast
Hi Muhammet Yazici

I'm very thank you when you sent the guide for me. However, I followed your script, it's not working. :(

---Helm chart solutions---
Metacontroller:
I1102 09:59:55.095918       1 controller.go:447] DecoratorController service-per-pod: sync Pod ihazelcast/hz-hazelcast-2
I1102 09:59:55.099264       1 manage_children.go:254] Pod ihazelcast/hz-hazelcast-2: creating Service hz-hazelcast-2
I1102 09:59:55.160399       1 controller.go:363] DecoratorController service-per-pod: Pod ihazelcast/hz-hazelcast-2: child Service hz-hazelcast-2 created or updated

Services:
NAME             TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
hz-hazelcast     NodePort   10.45.157.80   <none>        5701:30850/TCP   20m
hz-hazelcast-0   NodePort   10.45.44.96    <none>        5701:31520/TCP   20m
hz-hazelcast-1   NodePort   10.45.66.21    <none>        5701:32415/TCP   19m
hz-hazelcast-2   NodePort   10.45.202.17   <none>        5701:30298/TCP   19m

Describe service-0:
Annotations:       field.cattle.io/publicEndpoints:
                            [{"addresses":["worker1"],"port":31520,"protocol":"TCP","serviceName":"ihazelcast:hz-hazelcast-0","allNodes":true}]
                          metacontroller.k8s.io/decorator-controller: service-per-pod
                          metacontroller.k8s.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"metacontroller.k8s.io/decorator-controller":"service-per-pod"},"labels":{"...

Run client.py file
INFO:hazelcast.lifecycle:HazelcastClient 5.0.1 is STARTING
INFO:hazelcast.lifecycle:HazelcastClient 5.0.1 is STARTED
INFO:hazelcast.connection:Trying to connect to Address(host=10.163.15.75, port=30850)
INFO:hazelcast.lifecycle:HazelcastClient 5.0.1 is CONNECTED
INFO:hazelcast.connection:Authenticated with server Address(host=10.44.189.174, port=5701):beb9174d-ff88-4b89-b6f1-77743eb09815, server version: 5.0, local address: Address(host=0.0.0.0, port=55714)
INFO:hazelcast.cluster:

Members [3] {
        Member [10.44.189.174]:5701 - beb9174d-ff88-4b89-b6f1-77743eb09815
        Member [10.44.182.33]:5701 - a9ce97b5-ec33-48cf-96a3-f583b821a808
        Member [10.44.235.175]:5701 - 3122943a-d83e-4281-8678-bba6208fb6b4
}

INFO:hazelcast.connection:Connection(id=1, live=False, remote_address=None) closed. Reason: Connection timed out
INFO:hazelcast.connection:Connection(id=2, live=False, remote_address=None) closed. Reason: Connection timed out
INFO:hazelcast.client:Client started
Connection Successful!
Now, `map` will be filled with random entries.
INFO:hazelcast.connection:Connection(id=3, live=False, remote_address=None) closed. Reason: Connection timed out

---Hazelcast Operator solutions---
I don't know about Go language so I can't try and control it.

Thanks
Dũng.

Muhammet Yazici

unread,
Nov 2, 2021, 8:50:42 AM11/2/21
to Hazelcast
Hi Dũng,

The problem might be related to the Kubernetes cluster provider you are using. Could you tell me where you created the Kubernetes cluster? Can you also verify that all the nodes have external IPs by looking at the result of the following command:

kubectl get node -o wide

Best,

Muhammet

Dũng Lê

unread,
Nov 2, 2021, 10:38:32 AM11/2/21
to haze...@googlegroups.com
Hi Muhammet,

I'm using on-premise Kubernetes. I attached the photo file for "kubectl get node -o wide" cmd 

Thanks
Dũng

--
You received this message because you are subscribed to a topic in the Google Groups "Hazelcast" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hazelcast/1fjXVi7BBPU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/619832af-c9a7-42a4-8b42-a72d2a924ee3n%40googlegroups.com.
2021-11-02_21-34-12.png

Muhammet Yazici

unread,
Nov 3, 2021, 2:59:56 AM11/3/21
to Hazelcast
Hi Dũng,

The problem with you set-up is that the nodes do not have external IPs. Hazelcast members in Kubernetes find their public IPs using the services they are exposed with. In the case of NodePort, the nodes need to have external IP addresses that your client can access. You could solve the problem by assigning external IPs to the nodes that are routable from the host you run your client program.

Best,

Muhammet

Dũng Lê

unread,
Nov 4, 2021, 4:41:53 AM11/4/21
to Hazelcast
Hi Muhammet,
So I need to find a way to assign local worker or public IP to "external IP column" in kubectl get nodes -o wide.
Hazelcast will be used AddressPicker which be find public address (external IP) in k8s. Is that idea right?  
AddressPicker: 

Thanks
Dũng

Muhammet Yazici

unread,
Nov 4, 2021, 11:44:34 AM11/4/21
to Hazelcast
Hi Dũng,

Yes, that is the idea. The crucial thing is that `externalIP` must be routable from the host you run your client on.

Just for confirmation. This external IP situation is only required when your client has no direct access to Hazelcast member pods, hence external smart client. So a client running in a pod in the same Kubernetes cluster does not need the public addresses of the Hazelcast members. If the client is in the Kubernetes cluster, you can remove the `use_public_ip=True` from the client config and give the IP address or DNS name of the Hazelcast service to enable smart client, you could also give the IP address of one of the Hazelcast member pods.

I was not sure where you were running your client, so I felt the need for explaining that.

Best,

Muhammet

Dũng Lê

unread,
Nov 4, 2021, 11:27:34 PM11/4/21
to Hazelcast
Hi Muhammet.

Exactly, My clients don't directly access to Hazelcast Member pods so network policy of company.

When I read Hazelcast source code, I feel it was so clear about external IP.  In my opinion, the source code almost supports k8s cloud providers with Loadbancer service (pubicIP). However, I deploy on-premise k8s which is "internal IP" similar to "external IP" so I can't assign my public IP to "External-IP" column.  https://docs.bitnami.com/kubernetes/faq/troubleshooting/troubleshooting-ip-issues/

If your developer teams can custom source code with an option which is internal IP like external IP in the on premise k8s environment, many clients are thankful yours when they have also this problem.

Thanks
Dũng
Reply all
Reply to author
Forward
0 new messages