Facing performance issue in Hazelcast with Client-Server

[Pavan Mulani]

Hi Folks,

I am facing performance issues with hazelcast configured as client-server.

I have one K8S Cluster consisting of 5 Nodes and 1 Master Node. Each node is of 64 GB of RAM and 16 Core (Hazelcast Version 3.12.4)

Hazelcast server is deployed on K8S with one POD on one of the nodes available in cluster
My Client is deployed on K8S which is connected to above Hazelcast with smart client (Hazelcast discovery enabled for K8S). There are a total 10 PODs of my applications with each node consisting of 2 PODs of my application.

I am running different different APIs and performing load testing of my application (Approx 110 Threads at a time shared across all 10 PODs)

I am having the following piece of code in my application to get cache.

public Map<Object, Object> get(String cacheId, Long lTenantId) {
String strMethodName="get";
long t1 = System.currentTimeMillis();
Map<Object,Object> cacheDataMap=hazelcastInstance.getMap(cacheId);
long totalTimeTaken = (System.currentTimeMillis()-t1);
if(totalTimeTaken > 10){
logger.warnLog(CLASSNAME, strMethodName,"Total time taken by "+cacheId+" identifier for get operation is : "+totalTimeTaken+" ms");
}
return cacheDataMap;
}

The way my application uses this map varies like

1) map.get(key);
2) Set keys = map.keySet();
   Iterator iterator = keys.iterator(); //I changed to keyset iterator because entryset was causing lot of performance issues
   while (iterator.hasNext()) {
      // doing stuff
   }

When all My APIs are started for Load I am getting these logs printed in an application (Total time taken by....) where each cache access time is > 10 miliseconds and this is causing performance issues and hence I am not able to achieve my desired TPS for all APIs.

There are approx 300 Maps are stored in cache where total size of the cache is 4.22 MB

I am using near cache configuration and also on the management center it is showing effectiveness as 100%. (This was taken when hazelcast.client.statistics.enabled was enabled).

I have also tried with 8 PODs deployed on 4 Nodes and 1 dedicated node for Hazelcast server but the issue remains the same. There are no issues observed when I am connecting Hazelcast as embedded and I am able to achieve my desired TPS for all APIs.

Am I Missing any configuration or any other thing which is causing this problem?

Here is my hazelcast-client.xml


<hazelcast-client
xmlns="http://www.hazelcast.com/schema/client-config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.hazelcast.com/schema/client-config
                  http://hazelcast.com/schema/client-config/hazelcast-client-config-3.11.xsd">

<group>
<name>dev</name>
</group>
<instance-name>hazelcast</instance-name>
<properties>
<property name="hazelcast.client.shuffle.member.list">true</property>
<property name="hazelcast.client.heartbeat.timeout">600000</property>
<property name="hazelcast.client.heartbeat.interval">180000</property>
<property name="hazelcast.client.event.queue.capacity">1000000</property>
<property name="hazelcast.client.invocation.timeout.seconds">120</property>
<property name="hazelcast.client.statistics.enabled">false</property>
<property name="hazelcast.discovery.enabled">true</property>
<property name="hazelcast.map.invalidation.batch.enabled">false</property>
</properties>

<network>
<discovery-strategies>
<discovery-strategy enabled="true"
class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
<properties>
<property name="service-name"><service-name></property>
<property name="namespace"><namespace></property>
</properties>
</discovery-strategy>
</discovery-strategies>
<smart-routing>true</smart-routing>
<redo-operation>true</redo-operation>
<connection-timeout>90000</connection-timeout>
<connection-attempt-period>100</connection-attempt-period>
<connection-attempt-limit>0</connection-attempt-limit>

</network>

<near-cache name="default">
                <in-memory-format>OBJECT</in-memory-format>
                <serialize-keys>true</serialize-keys>
                <invalidate-on-change>true</invalidate-on-change>
                <eviction eviction-policy="NONE" max-size-policy="ENTRY_COUNT"/>
        </near-cache>
</hazelcast-client>


Here is my hazelcast.xml

<?xml version="1.0" encoding="UTF-8"?>

<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.11.xsd"
           xmlns="http://www.hazelcast.com/schema/config"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

  <management-center enabled="${hazelcast.mancenter.enabled}">${hazelcast.mancenter.url}</management-center>

</hazelcast>

[Rafal Leszko]

Hi Pavan,

I think the performance issue may be related to the fact that you run only one Hazelcast instance (you mention: "one POD on one of the nodes available in cluster"). Could you try to run at least 3 Hazelcast members as the server part? Perfectly, to have the same conditions as your embedded scenario, I'd run the same number of Hazelcast members for the server as you use for the embedded scenario. Then, I guess you should not observe any difference in terms of performance.

Cheers,

Rafał

STL - Sterlite Technologies Limited Disclaimer:

The content of this message may be legally privileged and confidential and are for the use of the intended recipient(s) only. It should not be read, copied and used by anyone other than the intended recipient(s). If you have received this message in error, please immediately notify the sender, preserve its confidentiality and delete it. Before opening any attachments please check them for viruses and defects. No employee or agent is authorised to conclude any binding agreement on behalf of Sterlite Technologies Limited with another party by email without express written confirmation by authorised person. Visit us at www.stl.tech

 Please consider environment before printing this email !

Registered office: E 1, MIDC Industrial Area, Waluj, Aurangabad, Maharashtra – 431 136 CIN – L31300MH2000PLC269261

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/7f994599-e1b2-48fe-a4eb-bd27c3affc40%40googlegroups.com.

--

Rafał Leszko Software Engineer       hazelcast®

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast

[Pavan Mulani]

Thanks for suggestion. Tried with 3, 4 and 5 Hazelcast members as server, But couldn't achieve my desired TPS. It is pretty much same as it was with Single POD