Hazelcast backup map performance

[shankar narayanan]

Hi,
We have started using hazelcast version 2.6 in our projects , previously we were using coherence cache. But ,after we started using hazelcast , the performance of the system has become very slow and after a couple of days , we have to restart the nodes. And the strange thing is , there are no errors. So , is there any reason why the system performance can be slower.
Here are the details:
Basically , we use it as a cache in one component and as an asynchronous update to the database in the other. Our environment is a clustered environment.

1).Component where it is used as cache:

Max entries : 5,000,000.
No of maps : 1
Contents of map : A POJO as a key with 3 variables. No value
Frequency of reading from cache : very high.

Configuration file :
 <bean id="XYZ" class="ClassName">
        <constructor-arg index="0"><ref bean="hazelcastInstance"/></constructor-arg>
        <constructor-arg index="1" value="Map" />
     </bean>
   
    <hz:hazelcast id="hazelcastInstance">
        <hz:config>
            <hz:group name="Groupname" password="password"/>
            <hz:properties>
                <hz:property name="hazelcast.merge.first.run.delay.seconds">5</hz:property>
                <hz:property name="hazelcast.merge.next.run.delay.seconds">5</hz:property>
                <hz:property name="hazelcast.version.check.enabled">false</hz:property>
            </hz:properties>
            <hz:network port="port" port-auto-increment="true">
                <hz:join>
                    <hz:multicast enabled="true"
                                  multicast-group="ip"
                                  multicast-port="port"/>
                </hz:join>
            </hz:network>
 
                   
             <hz:map name="Map"
                    backup-count="1"
                    read-backup-data="true"
                    cache-value="true"
                    time-to-live-seconds="86400"
                    max-size="5000000"
                    eviction-percentage="25"
                    eviction-policy="LRU"
                    eviction-delay-seconds="3"
                    merge-policy="hz.LATEST_UPDATE"/>
                   
        </hz:config>
    </hz:hazelcast>
 
2). Component where it is used as persistance store. We use a backup map and write the cache data to the database every 180 seconds and delete the entries in the cache.

Max entries : 5,000,000.
No of maps : 1
Contents of map : An integer as a key and a bigger object as value.
Frequency of reading from cache : high.
Frequency of writing to cache : high.

Configuration file :

<bean id="Cache" class="Classname">
        <constructor-arg index="0"><ref bean="hazelcastInstance"/></constructor-arg>
        <constructor-arg index="1" value="Map" />
     </bean>
   
    <hz:hazelcast id="hazelcastInstance">
        <hz:config>
            <hz:group name="cluster group" password="cluster.group.password"/>
            <hz:properties>
                <hz:property name="hazelcast.merge.first.run.delay.seconds">5</hz:property>
                <hz:property name="hazelcast.merge.next.run.delay.seconds">5</hz:property>
                <hz:property name="hazelcast.version.check.enabled">false</hz:property>
            </hz:properties>
            <hz:network port="cluster.instance.port" port-auto-increment="cluster.instance.autoincrement">
                <hz:join>
                    <hz:multicast enabled="cluster.group.multicast.enabled"
                                  multicast-group="cluster.group.multicast.ip"
                                  multicast-port="cluster.group.multicast.por}"/>
                </hz:join>            </hz:network>
 
                   
             <hz:map name="Map"
                    backup-count="1"
                    read-backup-data="true"
                    cache-value="true"
                    max-size="5000000"
                    eviction-percentage="25"
                    eviction-policy="LRU"
                    eviction-delay-seconds="3"
                    merge-policy="hz.LATEST_UPDATE">
                    <hz:map-store enabled="true" write-delay-seconds="180" class-name="DBCacheStore">
                    </hz:map-store>
                    </hz:map>
                   
        </hz:config>
    </hz:hazelcast>

I know , it is very difficult to know what the exact problem could be without any error occuring , but could it be the case where hazelcast consumes too much memory and garbage collection is not that efficient?

Regards,
Shankar

[Enes Akar]

How do you serialize your objects? Do you use default Java serialization (Serializable)?

I am suspicious of the eviction process (which begins when your map size reaches the maximum value). Can it be the case the time you experience performance problem is the time your map size reaches to maximum value?

Do you have a test environment, to reproduce the performance drop? If you have it, can you test your system with 3.0 which have different eviction mechanism?

Thanks

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
For more options, visit https://groups.google.com/groups/opt_out.

--

Enes Akar

Hazelcast | Open source in-memory data grid
Mobile: +90.507.150.56.71

[shankar narayanan]

Hi,
We use java serialization. It could be the case that the performance problem occured when map size reached maximum.
We do have a test environment, but this issue was not reproducible in that with the same load.

[shankar narayanan]

We had a restart of the nodes yesterday as well . They had captured the memory statistics and it revealed that the available memory during the time of crashing was extremely low. So , could it be the case that memory utilization is not efficient enough?

On Wednesday, September 25, 2013 4:28:58 PM UTC+2, Enes Akar wrote:

[Enes Akar]

We recommend to use DataSerializable as default java serialization is notably slower.

See here for info about serialization:

http://blog.hazelcast.com/blog/2013/06/13/comparing-hazelcast-3-serialization-methods-with-kryo-serialization/

For memory utilization:

If it is not possible to upgrade to 3.0; then you can try to decrease the max-size as much as possible. Also it will helpful to decrease the ttl ve idle-second values for evicting records as fast as possible.