Hazelcast heap memory usage

[karim slaimi]

Hello,

I installed hazelcast with 5GB heap memory in a dedicated tomcat server.

Now I have an issue with Hazelcast. Sometimes it's shutting down for high memory usage (97.27% memory used).

When I check the Memory usage I have:

1- 20% Map

2- 30% Free

3- 50% Other!!!

I do not know what "Other". Is it the management center? if so how it's possible it consume 50% of the memory?

Then how can I avoid this heap memory issue? because I am using hazelcast as a session clustering too. So when hazelcast is down, no one can connect to my application.

Also it's possible to have a history of the memory usage? Somewhere in the man-center may be?

Best regards

Karim SLAIMI

[Lucas Beeler]

Hi Karim,

Is it feasible for you to increase the size of your heap to 8GB and see if the issue persists? I'm trying to figure out if you're experiencing some kind of memory leak or if you're just under-provisioned. A 5GB heap really isn't that big by today's standards, so that could be the problem. Or it could be something else.

And, to answer your other question, "other" represents all class instances that are not data objects stored in Hazelcast's distributed data structures. So this could be the Hazelcast runtime itself or the Tomcat code and data. Are you running Hazelcast and Tomcat in the same JVM?

Take care,

Lucas

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAHSeLL2bmTics%2BOj-gmNCfFcd6grmC-MsJCRUOzHi1vH5C4VSA%40mail.gmail.com.

--

Lucas BEELER Principal Solutions Architect       hazelcast®     2 W 5th Ave, Ste 300 | San Mateo, CA 94402 | USA +1 (650) 521-5453 | hazelcast.com

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast

[karim slaimi]

Hi Lucas,

I will try to increase to heap Space to 8GB.

Actually i am in prod env and I got this heap space issue which is not good at all since all users were not anymore able to login into the web application!!!

Actually Iam using hazelcast as a tomcat session manager too.

I have a dedicated centos server for hazelcast. So Hazelcast and hazelcast man center are running into it only. So if I well understood the « other », which sometimes represents more than 50% of the heap is probably related to hazelcast system itself and to the man center web app ?

Thank you un advance for your recommandations and explanations.

Best 

Karim 

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAPT_Z_P6S_O5KPm4hDzu%3DJisPJFjg98D53WrUxVKW%3DbiOntapg%40mail.gmail.com.

[Lucas Beeler]

Hi Karim,

One thing to keep in mind is that the Hazelcast Management Center runs as its own JVM process. Indeed, if you simply do a ps -aux | grep java from the command line of your server, you should see two JVM processes running—one for the Hazelcast server itself and another for management center (and perhaps a third for your Tomcat server). When you get an out-of-memory exception, does it come from the core Hazelcast process or from the man center process or even from Tomcat?

Also, lemme know how it goes when you increase the heap space...

Take care,

Lucas

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAHSeLL1mjcTjhSt9A3m7RMZvvOWq%2B0G0ug2mPscbW0HiK65JCw%40mail.gmail.com.

[Lucas Beeler]

Hi Karim,

One addendum to my earlier note: you probably want to run ps -eaux | grep java assuming that all of your Java processes don't run as the same user.

Take care,

Lucas

[karim slaimi]

Hi Lucas,

It was from the core hazelcast process. I got « 97% of heap is used » then it shutdown all client connections and finally completely stopped.

I will increase the heap size and let you know how it is going.

Best,

Karim 

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAPT_Z_M8thcwMaCiMFkyz1-0RWnBu9MAVJKG1VnHsyNXT_-unw%40mail.gmail.com.

[karim slaimi]

Hi lucas,

I increased the memory of my Hazelcast server and it's giong better now.

I also added a second node to the hazelcast cluster.

I have 2 quick questions:

1- First, I kept backup to "1" so if I well understood each node data will be also duplicated into the second node. I set the backup value to zéro,  the idea is to set no backup to decrease the heap memory usage it's a good practice? what should I do?

2- When one node is down, I am getting an error in my web application "Error completing publication for publisher com.hazelcast.internal.metrics.jmx.JmxPublisher". Do you have any idea how I can fix it please?

Best regards

Karim

[Lucas Beeler]

Hi Karim,

A couple of things:

• First, the minimum recommended Hazelcast cluster size is 3 nodes. There are a number of reasons for this, but the simplest is explanation is that in the event of a node loss in a network partition (sometimes called a "split brain" scenario in distributed systems) in a system with only two nodes, you only have one node remaining and no guarantee that that remaining node has received the latest data, potentially leading to inconsistency. So, definitely run with at least 3 nodes.
• If you don't want to lose data when you bring down a node (which is a relatively routine maintenance operation to say, apply an operating system patch), you should set the backup count to 1. When you do this, your cluster will have two copies, one primary replica and one backup replica, on two separate nodes.
• Does the JMX publisher exception impede your application in any way? All the error says is that the system can't publish JMX metrics, which should not be critical. Your application still works, no?

Take care,

Lucas

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAHSeLL3tdovp%2BLY6oR1n%2BWOGdo%2BXDhDm%2BexZC1ji3pcpgZsVig%40mail.gmail.com.