Memory leak in client (v3.8.1 included)

[Jean Luc]

I've been doing reliability testing and found a leak with 3.8, also present with 3.8.1. The specific case I was testing was what happens in a Hazelcast client if the server is not available for a long time and the client keeps trying to connect. I had been running it for days with connections every few seconds and eventually it crashed with the heap full. I investigated the reason and found it to be as below.

In short, for every connection, there are 3 instances of java.lang.ThreadGroup that stay around. They belong to Hazelcast (their name is hz_client_NUMBER).

Further testing showed the leak occurs with successful connections as well.

A project to reproduce the problem in its simplest form, with instructions and screenshots from the profiler, is available at https://github.com/jl2008/hazelcast-client-test-threadgroup

The client is shut down in all cases, so I didn't expect a leak:

HazelcastInstance client = null;
try {
   client = HazelcastClient.newHazelcastClient(hazelcastClientConfig); //has to be inside try, the constructor throws an exception if the cluster is unavailable
} catch (Exception e) {
   //...
} finally {
   if (client != null) {
      client.shutdown();
   }
}

Let me know if I've done something wrong or I can assist with anything else,

JL

[M. Sancar Koyunlu]

Hi Jean, 
Thanks for the detailed work. I am looking into it. 

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAGz7LLkmPN35rHpKXmn%3DWf%2B72SHmbfX2EbBNov3d93OsciPXVQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Sancar Koyunlu

Software Engineer, Hazelcast

san...@hazelcast.com

[M. Sancar Koyunlu]

Hi Jean, 
I found the cause of the issue. It is because we were using ThreadGroup wrong. Since ThreadGroup is adding its own reference to parent thread group, they were not garbage collected. One has to call threadGroup.destroy() to remove it back from parent thread group. 

I have opened an issue for this with your report. 
https://github.com/hazelcast/hazelcast/issues/10394

Fixes
https://github.com/hazelcast/hazelcast/pull/10396 for 3.9
https://github.com/hazelcast/hazelcast/pull/10395 for 3.8.2

Thanks again. 
Sancar.

[Jean Luc]

Thanks for the quick fix, looking forward to 3.8.2.

JL

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAGz7LLkmPN35rHpKXmn%3DWf%2B72SHmbfX2EbBNov3d93OsciPXVQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Sancar Koyunlu

Software Engineer, Hazelcast

san...@hazelcast.com

--

Sancar Koyunlu

Software Engineer, Hazelcast

san...@hazelcast.com

--

You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAEj0St8p6MeQ945%2BUzDfOpSWbdfghndRzt-LK_4oLpWgt3KchA%40mail.gmail.com.

[Jean Luc]

Hi Sancar,

Is the release date for 3.8.2 known by any chance? We had an out-of-memory error in production and are really looking forward to having the fix in.

I looked at the 3.8.2. milestone in Github but there is no date given yet.

Thank you.

[M. Sancar Koyunlu]

Hi Jean,
There are no strong dates yet, but plan is to release within May. 

Regards, 

Sancar. 

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAGz7LLkmPN35rHpKXmn%3DWf%2B72SHmbfX2EbBNov3d93OsciPXVQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Sancar Koyunlu

Software Engineer, Hazelcast

san...@hazelcast.com

--

Sancar Koyunlu

Software Engineer, Hazelcast

san...@hazelcast.com

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAEj0St8p6MeQ945%2BUzDfOpSWbdfghndRzt-LK_4oLpWgt3KchA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAGz7LL%3DPo5WEAEOQwJOOyBYmVvyj0cRRVsRkYbnyojdcrKitVQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.