JVM is not exiting due to Hazelcast Client's non-daemon threads

234 views
Skip to first unread message

Pooja Pandey

unread,
Jul 26, 2019, 10:18:11 AM7/26/19
to Hazelcast
Hi,

I have a java application and during the course of action it gets an Hazelcast client. This hazelcast client causes the creation of a large number of non-daemon threads (~20). We have registered shutdown hook to shutdown the client, however at the end of main(), JVM is not exiting due to these non-daemon threads and since JVM is not exiting the shutdown hook for client doesn't get called. We don't want to call System.exit()/HazelcastClient.shutdownAll() from our main(). I wanted to check if there is any way/setting for Hazelcast client so that it will create daemon threads instead of non-daemon threads.

Thanks,
Pooja

Pooja Pandey

unread,
Jul 26, 2019, 10:32:34 AM7/26/19
to haze...@googlegroups.com, hazelcast/hazelcast, hazelcast/hazelcast
--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/7aeee9da-1463-4615-885d-47a70785369c%40googlegroups.com.

Pooja Pandey

unread,
Jul 28, 2019, 10:44:58 PM7/28/19
to haze...@googlegroups.com
Please let me know if you have any idea on this.
--

M. Sancar Koyunlu

unread,
Jul 29, 2019, 4:32:06 AM7/29/19
to Hazelcast
This is by design, both hazelcast member and clients continue to work unless shutdown is called. There is no option to create them as daemon threads.

I totally get that calling System.exit()/HazelcastClient.shutdownAll() is not a good option.  Why don't you call client.shutdown() when you are done with the client instead of a shutdown hook. 

On Mon, Jul 29, 2019 at 5:44 AM Pooja Pandey <pooja.ac...@gmail.com> wrote:
Please let me know if you have any idea on this.

On Friday, July 26, 2019, Pooja Pandey <pooja.ac...@gmail.com> wrote:
Hi,

I have a java application and during the course of action it gets an Hazelcast client. This hazelcast client causes the creation of a large number of non-daemon threads (~20). We have registered shutdown hook to shutdown the client, however at the end of main(), JVM is not exiting due to these non-daemon threads and since JVM is not exiting the shutdown hook for client doesn't get called. We don't want to call System.exit()/HazelcastClient.shutdownAll() from our main(). I wanted to check if there is any way/setting for Hazelcast client so that it will create daemon threads instead of non-daemon threads.

Thanks,
Pooja

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/7aeee9da-1463-4615-885d-47a70785369c%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CANy3jCUd3yDm4-4vwPn04N4WppOPz79zsUgjASghk8D8v82GiQ%40mail.gmail.com.


--
M. Sancar Koyunlu
Software Engineer, Hazelcast

Pooja Pandey

unread,
Jul 29, 2019, 4:53:42 AM7/29/19
to haze...@googlegroups.com
Hi Sancar,

Thank you very much for your response. For us calling HazelcastClient.shutdownAll()/client.shutdown() are same as we just have single client, however we have many many applications where we will require to do this change. Ideally we are not preferring to call for shutdown Hazelcast client from applications as Hazelcast client gets created in a deep down infrastructure of our project and we were looking for something where we can handle destruction of the Hazelcast client from internal infrastructure. 

Thanks,
Pooja

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAEj0St-J4sQ4O2GGNg3eFWCups6Ts5cwhX1-OdV%2BqjLWNqxdZA%40mail.gmail.com.

Guido Medina

unread,
Jul 29, 2019, 5:08:46 AM7/29/19
to Hazelcast
Hi Pooja,

The following answer has nothing to do with Hazelcast but with shutdown cycles, I have built complex -micro services based- systems for Linux deployed with different type of controllers, that be systemd and others.
The problem here is that Hazelcast nor any other type of client can hook into the JVM shutdown hook because things are meant to shutdown in the opposite order as they started, imagine a system with Akka, Hazelcast, RabbitMQ, etc etc.
The question is: How does Hazelcast knows that it needs to shutdown before or after Akka? or the logging system, etc etc, another problem that makes this even more difficult is that in Linux you have SIGTERM and SIGINT which are a PITA

How have I solved this problem:

Using this dependency:
<dependency>
  <groupId>org.agrona</groupId>
  <artifactId>agrona</artifactId>
  <version>1.0.3</version>
</dependency>

Here is an example of something that works better than JVM shutdown hooks, it will catch kill (SIGTERM) and catch Ctrl+C nicely (SIGINT),
I haven't tested this on Windows but on Linux it works like a charm, the changes only need to be done at your main method, you can also pass that barrier to any client and signal it yourself:

import org.agrona.concurrent.ShutdownSignalBarrier;

public class SomeMain
{
  public static void main(String[] args)
  {
    final ShutdownSignalBarrier barrier=new ShutdownSignalBarrier();
    try {
      // start client 1
      // start client 2
      // start client 3
      // you can also pass the signal down to a client and signal it yourself with barrier.signal()
      barrier.await();
    } finally{
      // stop client 3
      // stop client 2
      // stop client 1
    }
  }
}

Hope that helps,
Guido.


On Monday, July 29, 2019 at 9:53:42 AM UTC+1, Pooja Pandey wrote:
Hi Sancar,

Thank you very much for your response. For us calling HazelcastClient.shutdownAll()/client.shutdown() are same as we just have single client, however we have many many applications where we will require to do this change. Ideally we are not preferring to call for shutdown Hazelcast client from applications as Hazelcast client gets created in a deep down infrastructure of our project and we were looking for something where we can handle destruction of the Hazelcast client from internal infrastructure. 

Thanks,
Pooja

On Mon, Jul 29, 2019 at 2:02 PM M. Sancar Koyunlu <san...@hazelcast.com> wrote:
This is by design, both hazelcast member and clients continue to work unless shutdown is called. There is no option to create them as daemon threads.

I totally get that calling System.exit()/HazelcastClient.shutdownAll() is not a good option.  Why don't you call client.shutdown() when you are done with the client instead of a shutdown hook. 

On Mon, Jul 29, 2019 at 5:44 AM Pooja Pandey <pooja.ac...@gmail.com> wrote:
Please let me know if you have any idea on this.

On Friday, July 26, 2019, Pooja Pandey <pooja.ac...@gmail.com> wrote:
Hi,

I have a java application and during the course of action it gets an Hazelcast client. This hazelcast client causes the creation of a large number of non-daemon threads (~20). We have registered shutdown hook to shutdown the client, however at the end of main(), JVM is not exiting due to these non-daemon threads and since JVM is not exiting the shutdown hook for client doesn't get called. We don't want to call System.exit()/HazelcastClient.shutdownAll() from our main(). I wanted to check if there is any way/setting for Hazelcast client so that it will create daemon threads instead of non-daemon threads.

Thanks,
Pooja

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haze...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haze...@googlegroups.com.


--
M. Sancar Koyunlu
Software Engineer, Hazelcast

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haze...@googlegroups.com.

Guido Medina

unread,
Jul 29, 2019, 5:24:33 AM7/29/19
to Hazelcast
Another idea if you are using Spring which has service dependency; you can explicitly declare which services depend on which and have a @PreDestroy method in the service that is wrapping your Hazelcast client and there call the shutdown:

@Service
public class HazelcastService {
  
  @PreDestroy
  public void preDestroy() {
     hazelcastClient.shutdown();
  }
}

@Service
@DependsOn("HazelcastService")
public class SomeService {
}

The only problem with this is that you have to now explicitly declare which service each other service depend on if that is the case, but at least now you will have a proper orderly shutdown.

Pooja Pandey

unread,
Jul 29, 2019, 5:25:11 AM7/29/19
to oxy...@gmail.com, haze...@googlegroups.com
Thanks a lot Guido. We have also thought of similar kind of approach where we will notify to internal infrastructure about application's exit so that it can shutdown the client. This approach will require a modification in all the existing applications also and there are many many like applications in our product and we are trying to research an approach were we can handle shutdown of Hazelcast client internally in the Hazelcast infrastructure of our project, without touching those applications where this internal code is getting used, but as per your explanation  it looks like that by design Hazelcast clients are not meant to be shutdown automatically. So if we don't find any solution in limited time line then we might also use this notification kind of approach.

Thanks,
Pooja

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/2b26cd50-1b6c-42ef-bb1f-c88575331f7c%40googlegroups.com.

Pooja Pandey

unread,
Jul 29, 2019, 5:27:15 AM7/29/19
to haze...@googlegroups.com, oxy...@gmail.com
Thank you Guido, I will check if this suggestion is feasible for us.

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/41dd9cc9-a814-4457-9167-a4548f718da8%40googlegroups.com.

Guido Medina

unread,
Jul 29, 2019, 5:31:55 AM7/29/19
to Hazelcast
Actually, you don't need @DependsOn to be explicit if you are injecting the service that you depend on, Spring handles this automatically so the only thing you need on this approach is the @PreDestroy,
Then the only your main method has to do is to shutdown the Spring instance, in Spring boot you don't have to do this because this is done automatically,
shutdown hooks and order are not is not an easy problem to solve so I would be prepared to modify the code, etc

Pooja Pandey

unread,
Jul 29, 2019, 7:14:04 AM7/29/19
to haze...@googlegroups.com, Guido Medina
Hi,
I have figured out an approach for my case(Not tested yet completely) where I have started a non-daemon thread in internal infrastructure, this non-daemon thread monitors main thread and when it detects that main thread has exit then it shuts down the  Hazelcast client and exits. This non-daemon thread will sleep for a fixed interval in run() method.

I am not sure whether this approach will fit for other scenarios as well and in fact I need to look into all the aspects of this approach for case also.


Thanks,
Pooja



To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/9fd945c7-d091-4462-8888-95beafeaa106%40googlegroups.com.

Pooja Pandey

unread,
Aug 5, 2019, 2:55:17 PM8/5/19
to haze...@googlegroups.com, san...@hazelcast.com
Hi Sancar,

Calling client.shutdown()/ HazelcastClient.shutdownAll() from application’s main fixes the issue on Winsows10 and Windows Server 2016 but issue still persists on Windows Server 2012 R2 Standard.

Please let me know if you have idea on this OS specific issue.

Thanks,
Pooja

On Monday, July 29, 2019, M. Sancar Koyunlu <san...@hazelcast.com> wrote:
This is by design, both hazelcast member and clients continue to work unless shutdown is called. There is no option to create them as daemon threads.

I totally get that calling System.exit()/HazelcastClient.shutdownAll() is not a good option.  Why don't you call client.shutdown() when you are done with the client instead of a shutdown hook. 

On Mon, Jul 29, 2019 at 5:44 AM Pooja Pandey <pooja.ac...@gmail.com> wrote:
Please let me know if you have any idea on this.

On Friday, July 26, 2019, Pooja Pandey <pooja.ac...@gmail.com> wrote:
Hi,

I have a java application and during the course of action it gets an Hazelcast client. This hazelcast client causes the creation of a large number of non-daemon threads (~20). We have registered shutdown hook to shutdown the client, however at the end of main(), JVM is not exiting due to these non-daemon threads and since JVM is not exiting the shutdown hook for client doesn't get called. We don't want to call System.exit()/HazelcastClient.shutdownAll() from our main(). I wanted to check if there is any way/setting for Hazelcast client so that it will create daemon threads instead of non-daemon threads.

Thanks,
Pooja

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.


--
M. Sancar Koyunlu
Software Engineer, Hazelcast

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAEj0St-J4sQ4O2GGNg3eFWCups6Ts5cwhX1-OdV%2BqjLWNqxdZA%40mail.gmail.com.

M. Sancar Koyunlu

unread,
Aug 6, 2019, 3:59:48 AM8/6/19
to Pooja Pandey, haze...@googlegroups.com
Sorry, I could not help you with this. There should not be any difference between different operating systems. I would not suspect the problem caused by the different operationg system. 

On Mon, Aug 5, 2019 at 9:55 PM Pooja Pandey <pooja.ac...@gmail.com> wrote:
Hi Sancar,

Calling client.shutdown()/ HazelcastClient.shutdownAll() from application’s main fixes the issue on Winsows10 and Windows Server 2016 but issue still persists on Windows Server 2012 R2 Standard.

Please let me know if you have idea on this OS specific issue.

Thanks,
Pooja

On Monday, July 29, 2019, M. Sancar Koyunlu <san...@hazelcast.com> wrote:
This is by design, both hazelcast member and clients continue to work unless shutdown is called. There is no option to create them as daemon threads.

I totally get that calling System.exit()/HazelcastClient.shutdownAll() is not a good option.  Why don't you call client.shutdown() when you are done with the client instead of a shutdown hook. 

On Mon, Jul 29, 2019 at 5:44 AM Pooja Pandey <pooja.ac...@gmail.com> wrote:
Please let me know if you have any idea on this.

On Friday, July 26, 2019, Pooja Pandey <pooja.ac...@gmail.com> wrote:
Hi,

I have a java application and during the course of action it gets an Hazelcast client. This hazelcast client causes the creation of a large number of non-daemon threads (~20). We have registered shutdown hook to shutdown the client, however at the end of main(), JVM is not exiting due to these non-daemon threads and since JVM is not exiting the shutdown hook for client doesn't get called. We don't want to call System.exit()/HazelcastClient.shutdownAll() from our main(). I wanted to check if there is any way/setting for Hazelcast client so that it will create daemon threads instead of non-daemon threads.

Thanks,
Pooja

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.


--
M. Sancar Koyunlu
Software Engineer, Hazelcast

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

Guido Medina

unread,
Aug 22, 2019, 11:55:48 AM8/22/19
to Hazelcast
Were you able to figure this out? I was thinking to ask if you have your JVM up-to-date, if you are on Java 8 check which version you are using, also; how up-to-date your Windows servers are, etc.
Having daemon threads is not something most server services provide so I would doubt there is a problem with the current non-daemon threads behavior unless it is because some nasty bug on an old JVM version or even the OS itself.

Guido.


On Monday, August 5, 2019 at 7:55:17 PM UTC+1, Pooja Pandey wrote:
Hi Sancar,

Calling client.shutdown()/ HazelcastClient.shutdownAll() from application’s main fixes the issue on Winsows10 and Windows Server 2016 but issue still persists on Windows Server 2012 R2 Standard.

Please let me know if you have idea on this OS specific issue.

Thanks,
Pooja

On Monday, July 29, 2019, M. Sancar Koyunlu <san...@hazelcast.com> wrote:
This is by design, both hazelcast member and clients continue to work unless shutdown is called. There is no option to create them as daemon threads.

I totally get that calling System.exit()/HazelcastClient.shutdownAll() is not a good option.  Why don't you call client.shutdown() when you are done with the client instead of a shutdown hook. 

On Mon, Jul 29, 2019 at 5:44 AM Pooja Pandey <pooja.ac...@gmail.com> wrote:
Please let me know if you have any idea on this.

On Friday, July 26, 2019, Pooja Pandey <pooja.ac...@gmail.com> wrote:
Hi,

I have a java application and during the course of action it gets an Hazelcast client. This hazelcast client causes the creation of a large number of non-daemon threads (~20). We have registered shutdown hook to shutdown the client, however at the end of main(), JVM is not exiting due to these non-daemon threads and since JVM is not exiting the shutdown hook for client doesn't get called. We don't want to call System.exit()/HazelcastClient.shutdownAll() from our main(). I wanted to check if there is any way/setting for Hazelcast client so that it will create daemon threads instead of non-daemon threads.

Thanks,
Pooja

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haze...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haze...@googlegroups.com.


--
M. Sancar Koyunlu
Software Engineer, Hazelcast

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to haze...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages