Hazelcast instance shutdown on tomcat stop

[KZ]

Hi!

I searched a lot also in this google group, and all other parts of the internet, but did not found any useful information.

My problem is, that when I stop the tomcat server, the hazelcast instance stops immediately before any other service had a chance to stop before.

I have a Hazelcast WebFilter defined, and it is also throwing an exception saying the Hazelcast instance is already stopped.

My question would be, who stops my Hazelcast instance?

Second question: how can I prevent that Hazelcast is stopped before I want to be stopped? Means I would like to control when the Hazelcast instance can be stopped (even the WebFilter should not stop it!)

This is the log part, where I issued a tomcat stop, and right after the first entry is that the Hazelcast is going down

13-Mar-2015 11:27:57.154 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.

13-Mar-2015 11:27:57.155 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]

13-Mar-2015 11:27:57.206 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-nio-8009"]

13-Mar-2015 11:27:57.256 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina

13-Mar-2015 11:27:57.269 INFO [localhost-startStop-2] com.hazelcast.core.LifecycleService.null [10.1.1.138]:5000 [jbaf-cluster] [3.4.1] Address[10.1.1.138]:5000 is SHUTTING_DOWN

This is later the exception in the log, where the WebFilter tries to stop the Hazelcast

Exception in thread "Thread-21" com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!

at com.hazelcast.spi.impl.WaitNotifyServiceImpl.shutdown(WaitNotifyServiceImpl.java:221)

at com.hazelcast.spi.impl.NodeEngineImpl.shutdown(NodeEngineImpl.java:386)

at com.hazelcast.instance.Node.shutdown(Node.java:407)

at com.hazelcast.instance.LifecycleServiceImpl.shutdown(LifecycleServiceImpl.java:91)

at com.hazelcast.web.WebFilter.shutdownInstance(WebFilter.java:489)
        at com.hazelcast.web.WebFilter.destroy(WebFilter.java:480)

Is there any way, where I can tell to the instance that it should not stop unless I say so?

Configuration:

Hazelcast 3.4.1

The instance is created during server startup, and the WebFilter and the application uses the same instance, the WebFilter finds the existing instance by the instanceName. Everything seems to work properly, however since the instance is destroyed before spring destorys my services I am not able to stop the queue consumers, topic readers, etc, before the Hazelcast goes down, which can lead to exceptions.

Thanks for any help / suggestion.

Regards,

KZ

[KZ]

Update:

1.) I managed to turn off WebFilter shutdown (finally found the flag in code...)

2.) Tried to configure the Hazelcast instance through Spring, which shows a little progress... Now my Services do shutdown before Hazelcast starts its shutdown process, however still doing shutdown on his own, as now Spring throws exception upon destroying the Hazelcast instance...

Exception in thread "Thread-21" com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!

at com.hazelcast.spi.impl.WaitNotifyServiceImpl.shutdown(WaitNotifyServiceImpl.java:221)

at com.hazelcast.spi.impl.NodeEngineImpl.shutdown(NodeEngineImpl.java:386)

at com.hazelcast.instance.Node.shutdown(Node.java:407)

at com.hazelcast.instance.LifecycleServiceImpl.shutdown(LifecycleServiceImpl.java:91)

at com.hazelcast.instance.HazelcastInstanceProxy.shutdown(HazelcastInstanceProxy.java:237)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:483)

at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:350)

Regards,

KZ

[Hasan Çelik]

Hi KZ,

Hazelcast instance automatically shutdown when tomcat stopped in the other words jvm is killed. You can prevent this situation via flag at the GroupProperties.java:

PROP_SHUTDOWNHOOK_ENABLED = "hazelcast.shutdownhook.enabled" switch to disabled.

But this just guarantee that hazelcast instance not shutdown immediately and you can get response from other services. It does not provide complete manageability on instance shutdown.

Best Regards.

[KZ]

Hi Hasan,

thanks for your reply.

I tried the suggested flag, but is either not working, or I used it wrong.

This is my spring configuration for hazelcast:

    <hz:hazelcast id="hazelcastInstance">

        <hz:config>

            <hz:instance-name>hz-cluster</hz:instance-name>

            <hz:group name="cluster" password="${HZ_PASSWORD:}"/>

            <hz:properties>

                <hz:property name="hazelcast.shutdownhook.enabled">false</hz:property>

            </hz:properties>

            <hz:network port="${HZ_PORT:5701}" port-auto-increment="${HZ_PORT_AUTOINC:true}">

                <hz:join>

                    <hz:multicast enabled="${HZ_MULTICAST:true}"

                                  multicast-group="${HZ_MULTICAST_GROUP:224.2.2.3}"

                                  multicast-port="${HZ_MULTICAST_PORT:54327}"

                                  multicast-timeout-seconds="${HZ_MULTICAST_TIMEOUT:2}"

                                  multicast-time-to-live="${HZ_MULTICAST_TTL:32}"/>

                    <hz:tcp-ip enabled="${HZ_TCPIP:false}">

                        <hz:members>${HZ_TCPIP_MEMBERS:localhost}</hz:members>

                    </hz:tcp-ip>

                    <hz:aws enabled="${HZ_AWS:false}"

                            access-key="${HZ_AWS_ACCESSKEY:default}"

                            secret-key="${HZ_AWS_SECRETKEY:default}"

                            connection-timeout-seconds="${HZ_AWS_CONNTIMEOUT:5}"

                            host-header="${HZ_AWS_HOSTHEADER:ec2.amazonaws.com}"

                            region="${HZ_AWS_REGION:us-east-1}"

                            security-group-name="${HZ_AWS_SECUGROUP:}"

                            tag-key="${HZ_AWS_TAGKEY:}"

                            tag-value="${HZ_AWS_TAGVALUE:}"/>

                </hz:join>

            </hz:network>

        </hz:config>

    </hz:hazelcast>

I tried to add the shutdown hook = false to the properties, but still it stops before spring could stop it, and getting the same exception from spring in the log as in my previous post.

Any suggestion what I'm making wrong?

Thanks!

Regards,

KZ

[Mike Pilone]

I reported a similar issue a while back: https://github.com/hazelcast/hazelcast/issues/3950

I found the issue when using Jetty and a Spring application but setting the property did appear to work for me.

-mike

-- 
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/1555231c-f2e2-43ba-bfbf-6741400661bd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[KZ]

Hi Mike!

Thanks for sharing. I checked, and my property seems to be set properly, however for some reason the hazelcast is terminated by some unknown event. I tried to set a breakpoint to these shutdown parts, but I could not stop the JVM. Seems it already shuts down debugger ports at that point, when Hazelcast starts to be destroyed.

May be a fix would be, that in case the shutdownhook is not requested, then the shutdownhook thread is not registered in the Runtime at all, and not checking some property when the event is fired...

Still looking for a solution, as no matter I set the shutdownhook to enabled or disabled, it shuts down itself before spring could, which leads to an exception.

Thanks

KZ

[KZ]

Hi!

Finally I managed to debug out what is the problem. And of course it was my fault :)

So the shutdown was properly initiated by Spring, and the exception came because a queue was still read at that time.

I stopped the thread (and also interrupted) which read the queue (with poll(1, TimeUnit.SECONDS)), however it seems that it did not interrupted the poll, so the poll was still holding the queue, therefore the exception occured.

I changed my code to join the thread before continue with the shutdown, and now the exception disappeared!

Thanks for the helps and suggestions!

Regards

KZ