Hazelcast - java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already

[amand...@gmail.com]

Hi,

We are using hazelcast 3.3.3 as distributed cache manager with network join enabled for TCP/IP. We've two nodes of our application running in a private VPC and both these nodes are visible to each other. Sometimes, we are seeing below exception in our application logs and are wondering about the possible cause of this issue. Can you please help on identifying the reason for this exception? We deploy our application on Tomcat and use AWS for hosting our nodes.

Exception in thread "hz._hzInstance_1_dev.cached.thread-5" java.lang.IllegalStateException: Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already.  Could not load com.hazelcast.partition.impl.CheckReplicaVersion.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
        at java.lang.Throwable.initCause(Throwable.java:456)
        at org.apache.catalina.loader.WebappClassLoader.checkStateForClassLoading(WebappClassLoader.java:1335)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1216)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1177)
        at com.hazelcast.partition.impl.InternalPartitionServiceImpl$SyncReplicaVersionTask.run(InternalPartitionServiceImpl.java:1419)
        at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)
Caused by: java.lang.ClassNotFoundException
        at org.apache.catalina.loader.WebappClassLoader.checkStateForClassLoading(WebappClassLoader.java:1334)
        ... 9 more
Exception in thread "hz._hzInstance_1_dev.cached.thread-4" java.lang.IllegalStateException: Can't overwrite cause with java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already.  Could not load com.hazelcast.partition.impl.CheckReplicaVersion.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
        at java.lang.Throwable.initCause(Throwable.java:456)
        at org.apache.catalina.loader.WebappClassLoader.checkStateForClassLoading(WebappClassLoader.java:1335)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1216)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1177)
        at com.hazelcast.partition.impl.InternalPartitionServiceImpl$SyncReplicaVersionTask.run(InternalPartitionServiceImpl.java:1419)
        at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:209)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)

Any help in this regard shall be highly appreciated.

Thanks,

Aman

[Jaromir Hamala]

Hi Aman,

can't it be related to application redeployments? I assume it's a webapp running on Tomcat, is this correct? Is Hazelcast packaged inside WAR or it's in the (global) lib directory of your Tomcat installation? 

Cheers,

Jaromir

[Aman Jain]

Hi Jaromir,

Thanks for your response.

Yes, ours is an web application running on Tomcat. Hazelcast is packaged inside WAR. I am not sure whether it is related to application re-deployment or not. Our IT admin guys says that our application was up when this issue occurred, so I am assuming that it is not related to re-deployment.

I've few questions and it will be great if I can get answer to those:

- Does this issue cause any functional impact?

- What are possible reasons for this issue to happen?

- Does it make sense to shutdown hazelcast instance when my application is stopped? As of now, we are not doing it and I think it makes sense to do it. But I am not sure if this is the reason we are seeing this exception in our application.

PFA, code that we use in our application for Hazelcast.

Thanks,

Aman