"Singleton not set for org.glassfish.main.core.kernel" exception in cluster environment

[NamNVH]

1. Cluster structure

Load balancer -> web server -> 2 payara instances (instance 1 and instance 2)

Payara configuration:

• Payara Server 4.1.1.163
  
• Using Hazelcast for session replication
  
• JDK: 1.8.0_102
  

2. Problem

- Step 1: Access instance 1

- Step 2:Force the next request to go to instance 2 (by changing jsessionid or turn off instance 1)

- Step 3:The page goes BLANK, and there is exception in server log.

- Step 4:Hit F5, or Enter, the page displayed as normal, and no exception in server log.

At this moment, if I force the request to go to instance 1 again, same things happen as in step 3 and step 4.

Note:

I created another similar cluster, deployed the same war file, but this bug did not happen on this new cluster.

I can only reproduce the bug in the old cluster.

Do you have any suggestion to solve this problem?

3. Exception detail

java.lang.IllegalStateException: Singleton not set for org.glassfish.main.core.kernel [139]
 at org.glassfish.weld.ACLSingletonProvider$ACLSingleton.get(ACLSingletonProvider.java:112)
 at org.jboss.weld.Container.instance(Container.java:65)
 at org.jboss.weld.context.SerializableContextualFactory$SerializableContextualHolder.getContextualStore(SerializableContextualFactory.java:119)
 at org.jboss.weld.context.SerializableContextualFactory$SerializableContextualHolder.loadContextual(SerializableContextualFactory.java:144)
 at org.jboss.weld.context.SerializableContextualFactory$SerializableContextualHolder.get(SerializableContextualFactory.java:133)
 at org.jboss.weld.context.SerializableContextualFactory$AbstractSerializableBean.get(SerializableContextualFactory.java:165)
 at org.jboss.weld.context.SerializableContextualFactory$AbstractSerializableBean.delegate(SerializableContextualFactory.java:170)
 at org.jboss.weld.bean.ForwardingBean.attributes(ForwardingBean.java:41)
 at org.jboss.weld.util.bean.ForwardingBeanAttributes.getName(ForwardingBeanAttributes.java:55)
 at org.jboss.weld.bean.ForwardingBean.toString(ForwardingBean.java:36)
 at java.lang.String.valueOf(String.java:2994)
 at java.lang.StringBuilder.append(StringBuilder.java:131)
 at org.jboss.weld.context.SerializableContextualInstanceImpl.toString(SerializableContextualInstanceImpl.java:60)
 at java.lang.String.valueOf(String.java:2994)
 at java.lang.StringBuilder.append(StringBuilder.java:131)
 at org.glassfish.web.ha.session.management.BaseHASession.superToString(BaseHASession.java:247)
 at org.glassfish.web.ha.session.management.BaseHASession.toString(BaseHASession.java:258)
 at java.lang.String.valueOf(String.java:2994)
 at java.lang.StringBuilder.append(StringBuilder.java:131)
 at org.glassfish.web.ha.session.management.ReplicationManagerBase.findSession(ReplicationManagerBase.java:166)
 at org.apache.catalina.connector.Request.doGetSession(Request.java:3224)
 at org.apache.catalina.connector.Request.getSessionInternal(Request.java:3089)
 at org.apache.catalina.connector.Request.unlockSession(Request.java:4659)
 at org.apache.catalina.connector.CoyoteAdapter$CatalinaAfterServiceListener.onAfterService(CoyoteAdapter.java:1066)
 at org.glassfish.grizzly.http.server.Request.onAfterService(Request.java:617)
 at org.glassfish.grizzly.http.server.HttpServerFilter.afterService(HttpServerFilter.java:385)
 at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:260)
 at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
 at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
 at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
 at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
 at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
 at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
 at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:526)
 at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
 at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
 at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
 at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591)
 at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571)
 at java.lang.Thread.run(Thread.java:745)
]]

[Ondro Mihályi]

Hi,

Thanks for reporting this. It's better to raise this as an issue on our issue tracker though: https://github.com/payara/Payara/issues/

There we can continue investigatation and update you on the progress.

Ondro

[NamNVH]

Hi Ondro,

I created a  new issue:

https://github.com/payara/Payara/issues/2759

Please have a look.