Simple hazelcast cluster is working, but session replication fails

[Brady Vidovic]

I have Payara 172, full version, installed on Ubuntu using OpenJDK 8.

I have set up 2 standalone nodes, referencing the same configuration:

I have enabled Hazelcast replication in the config, and see all members in the screen:

I have configured Availability Service to be enabled and use Hazelcast

When deploying the app, i checked the "Availability" enabled:

In my test application (clusterjsp.war), session data is not replicated:

How can I troubleshoot this problem?

Steps taken so far:

• I have rebooted the servers, which did not fix the problem/
  
• I also added a `glassfish-web.xml` file to the war, with the relaxCacheVersionSemantics set to true.
  
• This did not help:

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE glassfish-web-app PUBLIC "-//GlassFish.org//DTD GlassFish Application Server 3.1 Servlet 3.0//EN" "http://glassfish.org/dtds/glassfish-web-app_3_0-1.dtd">

<glassfish-web-app error-url="">

  <session-config>

    <session-manager persistence-type="hazelcast">

      <manager-properties>

        <property name="relaxCacheVersionSemantics" value="true"/>

      </manager-properties>

    </session-manager>

  </session-config>

</glassfish-web-app>

[Brady Vidovic]

I have attached my log from node2. It shows that all members are in the hazelcast cluster:

Members [3] {

Member [10.69.6.20]:5901 - 5cde01bc-2f78-4506-85d9-4449b1150547

Member [10.69.6.20]:5900 - 8d659ddb-01e6-4554-8199-d9409eda18c2

Member [10.69.6.21]:5900 - c6a41488-1233-4248-996a-764bd5963dd9 this

}

 The only stacktrace I see is this:

[2017-08-12T11:11:46.952-0500] [Payara 4.1] [SEVERE] [] [] [tid: _ThreadID=89 _ThreadName=hz._hzInstance_1_production.cached.thread-4] [timeMillis: 1502554306952] [levelValue: 1000] [[

com.hazelcast.cache.CacheNotExistsException: Couldn't find cache config with name rest-jcache

at com.hazelcast.cache.impl.AbstractCacheService.checkCacheSimpleConfig(AbstractCacheService.java:213)

at com.hazelcast.cache.impl.AbstractCacheService.createDistributedObject(AbstractCacheService.java:193)

at com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.doCreateProxy(ProxyRegistry.java:174)

at com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.createProxy(ProxyRegistry.java:167)

at com.hazelcast.spi.impl.proxyservice.impl.operations.PostJoinProxyOperation$1.run(PostJoinProxyOperation.java:63)

at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:228)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)

at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)

]]

[Brady Vidovic]

Update: I have tested creating an Instance on the same node (instead of a different server), the session replication works for the 2 instances on the same server, but doesn't replicate to the remote server.

So that narrows it down to a problem with remote nodes not being able to replicate sessions.

The second node is on the same subnet as the first and neither node has a firewall enabled.

What could be causing this problem?  

[Steve Millidge]

Are you testing using a load balancer in front of Payara? If not when you browse directly to each node individually the host name will change and the jsessionid cookie will not be sent. This is because cookies are scoped via hostname. This is also why it will work when testing on the same host.

[Brady Vidovic]

Totaly overlook that! Sorry.  Using a LB in front, session replication is working well.