Hazelcast session replication not working with Apache + Tomcat + Session stickines

[hirsc...@gmail.com]

Hello everyone.

currently we are looking for session replication solution .

Our Configuration :

1 x Apache 2.0 - balancing the tomcat servers via mod_jk + sticky session enabled.
2 x Tomcat 7 + java version "1.7.0_45" -(each instance is part of VM machine consists of multiple tomcat instance for other purposes )

Our Web Application consists of :

1. RestEasy Servlet - for REST communication
2. Atmosphire Servlet - for Web socket communication
3. Spring integrated to the resteasy Servlet and charge of the Bean session lifecycle
4. some more infrastructure Filters like : NTLMFilter , SecurityFilter ...

Planning Hazelcast setup (Hazelcast 3.3 stable release .) :

1. each of the Tomcat servers will serve also as a Hazelcast node
2. the discovery method will be multicast with hazelcast group for each environment
   
3. Tomcat JVM settings are : -server -Xms512m -Xmx2048m -XX:MaxPermSize=512m
4. using session stickyness in order to boost performance.

i started testing the session replication using our Embedded Jetty 8 and its was ok.

the basic scenario i was testing is:

1.connecting via the Apache URL-> getting random tomcat server->getting session information :
JSessionid + HZ session id -> perform some REST operations.
2. forcefully kill -9 the relevant tomcat service -> try to execute another REST Service

But switching to our tomcat Test Environment i got some serious problems :

1. after working with the cluster for about an hour the CPU usage was about 80% +

2. getting the same member more then one time and some times not discovering the other member.
   how can i set a predefined cluster if i have static set of IP addresses ?

3. Debugging the WebFilter i saw :

   • each tomcat processing the doFilter and creating new hazelcastSession
     
     and sending back a cookie with the hazelcastSession session id 
     for some reason i cannot see it in the Chrome->resources->cookies , just the JSessionId

   • each tomcat server retrieving the relevant session properly

   • in each server i saw the hazelcastInstance.getMap(clusterMapName) for the same data

   • but after killing one the tomcat servers the apache redirected me to the second server
     
     and the WebFilter didnt retrieve the session.

I would very appreciate any thoughts / comments / ideas .....

Thanks in advance,
Elad Hirsch

------ catalina.out -------------

¡׳₪׳˜ 30, 2014 6:24:37 PM com.hazelcast.cluster.MulticastJoiner
INFO: [192.168.119.66]:5703 [dev] [3.3] Address[192.168.119.66]:5701 is added to the blacklist.
׳¡׳₪׳˜ 30, 2014 6:24:38 PM com.hazelcast.cluster.MulticastJoiner
INFO: [192.168.119.66]:5703 [dev] [3.3] Trying to join to discovered node: Address[192.168.119.66]:5702
׳¡׳₪׳˜ 30, 2014 6:24:38 PM com.hazelcast.nio.tcp.SocketConnector
INFO: [192.168.119.66]:5703 [dev] [3.3] Connecting to /192.168.119.66:5702, timeout: 0, bind-any: true
׳¡׳₪׳˜ 30, 2014 6:24:38 PM com.hazelcast.nio.tcp.SocketAcceptor
INFO: [192.168.119.66]:5702 [dev] [3.3] Accepting socket connection from /192.168.119.66:53956
׳¡׳₪׳˜ 30, 2014 6:24:38 PM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [192.168.119.66]:5703 [dev] [3.3] Established socket connection between /192.168.119.66:53956 and /192.168.119.66:5702
׳¡׳₪׳˜ 30, 2014 6:24:38 PM com.hazelcast.nio.tcp.TcpIpConnectionManager
INFO: [192.168.119.66]:5702 [dev] [3.3] Established socket connection between /192.168.119.66:5702 and /192.168.119.66:53956
׳¡׳₪׳˜ 30, 2014 6:24:44 PM com.hazelcast.cluster.ClusterService
INFO: [192.168.119.66]:5702 [dev] [3.3]

Members [2] {
Member [192.168.119.66]:5702 this
Member [192.168.119.66]:5703
}

׳¡׳₪׳˜ 30, 2014 6:24:44 PM com.hazelcast.cluster.ClusterService
INFO: [192.168.119.66]:5703 [dev] [3.3]

Members [2] {
Member [192.168.119.66]:5702
Member [192.168.119.66]:5703 this
}

here is the basic configuration

----------- Apache worker.properties ---------------

# TEST Node 1
worker.pearltest1.port=8009
worker.pearltest1.host=pearlapptest1
worker.pearltest1.type=ajp13
worker.pearltest1.socket_keepalive=true
worker.pearltest1.lbfactor=1
worker.pearltest1.connection_pool_size=50
worker.pearltest1.connect_timeout=5000
worker.pearltest1.prepost_timeout=5000

# TEST Node 2
worker.pearltest2.port=8009
worker.pearltest2.host=pearlapptest2
worker.pearltest2.type=ajp13
worker.pearltest2.socket_keepalive=true
worker.pearltest2.lbfactor=1
worker.pearltest2.connection_pool_size=50
worker.pearltest2.connect_timeout=5000
worker.pearltest2.prepost_timeout=5000

Load-balancing behaviour

worker.pearltestlb.type=lb
worker.pearltestlb.balance_workers=pearltest1,pearltest2
worker.pearltestlb.sticky_session=1

-------- Web.xml ----------

<?xml version="1.0" encoding="UTF-8"?>
xmlns="http://java.sun.com/xml/ns/javaee"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web app_3_0.xsd"
version="3.0" metadata-complete="false">
PearlWebApp

<!-- Spring Config -->
<context-param>
    <param-name>contextConfigLocation</param-name>
    <param-value>
        /WEB-INF/context/applicationContext.xml
    </param-value>
</context-param>

<absolute-ordering>
        <name>hazelcastWebFilter</name>
        <others/>
    </absolute-ordering>

<filter>
    <filter-name>hazelcastWebFilter</filter-name>
    <filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
    <init-param>
        <param-name>targetFilterLifecycle</param-name>
        <param-value>true</param-value>
    </init-param>
</filter>
<filter-mapping>
    <filter-name>hazelcastWebFilter</filter-name>
    <url-pattern>/*</url-pattern>
</filter-mapping>

<listener>
    <listener-class>com.hazelcast.web.SessionListener</listener-class>
</listener>

<filter>
    <filter-name>NTLMFilter</filter-name>
    <filter-class>com.idi.pearl.crm.war.filters.NTLMFilter</filter-class>
</filter>

<filter-mapping>
    <filter-name>NTLMFilter</filter-name>
    <url-pattern>/crm/*</url-pattern>
    <url-pattern>/rest/*</url-pattern>
</filter-mapping>

<filter>
    <filter-name>StaticFileFilter</filter-name>
    <filter-class>com.idi.pearl.crm.war.filters.CrmFileFilter</filter-class>
    <async-supported>true</async-supported>
    <init-param>
        <param-name>excludePattern</param-name>
        <param-value>rest|Resteasy;pubsub|Atmosphere</param-value>
    </init-param>
    <init-param>
        <param-name>loadOptimizedResources</param-name>
        <param-value>false/</param-value>
    </init-param>
</filter>

<filter-mapping>
    <filter-name>StaticFileFilter</filter-name>
    <url-pattern>/*</url-pattern>
</filter-mapping>

<filter>
    <filter-name>XSSFilter</filter-name>
    <filter-class>com.idi.pearl.framework.core.security.XSSFilter</filter-class>
</filter>

<filter-mapping>
    <filter-name>XSSFilter</filter-name>
    <url-pattern>/rest/*</url-pattern>
</filter-mapping>

<filter>
    <filter-name>SecurityTokenGenFilter</filter-name>
    <filter-class>com.idi.pearl.framework.core.security.SecurityTokenGenFilter</filter-class>
</filter>

<filter-mapping>
    <filter-name>SecurityTokenGenFilter</filter-name>
    <url-pattern>/rest/*</url-pattern>
</filter-mapping>

<filter>
    <filter-name>SecurityTokenValidateFilter</filter-name>
    <filter-class>com.idi.pearl.framework.core.security.SecurityTokenValidateFilter</filter-class>
</filter>

<filter-mapping>
    <filter-name>SecurityTokenValidateFilter</filter-name>
    <url-pattern>/rest/*</url-pattern>
</filter-mapping>


<!-- RESTEasy configuration: -->
<servlet>
    <servlet-name>Resteasy</servlet-name>
    <servlet-class>org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher</servlet-class>
    <load-on-startup>1</load-on-startup>
</servlet>

<servlet-mapping>
    <servlet-name>Resteasy</servlet-name>
    <url-pattern>/rest/*</url-pattern>
</servlet-mapping>


<servlet>
    <servlet-name>Atmosphere</servlet-name>
    <servlet-class>org.atmosphere.cpr.AtmosphereServlet</servlet-class>
    <init-param>
        <param-name>org.atmosphere.useNative</param-name>
        <param-value>false</param-value>
    </init-param>
    <load-on-startup>0</load-on-startup>
    <async-supported>true</async-supported>
</servlet>
<servlet-mapping>
    <servlet-name>Atmosphere</servlet-name>
    <url-pattern>/pubsub/*</url-pattern>
</servlet-mapping>


<!-- don't scan classpath -->
<context-param>
    <param-name>resteasy.scan</param-name>
    <param-value>false</param-value>
</context-param>

<!--todo need to check if using the builtin providers causes any problems -->
<context-param>
    <param-name>resteasy.use.builtin.providers</param-name>
    <param-value>false</param-value>
</context-param>

<context-param>
    <param-name>resteasy.providers</param-name>
    <param-value>
        org.jboss.resteasy.plugins.providers.StringTextStar, org.jboss.resteasy.plugins.providers.IIOImageProvider
        ,org.jboss.resteasy.plugins.providers.ByteArrayProvider
    </param-value>
</context-param>

<context-param>
    <param-name>resteasy.servlet.mapping.prefix</param-name>
    <param-value>rest</param-value>
</context-param>

<!-- use default factory -->
<context-param>
    <param-name>resteasy.use.deployment.sensitive.factory</param-name>
    <param-value>false</param-value>
</context-param>

<listener>
    <listener-class>org.jboss.resteasy.plugins.server.servlet.ResteasyBootstrap</listener-class>
</listener>

<listener>
    <listener-class>com.idi.pearl.framework.core.spring.SpringResteasyContextExLoaderListener</listener-class>
</listener>

<listener>
    <listener-class>org.springframework.web.context.request.RequestContextListener</listener-class>
</listener>

<listener>
    <listener-class>com.idi.pearl.framework.core.tabs.SessionTabFilter</listener-class>
</listener>

<session-config>
    <!--8 hours -->
    <session-timeout>480</session-timeout>
</session-config>

<welcome-file-list>
    <welcome-file>index.html</welcome-file>
</welcome-file-list>

-------- Spring applicationContext.xml ----------

<?xml version="1.0" encoding="UTF-8"?>
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:ehcache="http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring"
xmlns:hz="http://www.hazelcast.com/schema/spring"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring
http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring/ehcache-spring-1.2.xsd
http://www.hazelcast.com/schema/spring
http://www.hazelcast.com/schema/spring/hazelcast-spring.xsd">

<hz:hazelcast id="hazelcastSessionInstance">
    <hz:config>
        <hz:group name="test" password="123456"/>
        <hz:network port="5701" port-auto-increment="false">
            <hz:join>
                <hz:multicast enabled="false"
                              multicast-group="224.2.2.3"
                              multicast-port="54327"/>
                <hz:tcp-ip enabled="false">

                </hz:tcp-ip>
            </hz:join>
        </hz:network>

        <hz:map name="map"
                backup-count="2"
                max-size="0"
                eviction-percentage="30"
                read-backup-data="true"
                eviction-policy="NONE"
                merge-policy="com.hazelcast.map.merge.PassThroughMergePolicy"/>

    </hz:config>


</hz:hazelcast>



<bean id="hazelcastWebFilter" class="com.hazelcast.web.WebFilter"
      depends-on="hazelcastSessionInstance">
    <constructor-arg name="properties">
        <props>
            <prop key="map-name">pearl-sessions</prop>
            <prop key="sticky-session">true</prop>
            <prop key="cookie-name">hazelcast.sessionId</prop>
            <prop key="cookie-domain"></prop>
            <prop key="debug">true</prop>
            <prop key="instance-name">hazelcastSessionInstance</prop>
        </props>
    </constructor-arg>
</bean>

------- Hazelcast dependecies maven style ---------

    <!--Hazelcast-->

    <dependency>
        <groupId>com.hazelcast</groupId>
        <artifactId>hazelcast</artifactId>
        <version>3.3</version>
    </dependency>


    <dependency>
        <groupId>com.hazelcast</groupId>
        <artifactId>hazelcast-wm</artifactId>
        <version>3.3</version>
    </dependency>

    <dependency>
        <groupId>com.hazelcast</groupId>
        <artifactId>hazelcast-spring</artifactId>
        <version>3.3</version>
    </dependency>

    <dependency>
        <groupId>org.atmosphere</groupId>
        <artifactId>atmosphere-hazelcast</artifactId>
        <version>2.2.0</version>
    </dependency>

[steve...@gmail.com]

I think your issue is because you have sticky sessions turned on...

Try turning off sticky sessions.. Not sure you can make this work with sticky sessions, as when you kill one tomcat instance the proxy will redirect but not sure will keep the session.

We use the session replication with the filter and many tomcat instances serving hundreds of thousands of client requests per hour and session replication works fine.. but we don't use sticky sessions - so consecutive requests may go to any of the tomcat servers in the loadbalancer cluster.. not sure why the sticky session aids performance?


[Cleber Muramoto]

In theory sticky sessions should benefit from locality of reference, but I don't know if Hazelcast's partitioning mechanism works with sessions. From mancenter I could see that attributes got spread around members, so I set up my http-sessions map with near cache enabled.

I am using sticky sessions with apache and loadbalancing/failover both work fine. Once a node fails, the only thing that changes is the JSESSIONID suffix, but the hazelcast sessionid remains the same. E.g. if JSESSIONID=xyzw.nodeA and nodeA fails, apache should change the cookie to xyzw.nodeB and transparently failover to that node.

Did you set up the jvmRoute in your tomcat nodes?

[Cleber Muramoto]

Hello after further investigation I noticed that jvmRoute is not actually changing after failover, however the failover itself works, but affinity gets compromised.

Are you having experiencing this issue?

[Mesut Celik]

Hi Cleber,

Hazelcast Session Replication Module is container independent and does not update jvmRoute suffix of sessionId. Application developer needs to create a RenewSession Valve to do that. For container specific solution, You can check Hazelcast Enterprise - Tomcat Session Replication Module.

Elad,

If you kill your Tomcat with "kill -9" , there should not be a problem but if you are gracefully shutdown Tomcat Instance you might be suffering following issue.

https://github.com/hazelcast/hazelcast/issues/3362

On Sun, Oct 19, 2014 at 10:24 PM, Cleber Muramoto <cleber....@gmail.com> wrote:

Hello after further investigation I noticed that jvmRoute is not actually changing after failover, however the failover itself works, but affinity gets compromised.

Are you having experiencing this issue?

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at http://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/20c0e61b-b7f3-4456-849e-8487bd0340ec%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mesut Celik
Integration Team Lead 
Mahir İz Cad. No:35, Altunizade, İstanbul 
me...@hazelcast.com 

@mesutcelik

[hirsc...@gmail.com]

Hi Mesut.

thanks for the effort helping us.

1. there was a problem replicating Spring session beans , and for now i created workaround to solve this issue.

2. still - I configure the Hazelcast cluster and forcing kill -9 on one of the Tomcat servers and when Apache redirect the next client request to the other Tomcat i getting session not found exception.

3. strangely - its seems like testing Hazelcast on our Remote TEST Environment we dont getting the Hazelcast session cookie
    i think its may be the source of the problem .

10x

elad.