Tomcat 7, Spring, Spring Flex, BlazeDS and Flex Application with haProxy, Couchbase Server and MSM

[Raja Patil]

Hello,

I am trying to establish tomcat 7 cluster for Spring, Mybatis, flex web application stack.

I have configured tomcat 7 vertical cluster of 3 instances with haproxy, couchbase and MSM and tested with
small jsp where session replication and session attribute is tested. It works satisfactorily.

The above mentioned web application which has been extensively tested on
same tomcat version but different unclustered instance it works satisfactorily there. 

When I deployed that application clustered instances I got following error in tomcat log

can somebody help me out in this situation ?

Thanks and warm regards.

Raja Patil



! 23:29:45 ~ 703 ~ WARN    ~ ATC-1 ~ Found no validity info for session id 1FB71FD6D63EEC0B5358465073B424D4 ~ de.javakaffee.web.msm.LockingStrategyNone ~ 2012-12-30
! 23:29:45 ~ 734 ~ INFO    ~ ATC-1 ~ Could not store session 1FB71FD6D63EEC0B5358465073B424D4 in memcached. ~ de.javakaffee.web.msm.BackupSessionService ~ 2012-12-30
java.util.concurrent.ExecutionException: com.esotericsoftware.kryo.SerializationException: Unable to serialize object of type: java.util.concurrent.ConcurrentHashMap
    at de.javakaffee.web.msm.BackupSessionService$SynchronousExecutorService.submit(BackupSessionService.java:348)
    at de.javakaffee.web.msm.BackupSessionService.backupSession(BackupSessionService.java:205)
    at de.javakaffee.web.msm.MemcachedSessionService.backupSession(MemcachedSessionService.java:1059)
    at de.javakaffee.web.msm.RequestTrackingHostValve.backupSession(RequestTrackingHostValve.java:229)
    at de.javakaffee.web.msm.RequestTrackingHostValve.invoke(RequestTrackingHostValve.java:154)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:929)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1002)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)
Caused by: com.esotericsoftware.kryo.SerializationException: Unable to serialize object of type: java.util.concurrent.ConcurrentHashMap
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:524)
    at com.esotericsoftware.kryo.ObjectBuffer.writeObject(ObjectBuffer.java:251)
    at de.javakaffee.web.msm.serializer.kryo.KryoTranscoder.serializeAttributes(KryoTranscoder.java:272)
    at de.javakaffee.web.msm.TranscoderService.serializeAttributes(TranscoderService.java:153)
    at de.javakaffee.web.msm.BackupSessionTask.serializeAttributes(BackupSessionTask.java:175)
    at de.javakaffee.web.msm.BackupSessionTask.call(BackupSessionTask.java:109)
    at de.javakaffee.web.msm.BackupSessionTask.call(BackupSessionTask.java:50)
    at de.javakaffee.web.msm.BackupSessionService$SynchronousExecutorService.submit(BackupSessionService.java:346)
    ... 14 more
Caused by: com.esotericsoftware.kryo.SerializationException: Unable to serialize object of type: flex.messaging.HttpFlexSession
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:493)
    at com.esotericsoftware.kryo.serialize.MapSerializer.writeObjectData(MapSerializer.java:104)
    at com.esotericsoftware.kryo.Serializer.writeObject(Serializer.java:43)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:520)
    ... 21 more
Caused by: com.esotericsoftware.kryo.SerializationException: Unable to serialize object of type: java.security.ProtectionDomain
Serialization trace:
context (java.security.AccessControlContext)
acc (org.apache.catalina.loader.WebappClassLoader)
classLoader (flex.messaging.MessageBroker)
broker (flex.messaging.FlexSessionManager)
flexSessionManager (flex.messaging.HttpFlexSessionProvider)
sessionProvider (flex.messaging.HttpFlexSession)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:493)
    at com.esotericsoftware.kryo.serialize.ArraySerializer.writeArray(ArraySerializer.java:123)
    at com.esotericsoftware.kryo.serialize.ArraySerializer.writeObjectData(ArraySerializer.java:88)
    at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)
    at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
    at com.esotericsoftware.kryo.Serializer.writeObject(Serializer.java:43)
    at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:182)
    at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
    at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)
    at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
    at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)
    at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
    at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)
    at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
    at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)
    at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:489)
    ... 24 more
Caused by: com.esotericsoftware.kryo.SerializationException: Unable to serialize object of type: java.security.ProtectionDomain
Serialization trace:
context (java.security.AccessControlContext)
acc (org.apache.catalina.loader.StandardClassLoader)
classloader (java.security.ProtectionDomain)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:493)
    at com.esotericsoftware.kryo.serialize.ArraySerializer.writeArray(ArraySerializer.java:123)
    at com.esotericsoftware.kryo.serialize.ArraySerializer.writeObjectData(ArraySerializer.java:88)
    at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)
    at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
    at com.esotericsoftware.kryo.Serializer.writeObject(Serializer.java:43)
    at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:182)
    at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
    at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)
    at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:489)
    ... 40 more
Caused by: com.esotericsoftware.kryo.SerializationException: Serialization trace:
classes (sun.misc.Launcher$ExtClassLoader)
parent (sun.misc.Launcher$AppClassLoader)
classloader (java.security.ProtectionDomain)
    at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:191)
    at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
    at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)
    at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
    at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)
    at com.esotericsoftware.kryo.serialize.ReferenceFieldSerializer.writeObjectData(ReferenceFieldSerializer.java:52)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:489)
    ... 50 more
Caused by: java.util.ConcurrentModificationException
    at java.util.Vector$Itr.checkForComodification(Vector.java:1127)
    at java.util.Vector$Itr.next(Vector.java:1104)
    at com.esotericsoftware.kryo.serialize.CollectionSerializer.writeObjectData(CollectionSerializer.java:85)
    at com.esotericsoftware.kryo.serialize.FieldSerializer.writeObjectData(FieldSerializer.java:175)
    ... 56 more
! 23:29:45 ~ 746 ~ WARN    ~ ATC-1 ~ Found no validity info for session id BA946CB17032E497345622A69768270D ~ de.javakaffee.web.msm.LockingStrategyNone ~ 2012-12-30

[Thomas Andraschko]

Hi,

i think you would need an own serializer for flex.messaging.HttpFlexSession.
Could you please try it with java serialization?

Regards,
Thomas

2012/12/30 Raja Patil <kpr.r...@gmail.com>

[Raja Patil]

Thanks Thomas,

i think you would need an own serializer for flex.messaging.HttpFlexSession.
Could you please try it with java serialization?

Yes we changed our configuration and Now Application loads without any security Class serialization errors.

However now new error starts popping and we get a non-working application. The error is as Follows.

"Detected duplicate HTTP-based FlexSessions, generally due to the remote host disabling session cookies.
Session cookies must be enabled to manage the client connection correctly"
 
If anybody can help in this situation ?

Thanks and warm regards

Raja Patil

[Martin Grotzke]

HttpFlexSession does not support clustering as it seems, do a google search for both terms to learn about the related issues. You should find out if the flex features (e.g. messaging/subscriptions) you're using allow you at all to use clustering / failover. If so, Thomas' suggestion of creating a custom serializer is a good idea.

Happy new year,
Martin

[Raja Patil]

Thanks Martin Grotzke,

HttpFlexSession does not support clustering as it seems, do a google search for both terms to learn about the related issues. You should find out if the flex features (e.g. messaging/subscriptions) you're using allow you at all to use clustering / failover. If so, Thomas' suggestion of creating a custom serializer is a good idea.

I did googled for the same but information on blazeDS clustering seems to be bit scares. The only resource available is BlazeDS dev. Guide. At present studying that it do have information on load balancing and clustering let me see how it turns out.

Since your cances are more for coming across BlazeDS clustering I would like to request you to forward me the same.

Thanks and wish you very happy and prosperous new year.

warm regards

Raja Patil  

[Raja Patil]

Hello Martin Grotzke,

I got clustering working with Blazeds. The application context xml is as below.

I tested clustering with sticky and non sticky sessions. Both work fine except for little glitch like you have to login twice to load flex application in first attempt it logins to spring security and it redirects to page loading swf but this second request is sent to another instance by Haproxy and may be at that time session data is not available with that server so it waits but second login onward the data is available to all instances and login goes to swf page for normal functioning.

How can I make sure that session data is stored to Couchbase before response of first request is sent back ?

Secondly for sticky or non-sticky sessions I have to set sticky="true" only if i set it to false flex application starts showing error about session data not available and no data is transfered to flex application though at server side services execute properly.

Can you elaborate more on this parameter ?

Thanks and warm regards.

Raja Patil.
 

<?xml version="1.0" encoding="UTF-8"?>
<Context path="/Tej-EIS"
    docBase="/home/rsp/Backup/Appl/Web_Dev/Java_WorkSpace/Tej-EIS/WebContent"
    antiResourceLocking="false"
    allowLinking="true">

    <Loader className="org.apache.catalina.loader.VirtualWebappLoader"
              virtualClasspath="/Tej-EIS-Common/Tej-EIS-Lib/*.jar"/>

    <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
        memcachedNodes="http://localhost:8091/pools"
        username="tomcat"
        password="rsp123"
        sticky="true"
        sessionBackupAsync="false"
        memcachedProtocol="binary"
        requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
        transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFactory"
        />

</Context>

[Martin Grotzke]

On 01/06/2013 04:34 AM, Raja Patil wrote:
> Hello Martin Grotzke,
>
> I got clustering working with Blazeds. The application context xml is as
> below.
>
> I tested clustering with sticky and non sticky sessions. Both work fine
> except for little glitch like you have to login twice to load flex
> application in first attempt it logins to spring security and it
> redirects to page loading swf but this second request is sent to another
> instance by Haproxy and may be at that time session data is not
> available with that server so it waits but second login onward the data
> is available to all instances and login goes to swf page for normal
> functioning.
>
> How can I make sure that session data is stored to Couchbase before
> response of first request is sent back ?

sessionBackupAsync="false" is responsible for that.

You might also increase sessionBackupTimeout which default to 100 (millis).

Also check logs if there are timeouts during session backup.

>
> Secondly for sticky or non-sticky sessions I have to set *sticky="true"*

> only if i set it to false flex application starts showing error about
> session data not available and no data is transfered to flex application
> though at server side services execute properly.
>
> Can you elaborate more on this parameter ?

sticky="true" means sticky. Non-sticky sessions are supported with
sticky="false".

Stickyness is managed by your loadbalancer (haproxy in your case as it
seems), so you should check that/how your loadbalancer handles sticky
sessions.

With sticky="true" a session is stored in the responsible tomcat for the
whole lifetime of a session. memcached is only used for backup to
support failover - so that when this tomcat fails another tomcat can
takeover/load the session from memcached.

With sticky="false" the session is primarily stored in memcached, and
only loaded into tomcat for the duration of a request. At the end of a
request the session is removed from tomcats session map (as otherwise
you got stale/outdated session data).

Is this explanation helpful?

Cheers,
Martin

>
> Thanks and warm regards.
>
> Raja Patil.
>
>
> <?xml version="1.0" encoding="UTF-8"?>
> <Context path="/Tej-EIS"
>
> docBase="/home/rsp/Backup/Appl/Web_Dev/Java_WorkSpace/Tej-EIS/WebContent"
> antiResourceLocking="false"
> allowLinking="true">
>
> <Loader className="org.apache.catalina.loader.VirtualWebappLoader"
> virtualClasspath="/Tej-EIS-Common/Tej-EIS-Lib/*.jar"/>
>
> <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
> memcachedNodes="http://localhost:8091/pools"
> username="tomcat"
> password="rsp123"

> * sticky="true"*

> sessionBackupAsync="false"
> memcachedProtocol="binary"
> requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
>
> transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFactory"
> />
>
> </Context>
>

--
inoio gmbh - http://inoio.de
Breitenfelder Str. 13c, 20251 Hamburg
Amtsgericht Hamburg, HRB 123031
Geschäftsführer: Dennis Brakhane, Martin Grotzke, Ole Langbehn

[Raja Patil]

Thanks Martin Grotzke,

 

sessionBackupAsync="false" is responsible for that.

You might also increase sessionBackupTimeout which default to 100 (millis).

Also check logs if there are timeouts during session backup.

Please correct me if I have misunderstood it. What I have inferred from your post is I should set
sessionBackupAsync="true" and set sessionBackupTimeout to suitable value. Am I right ?

Somehow I could not get MSM logging working. What I have done is.

First I have set following in setenv.sh

JVM_OPTS="-Xmx1024M -Xms1024M -XX:NewSize=768M -XX:MaxNewSize=768M -Xss512K -XX:PermSize=128m -XX:MaxPermSize=128m -DtcInstance=$TC_INSTANCE -DsharedFolder=$SHARED_FOLDER -DconfigType=$CONFIG_TYPE -XX:+UseParallelGC -Xshare:off -Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.Log4JLogger"

Secondly in My application's Log4J-config.xml I have set following. This log4J config is being used by reference in WEB-INF/web.xml as follows

    <!--========================== Log4j Configuration ==========================-->
    <listener>
        <listener-class>org.springframework.web.util.Log4jConfigListener</listener-class>
    </listener>
   
    <context-param>
        <param-name>log4jConfigLocation</param-name>
        <param-value>file:${sharedFolder}/config/log4j-Config.xml</param-value>
    </context-param>

    <context-param>
        <param-name>log4jRefreshInterval</param-name>
        <param-value>1000</param-value>
    </context-param>

The Log4J-config.xml contains following.
    <!-- ====================================================================== -->

    <appender name = "OtherLog" class = "org.apache.log4j.DailyRollingFileAppender">
        <param name = "file" value = "${sharedFolder}/logs/Tej-EIS/${tcInstance}~Other"/>
        <param name = "threshold" value = "debug"/>
        <param name = "immediateFlush" value = "true"/>
        <param name = "append" value = "true"/>
        <param name = "DatePattern" value="'~'yyyy-MM-dd-HH'.logarc'"/>
        <layout class = "org.apache.log4j.PatternLayout">
            <param name = "conversionPattern"
                   value = "! %d{HH:mm:ss} ~ %d{SSS} ~ %-7p ~ ${tcInstance} ~ %m ~ %F ~ %L ~ Other ~ %X{loginID} ~ %X{sessionID} ~ %X{remoteHost} ~ %X{remoteAdrs} ~ %d{yyyy-MM-dd} %n"/>
        </layout>
    </appender>
   
    <!-- ====================================================================== -->

    <logger name = "net.spy.memcached" additivity = "false">
        <level value = "FINE"/>
        <appender-ref ref = "OtherLog"/>
    </logger>

    <logger name = "net.spy.memcached.MemcachedConnection" additivity = "false">
        <level value = "FINE"/>
        <appender-ref ref = "OtherLog"/>
    </logger>

I think that spymemcached logging mechanism is not aware of this log4j config file so is unable to put log messages.
The wiki says that there should be another JVM setting like -Djava.util.logging.config.file=logging.properties
but it do not explicitly states about whether it should be used with SunLogger or Log4JLogger or both.

Can you please guide me what Shall I do to use log4j xml config file ?
 

sticky="true" means sticky. Non-sticky sessions are supported with
sticky="false".

Stickyness is managed by your loadbalancer (haproxy in your case as it
seems), so you should check that/how your loadbalancer handles sticky
sessions.

With sticky="true" a session is stored in the responsible tomcat for the
whole lifetime of a session. memcached is only used for backup to
support failover - so that when this tomcat fails another tomcat can
takeover/load the session from memcached.

Yes this is happening. In sticky mode config I stopped Tomcat Instance to which application is connected still at browser level application continues to function normally though another TC instance gets charge of session and serves the amf requests.
 

With sticky="false" the session is primarily stored in memcached, and
only loaded into tomcat for the duration of a request. At the end of a
request the session is removed from tomcats session map (as otherwise
you got stale/outdated session data).

When sticky="false" is set Flex application starts giving session related errors. With help of your explanation I feel that reason for error might be due to essionBackupAsync and sessionBackupTimeout settings. Since session related data being removed at end of request and second amf request might be comming before session data being stored in Couchbase server, TC instance to which this second request lands, is unable to get HttpFlexSession data which is stored in Session so it might be firing session related errors.

I will get first logging working and make sure that in nonsticky mode session data is saved in couchbase server before first request returns. One of the blogs for BlazeDS clustering configuration also states that its necessary to have following setting in clustering config.  
Please refer : http://cogimp.blogspot.in/2009/07/hardware-load-balanced-blazeds-cluster.html

"Then in the server.xml in the Tomcat conf directly uncomment the tag. I also put the sendChannelOptions="4" attribute in because I wanted to make sure that the session replication occurred before the request returned (4 = synchronous while the default is 8 which is async)."

But in MSM cluster config  I am not where I should set "sendChannelOptions" value. If possible guide me.

Is this explanation helpful?

Off-course its a very valuable input for me. Many thanks for that.

[Raja Patil]

hi Martin Grotzke,

Please refer : http://cogimp.blogspot.in/2009/07/hardware-load-balanced-blazeds-cluster.html

"Then in the server.xml in the Tomcat conf directly uncomment the tag. I also put the sendChannelOptions="4" attribute in because I wanted to make sure that the session replication occurred before the request returned (4 = synchronous while the default is 8 which is async)."

But in MSM cluster config  I am not where I should set "sendChannelOptions" value. If possible guide me.

well I figured out where this sendChannelOptions is to be set by referring TC7 doc.

its in conf/server.xml engine/host section the tag is to be given as below.

 <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
                 channelSendOptions="4">

can you guide me about equivalent in MSM based clustering ?

[Martin Grotzke]

On 01/06/2013 06:16 PM, Raja Patil wrote:
> Thanks Martin Grotzke,
>
>
> sessionBackupAsync="false" is responsible for that.
>
> You might also increase sessionBackupTimeout which default to 100
> (millis).
>
> Also check logs if there are timeouts during session backup.
>
>
> Please correct me if I have misunderstood it. What I have inferred from
> your post is I should set
> sessionBackupAsync="true" and set sessionBackupTimeout to suitable
> value. Am I right ?

No. You must set sessionBackupAsync="false". This is the equivalent to
channelSendOptions="4" - perform *sync* session backup, so that the
backup is finished when the request returns.

You're confusing tomcat and webapp logging. You're using log4j in your
webapp, but this doesn't change anything for tomcat logging.

Please follow
https://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration#Configure_logging,
this should do the trick.

Cheers,
Martin

[Raja Patil]

Thanks Martin Grotzke,

No. You must set sessionBackupAsync="false". This is the equivalent to
channelSendOptions="4" - perform *sync* session backup, so that the
backup is finished when the request returns.

Ok.
 

You're confusing tomcat and webapp logging. You're using log4j in your
webapp, but this doesn't change anything for tomcat logging.

Please follow
https://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration#Configure_logging,
this should do the trick.

Good hint. I really got confused. I am already using Log4J for Tomcat logging also. There is Log4J.properties file in CATALINA-HOME/lib I have updated it as follows and I getting a nice memcached log. So I feel that setup is almost finished. May be some fine tuning might be required but today we will be starting beta testing on clients site.

Thanks a lot for quick and extensive help extended. As promised earlier I have already mailed you a how to Hope you might have got it. If not let me know I will mail you again.