Tomcat Session replication work working in my setup..
James P
I have a java (struts 2, string) web app. We need to host this app
over multiple datacenters.
I was hopping to configure 2 memcached server (Maybe 4). At least one
in each location.
We have hardware Loadbalancers on each datacenter to distribute the
traffic evenly across around 10 servers in each location.
The load balancer uses it own cookie to work out what server you were
last on to redirect you to the same one if it is available.
Then we have a round robin DNS between each datacenters.
Essentially we need session replication across both data centers.
However in my test setup i cant get the sessions to replicate at all.
On the one server it have :
<Manager
className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="dev2:*.*.*.11:11211 dev3:*.*.*.12:11211"
failoverNodes="dev3"
sessionBackupAsync="true"
transcoderFactoryClass="de.javakaffee.web.msm.serializer.javolution.JavolutionTranscoderFactory"
copyCollectionsForSerialization="false"
requestUriIgnorePattern=".*\.(png|gif|jpg|css|js)$"
/>
jvmroute=sv-dev-02
on the other
<Manager
className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="dev3:*.*.*.*:11211 dev2:*.*.*.*:11211"
failoverNodes="dev2"
sessionBackupAsync="true"
transcoderFactoryClass="de.javakaffee.web.msm.serializer.javolution.JavolutionTranscoderFactory"
copyCollectionsForSerialization="false"
requestUriIgnorePattern=".*\.(png|gif|jpg|css|js)$"
/>
jvmroute=sv-dev-03
As long as you stay on the server your session was created on its all
fine however if i turn off that server the other server doesn't share
the same session.
Memcache Stats show:
STAT pid 6835
STAT uptime 3460
STAT time 1282227998
STAT version 1.2.2
STAT pointer_size 32
STAT rusage_user 0.000000
STAT rusage_system 0.000000
STAT curr_items 0
STAT total_items 0
STAT bytes 0
STAT curr_connections 3
STAT total_connections 9
STAT connection_structures 4
STAT cmd_get 106
STAT cmd_set 0
STAT get_hits 0
STAT get_misses 106
STAT evictions 0
STAT bytes_read 1698
STAT bytes_written 5731
STAT limit_maxbytes 25165824
STAT threads 1
Session Id on one server is :
JSESSIONID=7C0----------------------AARW5-dev3.sv-dev-03;
I'm stuck Cant seem to make it work.
If i missed something you need let me know.
Martin Grotzke
do you see exceptions or s.th. else interesting in the logs? Do you
see that msm is initialized at startup?
To increase the log level for msm you should add some entries to
$CATALINA_HOME/conf/logging.properties:
de.javakaffee.web.msm.SessionTrackerValve.level = FINE
de.javakaffee.web.msm.MemcachedBackupSessionManager.level = FINE
de.javakaffee.web.msm.NodeAvailabilityCache.level = FINE
Perhaps you need also change the level of the ConsoleHandler like this:
java.util.logging.ConsoleHandler.level = FINE
Could you already get msm working on a locally installed tomcat, or
are you starting with this in the environment you described?
It's also possible to see msm in action in a local setup, just restart
a tomcat and see the session loaded from memcached on the next
request, or with two tomcats on different ports, where you can request
the session created on tomcat1 on tomcat2.
Cheers,
Martin
--
Martin Grotzke
http://www.javakaffee.de/blog/
James P
I managed to get it all working.
However have now come across a different issue.
The Session seems to exist across all of the servers. (the session
Session Creation Time seems to be shared correctly to all servers ).
However the actually data object stored in the session comes back as
null.
Strangely Some servers seem to share it. Other Show it as null.
I'm lost and don't know where to start to identify the problem.
Any help would be great.
Regards
James
On Aug 19, 7:59 pm, Martin Grotzke <martin.grot...@googlemail.com>
wrote:
Martin Grotzke
> Hi again,
>
> I managed to get it all working.
>
> However have now come across a different issue.
>
> The Session seems to exist across all of the servers. (the session
> Session Creation Time seems to be shared correctly to all servers ).
Are you referring to memcached nodes with "servers"? What's your setup
(tomcats, memcached etc.) and what do you do to create this scenario?
> However the actually data object stored in the session comes back as
> null.
> Strangely Some servers seem to share it. Other Show it as null.
Hmm. Is the data object directly stored in the session, or is it just
part of an object stored in the session?
Can you provide some simple sample that I can use to reproduce this issue?
What msm and serialization strategy are you using btw.?
Cheers,
Martin
--
Martin Grotzke
http://www.javakaffee.de/blog/
Martin Grotzke
please can you use the attached msm jar and replace the former one in
$CATALINA_HOME/lib/?
Also please set
sessionBackupAsync="false"
in your Manager configuration.
Can you then test session failover / deserialization of the session
and see if there are any exceptions in the logs?
Cheers,
Martin
James P
The objects that i was trying to place in the session seem to take to
long to serialize.
Meaning when the next server checks memcache they are not yet there.
I've setup a quick test of a diffrent approach and just serialiseing a
single string.
And it seem to always be there.
Would you say its a good idea to keep a user on a certain server for a
small amount of time to insure the session is persisted?
On Sep 7, 12:04 am, Martin Grotzke <martin.grot...@googlemail.com>
wrote:
> Hi James,
>
> $CATALINA_HOME/lib/?
> Also please set
> sessionBackupAsync="false"
> in your Manager configuration.
>
> Can you then test session failover / deserialization of the session
> and see if there are any exceptions in the logs?
>
> Cheers,
> Martin
>
> On Mon, Sep 6, 2010 at 10:12 PM, Martin Grotzke
>
>
>
> 119KViewDownload
Martin Grotzke
> After a little digging it think i've found the problem.
>
> The objects that i was trying to place in the session seem to take to
> long to serialize.
> Meaning when the next server checks memcache they are not yet there.
http://code.google.com/p/memcached-session-manager/wiki/JMXStatistics
msmStatAttributesSerializationInfo, msmStatEffectiveBackupInfo,
msmStatMemcachedUpdateInfo and msmStatCachedDataSizeInfo are
interesting stats to track this down.
To make things faster you might want to try kryo as serialization
strategy btw, you can have a look for a simple comparison:
http://code.google.com/p/memcached-session-manager/wiki/SerializationStrategyBenchmark
What's needed to use kryo is described on the SetupAndConfiguration page.
>
> I've setup a quick test of a diffrent approach and just serialiseing a
> single string.
>
> And it seem to always be there.
>
> Would you say its a good idea to keep a user on a certain server for a
> small amount of time to insure the session is persisted?
Depends on the avg time that's needed to push sessions to memcached
(requested number from above)
Just to be sure: you are using sticky sessions. Right?
Cheers,
Martin
James P
I tried to add the kryo however now no data in shared between my
servers.
Any Help you can give would be great.
Manager is:
<Manager
className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="dev3:*.*.*.12:11211"
sessionBackupAsync="false"
copyCollectionsForSerialization="false"
transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderF
$
/>
One server shows the following:
09-Sep-2010 17:36:58
de.javakaffee.web.msm.MemcachedBackupSessionManager createSession
FINE: Created new session with id 335D69ED437FA06A035E8EF486667D45-
dev3
09-Sep-2010 17:36:58 de.javakaffee.web.msm.SessionTrackerValve
backupSession
FINE: Have a session: true
09-Sep-2010 17:36:58 de.javakaffee.web.msm.SessionTrackerValve
logDebugResponseCookie
FINE: Finished, javax.servlet.http.Cookie[name=JSESSIONID,
value=335D69ED437FA06A035E8EF486667D45-dev3, domain=null, path=/,
maxAge=-1, secure=false, version=0
09-Sep-2010 17:36:58 de.javakaffee.web.msm.SessionTrackerValve
backupSession
FINE: Have a session: true
09-Sep-2010 17:36:58 de.javakaffee.web.msm.SessionTrackerValve
logDebugResponseCookie
FINE: Finished, null
.
.
.
09-Sep-2010 17:37:04
de.javakaffee.web.msm.MemcachedBackupSessionManager
updateExpirationInMemcached
FINE: Checking session 335D69ED437FA06A035E8EF486667D45-dev3:
- isValid: true
- isExpiring: false
- isBackupRunning: false
- isExpirationUpdateRunning: false
- wasAccessedSinceLastBackup: true
- memcachedExpirationTime: 3596
And the other server shows:
09-Sep-2010 17:36:58
de.javakaffee.web.msm.MemcachedBackupSessionManager loadFromMemcached
FINE: Loading session from memcached: 335D69ED437FA06A035E8EF486667D45-
dev3
09-Sep-2010 17:36:58
de.javakaffee.web.msm.MemcachedBackupSessionManager loadFromMemcached
FINE: Found session with id 335D69ED437FA06A035E8EF486667D45-dev3
09-Sep-2010 17:36:58
de.javakaffee.web.msm.MemcachedBackupSessionManager loadFromMemcached
FINE: Found session with id 335D69ED437FA06A035E8EF486667D45-dev3
09-Sep-2010 17:36:58
de.javakaffee.web.msm.MemcachedBackupSessionManager loadFromMemcached
FINE: Found session with id 335D69ED437FA06A035E8EF486667D45-dev3
09-Sep-2010 17:36:58
de.javakaffee.web.msm.MemcachedBackupSessionManager loadFromMemcached
FINE: Found session with id 335D69ED437FA06A035E8EF486667D45-dev3
09-Sep-2010 17:36:58 de.javakaffee.web.msm.SessionTrackerValve
backupSession
FINE: Have a session: true
09-Sep-2010 17:36:58 de.javakaffee.web.msm.SessionTrackerValve
backupSession
FINE: Have a session: true
09-Sep-2010 17:36:58 de.javakaffee.web.msm.SessionTrackerValve
logDebugResponseCookie
FINE: Finished, null
09-Sep-2010 17:36:58 de.javakaffee.web.msm.SessionTrackerValve
logDebugResponseCookie
.
.
.
09-Sep-2010 17:37:03
de.javakaffee.web.msm.MemcachedBackupSessionManager
updateExpirationInMemcached
FINE: Checking session 335D69ED437FA06A035E8EF486667D45-dev3:
- isValid: true
- isExpiring: false
- isBackupRunning: false
- isExpirationUpdateRunning: false
- wasAccessedSinceLastBackup: true
- memcachedExpirationTime: -1284050304
09-Sep-2010 17:37:05 de.javakaffee.web.msm.NodeAvailabilityCache
updateIsNodeAvailable
FINE: CacheLoader returned node availability 'true' for node 'dev3'.
09-Sep-2010 17:37:05 de.javakaffee.web.msm.SessionTrackerValve
backupSession
FINE: Have a session: true
09-Sep-2010 17:37:05 de.javakaffee.web.msm.SessionTrackerValve
logDebugResponseCookie
FINE: Finished, null
Thanks
James
James P
I'm using a hardware load balancer to balance requests.
Server timeouts are set to around 10 seconds so after that you may get
forwarded to a different server.
In the hope that your session is persisted across them all.
I've also now changed all our code to only store a single sting in the
session instead of a java object.
Incase that was the problem.
If you need any more let me know.
Martin Grotzke
> To Answer your other questions.
>
> I'm using a hardware load balancer to balance requests.
>
> Server timeouts are set to around 10 seconds so after that you may get
> forwarded to a different server.
> In the hope that your session is persisted across them all.
>
> I've also now changed all our code to only store a single sting in the
> session instead of a java object.
> Incase that was the problem.
So if you store only a single string, this session is correctly
restored on another server, but not if you store a more complex java
object?
Cheers,
Martin
>
> If you need any more let me know.
--
Martin Grotzke
http://www.javakaffee.de/blog/
James P
should stick to a server as long as the user is not inactive for
around that time.
Since i cant seem to get it to store anything now. The only thing that
seems to be the same on all the servers in the HttpSession.getId().
I did notice a minus memcachedExpirationTime:
> - memcachedExpirationTime: -1284050304
i just tried to add a simple random string to the session but can only
see it populated on the server that added it.
When i force the requests to any other server it is null.
Also on the sticky session. How long do they need to stick for?
Thanks again for all the help.
Regards
James
On Sep 9, 8:47 pm, Martin Grotzke <martin.grot...@googlemail.com>
wrote:
Martin Grotzke
>> Is the load balancer configured to use sticky-sessions?
> Yes it is, the session timeout is set to about 10 seconds so requests
> should stick to a server as long as the user is not inactive for
> around that time.
session configuration?
Is the loadbalancer using cookies for session stickyness and cookies
get set a expires/max-age of 10 seconds?
Otherwise I wonder why you answer with a "session timeout" when I ask
for session stickyness.
Or dou you really have a session timeout of 10 seconds configured in
tomcat - what I can't believe honestly?
>
> Since i cant seem to get it to store anything now. The only thing that
> seems to be the same on all the servers in the HttpSession.getId().
>
> I did notice a minus memcachedExpirationTime:
>> - wasAccessedSinceLastBackup: true
>> - memcachedExpirationTime: -1284050304
>
> Could this be the cause?
No.
>
> i just tried to add a simple random string to the session but can only
> see it populated on the server that added it.
> When i force the requests to any other server it is null.
How do you "force the requests to any other server"?
>
>
> Also on the sticky session. How long do they need to stick for?
Forget about this.
The problem for me with your issue is that it's not reproducable at
all - or that I don't know what I had to do to reproduce it.
The only way to solve this that I currently see is that you provide a
very simple struts2 webapp that shows your issue, so that I can debug
into the code and see what's going wrong there.
I started playing with a struts2 example yesterday btw, by just
putting struts2-showcase-2.2.1.war (from the struts2 downloads page)
into the webapps directory of the msm sample webapp
(http://github.com/magro/msm-sample-webapp ) and slightly changing the
msm configuration.
The result was that I got a NotSerializableException:
com.opensymphony.xwork2.inject.ContainerImpl$ConstructorInjector, and
this class could also not be serialized with kryo.
As you don't report this exception, I really wonder what your "setup"
is and how your webapp uses struts. This is another reason why I want
to see a sample webapp that reflects the environment that shall be
supported by msm.
Cheers,
Martin
>
> Thanks again for all the help.
>
> Regards
> James
>
>
>
> On Sep 9, 8:47 pm, Martin Grotzke <martin.grot...@googlemail.com>
> wrote:
>> On Thu, Sep 9, 2010 at 7:11 PM, James P <jsparry.gt...@gmail.com> wrote:
>> > To Answer your other questions.
>>
>> > I'm using a hardware load balancer to balance requests.
>>
>> Is the load balancer configured to use sticky-sessions?
>>
>>
>>
>> > Server timeouts are set to around 10 seconds so after that you may get
>> > forwarded to a different server.
>> > In the hope that your session is persisted across them all.
>>
>> > I've also now changed all our code to only store a single sting in the
>> > session instead of a java object.
>> > Incase that was the problem.
>>
>> So if you store only a single string, this session is correctly
>> restored on another server, but not if you store a more complex java
>> object?
>>
>> Cheers,
>> Martin
>>
>>
>>
>> > If you need any more let me know.
>>
>> --
>> Martin Grotzkehttp://www.javakaffee.de/blog/
--
Martin Grotzke
http://www.javakaffee.de/blog/
James P
I dont think its a bug in the code. more a problem with my setup.
I'll start for the beginning and see if i can explain the issues
better.
Diagram bellow shows what i was trying to achieve,
DNS (RR) (TTL set to about 1 minute
will also only )
| (Each LB is in a diffrent physical
DC.)
------------------------------------------
| |
LB LB
| |
------------------ ------------------
| | | | | |
S1 S2 S3 S4 S5 S6
| | | | | |
------------------ --- ( VPN ) --- ------------------ (so
Servers can talk to both MC's)
| |
MC1 MC2
Im currently only trying to setup the one data centre. So just 1 LB
and 3 servers and one MC server for testing.
Before i managed to get it 1/2 working. And i think i was only
experiencing issues because of the large objects i was trying to place
in the session.
So i've adjusted our implementation so that i only store the id of the
object in the session. I also upgraded the Jar as you suggested. and
then chaned to the Kyro serialization. (was using basic java before.)
However since doing that it now appears that the other servers don't
see the Session contents at all.
Sticky Sesisons:
The Load balancers will be configured with sticky sessions (More
sticky connection (Seems to be IP based and stored internaly on the LB
rather than Cookie bassed)) and just preforming a simple TCPProxy
redirect.
Currently for testing this LB session is set to stick you to a server
for only 10 seconds (just for testing to i can easily test failover.)
In the production setup i plan to have this set at about a minutes.
Therefore after 10 seconds without any connection the LB should
allocate you a new server. (This is where i'm hopping MSM comes in and
then the new server is able to retrieve the old session from the MC
server.) Also In the event the server dies the LB should allocate the
use to a new one.
So for testing i'm forcing the user to a new server by disabling the
server the user is allocated in the LB.
I'm about to start setting it back to the Java serialization to see
its its the Kryo or something else.
Hope that clears things up better.
Sorry for being such a pain i should of explained it fully the 1st
time.
James
Martin Grotzke
thanx for this explanation of your setup. My last point is still true:
I cannot reproduce your issue, even not with this explanation
(obviously I don't know/have your LBs, application, etc.). So please
tell me how I can help you. Alternatively you can follow my suggestion
and provide a simple struts2 sample webapp that I can use to reproduce
your issue. As it's not working in the big/complex setting you have to
start with a small/simple setting to see where it's starting to go
wrong.
Cheers,
Martin
--
Martin Grotzke
http://www.javakaffee.de/blog/
James P
It thing to do a little debuging looking at the stats.
I'm getting:
Server A
> msmStatNumBackupFailures: 11
Server B
> msmStatNumBackupFailures: 41
is there anyway to find out what causes the failure?
Thanks
James
James P
It was the sessionBackupTimeout.
Setting it slightly longer fixed all the errors and all servers see
the sessions perfectly.
Martin Many Thanks for all your help.
Regards
James
Martin Grotzke
> Good news. I finally cracked it.
>
> It was the sessionBackupTimeout.
>
> Setting it slightly longer fixed all the errors and all servers see
> the sessions perfectly.
To which value have you set it now? It's only relevant if you have set
sessionBackupAsync=false btw.
Is your session failover now also working with your complex objects,
or are you only storing ids in the session (what I would recommend
:-))?
I noticed one thing that seems to have changed in your setup: In one
of your first mails you mentioned a session id like
7C0----------------------AARW5-dev3.sv-dev-03
so you had configured a jvmRoute "sv-dev-03".
In one of your last postings your session id was e.g.
335D69ED437FA06A035E8EF486667D45-dev
so there was no jvmRoute included.
The first thing will fail with the current version of msm, a jvmRoute
like "svdev03" (not containing a dash) would be ok.
I just fixed this issue, it's now in the repo (if you need a release just ask).
To see more info about session backup in the logs you might add
de.javakaffee.web.msm.BackupSessionService.level = FINE
de.javakaffee.web.msm.BackupSessionTask.level = FINE
to the logging.properties.
Cheers,
Martin
>
> Martin Many Thanks for all your help.
>
> Regards
> James
--
Martin Grotzke
http://www.javakaffee.de/blog/
James P
Is there any benefit from having the jvmRoute if its not being used by
the Loadbalancer?
If there is a benefit would it be possible you could create me a
release?
Many thanks for all your help.
James
On Sep 10, 5:29 pm, Martin Grotzke <martin.grot...@googlemail.com>
wrote:
Martin Grotzke
> That might explain the reason it wasn't working with the jvmRoute.
>
> Is there any benefit from having the jvmRoute if its not being used by
> the Loadbalancer?
>
> If there is a benefit would it be possible you could create me a
> release?
>
> Many thanks for all your help.
You're welcome :-)
Cheers,
Martin
--
Martin Grotzke
http://www.javakaffee.de/blog/