Tomcat + EC2 + ELB + memcached-session-manager

[DALDEI]

I've configured tomcat on 2 EC2 instances behind an ELB set for sticky
sessions.
I first experimented with JDBC storage of sessions and it worked OK.
Then I went for memcached. The problem I'm seeing is that every
request is generating a brand new session ID. This is causing the
ELB to switch over to the other host toggling back and forth between
hosts with fresh sessions every request.

I've configured context.xml like this :


<Manager
className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:localhost:11211,n2:10.195.218.100:11211"
failoverNodes="n1"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"

transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFactory"
/>


With the coresponding other tomcat similarly. Each EC2 node runs 1
tomcat + 1 memcached and the memcachedNodes point the opposite server.

I am not seeing errors in the log files. Requests are coming through
fine and appear to be being stuck into memcached, except if I watch
the JSESSIONID (with chrome developer tools) it changes on every
request.

Any suggestions ?

Thanks
-david lee

[Martin Grotzke]

Hi David,

does the ELB stickyness work when you remove the msm configuration?
If it does the it's really likely that there's an issue

When you try to get some debug logging from msm you can follow

http://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration#Configure_logging
You can post the msm log output from catalina.out (for startup and
requests) here so that I can have a look at it.

Just in the case you don't know: msm can also do non-sticky sessions,
just set sticky="false" in the config.

Cheers,
Martin

[DALDEI]

Yes ELB does stickyness correctly if I comment out the manager
completely, or if I use the JDBC tomcat Store manager.
But when I add the memcached manager stickyness stops working and
every request generates a new JSESSIONID cookie with different values
There's a LOT of debugging ... how much do you want ?
Here's a snippet ... let me know if I'm at all close ...











FINE: Session 30DB4BBB41175F758D7288944638C4C9-n2 not found in
memcached.
Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.SessionTrackerValve
invoke
FINE: >>>>>> Request starting: /odd_admin/Login.jsp ==================
Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.MemcachedSessionService
createSession
FINE: createSession invoked: null
Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.SessionIdFormat
createSessionId
FINE: Creating new session id with orig id
'09720EC692210B9AB9EF05908961BC01' and memcached id 'n2'.
Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.MemcachedSessionService
createSession
FINE: Created new session with id 09720EC692210B9AB9EF05908961BC01-n2
Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.BackupSessionService
backupSession
FINE: Starting for session id 09720EC692210B9AB9EF05908961BC01-n2
Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.BackupSessionTask call
FINE: Starting for session id 09720EC692210B9AB9EF05908961BC01-n2
Oct 11, 2011 5:01:04 AM
de.javakaffee.web.msm.JavaSerializationTranscoder writeAttributes
FINE: Ignoring attribute 'ODD_NLS_RESOURCE' as it does not implement
Serializable
Oct 11, 2011 5:01:04 AM
de.javakaffee.web.msm.JavaSerializationTranscoder writeAttributes
FINE: storing attribute 'characterEncoding' with value 'utf-8'
Oct 11, 2011 5:01:04 AM
de.javakaffee.web.msm.JavaSerializationTranscoder writeAttributes
FINE: storing attribute 'UserName' with value 'admin'
Oct 11, 2011 5:01:04 AM
de.javakaffee.web.msm.JavaSerializationTranscoder writeAttributes
FINE: storing attribute 'UserID' with value '1'
Oct 11, 2011 5:01:04 AM
de.javakaffee.web.msm.JavaSerializationTranscoder writeAttributes
FINE: storing attribute 'sLoginErr' with value 'sendRedirect'
Oct 11, 2011 5:01:04 AM
de.javakaffee.web.msm.JavaSerializationTranscoder writeAttributes
FINE: storing attribute 'UserRights' with value '3'
Oct 11, 2011 5:01:04 AM
de.javakaffee.web.msm.JavaSerializationTranscoder writeAttributes
FINE: storing attribute 'ODD_LOCALE' with value 'en_US'
Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.BackupSessionTask
doBackupSession
FINE: Trying to store session in memcached:
09720EC692210B9AB9EF05908961BC01-n2
Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.SessionTrackerValve
logDebugRequestSessionCookie
FINE: Have request session cookie: domain=null, maxAge=-1, path=null,
value=30DB4BBB41175F758D7288944638C4C9-n2, version=0, secure=false
Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.SessionTrackerValve
logDebugResponseCookie
FINE: Request finished, with Set-Cookie header:
JSESSIONID=09720EC692210B9AB9EF05908961BC01-n2; Path=/odd_admin;
Secure
Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.SessionTrackerValve
invoke
FINE: <<<<<< Request finished: /odd_admin/Login.jsp ==================














On Oct 11, 8:18 am, Martin Grotzke <martin.grot...@googlemail.com>
wrote:

> Hi David,
>
> does the ELB stickyness work when you remove the msm configuration?
> If it does the it's really likely that there's an issue
>
> When you try to get some debug logging from msm you can follow
>

> http://code.google.com/p/memcached-session-manager/wiki/SetupAndConfi...

>  signature.asc
> < 1KViewDownload

[Martin Grotzke]

It seems as if this was the output of the first request, was it? Or was
this a request that already had a session?
Please also the log output of two requests for the same session (logs
from different tomcats if they're both involved).

Can you please also post log output when msm starts?

Can you please configure sessionBackupAsync="false" and try + send logs
again?

Cheers,
Martin

[DALDEI]

I couldnt see a way to attach files here so I put them on FTP.
If you want me to copy them I will but they are 100k each.
This is the catalina.out from a fresh run of 2 tomcat servers while I
hit the load balencer trying to login at odd_admin/Login.jsp

ftp://lee.calldei.com/pub/catalina.out.1
ftp://lee.calldei.com/pub/catalina.out.2


Here's my config
tomcat1:

<Manager
className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:localhost:11211,n2:10.195.218.100:11211"
failoverNodes="n1"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"

transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFactory"

sessionBackupAsync="false"
/>


tomcat2
<Manager
className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:localhost:11211,n2:10.116.79.43:11211"

failoverNodes="n1"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"

transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFactory"

sessionBackupAsync="false"
/>

Let me know if there's anything else I can try , and thanks for the
help !

-David




On Oct 11, 8:57 am, Martin Grotzke <martin.grot...@googlemail.com>
wrote:

>  signature.asc
> < 1KViewDownload

[Martin Grotzke]

There's an issue with your memcachedNodes config, as n1 and n2 point to
different memcached servers. memcachedNodes should be the same for both
tomcats, just with different failoverNodes.

E.g.
tomcat1:
memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
failoverNodes="n1"
tomcat2:
memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
failoverNodes="n2"

There also seems to be an issue with the stickyness configuration. The
first request was served by tomcat2, from catalina.out.2:
Oct 11, 2011 6:04:19 AM de.javakaffee.web.msm.SessionTrackerValve

logDebugResponseCookie
FINE: Request finished, with Set-Cookie header:

JSESSIONID=D81FCF8A0B8F939B7C359C8AF8481D49-n2; Path=/odd_admin; Secure

The next request was served by tomcat1, from catalina.out.1:
Oct 11, 2011 6:04:26 AM de.javakaffee.web.msm.MemcachedSessionService
loadFromMemcached
FINE: Loading session from memcached: D81FCF8A0B8F939B7C359C8AF8481D49-n2

For this request on tomcat1 a new session was created:
Oct 11, 2011 6:04:26 AM de.javakaffee.web.msm.SessionTrackerValve

logDebugResponseCookie
FINE: Request finished, with Set-Cookie header:

JSESSIONID=B943B81E6153A6B015FA65363689C6D8-n2; Path=/odd_admin; Secure

There's also one thing that looks somehow strange: the session cookie
that was sent to tomcat1 does not have the Path set (which was set as
you can see from the logged Set-Cookie header from tomcat2) and it was
not flagged as secure:
Oct 11, 2011 6:04:26 AM de.javakaffee.web.msm.SessionTrackerValve

logDebugRequestSessionCookie
FINE: Have request session cookie: domain=null, maxAge=-1, path=null,

value=D81FCF8A0B8F939B7C359C8AF8481D49-n2, version=0, secure=false

Not sure if this is a problem, but perhaps it indicates an issue related
to SSL/Non-SSL (see e.g.
https://forums.aws.amazon.com/message.jspa?messageID=201320)

Another thing that I noticed: you should exclude heartbeat from msm
tracking by adding it to the requestUriIgnorePattern (s.th. like
requestUriIgnorePattern="/heartbeat|.*\.(ico|png|gif|jpg|css|js)$").

Cheers,
Martin

[Martin Grotzke]

Btw, another observation was that you have lots of webapps deployed.
Shall all be run with msm? If not you should configure msm only for the
relevant webapps (in META-INF/context.xml or
conf/[enginename]/[hostname]/[context].xml, see also
http://tomcat.apache.org/tomcat-6.0-doc/config/context.html#Introduction).

Cheers,
Martin

[DALDEI]

Thanks for the help ! I was about to give up when I finally got it to
work.
The final fix was I was mucking around with the ELB settings trying
out both app level ("JSESSIONID") sticky cookies and ELB generated
cookies.
A complexity is I'm using the SSL terminated on ELB ... I had port 80
and 443 configured to different sticky settings.
I set them both to JSESSIONID and then NOTHING worked ... even without
any special Manager ELB wasnt being sticky !
So I switched it back to ELB managed cookies then stickyness started
working again. Then I tried both JDBC and memcached and both are
working great.

Need to do some more tests now but the concept is working.

Question: is there and advantage in this setup of using memcached
instead of Tomcat clustering (I know I cant use multicast on EC2 but I
found a way to force the IP for the clustering). What advantage does
using memcached have over the builtin tomcat clustering ?

Thanks again for the help !

Going to try to get off the need for sticky sessions but I still have
some state information for back-to-back requests (a temp file being
made ...) that needs stickyness. If the server really fails its OK
for the user to retry - just not to have to login again.



On Oct 11, 10:11 am, Martin Grotzke <martin.grot...@googlemail.com>
wrote:

[DALDEI]

One more question I'm a little confused about.
I have a 2 server configuration like in the docs for testing.

Server A: Tomcat 1 , Memcached 1
Server B : Tomcat 2 , Memcached 2

The Manager is configured to know both memcached's but "ignore"
itself. Like you suggested:

<Manager
className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:localhost:11211,n2:10.195.218.100:11211"
failoverNodes="n1"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFa
ctory"
sessionBackupAsync="false"
/>
tomcat2
<Manager
className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:localhost:11211,n2:10.116.79.43:11211"
failoverNodes="n1"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFa
ctory"
sessionBackupAsync="false"
/>

=================





I tried shutting down the active tomcat (sticky session) and it
properly failed over to the other tomcat.
Great. Then I started it up again and waited for it to stabilize.

Then I shutdown both tomcat and memcached on the same system. This
would simulate a host going down.
Failover occured but not session information ! I had to login
again ...

My guess is that the state was

Session -> Active on Tomcat1 , stored on Memcached 2

When I kill the host (tomcat1 , memcached1 ) the load balencer hits
Tomcat2 ... But tomcat2 was told to ignore Memcached 2 so it looks in
Memcached1 and cant find the session !

How do I arrange for this kind of failover to work ?

I think I'm missing something fundimental.

Thanks for any suggestions.

-David

[Martin Grotzke]

On 10/11/2011 05:08 PM, DALDEI wrote:
> Thanks for the help ! I was about to give up when I finally got it to
> work.

Great!

> So I switched it back to ELB managed cookies then stickyness started
> working again. Then I tried both JDBC and memcached and both are
> working great.

Out of couriosity: how does ELB-managed differ from app-managed? Also in
ELB-managed mode I'd expect the app to set the JSESSIONID cookie when
it's needed, as ELB does not know when to set the cookie (and which value).

> Question: is there and advantage in this setup of using memcached
> instead of Tomcat clustering (I know I cant use multicast on EC2 but I
> found a way to force the IP for the clustering). What advantage does
> using memcached have over the builtin tomcat clustering ?

The advantage is better scalability. Tomcat session replication is an
all-to-all replication and thus is not very scalable. The backup session
replication is/was not recommended for production (now this is changed
to "Downside of the BackupManager: not quite as battle tested as the
delta manager", see [1]).

Cheers,
Martin

[1] http://tomcat.apache.org/tomcat-6.0-doc/cluster-howto.html

[Martin Grotzke]

The failoverNodes of tomcat2 should be n2, not n1.

> I tried shutting down the active tomcat (sticky session) and it
> properly failed over to the other tomcat.
> Great. Then I started it up again and waited for it to stabilize.
>
> Then I shutdown both tomcat and memcached on the same system. This
> would simulate a host going down.

Right, the previous restart of tomcat1 was not really necessary for
simulating a server crash...

> Failover occured but not session information ! I had to login
> again ...
>
> My guess is that the state was
>
> Session -> Active on Tomcat1 , stored on Memcached 2
>
> When I kill the host (tomcat1 , memcached1 ) the load balencer hits
> Tomcat2 ... But tomcat2 was told to ignore Memcached 2 so it looks in
> Memcached1 and cant find the session !

msm only moves the session to another memcached if a memcached goes down
that should store a session.

The memcached nodeId is stored in the sessionId, so you can check for
each request in which memcached the session is stored.

> How do I arrange for this kind of failover to work ?

Actually it should work. You can send logs so that I can have a look at
what happened.

Cheers,
Martin

[Martin Grotzke]

As I forgot: another advantage of msm is pluggable serialization, you're not restricted to java serialization (e.g. kryo is faster and you can provide custom serializers which is especially useful for 3rd party classes stored in the session).

Cheers,
Martin

[DALDEI]

Thanks for all the tips. Custom serialization might be really
valuable as I'm using for some apps a codebase that does not implement
Serilalizable !!!!

Here's links to the 2 logfiles (catalina.out)
Failure case

Start both
Login to 1 - works great
Kill 1 - wait 30 sec
Click around - Failover works great !!!
restart #1 ... kill #2 - wait 30 sec
click ... requires login (session data must have been lost)


ftp://lee.calldei.com/pub/catalina.out.1a
ftp://lee.calldei.com/pub/catalina.out.2a


As for what ELB differs between ELB sticky cookies and App specified
cookies ?
Here's the link ... but I cant get the App specified cookie
(JSESSIONID) to actually work.
http://stackoverflow.com/questions/5093309/amazon-load-balancer-sticky-sessions-configuration-for-jsessionid-in-url

-David
(again thanks for spending the time on this !)


On Oct 11, 12:49 pm, Martin Grotzke <martin.grot...@googlemail.com>
wrote:

> As I forgot: another advantage of msm is pluggable serialization, you're not
> restricted to java serialization (e.g. kryo is faster and you can provide
> custom serializers which is especially useful for 3rd party classes stored
> in the session).
>
> Cheers,
> Martin
>

[Martin Grotzke]

On 10/11/2011 07:02 PM, DALDEI wrote:
> Here's links to the 2 logfiles (catalina.out)
> Failure case
>
> Start both
> Login to 1 - works great
> Kill 1 - wait 30 sec
> Click around - Failover works great !!!
> restart #1 ... kill #2 - wait 30 sec
> click ... requires login (session data must have been lost)

Can you please be more specific if/when you're killing tomcat, memcached
or both?

Again, as written previously in a previous mail: It seems your
failoverNodes are wrong. The failoverNode should point to the memcached
running on the same machine. This also explains why your session was
lost on host crash.

It should be:
tomcat1:
memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
failoverNodes="n1"

tomcat2:
memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
failoverNodes="n2"

> ftp://lee.calldei.com/pub/catalina.out.1a
> ftp://lee.calldei.com/pub/catalina.out.2a
I'm getting "550 Failed to change directory.".

> As for what ELB differs between ELB sticky cookies and App specified
> cookies ?
> Here's the link ... but I cant get the App specified cookie
> (JSESSIONID) to actually work.
> http://stackoverflow.com/questions/5093309/amazon-load-balancer-sticky-sessions-configuration-for-jsessionid-in-url

Thanx, I'll read it later.

Cheers,
Martin

[DALDEI]

Hi sorry for the bad links forgot to do the chmod.
They should work now

1) when I say "kill tomcat and memcached" I mean on 'server n1' I do
service tomcat stop
service memcached stop

2) as for the config, I have it fixed as you initially mentioned (and
have for the last 3 posts :).
The failoverNode refers to the same machine as the current machine
'aka localhost'.
For example

>    memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
>    failoverNodes="n1"

This is on the machine with IP=10.116.79.43

and

>    memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
>    failoverNodes="n2"
>

This is on the machine with
IP=10.195.218.100




Thanks !
-David

[Martin Grotzke]

Hi David,

to which tomcat belongs catalina.out.1a? It says

configured nodes definition
n1:10.116.79.43:11211,n2:10.195.218.100:11211, failover nodes n2

and therefore should belong to tomcat2. Is this right?

Cheers,
Martin

--
Dennis Brakhane, Martin Grotzke und Ole Langbehn GbR
Breitenfelder Str. 13c, 20251 Hamburg

[DALDEI]

Yes that is right.


On Oct 11, 8:09 pm, Martin Grotzke <martin.grot...@googlemail.com>
wrote:

[Martin Grotzke]

Now I had a look at the logs, this is what happened:

1) tomcat2: created session C57F53867C23BECF9EE0F566EB4594CB-n1 (n1 ->
stored in memcached1)
2) tomcat2: shutdown
3) tomcat1: take over C57F53867C23BECF9EE0F566EB4594CB-n1
4) tomcat1, memcached1: shutdown,
5) tomcat2: start, cannot connect to memcached1, says: "The node n1 is
not available, therefore C57F53867C23BECF9EE0F566EB4594CB-n1 cannot be
loaded from this memcached."

So you have created the (improbable?) situation that both the tomcat
holding a session and the memcached that keeps the backup go both down
at the same moment.

Even if you had a persistent store this would not have helped you in
this situation, if it were running on machine1. For this situation a
membase cluster with servers on machine1 and machine2 and replication
enabled would be the solution.

Cheers,
Martin

[DALDEI]

Thank you, I appreciate the time you took analizing this and I
agree ... I am trying to stress the limits in order to understand how
it works.
( you never really know how well something works until you break
it ! )

Question: "replication enabled would be the solution. "

Does msm support replication ? I cant find any reference to it.

Thanks.

And thanks again for spending so much time helping me out, I really
appreciate it (and so will the company when I get it working
properly !)

-David

[Martin Grotzke]

With "replication enabled" I referred to membase. It allows to configure replication per bucket.

Cheers,
Martin