Re: Need some information and help in using the memcached-session-manager
Martin Grotzke
I'm cc'ing the mailing list, as such questions are of general interest
and might be asked several times (we take it over to the list completely
if you're subscribed). I hope this is ok for you.
On 04/19/2011 08:41 PM, Reinwald Warapen wrote:
> Hey Martin,
> We tried the Tomcat Persistent Manager (JDBC Store) and it does not work
> for our application for the following reasons.
> ** We have 6 instances of tomcat each receiving approximately 200
> concurrent requests. Since there were around 1200 writes at any time to
> the mysql store it took around 4 to 5 secs for the background thread
> operation to complete. Now at this point of time if we faced a server
> restart and the request was routed to another tomcat server we faced an
> issue since the session does not exist in the database (request to the
> new server took place before the old could write to database ) . We came
> to the conclusion that the Tomcat Persistent Manager does not scale for
> our application. :(
>
> We plan on testing the memcache session manager for our web app.Some
> questions I wanted to ask you were :
>
> A) Based on the above ,do you think that the memcache approach would
> work for our application?
Hard to answer this theoretically. There are several things that impact
this, e.g.:
- used hardware for tomcat host machines, memcached servers
- networking infrastructure
- session size
- percentage of requests that modify the session (vs. readonly sessions)
>
> B)You also mention in the documentation that "after a request was
> finished, the session (only if existing) is additionally sent to a
> memcached node for backup". I assume a backup would be needed only in
> the case of sticky .
I chose the term "backup" as for sticky sessions the session stored in
memcached only takes the role of a backup (sessions are by default
served from the local session map).
For non-sticky sessions memcached (or membase, memcachedb) is the
primary session repository, sessions are (when requested by the app)
loaded from memcached (only once per request). For non-sticky sessions
the session is additionally stored in another/secondary memcached node
for "backup" (if the primary memcached node fails).
> So does this happen the same way like in Persistent
> Manager (background thread which runs every 10 sec) or in a different
> manner (may be after every request) ?
It's different: sessions are stored directly in memcached. You can
control if this shall by done asynchronously (in a separate thread, when
sessionBackupAsync="true") or in the request thread
(sessionBackupAsync="false").
For sessionBackupAsync="true" you can specify the number of threads for
session backup via the backupThreadCount attribute.
For sessionBackupAsync="false" you can specify the timeout to wait for
the session backup to complete via sessionBackupTimeout.
Those configuration attributes are specified in the documentation:
http://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration
>
> C) Can you please elaborate on how the session replication happens
> across the memcache nodes in the case of non-sticky? As in how can we be
> sure that the next request when sent to a different server in a fraction
> of a second would get the session from memcache?
Set sessionBackupAsync="false" and an appropriate sessionBackupTimeout.
Though, I have still one thing to fix/release for this (with non-sticky
sessions), see this thread:
http://groups.google.com/group/memcached-session-manager/browse_thread/thread/2bd165916ed24a2f/02d045967659a67a?#02d045967659a67a
The last message also contains a msm jar (tomcat 6 version) with a fix.
>
> D) Do you think the non-sticky session version of memcache is stable
> enough for production ?
The same as above.
With this fix I'm not aware of any open issues.
Still, it's a relatively new feature and I'm not sure how many people
are running msm with non-sticky sessions, so it's hard to tell. From
some questions/discussions some seem to use it, so you should not be the
first.
>
> E)/sessionBackupAsync/ ,/backupThreadCount/ ,/sessionBackupTimeout/ are
> not not used in a sticky environment.
They are used, in BackupSessionService and BackupSessionTask.
Why do you think they are not used?
>
> F) Can you please provide me
> ** The link where I can check out the code of the stable release for
> both sticky and non sticky (since I cannot see a sticky branch on github)
master is the stable / release branch for tomcat6, tomcat7 branch is for
tomcat7. When there are bigger features there are separate feature
branches used.
> ** Downloadable Jars or can I assume the ones got from here
> http://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration
> are the latest
http://code.google.com/p/memcached-session-manager/downloads/list
Cheers,
Martin
>
> Thanks in advance
>
>
> On 2/7/2011 9:03 PM, Martin Grotzke wrote:
>> Hi Reinwald,
>>
>> On Sat, Feb 5, 2011 at 8:03 AM, Reinwald Warapen <reinw...@directi.com> wrote:
>>> Hey Martin,
>>> You had mentioned that the background thread which calls the method in the
>>> Persistent Manger runs after every 10 sec. After reading up the docs its
>>> mentioned that the time delay can be set by setting
>>> "backgroundProcessorDelay"
>>> (http://tomcat.apache.org/tomcat-6.0-doc/config/context.html) when defining
>>> the context in the config file.
>>> So by setting this to a value say 1 , would the thread be spawned every 1
>>> sec instead of the default 10 ?
>> Yes, the method backgroundProcess should be called every second.
>>
>>> Your thoughts on this?
>> One question is if 1 second delay of sessions being updated is ok for you.
>> Another question would be which other things are executed each second
>> then and if there's anything that's considered to be too expensive for
>> this.
>> You'd need to check tomcat sources for this
>> (ContainerBase.ContainerBackgroundProcessor is the starting point).
>>
>> When sessions are swapped out every second it's getting more probable
>> that a session is accessed by the webapp during swapOut, therefore you
>> should check that you use thread safe collections for collections put
>> in the session (if there are any).
>>
>> Cheers,
>> Martin
>>
>> Cheers,
>> Martin
>>
>>> On 1/19/2011 2:38 AM, Martin Grotzke wrote:
>>>
>>> On Tue, Jan 18, 2011 at 6:01 PM, Reinwald Warapen <reinw...@directi.com>
>>> wrote:
>>>> I just had a discussion with my team and we are keen on going the
>>>> Non-Sticky way. We are not worried about these issues you had mentioned
>>>> before :
>>>> [1] http://code.google.com/p/memcached-session-manager/issues/detail?id=71
>>>> [2] http://code.google.com/p/memcached-session-manager/issues/detail?id=72
>>>>
>>>> Other than the above mentioned issues can we safely use the
>>>> memcached-session-manager (non-sticky) ?
>>> This is a non-sticky version:
>>> https://github.com/magro/msm-sample-webapp/raw/nonsticky-sessions/runtime/apache-tomcat-6.0.20/lib/memcached-session-manager-1.3.3.jar
>>> I can also push the latest non-sticky version / provide built jars soon,
>>> this will contain proper handling for
>>> http://code.google.com/p/memcached-session-manager/issues/detail?id=74
>>>
>>>> One more solution we thought would be easy :
>>>> 1) We will use the HttpServletRequestWrapper and override the getSession
>>>> method to return a custom session object.Session object would always save
>>>> and load from a database or memcache.
>>>> For this to work we would create a filter listening on an url-pattern of
>>>> /* .
>>> Perhaps this could work. There are some things that come to mind:
>>> - Session lifecycle listeners won't work anymore (no session invalidation
>>> event when session timeout is reached)
>>> - You need to take care of invalidated session in the database and clean up
>>> regularly
>>> - You need to take care of session security issues by yourself that would
>>> otherwise be covered by the container
>>> Cheers,
>>> Martin
>>>
>>
>>
>
--
Martin Grotzke
http://twitter.com/martin_grotzke
Martin Grotzke
> On 04/19/2011 08:41 PM, Reinwald Warapen wrote:
>> C) Can you please elaborate on how the session replication happens
>> across the memcache nodes in the case of non-sticky? As in how can we be
>> sure that the next request when sent to a different server in a fraction
>> of a second would get the session from memcache?
> Set sessionBackupAsync="false" and an appropriate sessionBackupTimeout.
>
> Though, I have still one thing to fix/release for this (with non-sticky
> sessions), see this thread:
> http://groups.google.com/group/memcached-session-manager/browse_thread/thread/2bd165916ed24a2f/02d045967659a67a?#02d045967659a67a
>
> The last message also contains a msm jar (tomcat 6 version) with a fix.
http://code.google.com/p/memcached-session-manager/issues/detail?id=98
and pushed the fix to master and tomcat7 branch.
Cheers,
Martin
Reinwald
On Apr 20, 1:24 am, Martin Grotzke <martin.grot...@googlemail.com>
wrote:
A) You mention that the backup takes place in a separate thread (if we
set sessionBackupAsync="true") .Does the thread get spawned by each
request or is it done on a timer basis (like in Persistent Manager).
and how does the "backupThreadCount" come into the picture here.
B) If we specify say 3 memcached nodes (non-sticky environment)
1) When a session needs to be stored on what basis do you select
the primary node?
2) Can we assume at any point of time all memcache nodes will all
sessions?
>
> > So does this happen the same way like in Persistent
> > Manager (background thread which runs every 10 sec) or in a different
> > manner (may be after every request) ?
>
> It's different: sessions are stored directly in memcached. You can
> control if this shall by done asynchronously (in a separate thread, when
> sessionBackupAsync="true") or in the request thread
> (sessionBackupAsync="false").
> For sessionBackupAsync="true" you can specify the number of threads for
> session backup via the backupThreadCount attribute.
> For sessionBackupAsync="false" you can specify the timeout to wait for
> the session backup to complete via sessionBackupTimeout.
>
>
>
> > C) Can you please elaborate on how the session replication happens
> > across the memcache nodes in the case of non-sticky? As in how can we be
> > sure that the next request when sent to a different server in a fraction
> > of a second would get the session from memcache?
>
> Set sessionBackupAsync="false" and an appropriate sessionBackupTimeout.
>
> Though, I have still one thing to fix/release for this (with non-sticky
> The last message also contains a msm jar (tomcat 6 version) with a fix.
>
>
>
> > D) Do you think the non-sticky session version of memcache is stable
> > enough for production ?
>
> The same as above.
>
> With this fix I'm not aware of any open issues.
>
> Still, it's a relatively new feature and I'm not sure how many people
> are running msm with non-sticky sessions, so it's hard to tell. From
> some questions/discussions some seem to use it, so you should not be the
> first.
>
>
>
> > E)/sessionBackupAsync/ ,/backupThreadCount/ ,/sessionBackupTimeout/ are
> > not not used in a sticky environment.
>
> They are used, in BackupSessionService and BackupSessionTask.
>
> Why do you think they are not used?
>
>
>
> > F) Can you please provide me
> > ** The link where I can check out the code of the stable release for
> > both sticky and non sticky (since I cannot see a sticky branch on github)
>
> master is the stable / release branch for tomcat6, tomcat7 branch is for
> tomcat7. When there are bigger features there are separate feature
> branches used.
>
> > ** Downloadable Jars or can I assume the ones got from here
>
> http://code.google.com/p/memcached-session-manager/downloads/list
>
> Cheers,
> Martin
>
>
>
>
>
>
>
>
>
>
>
> > Thanks in advance
>
> > On 2/7/2011 9:03 PM, Martin Grotzke wrote:
> >> Hi Reinwald,
>
> >>>> I just had a discussion with my team and we are keen on going the
> >>>> Non-Sticky way. We are not worried about these issues you had mentioned
> >>>> before :
> >>>> [1]http://code.google.com/p/memcached-session-manager/issues/detail?id=71
> >>>> [2]http://code.google.com/p/memcached-session-manager/issues/detail?id=72
>
> >>>> Other than the above mentioned issues can we safely use the
> >>>> memcached-session-manager (non-sticky) ?
> >>> This is a non-sticky version:
> >>> this will contain proper handling for
> >>>http://code.google.com/p/memcached-session-manager/issues/detail?id=74
>
> >>>> One more solution we thought would be easy :
> >>>> 1) We will use the HttpServletRequestWrapper and override the getSession
> >>>> method to return a custom session object.Session object would always save
> >>>> and load from a database or memcache.
> >>>> For this to work we would create a filter listening on an url-pattern of
> >>>> /* .
> >>> Perhaps this could work. There are some things that come to mind:
> >>> - Session lifecycle listeners won't work anymore (no session invalidation
> >>> event when session timeout is reached)
> >>> - You need to take care of invalidated session in the database and clean up
> >>> regularly
> >>> - You need to take care of session security issues by yourself that would
> >>> otherwise be covered by the container
> >>> Cheers,
> >>> Martin
>
> --
> Martin Grotzkehttp://twitter.com/martin_grotzke
>
> < 1KViewDownload
Martin Grotzke
> Thanks for the clarification. But some more doubts .
> A) You mention that the backup takes place in a separate thread (if we
> set sessionBackupAsync="true") .Does the thread get spawned by each
> request or is it done on a timer basis (like in Persistent Manager).
> and how does the "backupThreadCount" come into the picture here.
Threads are managed by a ExecutorService/ThreadPoolExecutor created via
Executors.newFixedThreadPool( backupThreadCount ).
From the request thread the Callable that actually handles interaction
with spymemcached (BackupSessionTask) is submitted to the executor:
https://github.com/magro/memcached-session-manager/blob/master/core/src/main/java/de/javakaffee/web/msm/BackupSessionService.java#L208
There's an additional asynchronous decoupling, in spymemcached during
memcached.set(session): spymemcached uses a separate IO-thread for each
memcached server:
http://code.google.com/p/spymemcached/wiki/Optimizations
The additional concurrency in BackupSessionService/BackupSessionTask is
there to decouple serialization from the request thread, as this may
take a considerable amount of time.
For sessionBackupAsync="false" both things are tied to the request
thread: the BackupSessionTask is executed in the request thread and it
waits for the result of memcached.set(session) (via future.get(
sessionBackupTimeout )).
>
> B) If we specify say 3 memcached nodes (non-sticky environment)
> 1) When a session needs to be stored on what basis do you select
> the primary node?
Random, see NodeIdService.getMemcachedNodeId():
https://github.com/magro/memcached-session-manager/blob/master/core/src/main/java/de/javakaffee/web/msm/NodeIdService.java#L178
> 2) Can we assume at any point of time all memcache nodes will all
> sessions?
[Assuming you meant "will hold all session"]
No. In non-sticky mode a session is stored in the primary node following
sessionBackupAsync concurrency.
Additionally, the session is stored another, secondary memcached node
(the "next" from the memcachedNodes configuration after the primary
node). This is always done asynchronously.
So at any point of time a session will be available in two memcached
nodes (while after initial session creation there may be a little delay
until the secondary node received the session).
You also have to take into account that memcached is basically a cache,
so it should have enough memory assigned to be able to hold all
sessions. You should check memcached stats/monitoring therefore. It
might also be a good idea to run memcached with "-M": then it returns an
error on memory exhausted (rather than removing items).
Alternatively there are also tools like memcachedb or membase.
Regarding optimization I'd suggest the following:
- Use more memcached nodes to better spread I/O (e.g. one memcached node
per tomcat)
- Optimize session usage:
- make the session small (session size can be checked via jmx, see
http://code.google.com/p/memcached-session-manager/wiki/JMXStatistics)
- touch/modify it only if necessary (so that no update of the whole
session in memcached is required)
- Use kryo serialization as it's generally faster than java ser
- Run performance/load tests to see how it all works, while checking jmx
stats, monitor system/tomcat/memcached stats
I'd also check if non-sticky sessions are a must, or if sticky session
were also possible - as they relax the need for immediate session
replication and allow shorter request/response times.
Cheers,
Martin
Reinwald
On Apr 21, 1:44 am, Martin Grotzke <martin.grot...@googlemail.com>
wrote:
>
> > Thanks for the clarification. But some more doubts .
> > A) You mention that the backup takes place in a separate thread (if we
> > set sessionBackupAsync="true") .Does the thread get spawned by each
> > request or is it done on a timer basis (like in Persistent Manager).
> > and how does the "backupThreadCount" come into the picture here.
>
> Threads are managed by a ExecutorService/ThreadPoolExecutor created via
> Executors.newFixedThreadPool( backupThreadCount ).
> From the request thread the Callable that actually handles interaction
> There's an additional asynchronous decoupling, in spymemcached during
> memcached.set(session): spymemcached uses a separate IO-thread for each
> memcached server:http://code.google.com/p/spymemcached/wiki/Optimizations
>
> The additional concurrency in BackupSessionService/BackupSessionTask is
> there to decouple serialization from the request thread, as this may
> take a considerable amount of time.
>
> For sessionBackupAsync="false" both things are tied to the request
> thread: the BackupSessionTask is executed in the request thread and it
> waits for the result of memcached.set(session) (via future.get(
> sessionBackupTimeout )).
>
>
>
> > B) If we specify say 3 memcached nodes (non-sticky environment)
> > 1) When a session needs to be stored on what basis do you select
> > the primary node?
>
> > 2) Can we assume at any point of time all memcache nodes will all
> > sessions?
>
> [Assuming you meant "will hold all session"]
> No. In non-sticky mode a session is stored in the primary node following
> sessionBackupAsync concurrency.
> Additionally, the session is stored another, secondary memcached node
> (the "next" from the memcachedNodes configuration after the primary
> node). This is always done asynchronously.
>
> So at any point of time a session will be available in two memcached
> nodes (while after initial session creation there may be a little delay
> until the secondary node received the session).
> You also have to take into account that memcached is basically a cache,
> so it should have enough memory assigned to be able to hold all
> sessions. You should check memcached stats/monitoring therefore. It
> might also be a good idea to run memcached with "-M": then it returns an
> error on memory exhausted (rather than removing items).
> Alternatively there are also tools like memcachedb or membase.
and n2. Both instances were up .I made a request and to verify that
the session was stored in the primary node and also backed up ,I
telnet to the nodes and tried to get by key (in this case xxx-n1). n1
had the information but n2 did not. Now when I stopped n1 and made a
another request my session id changed to xxx-n2 and then on checking
with the n2 server (telnet way) I see the session there.Even though
the switch to n2 happened successfully when n1 was down, I am confused
why I did not see the session stored in both nodes (telnet way) when
both the nodes were up.
> Regarding optimization I'd suggest the following:
> - Use more memcached nodes to better spread I/O (e.g. one memcached node
> per tomcat)
> - Optimize session usage:
> session in memcached is required)
> - Use kryo serialization as it's generally faster than java ser
> - Run performance/load tests to see how it all works, while checking jmx
> stats, monitor system/tomcat/memcached stats
>
> I'd also check if non-sticky sessions are a must, or if sticky session
> were also possible - as they relax the need for immediate session
> replication and allow shorter request/response times.
>
> Cheers,
> Martin
>
> < 1KViewDownload
Martin Grotzke
> I tried to test this functionality out. I have two memcached nodes n1
> and n2. Both instances were up .I made a request and to verify that
> the session was stored in the primary node and also backed up ,I
> telnet to the nodes and tried to get by key (in this case xxx-n1). n1
> had the information but n2 did not. Now when I stopped n1 and made a
> another request my session id changed to xxx-n2 and then on checking
> with the n2 server (telnet way) I see the session there.Even though
> the switch to n2 happened successfully when n1 was down, I am confused
> why I did not see the session stored in both nodes (telnet way) when
> both the nodes were up.
Probably you didn't see the session in the secondary node (backup) as
you looked up by the sessionId, while the backup sessionId is prefixed
with "bak:". This is necessary so that the NodeLocator knows that it has
to select another node for it, not the one specified by the "-n1"
suffix. The session validity info is prefixed with "validity:" btw.
(getting the same node as the session), so that the backup for validity
info becomes "bak:validity:<sessionId>-<nodeId>".
To see what's going into memcached (or what's requested, not available
etc.) you can also run memcached with -vv.
For debugging msm it's also a good idea to enable debug logging:
http://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration#Logging
Cheers,
Martin
Reinwald
SEVERE: An exception or error occurred in the container during the
request processing
java.lang.IllegalArgumentException: The given node id is not part of
this list
--
at
de.javakaffee.web.msm.SuffixBasedNodeLocator.getPrimary(SuffixBasedNodeLocator.java:
93)
at
net.spy.memcached.MemcachedConnection.addOperation(MemcachedConnection.java:
558)
at
net.spy.memcached.MemcachedClient.addOp(MemcachedClient.java:277)
at
net.spy.memcached.MemcachedClient.asyncGet(MemcachedClient.java:797)
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:
917)
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:
939)
at
de.javakaffee.web.msm.LockingStrategy.loadSessionValidityInfoForValidityKey(LockingStrategy.java:
325)
at
de.javakaffee.web.msm.LockingStrategy.onBackupWithoutLoadedSession(LockingStrategy.java:
220)
at
de.javakaffee.web.msm.MemcachedBackupSessionManager.backupSession(MemcachedBackupSessionManager.java:
867)
at
de.javakaffee.web.msm.SessionTrackerValve.backupSession(SessionTrackerValve.java:
238)
at
de.javakaffee.web.msm.SessionTrackerValve.invoke(SessionTrackerValve.java:
149)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:
470)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:
127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:
102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:
554)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:
109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
298)
at
org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:
864)
at org.apache.coyote.http11.Http11AprProtocol
$Http11ConnectionHandler.process(Http11AprProtocol.java:579)
at org.apache.tomcat.util.net.AprEndpoint
$SocketWithOptionsProcessor.run(AprEndpoint.java:2142)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Apr 25, 2011 12:43:44 PM org.apache.catalina.connector.CoyoteAdapter
service
On Apr 22, 4:13 pm, Martin Grotzke <martin.grot...@googlemail.com>
wrote:
>
> Cheers,
> Martin
>
> signature.asc
> < 1KViewDownload
Martin Grotzke
Which version of msm are you using? What's your manager configuration?
What do you do to reproduce this? Is it possible to reproduce it with a
simple sample?
Can you try the attached 1.4.2-SNAPSHOT, with sessionBackupAsync="false"?
Cheers,
Martin
>> For debugging msm it's also a good idea to enable debug logging:http://code.google.com/p/memcached-session-manager/wiki/SetupAndConfi...
>>
>> Cheers,
>> Martin
>>
>> signature.asc
>> < 1KViewDownload
--
Martin Grotzke
http://twitter.com/martin_grotzke
Martin Grotzke
one obvious reason for what you're experiencing is when there are
tomcats running with different memcachedNodes configurations. Then a
tomcat would get a sessionId containing a memcached nodeId he has not in
his list.
Cheers,
Martin
Reinwald
We were actually running Tomcat and Resin together behind a load
balancer. Hence (for some other reason load balancer not being sticky)
a request which was supposed to go to Resin landed up on Tomcat and
hence we faced the issue.
On Apr 26, 4:35 am, Martin Grotzke <martin.grot...@googlemail.com>
wrote:
> < 1KViewDownload