Non Sticky sessions + MSM + Elasticache / Memcached + Non Sticky Elastic Load Balancer + AWS Beanstalk + AWS Linux AMI + Tomcat 7 --- Setup issue

[S Shah]

Hi Martin,

Thanks for the hard work you have put in. 

We are trying to setup the non-sticky sessions with autoscaled AWS instances with elasticache/memcached, but are running into walls.  There seems to be something wrong with the config.  

Non Sticky sessions + MSM + Elasticache / Memcached + Non Sticky Elastic Load Balancer + AWS Beanstalk + AWS Linux AMI + Tomcat 7 --- Setup issue

In our case, the users login to our app and for the most part their activity means low cpu-utilization.  But there are activities that a logged in user can start that will require higher cpu-resources.  So you can imagine that on our single instance, we could admit 50 users with low cpu-resource-needs initially.. but later on they all might need more cpu-resources and all of their sessions could 'swell' inside that single instance leading to 100% cpu utilization (and the problems related to that).  We do autoscale the instances based on cpu utilization, but that only affects new sessions in 'sticky sessions' case.  

That is why, what we are trying to get to is a non-sticky session setup where we are able to add/delete autoscaled EC2 instances without loosing the sessions of logged in users.  From your other posts, its clear how the non-sticky mode does not save the sessions on the tomcat instance at all, and saves the sessions externally in the memcached nodes.  That way, any logged in user could use any instance to continue with the same session.  

1. Below are the jar files we are using: 
   
• memcached-session-manager-1.9.2.jar 
• memcached-session-manager-tc7-1.9.2.jar 
• spymemcached-2.10.2.jar 
2. AWS ELB session stickyness is NOT Enabled (unchecked).
   
3. We are using the default Java serialization so left the serialization strategy in the config as the default 'text'.  (Could this be the issue?)
   
4. We are thinking the lockingmode=auto might work, but not sure if we need to do anything additional so that MSM will be able to detect the read-only requests as opposed to modification requests.  
5. Here is the config we are using inside our context.xml:
   

      <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"

             memcachedNodes="memcached.l59tff.0001.usw1.cache.amazonaws.com:11211"

             sticky="false"

             lockingMode="auto"

             sessionBackupAsync="false"

             copyCollectionsForSerialization="false"

             requestUriIgnorePattern=".*\.(gif|jpg|jpeg|png|wmv|avi|mpg|mpeg|mp4|htm|html|js|css|mp3|swf|ico|flv)$"

      />

There is a bit of confusion about which jar to use for the memcached in our setup, spymemcached or the 'AmazonElastiCacheClusterClient-1.0.1'.  We tried both and neither worked, but not sure which one we should be using.

( Source : https://github.com/awslabs/aws-elasticache-cluster-client-memcached-for-java    ;  Amazon ElastiCache Cluster Client is an enhanced Java library to connect to ElastiCache clusters. This client library has been built upon Spymemcached and is released under the Amazon Software License. )

( AWS Description for 'AmazonElastiCacheClusterClient-1.0.1: ElastiCache Cluster Client is a Memcached client that supports Auto Discovery which allows it to automatically discover Cache Nodes. This means that scaling the number of nodes in a cluster no longer requires updating the static list of endpoints in the client configuration.  )

We are still not able to get this to work.  We are not able to get past the login page in our app.    

Could you please advise if there is something wrong with our config or if we are not using the correct combination of jars, or maybe wrong values for the parameters here?  Thanks in advance.  Cheers!

[Martin Grotzke]

Hi,

At first, I'm sorry to say that I haven't used this setup and therefore cannot share own experience.

I'd say that you have to use the AmazonElastiCacheClusterClient jar and remove the spymemcached jar. Your configuration looks good so far.

Have you seen https://github.com/magro/memcached-session-manager/wiki/ConfigureElastiCache? Perhaps this is helpful.

If you still don't get it to work I'd recommend to start with a very simple sample app to see if the setup is working at all and if issues are related to your specific app. You could activate debug logging and see if this provides insights.

Cheers,
Martin

--

---
You received this message because you are subscribed to the Google Groups "memcached-session-manager" group.
To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[S Shah]

Thanks for that prompt reply Martin.  

On Tuesday, May 31, 2016 at 1:01:44 PM UTC-7, Martin Grotzke wrote:

Hi,

At first, I'm sorry to say that I haven't used this setup and therefore cannot share own experience.

I was wondering what you meant by saying you have not used this setup - did you mean you have not tested that with AmazonElastiCacheClusterClient or with AWS? 

 

I'd say that you have to use the AmazonElastiCacheClusterClient jar and remove the spymemcached jar. Your configuration looks good so far.

 Do you have an example of non-sticky session setup for a java app that we can refer?  It seems the jar file versions have evolved since some of these articles were written.  

Have you seen https://github.com/magro/memcached-session-manager/wiki/ConfigureElastiCache? Perhaps this is helpful.

Yes we did see that article.  It says AmazonElastiCacheClusterClient needs to be used to to properly support Amazon ElastiCache Auto Discovery, but we are not sure if that Auto discovery feature is required with MSM.  We are thinking we will have only one memcached node for now and so instead of giving the elasticache config endpoint, we can just give directly the node endpoint.  

If you still don't get it to work I'd recommend to start with a very simple sample app to see if the setup is working at all and if issues are related to your specific app. You could activate debug logging and see if this provides insights.

We will try debugging in the mean time.   

Cheers,
Martin

Also, in our earlier post, we tried to explain how we understood the application of MSM for non-sticky... 

( In our case, the users login to our app and for the most part their activity means low cpu-utilization.  But there are activities that a logged in user can start that will require higher cpu-resources.  So you can imagine that on our single instance, we could admit 50 users with low cpu-resource-needs initially.. but later on they all might need more cpu-resources and all of their sessions could 'swell' inside that single instance leading to 100% cpu utilization (and the problems related to that).  We do autoscale the instances based on cpu utilization, but that only affects new sessions in 'sticky sessions' case.  

That is why, what we are trying to get to is a non-sticky session setup where we are able to add/delete autoscaled EC2 instances without loosing the sessions of logged in users.  From your other posts, its clear how the non-sticky mode does not save the sessions on the tomcat instance at all, and saves the sessions externally in the memcached nodes.  That way, any logged in user could use any instance to continue with the same session.   )

... does that seem to be correct way to be using MSM?   

Also, regarding the lockingmode...

We are thinking the lockingmode=auto might work, but not sure if we need to do anything additional so that MSM will be able to detect the read-only requests as opposed to modification requests.  

Thanks a lot again.  

 

To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-manager+unsub...@googlegroups.com.

[Martin Grotzke]

S Shah <sehul...@gmail.com> schrieb am Di., 31. Mai 2016, 22:27:

Thanks for that prompt reply Martin.  

On Tuesday, May 31, 2016 at 1:01:44 PM UTC-7, Martin Grotzke wrote:

Hi,

At first, I'm sorry to say that I haven't used this setup and therefore cannot share own experience.

I was wondering what you meant by saying you have not used this setup - did you mean you have not tested that with AmazonElastiCacheClusterClient or with AWS? 

Both. I've not used msm on AWS with elasticache and therefore also not the AmazonElastiCacheClusterClient. 

 

I'd say that you have to use the AmazonElastiCacheClusterClient jar and remove the spymemcached jar. Your configuration looks good so far.

 Do you have an example of non-sticky session setup for a java app that we can refer?  It seems the jar file versions have evolved since some of these articles were written.  

You could have a look at the simpleservlet sample from https://github.com/magro/memcached-session-manager/tree/master/samples to get a simple sample app for testing.

You're saying that jar file versions have evolved - does this mean that the AmazonElastiCacheClusterClient is not compatible with the latest msm version? 

Have you seen https://github.com/magro/memcached-session-manager/wiki/ConfigureElastiCache? Perhaps this is helpful.

Yes we did see that article.  It says AmazonElastiCacheClusterClient needs to be used to to properly support Amazon ElastiCache Auto Discovery, but we are not sure if that Auto discovery feature is required with MSM.  We are thinking we will have only one memcached node for now and so instead of giving the elasticache config endpoint, we can just give directly the node endpoint.  

msm does not require auto discovery. It would "only" allow you to reconfigure / scale memcached nodes up or down. If you don't need that or for testing you can also used statically configured nodes. 

If you still don't get it to work I'd recommend to start with a very simple sample app to see if the setup is working at all and if issues are related to your specific app. You could activate debug logging and see if this provides insights.

We will try debugging in the mean time.   

Cheers,
Martin

Also, in our earlier post, we tried to explain how we understood the application of MSM for non-sticky... 

( In our case, the users login to our app and for the most part their activity means low cpu-utilization.  But there are activities that a logged in user can start that will require higher cpu-resources.  So you can imagine that on our single instance, we could admit 50 users with low cpu-resource-needs initially.. but later on they all might need more cpu-resources and all of their sessions could 'swell' inside that single instance leading to 100% cpu utilization (and the problems related to that).  We do autoscale the instances based on cpu utilization, but that only affects new sessions in 'sticky sessions' case.  

That is why, what we are trying to get to is a non-sticky session setup where we are able to add/delete autoscaled EC2 instances without loosing the sessions of logged in users.  From your other posts, its clear how the non-sticky mode does not save the sessions on the tomcat instance at all, and saves the sessions externally in the memcached nodes.  That way, any logged in user could use any instance to continue with the same session.   )

... does that seem to be correct way to be using MSM?   

Yes, makes sense. 

Also, regarding the lockingmode...

We are thinking the lockingmode=auto might work, but not sure if we need to do anything additional so that MSM will be able to detect the read-only requests as opposed to modification requests.  

That depends on your application. If you don't expect parallel/concurrent requests for the same session (e.g. caused by ajax or tabbed browsing) you could disable locking completely. 

Cheers, 

Martin 

Thanks a lot again.  

 

To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "memcached-session-manager" group.

To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-...@googlegroups.com.

[S Shah]

Thanks again Martin.  

Also, have you tried using default Java serialization with MSM?  

We will try some more to make this work and keep you posted.  

Not sure if you are able to ask around in your colleagues if anyone has done this setup.  It seems to be a fairly common need but for some reason, there doesnt seem to be much written about nonsticky sessions with aws.  Please let us know if you find some example java setup for MSM with AWS Elasticache.  We'd greatly appreciate it. 

Cheers!

On Tuesday, May 31, 2016 at 1:53:06 PM UTC-7, Martin Grotzke wrote:

Thanks a lot again.  

 

To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-manager+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "memcached-session-manager" group.

To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-manager+unsub...@googlegroups.com.

[Martin Grotzke]

S Shah <sehul...@gmail.com> schrieb am Di., 31. Mai 2016, 23:06:

Thanks again Martin.  

Also, have you tried using default Java serialization with MSM?  

Yes, it's often a safe default. 

We will try some more to make this work and keep you posted.  

Great! :-) 

Not sure if you are able to ask around in your colleagues if anyone has done this setup.  

Unfortunately I don't know anybody personally running this setup. 

It seems to be a fairly common need but for some reason, there doesnt seem to be much written about nonsticky sessions with aws.  Please let us know if you find some example java setup for MSM with AWS Elasticache.  We'd greatly appreciate it. 

Sure, I will. Perhaps you get a modified version of the simpleservlet sample up and running with elasticache, would be great to have that in msm :-) 

Cheers, 

Martin 

Thanks a lot again.  

 

To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "memcached-session-manager" group.

To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "memcached-session-manager" group.

To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-...@googlegroups.com.

[Sehul Shah]

Hi Martin,

Just wanted to update you and the friends in this group.  We were not able to make the default java serialization work, but ended up going with Kryo and it works fine so far.  

Thank you very much for all your support.  We really appreciate it.  

Cheers!

On Tue, May 31, 2016 at 2:20 PM, 'Martin Grotzke' via memcached-session-manager <memcached-se...@googlegroups.com> wrote:

--

---
You received this message because you are subscribed to a topic in the Google Groups "memcached-session-manager" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/memcached-session-manager/tFPjSbm9Uwc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to memcached-session-...@googlegroups.com.

[Martin Grotzke]

Great!

Cheers,
Martin

[S Shah]

Martin,

Seems like our non-sticky sessions are not working properly.  Below is our config:

<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
            memcachedNodes="dzmemcached-cluster.lo9mzf.cfg.usw1.cache.amazonaws.com:11211"
            memcachedProtocol="binary"
            sticky="false"
            lockingMode="all"
            sessionBackupAsync="false"
            sessionBackupTimeout="2000"
            copyCollectionsForSerialization="true"
            requestUriIgnorePattern=".*\.(gif|jpg|jpeg|png|wmv|avi|mpg|mpeg|mp4|htm|html|css|js|mp3|swf|ico|flv)$"
            transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"
     />

Seems like 

• our reads are working fine, but when we do updates/CRUD operation, the session is supposed to refresh with the new content... but instead, the new content is not presented.  
• We tried the all/none/auto for lockingmode, but no difference.

Not sure where we can go from here.  Any help would be appreciated.  

Thanks again.

Great!

Cheers,
Martin

Thanks a lot again.  

 

To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-manager+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "memcached-session-manager" group.

To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-manager+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "memcached-session-manager" group.

To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-manager+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to a topic in the Google Groups "memcached-session-manager" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/memcached-session-manager/tFPjSbm9Uwc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to memcached-session-manager+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "memcached-session-manager" group.

To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-manager+unsub...@googlegroups.com.

[Martin Grotzke]

Hi,

I'd suggest that you activate debug logging and inspect/share the log
output.

Cheers,
Martin

On 06/06/2016 11:04 AM, S Shah wrote:
> Martin,
>
> Seems like our non-sticky sessions are not working properly. Below is
> our config:
>
> <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
>
> memcachedNodes="dzmemcached-cluster.lo9mzf.cfg.usw1.cache.amazonaws.com:11211"
> memcachedProtocol="binary"
> sticky="false"
> lockingMode="all"
> sessionBackupAsync="false"
> sessionBackupTimeout="2000"
> copyCollectionsForSerialization="true"
>
> requestUriIgnorePattern=".*\.(gif|jpg|jpeg|png|wmv|avi|mpg|mpeg|mp4|htm|html|css|js|mp3|swf|ico|flv)$"
>
> transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"
> />
>
> Seems like
>

> * our reads are working fine, but when we do updates/CRUD operation,

> the session is supposed to refresh with the new content... but
> instead, the new content is not presented.

> * We tried the all/none/auto for lockingmode, but no difference.

>
> Not sure where we can go from here. Any help would be appreciated.
>
> Thanks again.
>
>
>
>
> On Saturday, June 4, 2016 at 12:14:20 AM UTC-7, Martin Grotzke wrote:
>
> Great!
>
> Cheers,
> Martin
>
>

> Sehul Shah <sehul...@gmail.com <javascript:>> schrieb am Sa., 4.

> Juni 2016, 08:06:
>
> Hi Martin,
>
> Just wanted to update you and the friends in this group. We
> were not able to make the default java serialization work, but
> ended up going with Kryo and it works fine so far.
>
> Thank you very much for all your support. We really appreciate
> it.
>
> Cheers!
>
>
> On Tue, May 31, 2016 at 2:20 PM, 'Martin Grotzke' via
> memcached-session-manager <memcached-se...@googlegroups.com

> <javascript:>> wrote:
>
>
>
> S Shah schrieb am Di., 31. Mai 2016, 23:06:
>
> Thanks again Martin.
>
> Also, have you tried using default Java serialization
> with MSM?
>
>
> Yes, it's often a safe default.
>
> We will try some more to make this work and keep you
> posted.
>
>
> Great! :-)
>
>
>
> Not sure if you are able to ask around in your
> colleagues if anyone has done this setup.
>
>
> Unfortunately I don't know anybody personally running this
> setup.
>
> It seems to be a fairly common need but for some reason,
> there doesnt seem to be much written about nonsticky
> sessions with aws. Please let us know if you find

> someexample java setup for MSM with AWS Elasticache.

> <https://github.com/magro/memcached-session-manager/wiki/ConfigureElastiCache>?

> a *non-sticky session setup* where we are able

> to is a *non-sticky session setup* where

> we are able to add/delete autoscaled EC2
> instances without loosing the sessions
> of logged in users. From your other
> posts, its clear how the non-sticky mode
> does not save the sessions on the tomcat
> instance at all, and saves the sessions
> externally in the memcached nodes. That
> way, any logged in user could use any
> instance to continue with the same
> session.
>

> 1. Below are the jar files we are using:
> * memcached-session-manager-1.9.2.jar
> * memcached-session-manager-tc7-1.9.2.jar
> * spymemcached-2.10.2.jar
> 2. AWS ELB session stickyness is NOT
> Enabled (unchecked).
> 3. We are using the default Java

> serialization so left the
> serialization strategy in the config
> as the default 'text'. (Could this
> be the issue?)

> 4. We are thinking the lockingmode=auto

> might work, but not sure if we need
> to do anything additional so that
> MSM will be able to detect the
> read-only requests as opposed to
> modification requests.

> 5. Here is the config we are using

> inside our context.xml:
>
> <Manager
> className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
>
> memcachedNodes="memcached.l59tff.0001.usw1.cache.amazonaws.com:11211

> <http://memcached.l59tff.0001.usw1.cache.amazonaws.com:11211>"

> memcached-session-...@googlegroups.com.

> For more options, visit
> https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
>
> ---
> You received this message because you are
> subscribed to the Google Groups
> "memcached-session-manager" group.
> To unsubscribe from this group and stop
> receiving emails from it, send an email to

> memcached-session-...@googlegroups.com.

> For more options, visit
> https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
>
> ---
> You received this message because you are subscribed to
> the Google Groups "memcached-session-manager" group.
> To unsubscribe from this group and stop receiving emails
> from it, send an email to

> memcached-session-...@googlegroups.com
> <javascript:>.

> For more options, visit
> https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
>
> ---
> You received this message because you are subscribed to a
> topic in the Google Groups "memcached-session-manager" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/memcached-session-manager/tFPjSbm9Uwc/unsubscribe

> <https://groups.google.com/d/topic/memcached-session-manager/tFPjSbm9Uwc/unsubscribe>.

> To unsubscribe from this group and all its topics, send an
> email to

> memcached-session-...@googlegroups.com
> <javascript:>.

>
>
> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
>
> ---
> You received this message because you are subscribed to the
> Google Groups "memcached-session-manager" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to

> memcached-session-...@googlegroups.com
> <javascript:>.

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "memcached-session-manager" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to memcached-session-...@googlegroups.com
> <mailto:memcached-session-...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.

--

inoio gmbh - http://inoio.de
Schulterblatt 36, 20357 Hamburg
Amtsgericht Hamburg, HRB 123031
Geschäftsführer: Dennis Brakhane, Martin Grotzke, Ole Langbehn

[S Ko]

Hi S Shah,

I had almost the same stack as you do except mine is managed by OpsWorks instead of Beanstalk, and I'm using Kryo serializer. My goal is to disable sticky session on ELB and manage with Elasticache/Memcached as well.

Session didn't keep when I follow the MSM wiki config, after some T&R I found your config post above and gave it a try, and it solved my problem, I'm able to log in to apps now. The key config I was missing is the "memcachedProtocol="binary", which didn't exist in your first config and seemed to be working in your June 4th post, but you have it in your June 6 config and it didn't work.

Thanks!