intermittent service ticket not found in memcached, but not memcached issue

[Yan Zhou]

Hello,

We are experiencing intermittent ticket error issue with CAS 4.1.7 overlay setup. The same issue exists in our app based on CAS 3.1.5. I am not saying that is JASIG CAS issue, most likely it is something in our configuration. But I cannot find out why. 

We have two servers running CAS on active-active setup with load balancer setup for session affinity.  We have one memcached instance running on each CAS server, thus two memcached instance running along with two CAS servers.

Intermittently we see that /serviceValidate fail when validating service ticket, because MemcachedTicketRegistry failed to fetch the ticket. I do not think that is a memcached issue, because it works fine if I shutdown on CAS server, leaving one single CAS server running to handle all the traffic. This is why I am not posting this on memcached mailing list. 

But I do not understand why looking up ticket when we have two CAS servers running would fail intermittently in MemcachedTicketRegistry. What I did notice is that, when it fails, it is usually Server01 is looking up the ticket stored on memcached instance of Server02, or Server02 is looking up ticket stored on memcached instance on Server 01.

Any suggestions?

Thx!

Yan

This is my memcached configuration, both servers have identical setting as follows.

memcached.servers=server01.dev.medplus.com:11211,server02.dev.medplus.com:11211

memcached.hashAlgorithm=FNV1_64_HASH

memcached.protocol=BINARY

memcached.locatorType=ARRAY_MOD

memcached.failureMode=Redistribute

This is the log I see when it failed.

On server02

 

172.18.4.136 - - [16/May/2016:20:28:47 +0000] "POST /cas/login?service=https%3A%2F%2Fintcas.dev.medplus.com%2Fcas-admin%2Fj_spring_cas_security_check HTTP/1.1" 302 -

172.18.4.136 - - [16/May/2016:20:28:49 +0000] "GET /cas-admin/j_spring_cas_security_check?ticket=ST-1-WLE4H2PcgDuff51TUYnG-dcasde02.dev.medplus.com HTTP/1.1" 302 –

 

This is seen on server01 (request now is directed to server01)

 

172.18.4.136 - - [16/May/2016:20:28:49 +0000] "GET /cas/serviceValidate?ticket=ST-1-WLE4H2PcgDuff51TUYnG-dcasde02.dev.medplus.com&service=https%3A%2F%2Fintcas.dev.medplus.com%2Fcas-admin%2Fj_spring_cas_security_check HTTP/1.1" 200 271

 

Cannot find this ticket, therefore, goes back to /login page.  If ticket was found, it should redirect to the App's landing page. But it does not.

 

172.18.4.136 - - [16/May/2016:20:28:49 +0000] "GET /cas/login?service=https%3A%2F%2Fintcas.dev.medplus.com%2Fcas-admin%2Fj_spring_cas_security_check HTTP/1.1" 302 -

172.18.4.136 - - [16/May/2016:20:28:49 +0000] "GET /cas/serviceValidate?ticket=ST-2-CHAHXB1PAlYxUZ5Ybcu0-dcasde02.dev.medplus.com&service=https%3A%2F%2Fintcas.dev.medplus.com%2Fcas-admin%2Fj_spring_cas_security_check HTTP/1.1" 200 213

[Ray Bon]

Yan,

This looks like a race condition between ST validation and memcached replication. We had this issue with 3.5.2.1 and ehcache. The service would try to validate the ticket before ehcache could replicate it. This happened even with a sticky load balancer.
Perhaps you could turn on memcached logging in the CAS logs to see when replication and validation are occurring (as well as which server is handling the request).
Apache's 1 sec log resolution is not fine enough.

Ray

--
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To post to this group, send email to cas-...@apereo.org.
Visit this group at https://groups.google.com/a/apereo.org/group/cas-user/.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/bffc1502-6907-4381-be4e-a95bf7e52381%40apereo.org.
For more options, visit https://groups.google.com/a/apereo.org/d/optout.

[Zhou, Yan]

Thanks, but, I do not think Memcached does replication, though.   That is an interesting thought, maybe I do need to look into memcached closer.

To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/18192a83-c675-8e35-642b-637cb0545644%40uvic.ca.