Memcache cas never updating

[adriano_b...@qmagico.com.br]

I have an application in which I use the memcache to avoid race conditions by retrieving keys using get(..., for_cas=True) and saving using cas. However, ever since yesterday 18:00 (GMT-02) cas is always returning False for me. Even a code as simple as this:

memcache.add('key', 'A')
memclient = memcache.Client()
dict = memclient.get('key', for_cas=True)
result = memclient.cas('key', 'B')

print result

always returns False on a clean memcache. I have tried to run this code in different applications and result is always False. If i change the code to use traditional get() and set(), result returns True. I havent done a deploy in three days, so the code haven't changed as the problem started to appear. Did something change in the sdk?

[Nicholas (Google Cloud Support)]

Hey adriano_brasileiro,

I was not able to reproduce the results you describe.  This might require additional investigation which is better suited to the App Engine public issue tracker.  I recommend that you file an issue there and posting a link to that issue here so that others may follow.

When doing so, please include the following:

• Timestamps for when you first and last noticed this issue occurring
  
• Does it happen only when reaching a certain load?
  
• Are there any other modules contending with this memcache?

Also, I'd suggest using the memcache.Client().add() rather than the static memcache.add().  Lastly, the code you provide suggests that memcache.Client().get() returns a dict but it in fact returns the value of a key or None.

I look forward to seeing you issue report.

[Carlos]

For what it's worth, I have an application where the "client.cas(key, counter+1)" which has worked for years, and it practically a direct copy of Guido's code on the page below, is suddenly failing in the past few days.  

http://neopythonic.blogspot.com/2011/08/compare-and-set-in-memcache.html

I've debugged the code extensively.  It isn't that complex.  But something certainly changed on App Engine.

I've tried variations of setting the initial value.

I've tried cas_reset().

I've tried flushing the cache just before the request.

"client.cas(key, counter+1)" is simply returning False every time during the while True loop, eventually timing out after 30 seconds.

[Nicholas (Google Cloud Support)]

I appreciate the extra troubleshooting. We've noticed an internal issue with memcache that may be related but require additional information to confirm.

Are you using namespaces?  If so, can you try setting the namespace in each method call explicitly?

from google.appengine.api import memcache
ns = "simple-namespace"
memclient = memcache.Client()
memclient.add("key", "A", namespace=ns)
val = memclient.get("key", for_case=True, namespace=ns)
result = memclient.cas("key", "B", namespace=ns)
print result

The purpose is to test the above instead of using:

from google.appengine.api import namespace_manager
namespace_manager.set_namespace(ns)

I look forward to your feedback while I attempt a few more tests of my own.

[adriano_b...@qmagico.com.br]

I do use namespaces. Indeed, I tried to set it explicitally as you said and it worked!

So will the issue be resolved? Should I set the namespace explicitally on my code just in case?

[Nicholas (Google Cloud Support)]

Thanks for the quick response.  I will also await feedback from Carlos about this workaround as well.

For the time being, I would suggest setting it explicitly for calls to memcache.Client().cas() to workaround this known issue and use memcache to your advantage.  Know that we are aware of this issue and working towards its root cause.  The aforementioned suggestion is the recommended workaround.

[Carlos]

Good morning!  Indeed, I use namespaces too, and explicitly setting the namespace now did make it function again.

I was about to proceed with refactoring to move away from cas() and instead use incr(), since I may have over-engineered my race condition need.  But incr() also has namespace as a possible parameter.  Any advice on whether we should be explicitly setting namespace for both functions?  Or which function is better to rely on right now?

P.S.  Your code above has a small error:  "for_case" should be "for_cas" in case anyone will see this and cut and paste code.

[Nicholas (Google Cloud Support)]

Thanks for confirming that this workaround applies to you.  Given your responses, I think it safe to assume you both are affected by the aforementioned issue relating to memcache.Client().cas().  This should not affect other memcache methods so namespace setting should work as expected there.

I've filed an external issue on the App Engine public issue tracker so please star said issue if this affects you.  Status updates will be posted there.