Memcache API Timing out

[Gayle Laakmann]

Despite what the post at the link below says, memcache is actually
timing out and throwing a DeadlineExceededError. Furthermore, it
hardly seems accurate to say that our apps should continue serving
normally. The quota limits are so ridiculously low that we CAN'T
serve pages without caching.

http://groups.google.com/group/google-appengine-downtime-notify/browse_thread/thread/77cfc8980acd6873/28a397a9fc65420f#28a397a9fc65420f
We will be taking memcache offline tomorrow morning from 9-10am PST
(GMT-8) for routine maintenance. Calls to the memcache API will *not*
throw exceptions but will instead return false for set() calls and
None for get() calls (just like any other cache miss.)

Your app should continue serving normally during this period, and
we'll keep you updated on our progress. "


Additonally, I'd like to make two suggestions:
1. If you're going to take down memcache, wouldn't it make sense to
remove the quota limits? It hardly seems fair to penalized for
exceeding quota when caching is disabled.

2. If you have to take down our apps for an hour, can you pick
something like 2am - 3am? I know you don't want to go to work at 2am,
but it's not really ok to take down our apps for an hour during prime-
time.

[Pete Koomen]

Hi Gayle,

Would you mind sending either your app_id or a stack trace to
pko...@google.com? We haven't seen this globally and would like to
look into it.

Thanks!
Pete, App Engine Team

On Nov 13, 9:29 am, Gayle Laakmann <gay...@gmail.com> wrote:
> Despite what the post at the link below says, memcache is actually
> timing out and throwing a DeadlineExceededError. Furthermore, it
> hardly seems accurate to say that our apps should continue serving
> normally. The quota limits are so ridiculously low that we CAN'T
> serve pages without caching.
>

> http://groups.google.com/group/google-appengine-downtime-notify/brows...

[Pratham]

They're timing out for me as well

App-ID - lifesizedpicx

<class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
File "/base/data/home/apps/lifesizedpicx/4.26/home.py", line 1316,
in main
run_wsgi_app (application)
File "/base/python_lib/versions/1/google/appengine/ext/webapp/
util.py", line 76, in run_wsgi_app
result = application(env, _start_response)
File "/base/python_lib/versions/1/google/appengine/ext/webapp/
__init__.py", line 499, in __call__
handler.get(*groups)
File "/base/data/home/apps/lifesizedpicx/4.26/home.py", line 1078,
in get
memcache.set ('pic'+id, pix, CACHE_EXPIRE)
File "/base/python_lib/versions/1/google/appengine/api/memcache/
__init__.py", line 539, in set
return self._set_with_policy(MemcacheSetRequest.SET, key, value,
time=time)
File "/base/python_lib/versions/1/google/appengine/api/memcache/
__init__.py", line 609, in _set_with_policy
self._make_sync_call('memcache', 'Set', request, response)
File "/base/python_lib/versions/1/google/appengine/api/
apiproxy_stub_map.py", line 46, in MakeSyncCall
stub.MakeSyncCall(service, call, request, response)
File "/base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py", line 245, in MakeSyncCall
rpc.Wait()
File "/base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py", line 161, in Wait
rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self)

On Nov 13, 10:37 pm, Pete Koomen <pkoo...@google.com> wrote:
> Hi Gayle,
>
> Would you mind sending either your app_id or a stack trace to

> pkoo...@google.com?  We haven't seen this globally and would like to

[Denis]

Pete,

Please also check this post:
http://groups.google.com/group/google-appengine/browse_thread/thread/d8a3a54df62e1b04

I guess, you've messed something up with versions.

Denis

[gee]

I second Gayle's points entirely.

We're getting hit with a huge number of deadline exceeded errors at
the moment, all due more or less to cache misses.

Site is basically unusable for our users.

Gee

On Nov 13, 9:29 am, Gayle Laakmann <gay...@gmail.com> wrote:

> Despite what the post at the link below says, memcache is actually
> timing out and throwing a DeadlineExceededError.  Furthermore, it
> hardly seems accurate to say that our apps should continue serving
> normally.  The quota limits are so ridiculously low that we CAN'T
> serve pages without caching.
>

> http://groups.google.com/group/google-appengine-downtime-notify/brows...

[Brett]

Hi Gayle,

I'm sorry to hear you're seeing issues. It's important to note that
there are two types of DeadlineExceededErrors. One is defined in
apiproxy_errors.py. The other is in runtime/__init__.py:

http://code.google.com/p/googleappengine/source/browse/trunk/google/appengine/runtime/__init__.py

The names are the same, which makes this confusing. The latter error
is raised when the request has been processing for too long (~10
seconds). Perhaps what's happening here is that the cache misses are
causing a lot of extra work to be done, causing your application to
hit this other deadline.

The traceback sent by Pratham confirms this behavior. More detail on
how to deal with this is here, in the section entitled "The Request
Timer":

http://code.google.com/appengine/docs/python/requestsandcgi.html



-Brett

On Nov 13, 9:29 am, Gayle Laakmann <gay...@gmail.com> wrote:

> Despite what the post at the link below says, memcache is actually
> timing out and throwing a DeadlineExceededError.  Furthermore, it
> hardly seems accurate to say that our apps should continue serving
> normally.  The quota limits are so ridiculously low that we CAN'T
> serve pages without caching.
>

> http://groups.google.com/group/google-appengine-downtime-notify/brows...

[Gayle Laakmann]

It was timing out within memcache.get and memcache.set -
consistently. If I understand you correctly - you're saying that
memcache isn't technically timing out, on the grounds that it would
complete if given enough time. But, the cache misses take SO much
time that it consistently timeouts during the operations of
memcache.get and memcache.set.

Ok, sure, that may be the case. I'm afraid I don't really see the
difference between that and timing out. Either way, memcache.get and
memcache.set aren't exactly just "return false and letting your
application proceed." The fact is they are causing timeouts.

obj = memcache.get(cache_key)
File "/base/data/home/apps/careercup/13.329261908074364999/memcache/
__init__.py", line 362, in get
self._make_sync_call('memcache', 'Get', request, response)

File "/base/python_lib/versions/1/google/appengine/api/
apiproxy_stub_map.py", line 46, in MakeSyncCall
stub.MakeSyncCall(service, call, request, response)
File "/base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py", line 245, in MakeSyncCall

rpc.Wait()
File "/base/python_lib/versions/1/google/appengine/runtime/
apiproxy.py", line 161, in Wait
rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self)

On Nov 13, 9:45 am, Brett <brett-a...@google.com> wrote:
> Hi Gayle,
>
> I'm sorry to hear you're seeing issues. It's important to note that
> there are two types of DeadlineExceededErrors. One is defined in
> apiproxy_errors.py. The other is in runtime/__init__.py:
>

> http://code.google.com/p/googleappengine/source/browse/trunk/google/a...

[Brett (Google)]

Hi Gayle,

Yes, the *request* times out during the API call. It's not the API
call itself that's timing out. But there is no effective difference to
your application if you're caught in this situation (as you pointed
out). I'd assume the first few calls you make to memcache during a
request actually return False and None. But later on in your request
handler, so much time has already passed that your API call gets
interrupted by the DeadlineExceededException for the entire request.

I think for now your best bet is to catch the runtime
DeadlineExceededException and immediately return with some kind of
error code. We're looking into the cause of the delayed memcache calls
on failure. They should be immediately failing instead of waiting up
to a second to return. That's the root cause of the issue here.

I'm sorry for the trouble this is causing for your application. We're
working right now to address this issue. Someone from the team will
follow up when everything has been resolved.

-Brett

On Nov 13, 9:57 am, Gayle Laakmann <gay...@gmail.com> wrote:
> It was timing out within memcache.get and memcache.set -
> consistently.  If I understand you correctly - you're saying that
> memcache isn't technically timing out, on the grounds that it would
> complete if given enough time.  But, the cache misses take SO much
> time that it consistently timeouts during the operations of
> memcache.get and memcache.set.
>
> Ok, sure, that may be the case.  I'm afraid I don't really see the
> difference between that and timing out.  Either way, memcache.get and
> memcache.set aren't exactly just "return false and letting your
> application proceed."  The fact is they are causing timeouts.
>
> obj = memcache.get(cache_key)
>   File "/base/data/home/apps/careercup/13.329261908074364999/memcache/
> __init__.py", line 362, in get
>     self._make_sync_call('memcache', 'Get', request, response)
>
>   File "/base/python_lib/versions/1/google/appengine/api/
> apiproxy_stub_map.py", line 46, in MakeSyncCall
>     stub.MakeSyncCall(service, call, request, response)
>   File "/base/python_lib/versions/1/google/appengine/runtime/
> apiproxy.py", line 245, in MakeSyncCall
>
>     rpc.Wait()
>   File "/base/python_lib/versions/1/google/appengine/runtime/
> apiproxy.py", line 161, in Wait
>     rpc_completed = _apphosting_runtime___python__apiproxy.Wait(self)
>

[Brett (Google)]

Memcache calls should now be failing fast. That should significantly
reduce the number of request deadlines people are seeing. Otherwise,
we're working to bring the memcache API back online for all
applications as soon as possible. Thanks again for your patience,
everyone!

On Nov 13, 10:12 am, "Brett (Google)" <brett-appeng...@google.com>
wrote:

[gee]

Hi Brett,

During this time, it would have been very extremely helpful if CPU/
response time quotas were relaxed.

All our cache misses are adding up, and we're now over quota.

We've had significant downtime over the last two days due to the
scheduled downtimes going awry (our app was one of those taken down
unexpectedly Tuesday night), and it's definitely hurting our users'
experience with our growing service. Any help would be much
appreciated.

Thanks
Gee






On Nov 13, 10:30 am, "Brett (Google)" <brett-appeng...@google.com>

[Marzia Niccolai]

Hi Gee,

Can you please reply to me with your app ID and I can look in to it.

-Marzia

[Brett (Google)]

Hi Gee,

Definitely understood that there's been quite a bit of trouble the
last two days, especially for you and a few other apps. We're sorry
you guys have been hurting from this. I can imagine it's especially
frustrating for a growing service.

In the specific case of this memcache maintenance, the fail-fast issue
lead to high CPU and response time problems for applications that rely
heavily on memcache. We're working to resolve this fail-fast issue so
it will not happen again. We also came up with an interim solution
that will prevent this problem from happening during future memcache
maintenances.

Otherwise, I understand the pain you're feeling here, and relaxing
response times and CPU times would be one way to alleviate that. Our
team is working hard to improve this in general. Feel free to email
Pete, Paul, or me if you have any more questions.

The good news is that memcache is now back up and serving. Please let
us know if you encounter any more issues!

-Brett

[Gayle Laakmann]

I'm now hitting TONS of "over quota" errors, and I wasn't before.

This is really hurting my revenue today.

http://www.careercup.com

On Nov 13, 11:09 am, "Brett (Google)" <brett-appeng...@google.com>

[Jeff S]

Hi Gayle,

Could you email your app id (x.appspot.com) ? I'd be happy to look
into this for you.

Cheers,

Jeff