URGENT: is GAE Standard memcache down????

191 views
Skip to first unread message

PK

unread,
Nov 6, 2017, 3:56:52 PM11/6/17
to Google App Engine, Google Cloud Insiders
This started returning False 15 minutes ago. My app is down because of it.

from google.appengine.api.capabilities import CapabilitySet
print CapabilitySet('memcache', methods=['set']).is_enabled()



bFlood

unread,
Nov 6, 2017, 4:00:44 PM11/6/17
to Google App Engine
agreed, memcache is down

Adam Sah

unread,
Nov 6, 2017, 4:03:24 PM11/6/17
to Google App Engine
confirmed - that code is returning False for me too.

PK

unread,
Nov 6, 2017, 4:11:12 PM11/6/17
to Wade Piotrowski, google-a...@googlegroups.com, Google Cloud Insiders
Yes it is here: https://issuetracker.google.com/issues/68946028

I also see now a message in the Google Cloud Status Console about this.

On Nov 6, 2017, at 1:07 PM, Wade Piotrowski <wa...@google.com> wrote:

Yes, Posting shortly. Please file a support case for the most up-to-date information if you haven't already.


--
You received this message because you are subscribed to the Google Groups "Google Cloud Insiders" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-ins...@googlegroups.com.
To post to this group, send email to google-clo...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-cloud-insiders.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-insiders/C170C4A6-5355-428B-B010-4507B5148F3C%40gae123.com.
For more options, visit https://groups.google.com/d/optout.

Moe Adham

unread,
Nov 6, 2017, 4:27:34 PM11/6/17
to Google App Engine
Something pretty catastrophic is going on here, all our services on app-engine got a stop request and we can't bring them back up.


On Monday, November 6, 2017 at 4:11:12 PM UTC-5, PK wrote:
Yes it is here: https://issuetracker.google.com/issues/68946028

I also see now a message in the Google Cloud Status Console about this.
On Nov 6, 2017, at 1:07 PM, Wade Piotrowski <wa...@google.com> wrote:

Yes, Posting shortly. Please file a support case for the most up-to-date information if you haven't already.


On Mon, Nov 6, 2017 at 12:56 PM PK <p...@gae123.com> wrote:
This started returning False 15 minutes ago. My app is down because of it.

from google.appengine.api.capabilities import CapabilitySet
print CapabilitySet('memcache', methods=['set']).is_enabled()



--
You received this message because you are subscribed to the Google Groups "Google Cloud Insiders" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-insiders+unsub...@googlegroups.com.

PK

unread,
Nov 6, 2017, 4:27:45 PM11/6/17
to Wade Piotrowski, google-a...@googlegroups.com, Google Cloud Insiders
Showing that Google App Engine is green when the following code fails in both assertions is very misleading. If any of datastore, memcache and task queues are down you need to take app engine status to yellow if not red.

from google.appengine.api.capabilities import CapabilitySet
assert CapabilitySet('memcache', methods=['set']).is_enabled()

from google.appengine.api import memcache
memcache.set('foo123', 'bar', time=60)
assert(memcache.get('foo123') == 'bar’)

PK

unread,
Nov 6, 2017, 4:30:59 PM11/6/17
to Wade Piotrowski, google-a...@googlegroups.com, Google Cloud Insiders
I acknowledge, I see it red now too. Earlier was green with just this at the bottom. Thanks


On Nov 6, 2017, at 1:28 PM, Wade Piotrowski <wa...@google.com> wrote:

Hello,

I see status.cloud.google.com as red for app engine. Please refresh if you do not.

Best,
Wade
Google Cloud Platform Support

Yannick (Cloud Platform Support)

unread,
Nov 6, 2017, 4:32:39 PM11/6/17
to Google App Engine
There effectively is an ongoing Memcache outage, you can follow the incident details for periodic updates on this issue as we work on resolving it.


On Monday, November 6, 2017 at 4:11:12 PM UTC-5, PK wrote:
Yes it is here: https://issuetracker.google.com/issues/68946028

I also see now a message in the Google Cloud Status Console about this.
On Nov 6, 2017, at 1:07 PM, Wade Piotrowski <wa...@google.com> wrote:

Yes, Posting shortly. Please file a support case for the most up-to-date information if you haven't already.


On Mon, Nov 6, 2017 at 12:56 PM PK <p...@gae123.com> wrote:
This started returning False 15 minutes ago. My app is down because of it.

from google.appengine.api.capabilities import CapabilitySet
print CapabilitySet('memcache', methods=['set']).is_enabled()



--
You received this message because you are subscribed to the Google Groups "Google Cloud Insiders" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-insiders+unsub...@googlegroups.com.
To post to this group, send email to google-cloud-insiders@googlegroups.com.

Ani Hatzis

unread,
Nov 6, 2017, 4:41:04 PM11/6/17
to google-a...@googlegroups.com

Hm, no issues here in my apps, eu-west. But only use memcache explicitly with Datastore for an admin task, but it works either case. Maybe it is the Capability API, I don't use it here.


On Mon, Nov 6, 2017, 22:32 'Yannick (Cloud Platform Support)' via Google App Engine, <google-a...@googlegroups.com> wrote:
There effectively is an ongoing Memcache outage, you can follow the incident details for periodic updates on this issue as we work on resolving it.


On Monday, November 6, 2017 at 4:11:12 PM UTC-5, PK wrote:
Yes it is here: https://issuetracker.google.com/issues/68946028

I also see now a message in the Google Cloud Status Console about this.
On Nov 6, 2017, at 1:07 PM, Wade Piotrowski <wa...@google.com> wrote:

Yes, Posting shortly. Please file a support case for the most up-to-date information if you haven't already.


On Mon, Nov 6, 2017 at 12:56 PM PK <p...@gae123.com> wrote:
This started returning False 15 minutes ago. My app is down because of it.

from google.appengine.api.capabilities import CapabilitySet
print CapabilitySet('memcache', methods=['set']).is_enabled()



--
You received this message because you are subscribed to the Google Groups "Google Cloud Insiders" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-ins...@googlegroups.com.
To post to this group, send email to google-clo...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/ab1bc058-f3d4-4c3f-b4ea-e9951177b2e1%40googlegroups.com.

Justin Yip

unread,
Nov 6, 2017, 5:32:00 PM11/6/17
to Google App Engine
We cannot bring up any "vm: true" for the past couple of hours too.

(Maybe the memcache service is also on "vm: true", which all gets turned down :)

Alan deLespinasse

unread,
Nov 6, 2017, 6:12:00 PM11/6/17
to Google App Engine
I'm also unable to deploy a "vm: true" service. My app doesn't even use memcache, but maybe "vm: true" depends on memcache somehow?

The error message on my "gcloud app deploy" command said "Timed out when starting VMs." I can' t find anything in the logs that means anything to me.

Alan deLespinasse

unread,
Nov 6, 2017, 6:13:56 PM11/6/17
to Google App Engine
I maybe should mention that the old version of my service was working fine, as far as I can tell. I just can't deploy a new version.

Yannick (Cloud Platform Support)

unread,
Nov 7, 2017, 11:18:10 AM11/7/17
to Google App Engine
This incident was resolved yesterday. See the incident page for a timeline.

Alan deLespinasse

unread,
Nov 7, 2017, 11:24:36 AM11/7/17
to Google App Engine
I also noticed that my failed instances were still running; see my other thread for details. This could have cost my company money if I hadn't noticed it.
Reply all
Reply to author
Forward
0 new messages