Massive GAE latency since 10:00 GMT

[Keith Marsh]

Hi

On a project's default module that normally returns sub 100mS latency, I'm seeing latency of > 6,000mS since 10am this morning when cloud monitoring reported problems (and has been on and off since).

I've switched from using shared memcache to private to see if that was the issue.

Tracing isn't reporting RPC as the issue.  Task queues are fine, and other modules are fine.

The project hasn't been updated since Nov 15.

I have other projects that aren't seeing any latency issues, and wondered if I'm in an A/B test of some sort.

Anyone else suffering latency issues ?

Thanks

Keith

[Keith Marsh]

Moving this to the issue tracker.

[blackpawn]

yep crazy slow.  all requests are taking like 10 seconds O_O

[Adi Mor Barak]

Any idea what is happing there?

My HTTP request takes 5-7 seconds I cannot work like that.

[Nick (Cloud Platform Support)]

Thanks for moving this to the issue tracker, since it's the best forum to get such support. I wish you luck in having the issue triaged quickly.

[Chris Ramsdale]

If you can email me application IDs, we can take a look.

-- Chris

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/afe6d220-7745-432d-82b3-1b28f6c225c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeff Deskins]

One of my projects appears to have high latency also.

After some brief investigation, for the requests that are longer than normal, I am seeing "cloud_debugger.DebugletStarted" in the cloud trace.  Other requests to the same path that have normal latency do not have the debugger entry in the trace.

Jeff

[Keith Marsh]

The problem looks to have been resolved at 21:00 GMT.  I spoke with a member of the trace team, and my issue was answered with requests for my source.  

Jeff's observation about cloud debugger is very interesting, though I didn't see that in my traces.  I just saw a 6 second delay before my first RPC.  

I was suffering 500 (error 121) on about 5% of my requests.

Keith

[Jeff Deskins]

After thinking about it more - the "cloud_debugger.DebugletStarted" I was seeing in the responses with higher latency is probably part of the normal startup script for an instance.  The underlying problem - for my case - is why the instances were being restarted so aggressively.  

Normally, I may see an instance run for a few days.  Yesterday, it seemed instances were being stopped/started almost every minute - with minimal traffic.

Jeff

[Keith Marsh]

Ugh, spoke too soon.  Latency and 500s back.

[Jeff Deskins]

Not sure if mine is same issue.  I am still seeing latency on one of my apps - but appears to be coming from instances stopping/restarting.  Sometimes the instance is stopped less than a minute after starting and is only occurring with one of my apps - the others are fine.

This is during an expected low traffic time for my app - when the instance chart would normally show a flat line with one instance handling everything.  Now it is a sawtooth chart - bringing down an instance and starting another about every minute.  

I reverted to previous version of the app just in case something odd got introduced - but no change.

Jeff

[Keith Marsh]

Yes, I'm seeing instances churning heavily too.  I'm supplying info to Google via the issue tracker.  Keep you posted.

Do you have warmup set?

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/ZXx-VhK7CQU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.

To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/6c15640d-507c-4a6b-b5b9-0989b679e68a%40googlegroups.com.

[Jeff Deskins]

Yes - warmup request is setup.

[Mauricio Lumbreras]

Hello

from past two days there was a constant recycling of instances

I filled a ticket to support but the answer is from the book

Now I'm suffering instances that are killed about 2 or 3 minutes after are created in spite there is some traffic

Regards

Mauricio

[Keith Marsh]

My failures are getting more serious with backend modules reporting 503 with BigQuery and Transient Queue Errors.  Considering migrating my app to another region.

--

You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/ZXx-VhK7CQU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/8a347629-6810-4663-9069-3005e6baba3e%40googlegroups.com.

[Jeff Deskins]

Here is snapshot I took earlier of one of my apps struggling to keep one instance running during low traffic.  

This graph normally shows straight line for one instance running smoothly.  It's now choppy with instances constantly stopping and starting.

[Nick (Cloud Platform Support)]

Hey Folks,

Just wanted to reiterate: the best way to get support for a production issue is to file a Public Issue Tracker issue or to open a support case. The information provided here should be grouped into such a form so that it can be usefully included as part of a formal process to get the issue looked at.

[Keith Marsh]

Hi Nick.

Understood.  Get your issues in.

That said, because the tracker is private, this is good platform for 

• us to share symptoms so we can better glimpse the big picture
• understand you're not the only one going through the pain
• people to find in the future when they encounter similar issues down the road
• visibility; a GCP manager reached out to me directly because of this thread (which was greatly appreciated).

So from my experience on this thread, I think 6 of one and half a dozen of the other works well.

Keith

[Nick (Cloud Platform Support)]

Hey Keith,

Excellent points and a good description of the value of posting parallel to the more formal issue tracker process. I hope everyone is getting assistance in the proper place, and this thread can definitely fulfill a useful purpose for users to compare situations, etc.

Best wishes,

Nick 

[Jeff Deskins]

Looks like the instance churning issue has improved over the last several hours.  One of my instances has been live and running smooth for over two hours.  Latency for requests are back to normal.

Thanks,
Jeff

[Mauricio Lumbreras]

Hello
I posted a case at support but it seems the answers comes from a help desk telephone guy
I explained the churning instance situation and they explain me it is normal and give us the speech I heard tons of times
I think this list at least serve to discuss what is happening. No one good gauge console will show the real health of the system, the only place to review the status of this unstability seems this list and heard real user feedback
Regards
Mauricio

[Keith Marsh]

My latency and churning seems to have returned to normal (I hope that hasn't jinxed it!).  I didn't get any response from the issue tracker, just a "thanks, we've passed it on.  Status updates will be posted here too"

This issue has made me think about how losing a appengine system in a region can be mitigated.  If it's standalone, it's pretty straight forward to redeploy, but if you're using datastore or other project resources like BigQuery, Cloud Storage etc, I can't see an easy way to shift your processing from us-central to us-east or eu-west.  And the issue could also affect those resources.   I guess regular copies from one project to another can be done using the command line.  It might be neat for Google to offer migrating an entire project from one region to another, though Compute would offer challenges.

A member from Cloud Trace asked me how it could be improved to help in this situation.  Does anyone have any thoughts on that?  It helped identify that there was a delay before the first RPC call, (see second pic).  

Cheers

Keith

[Anthony Shapley]

Region migration would be ace. When I started some of my Apps, US was the only region available - and all of my 'gear' would be far better suited to the EU, but no migration options are available and I do not want to have to do it manually.

[Nick (Cloud Platform Support)]

Hey Folks,

So, this issue has officially subsided and is tracked over at the Cloud Status Dashboard. 

Couldn't agree more that region-migration is a great feature request. I took the time to fill out the request and it's currently being tracked here. You can star that thread to receive updates on its progress.

Regards,

Nick