Sporadically terrible performance

[Devel63]

I haven't been paying close attention, but suddenly(?) today I'm seeing sporadically awful performance.  It took 20 seconds just to serve a static HTML page with no one else using this instance, and no cold start.  Normally this takes about 2ms, so 10000 times worse. Let alone my actual datastore interactions, which sometimes worked, sometimes took orders of magnitude longer than usual, and sometimes issued a 500 with no corresponding log entry.  What is going on with GAE?  Is anyone else seeing problems?  

Or is this just normal?

[Ed]

This is unusual, but we are seeing the same behavior.

Our static handlers and trivial health-check handlers are seeing 30s+ latency. We've escalated with Google support, and we have an open ticket on the issue.

We'd been seeing strange behavior for a few days, including 30s+ on some modules that do nontrivial work. But static files and trivial handlers taking this long is new, and only started for us about a half hour ago.

Ed

[Ed]

Fyi -- this issue was resolved for us around 1:15 am pacific (8:15 am UTC) this morning.

Ed

[Kaan Soral]

As usual I've experienced a lot of aftermath confusion caused by missing push queue tasks

This kind of performance flukes are also pretty regular, yet they usually happen spontaneously to screw things up

[Devel63]

Ed, was this something that Google fixed for you specifically, or part of a larger issue?  

I haven't seen the problem again today in my extremely limited testing, but frankly I have no idea what the real situation is.  I do see, though, that some interactions that "always" were very fast may need additional client logic to handle hugely slower interactions, and I'm wondering how often that occurs.  Occasionally on an ongoing basis, in batches on rare occasions, etc.  I was warned not to deploy on GAE in part for this reason, but I had hoped those issues were gone.

[Ed]

As far as I know, it was a systemic issue, but I don't know how many people it affected.

I asked for a detailed explanation, but I'm told that the engineering investigation is still ongoing, so information is not yet available.

Ed

[Kaan Soral]

I inspected the issue further, 50% of my issues were caused by failed ndb.transaction's - other half was left uncovered for me, but I'm guessing similar issues

I improved things on my end to handle such failures better