Serving errors and latency, Tuesday Nov 29th

[Kenneth]

I've been experiencing serious latency and 10 second startup error messages for about the past hour.  I put in a production issue but no response yet.

Is anyone else seeing errors or is it just in my little part of app engine?

[Brandon Wirtz]

None of my apps have any issues at present. HR or MS?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/vcmUlxwDgOYJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Kenneth]

Go on, take a guess!  

MS of course.  

Still waiting for that blobstore auto copy migration magic...

[Brandon Wirtz]

My Ms apps are behaving.

 

Moving blobstore between apps is only a couple of lines of code, and a one time computation/bandwidth fee.  Bite the bullet.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/0aosKDpydOgJ.

[Kenneth]

A couple of lines of code?  Really?  Can you send me those line?  I've seen a couple of examples and they run to hundreds of lines and scare the bejesus out of me.

If it is a couple of lines I'd be a very happy camper.

[André Pankraz]

I think this is really a little bit optimistic. At least the Java version has some problems if you copy multiply 1000 blobs. 1 out of 100 writes has timeouts (even with complete pre-caching of data and write as fast as possible into Channel) or there are also some memory leaks for long running blob-creation loops (don't cache BlobstoreService...). So the exception handling and BlobKey handling is for sure more than a couple of lines?!

[Kenneth]

DeadlineExceededError: The API call blobstore.CreateUploadURL() took too long to respond and was cancelled.

That's a new one!

I've cranked up the resident instance to about 3x my normal instance count so errors have gone down from once every 5 seconds to once every 2 minutes or so.  Gonna cost me a bomb for something that isn't my problem.

[Joshua Smith]

I see this now and then. In my case, I just catch the error and "forget" to put the upload button onto the page in this case (users rarely use the upload button in my application).

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Ueqwx-K8y0gJ.

[Kenneth]

Wow, that's pretty bad, it sounds like your users are pretty forgiving!  

In my case it is symptomatic of something having broken in the datastore behind my application.  Queries are taking about 3x longer to execute today and the whole app is bogging down.

[C Sowa]

That part of App Engine includes me; milliseconds per request chart attached. Started about midnight PST.  Yet another scheduler change?  Also, instance activity shot up as well.  Will impact costs/quota in terms of chargeable instance use.

 

[Kenneth]

I reckon it's a problem with our little corner of bigtable.  Star this issue if you want:  http://code.google.com/p/googleappengine/issues/detail?id=6428  I'm surprised no one at Google has looked at the issue in over 7 hours, I guess they're just waking up.

[johnP]

My corner of the cloud is affected this morning.

So +1 from me on this issue.

johnP

[Brandon Wirtz]

80 ish lines of python. I consider that a few. Especially since it is a good exercise in working with blobstore.

 

I’ll see where I put my python version.

 

The problem with the Java example is that it is too complicated.

 

The logic is:

 

“get blob list” on OLD appid.  (6 lines excluding the app.yaml)

 

On new app, request blob, write blob, if error note, and continue error or not. Do next 20 blobs (or how ever many depending on the average size)  (30 lines)

 

When you reach the end of the blob list do the same thing for the error list. (6 lines)

 

Use scheduled tasks as a Cron to fire these off over a couple of days. If you want it on the cheap be ok with it taking a week using as much free quota as possible. (few minutes in configuration)

 

 

 

 

 

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of André Pankraz
Sent: Tuesday, November 29, 2011 4:34 AM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] Serving errors and latency, Tuesday Nov 29th

 

I think this is really a little bit optimistic. At least the Java version has some problems if you copy multiply 1000 blobs. 1 out of 100 writes has timeouts (even with complete pre-caching of data and write as fast as possible into Channel) or there are also some memory leaks for long running blob-creation loops (don't cache BlobstoreService...). So the exception handling and BlobKey handling is for sure more than a couple of lines?!

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/WI1pdtSkOeAJ.

[C Sowa]

FYI, appears to have resolved about an hour ago. ~16 hours total for this event.