"Request was aborted after waiting too long to attempt to service your request." sprees

[Kaan Soral]

"Request was aborted after waiting too long to attempt to service your request."

I've been seeing these messages a lot lately, momentarily many requests log these errors, it floods the error logs

Anyone else?

[Mario]

In order to help you further, you'd need to provide more information about when you see those messages, kind of instances you're using, programming language, etc.

[Cody Landgrebe]

Kaan,

Looking at the SDK release notes and current version of GAE console app engine release 1.9.19 notes have not been published yet; but from research my assumption is that the logging level was changed from info to error in the latest version. If you look through your logs prior to the move to .19 I assume that you will have the same messages but logged as info.

[Kaan Soral]

Hi Cody

That makes an extreme amount of sense, it would also explain a lot of the inconsistent behaviours of appengine, especially invisibly exhausted taskqueue retries

I checked the logs, however, although my log storage is more than enough, for some reason the phrase "aborted"/".*aborted.*" didn't produce any results, didn't pursue the issue further, the log routines are annoying at best

Mario, the instances are:

instance_class: F2

automatic_scaling:

  min_idle_instances: 1

  max_idle_instances: 1

  max_pending_latency: 900ms

The issue has been happening in bursts lately

[Mario]

Hi Kaan,

This error is sometimes created by your requests going over the memory limit of your instance. You could try to update the instance to F4 or to make your requests to process data in smaller chunks. 

Mario

[Kaan Soral]

I started regularly getting 200-1000 of these in batches

Not sure what to do about them

My other app that handles more requests never has them, I'm guessing they are related to instance bursts, the app that experiences the issue probably bursts instances too much

I don't think the issue is memory related, as in that case, the instances usually die with the critical memory error

[Mario]

Hi,

What are those requests that result in errors?

[Kaan Soral]

They are all taskqueue tasks on a module with F2's

I think out of the 3, the module aspect might be the cause, as I don't experience the issue on other systems

[Alexandra]

Hi,

Did you find the reason of these errors?

I also see a lot of them. The time which is elapsed is about 10 min, so it seems that there are tasks that are stuck for some reason in the task queue.

Any insights on that?

Thanks

Alexandra

[Nick (Cloud Platform Support)]

Hey Alexandra,

If you're experiencing an issue on the platform, the best way to report that issue and get help with it would either be to post on a Stack Exchange site (Stack Overflow for example), in the case that you think the issue is related to your own code, or on the public issue tracker for App Engine, if, having followed all documentation and fully understanding the issue could not be in your code, you suspect that the issue might be on our side, which we can fix. 

Unfortunately, this thread contains next-to-zero useful debugging information other than some instance stats (F2 instance) and a vague indication that the error coincides with bursts of activity, so this thread can't really function as an issue report which is actionable.

So, I wish you or anybody else experiencing issues who reads this thread the best of luck in reporting your issue through one of the forms above, and hopefully with providing enough information, you can have someone help you out with a solution for your code or a tracking number for a solution to be patched, depending on the situation.

[Kaan Soral]

Yes, AppEngine can't handle bursts well, so the solution is to slow things down

Actually, there are speed limits on a lot of things that are usually not advertised well, if you are getting these kind of errors, try to slow things down and see how it goes

Slowing things down is also pretty challenging, much harder than building stuff that will scale

[Jason Collins]

More min_idle_instances (at an increased cost of course) will help service bursts and avoid these pending queue timeouts.

[Nick (Cloud Platform Support)]

@Kaan Soral

I think the issue cannot be adequately narrowed-down from the information provided, and with only the commonality of seeing the "Request was aborted after waiting too long to attempt to service your request." error, it's impossible to determine that the same issue is being reported by Alexandra - that's the reason why a proper issue report is needed (see in that link a template).

There doesn't seem to be a basis for inferring the general statement that "AppEngine can't handle bursts well". This is especially so when the justification for the statement refers to a very specific use-case related to errors seen in a task queue with specific scaling settings. These scaling settings might be at the heart of the original issue reported at the top of the thread. Having a minimum and maximum of 1 idle instance doesn't exactly lend itself to managing every possible burst.

Depending on the request pattern and start-up time, you could see strange scenarios where requests are building up in pending queues, instances are firing up and then shutting down after becoming idle, with the pending queues migrating and merging among these rapidly spawning and de-spawning instances. Requests in a rather short pending queue dealing with this potentially-volatile setup would not be seen to scale well, and many could in fact reach a timeout without getting to an instance. Again, this is a plausible explanation, but not confirmed by any real data. You can read in the docs some advice on how the scaling settings work, and how to best design for scaling events. Of course, as usual, if you have a specific issue, feel free to provide a report in the issue tracker.