App Engine incorrectly reporting high number of tasks in the task queue

[Gwyn Howell]

Just checked the app engine console to alarmingly find over 20k tasks sitting in the task queue and 25 instances running. Our app normally runs on 4-5 instances. I paused the tasks queue, so I could investigate. I found that the tasks were making calls to the Google Calendar API and throwing "DeadlineExceededError: ApplicationError: 5" exception presumably UrlFetch is timing out when communicating with the API. Regardless of this, there still should not be that many tasks, so I purged the queue and started the job again and sat there watching the tasks spawn - 1, 2, 3, 4, 5 all the way to about 20 tasks then suddenly 13,000 of them. Clearly this is incorrect. My app does not contain code to spawn that many tasks in such little time.

What is going on here? Why is App Engine reporting so many tasks?

[Nicholas Verne]

Can you please give us an app id so we can investigate?

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/8sfyiYFqmM0J.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Gwyn Howell]

the app id is appogee-leave, but it seems to have sorted itself out now. yesterday from about 10am - 3pm utc the task queue was out of control, and we now have a bill to pay to cover it

On Tuesday, 27 March 2012 22:51:11 UTC+1, nverne wrote:

Can you please give us an app id so we can investigate?

On Tue, Mar 27, 2012 at 9:27 PM, Gwyn Howell <gwyn....@appogee.co.uk> wrote:
> Just checked the app engine console to alarmingly find over 20k tasks
> sitting in the task queue and 25 instances running. Our app normally runs on
> 4-5 instances. I paused the tasks queue, so I could investigate. I found
> that the tasks were making calls to the Google Calendar API and throwing
> "DeadlineExceededError: ApplicationError: 5" exception presumably UrlFetch
> is timing out when communicating with the API. Regardless of this, there
> still should not be that many tasks, so I purged the queue and started the
> job again and sat there watching the tasks spawn - 1, 2, 3, 4, 5 all the way
> to about 20 tasks then suddenly 13,000 of them. Clearly this is incorrect.
> My app does not contain code to spawn that many tasks in such little time.
>
> What is going on here? Why is App Engine reporting so many tasks?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/8sfyiYFqmM0J.

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

[Kenneth]

Have a look at this issue:  http://code.google.com/p/googleappengine/issues/detail?id=6481 

Note that this is only a reporting issue only and doesn't affect what's actually going on, so if you are experiencing cost increases that's something else.  Make sure you set larger timeouts on your url fetches.

[Gwyn Howell]

That would explain the task queue issue. I did notice a larger number of instances than usual, but that could have been as a result of the Calendar API issue.

Regarding your advice to set larger timeouts - do you have best practice for doing this when using the gdata library?

[Kenneth]

I'm offline right now so I'm not certain if the deadline was already there but in our copy of gdata we have an increased deadline set in alt/appengine.py line 146.

We also make sure we never block on gdata, so either we do it in a task or on an ajax call where we can have a pretty spinner and handle errors in a friendly manner.