Taskqueue execution latency up by 1000x

[Dan]

Hi Goggle,

Is it possible to have an update as to what's going on with the taskqueue execution latency. According to the status monitor it has been going on since midnight on 10th November and is still continuing.

There are no notes to say you are looking into it so I just wanted to make sure you are aware of it.

The pattern I am seeing is that multiple tasks are delayed for over a minute and then execute all at once. My app is used to an execution latency in the order of < 10ms but I am seeing delays of over 60,000ms.

Best wishes,

Dan

[Dan]

Looking at the logs I was a factor of 10 out. It is more like a 100x latency increase, not 1000x as taskqueue execution latency is usually < 100ms, not 10ms . 1000x is much better for a post title though. ;-)

[Marcel Manz]

Can someone from Google please respond to this issue?

The execution latency issue exists now for more than 48 hours and still no word or alerts about it. Can Google please allocate more human resources in order to communicate better? This is a business oriented service and such lack of communication must just not happen. And we don't care if such issues happen on weekends, App Engine needs to be stable 24/7.

- Marcel

[Dan]

The system seems to be returning to normal after 60 hours of 60,000 - 120,000 ms latency in taskqueue execution instead of 60 - 120 ms.

I am surprised users were not more vocal about this and no acknowledgement / explanation from Google. Maybe people don't use taskqueues the way I use them - process some data on a taskqueue and then push it back to the UI. For this period of increased latency my users were getting their asynchronous data back in the UI one to two minutes after they had submitted it instead of the usual less than two seconds.

I imagine App Engine uses taskqueues as a way of smoothing out server traffic as they can stack them when load is high and run them when it is low and for the last 60 hours load has been continually high...

[stevep]

This issue has been raised before (I believe this is what you are seeing). Many moons ago vlad was its champion in these groups. His conclusion (link below): "I have been screaming about this for a looong time. The offcial position of GAE team is - we don't guarantee on time execution. Even though this issue is not about on time execution but rather about bugs/inefficiency of their scheduler." 

Sadly I think this is never going away, so best to build your client with the anticipation of such delays -- TQ tasks as part of handling client calls are too powerful/necessary to avoid entirely. HTH -stevep

Link:

https://groups.google.com/forum/?fromgroups=#!topic/google-appengine/nwnq7cjABhY

[Marcel Manz]

That issue seems related to task delays occuring for tasks that get scheduled from within tasks.

I assume that the taskqueue execution latency chart as per the status page does not relate to that issue and happened to be a generic task execution delay issue during the last days.

[Dan]

Thanks for the info.

Looks like I spoke too soon as well, taskqueue execution latency is back up again - awesome. If only the latency increase was in the order of 10x not a massive 100x - 1000x...

[vlad]

I ditched TaskQueue for all requests with timeouts of less than 60 sec. It was simply too unreliable. Btw, there is a ticket filed for feature requesting a TaskQueue with timing guarantees. Feel free to star it. Sorry, too lazy to dig out a link.

[Doug Anderson]

Have you tried increasing the bucket_size above the default value of 5 and/or the processing rate above 5/s?  I'm not saying there isn't a bug... I'm just curious if there's a workaround or if perhaps the problem is triggered by exceeding some threshold from which task queue manager never properly recovers.

[Saurabh]

Hi Google,

The TaskQueue is experiencing HUGE latency in execution (as of 15:00 EST): it is totally messing up our mobile app: usually, TQ has been rock-solid reliable in chewing through the tasks, but it is really slow to execute things today.

Please address / acknowledge this since it is really wreaking havoc with the async apps that needed to get a lot of data processed for users!

Best regards,

Saurabh

[stevep]

I do not think the origination point of a TQ task (on-line handler, deferred function, or other TQ task) will affect its scheduling. There is a tiny bit of info about the TQ scheduler (it differs from "The Instance Scheduler"). Task scheduling is not always intuitive nor easily understood was my docs takeaway. HTH -stevep

[stevep]

Yet an architecture with such potential to do good. Someone call the Justice League!!! :-) -stevep

[Saurabh Gupta]

Latency went down for a bit but has risen again:

http://code.google.com/status/appengine/detail/taskqueue/2012/11/13#ae-trust-detail-taskqueue-execution-latency

I've moved stuff over to execute on a backend: hope that can hold up until TQ latency becomes reasonable again.

Is this latency related to the nexus launch? Isn't the TQ designed to mitigate exactly this situation?

{SG}

On Sunday, November 11, 2012 5:43:06 AM UTC-5, Dan wrote:

[Julie Smith]

FYI, it is issue #4901 at

http://code.google.com/p/googleappengine/issues/detail?id=4901

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/rxIPu2eId4wJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Saurabh Gupta]

Thanks, Julie! That reference is really helpful!