Task queue not executing tasks / backlogged

[Hugo Visser]

Since last night or so I'm seeing this issue where tasks are being queued up for some time and not being executed on an app that has been running like this for quite a while. This results in a huge pile of tasks that get backlogged for my app.

Cloud status dashboard says everything is OK. Anyone else seeing this?

Hugo

[Hugo Visser]

I'm seeing tasks in my queue that are there for 6 minutes without any tasks running at all. There doesn't seem to be much I can do about this which is really frustrating.

[Nick (Cloud Platform Support)]

Hey Hugo,

Without knowing more about A) the configuration of the queue, B) the rate of task creation, C) the latency statistics on the task handler, it's difficult to say much on why this might be happening. If you could gather all those types of information, myself or someone else may be able to leave a more insightful comment.

Cheers,

Nick
Cloud Platform Community Support

[Hugo Visser]

It looks like the queues are back to normal. I've setup alerts in stack driver that triggered for a while but are resolved now. Will report back if that changes.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/MYxDL8sihN4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/370b31f3-e1fa-4a2f-9287-652aeac0500c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nick (Cloud Platform Support)]

Thanks for coming back to update. Feel free to post again with the information mentioned if you notice anything in the future. 

Best wishes,

Nick
Cloud Platform Community Support

On Wednesday, May 11, 2016 at 4:33:36 PM UTC-4, Hugo Visser wrote:

It looks like the queues are back to normal. I've setup alerts in stack driver that triggered for a while but are resolved now. Will report back if that changes.

On Wed, May 11, 2016, 21:08 'Nick (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com> wrote:

Hey Hugo,

Without knowing more about A) the configuration of the queue, B) the rate of task creation, C) the latency statistics on the task handler, it's difficult to say much on why this might be happening. If you could gather all those types of information, myself or someone else may be able to leave a more insightful comment.

Cheers,

Nick
Cloud Platform Community Support

On Tuesday, May 10, 2016 at 4:54:00 AM UTC-4, Hugo Visser wrote:

I'm seeing tasks in my queue that are there for 6 minutes without any tasks running at all. There doesn't seem to be much I can do about this which is really frustrating.

On Tuesday, May 10, 2016 at 8:54:33 AM UTC+2, Hugo Visser wrote:

Since last night or so I'm seeing this issue where tasks are being queued up for some time and not being executed on an app that has been running like this for quite a while. This results in a huge pile of tasks that get backlogged for my app.

Cloud status dashboard says everything is OK. Anyone else seeing this?

Hugo

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/MYxDL8sihN4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

[Hugo Visser]

Just got an alert again. The task rate is 25 tasks/sec with a bucket size of 10. Average latency of the handlers is varying depending on the task, but anywhere between 1 and 8 seconds.

The thing that I'm seeing is that the task queue shows 120 tasks in the queue, 0 running, oldest task ETA 6 minutes ago. That doesn't seem right to me, and it looks like these tasks are just not being executed. I see nothing in the logs of my task handlers that indicate that a request is being executed. 

Obviously this task execution thing is pretty essential to my app.

Hugo

On Wednesday, May 11, 2016 at 11:34:34 PM UTC+2, Nick (Cloud Platform Support) wrote:

Thanks for coming back to update. Feel free to post again with the information mentioned if you notice anything in the future. 

Best wishes,

Nick
Cloud Platform Community Support

On Wednesday, May 11, 2016 at 4:33:36 PM UTC-4, Hugo Visser wrote:

It looks like the queues are back to normal. I've setup alerts in stack driver that triggered for a while but are resolved now. Will report back if that changes.

On Wed, May 11, 2016, 21:08 'Nick (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:

Hey Hugo,

Without knowing more about A) the configuration of the queue, B) the rate of task creation, C) the latency statistics on the task handler, it's difficult to say much on why this might be happening. If you could gather all those types of information, myself or someone else may be able to leave a more insightful comment.

Cheers,

Nick
Cloud Platform Community Support

On Tuesday, May 10, 2016 at 4:54:00 AM UTC-4, Hugo Visser wrote:

I'm seeing tasks in my queue that are there for 6 minutes without any tasks running at all. There doesn't seem to be much I can do about this which is really frustrating.

On Tuesday, May 10, 2016 at 8:54:33 AM UTC+2, Hugo Visser wrote:

Since last night or so I'm seeing this issue where tasks are being queued up for some time and not being executed on an app that has been running like this for quite a while. This results in a huge pile of tasks that get backlogged for my app.

Cloud status dashboard says everything is OK. Anyone else seeing this?

Hugo

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/MYxDL8sihN4/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

[Hugo Visser]

Hitting the "run now" button makes the task run by the way. I don't see any relation to error rates on the task handler instances either. Looks like they are just stuck there waiting for execution.

[Nick (Cloud Platform Support)]

Thanks for the update. Barring a very low setting for "max concurrent tasks", I don't see why this could occur, so it appears this could be a production issue. In that case, I'd recommend opening a Public Issue Tracker thread with label "Restrict-View-EditIssue" (to protect your information) and the following information:

* your app id
* the name of the queue
* the queue settings
* some queue statistics

We may be able to determine the root cause of this at that point. 

Sincerely,

Nick
Cloud Platform Community Support

[Hugo Visser]

This has been happening on and off but today it's really off the charts. Currently have 28k tasks and none of them are executed. I filed #13118 for that.

[Pasha Seliverstov]

Hi Nick,

I have the same issue with queue not executing tasks at rate set in configuration. It looks really strange because the execution works correctly most of the days, but once per week the queue does not execute at correct rate. I experience the issue on  Sep 30th and today Oct 6.

I tried different configuration settings, but the configuration changes did affect the real execution rate. The execution is way too slow than set in configuration.

Best regards,

Pasha

[Nick (Cloud Platform Support)]

Hey Pasha,

I would suggest posting a more detailed technical report to the Public Issue Tracker. The Public Issue Tracker is much better suited to issues like this, while this forum generally should be geared toward high-level discussion of the platform and services, design patterns, etc.

You should try to include some or all of the following:

1. your project ID
2. the task queue names
3. their configurations
4. the code for and/or a description of the task handler
5. the time-frames when the issue is observed to occur within the past few days (since time is of the essence in making sure the logs still exist in the system)
6. a description of the average load and the frequency distribution of different sized loads
7. the frequency distribution of response latencies for the task handlers

This will enable us to get started on reasoning about the issue and looking into it. Feel free to post a link to the thread here once you've done that. I'll be happy to assist.

Cheers,

Nick
Cloud Platform Community Support

[Thanasis Delenikas]

Hi Nick / Google,

During the last 1-2 weeks, I've had the same problem. I've *never* encountered a backlog on my queues (load is small and processing generally fast). Then, suddenly and out of the blue, with *no* application or other changes, I get the behavior displayed on the attached graph (graph from Stackdriver, queue latency, 1month graph). It really crippled me!

I don't believe that this issue is related to my project. It must be a generic issue that you should be aware of. I've even tried with different AppEngine instance classes, but this "delay" had nothing to do with the instance "power".

I don't think opening tickets will do any good. I've done it in the past, when I reported an issue with task queue rate (Issue #12620, acknowledged since December 17, 2015). Resolution -> NONE.

I really love the AppEngine concept and development environment. But I doubt Google's devotion to it and that makes me really sad.

[Nick (Cloud Platform Support)]

Hey Thanasis,

Does that latency correspond to the time spent in the handlers that deal with the task queue request itself? Or, if not, what underlies the metric?

Cheers,

Nick
Cloud Platform Community Support

[Thanasis Delenikas]

Hello Nick,

No, they do not relate with the processing time. The later is quite constant - not very fast, but constant.

Tasks just waited (and waited and waited) in the queue without getting processed. When they did find their way in the processing handlers, they took the same time as always.

The graphs you see skyrocketed not because the processing time went nuts, but because GAE "forgot" to consume the tasks.

This is a budget-enabled project - but GAE made no attempt to launch even one more instance in order to take some tasks off the queues. Queues are set quite high: Token Bucket:500 and Rate: 10/s

Since the last post, I've moved my project to the European DCs (I was planning to do that anyway, since it's closer to me).

I am experiencing similar issues here as well. I mean, I've seen tasks staying in the queue for 5 minutes without any obvious reason. On other occasions, queue consumption is instant.

Thanks.

[Nick (Cloud Platform Support)]

Hey Thanasis,

Could you email me directly with your project ID and task queue names, along with timeframes in recent days when this has occurred? Or if it isn't occurring at present, send an email immediately upon viewing it in future? You could also create a Public Issue Tracker issue with label "Type-Production". We should be able to look into this to determine what's going on.

Cheers,

Nick
Cloud Platform Community Support

[Asim Pereira]

I know this issue was raised years ago, but I am facing the exact issue since quite many months.

I have tried various queue configurations, different instance classes, memory optimization but to no avail.

I even broke my backend and move a part of it to Frontend and Queues, but even the Queues backed by frontend stalls (apart from the backend)

What solution worked for you guys?

[Alan Gregg]

I find this and I struggle to upload things usually because memory wont let me put through things.

Yet you have places saying you can upload and make apps with very little memory and storage.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/d66b56dc-5e5f-4336-bdb9-688a7afc9589%40googlegroups.com.

[Ying Li]

For production issue, please post a proper report on our Public Issue Tracker [1] so we can investigate into it. Google Groups is generally used for simple product inquires and discussion and not actual issue reports, thanks.

[1] https://cloud.google.com/support/docs/issue-trackers#search_for_or_create_bugs_and_feature_requests_by_product

On Friday, May 24, 2019 at 4:52:15 PM UTC-4, Alan Gregg wrote:

I find this and I struggle to upload things usually because memory wont let me put through things.

Yet you have places saying you can upload and make apps with very little memory and storage.

On Fri, 24 May 2019, 21:30 Asim Pereira, <perei...@gmail.com> wrote:

I know this issue was raised years ago, but I am facing the exact issue since quite many months.

I have tried various queue configurations, different instance classes, memory optimization but to no avail.

I even broke my backend and move a part of it to Frontend and Queues, but even the Queues backed by frontend stalls (apart from the backend)

What solution worked for you guys?

On Tuesday, May 10, 2016 at 12:24:33 PM UTC+5:30, Hugo Visser wrote:

Since last night or so I'm seeing this issue where tasks are being queued up for some time and not being executed on an app that has been running like this for quite a while. This results in a huge pile of tasks that get backlogged for my app.

Cloud status dashboard says everything is OK. Anyone else seeing this?

Hugo

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.

[Javier Martin]

I'm experiencing the same problem. 

Every 30-45 minutes, about 10-20 tasks in queue are just ignored for some minutes. It takes about 7-9 minutes for the queue to dispatch them. All the rest of tasks are dispatched normally : (

I already filed an issue on the tracker https://issuetracker.google.com/issues/135471809

Did you find any solution?

[Nicolas (Google Cloud Platform Support)]

Hi,

 

Thanks for reporting this and thanks for opening the issue tracker ticket, that was the right thing to do.

 

Further communications will be made in this thread, however we cannot promise an ETA for the resolution of this issue.

[Govind Rai]

Hi Nicolas,

I am experiencing the same issue, and no one has actually responded to that open ticket. It's stopping up from deploying to production because the Queue keeps getting backlogged because Task Queues slows down so much. Can you bring that issue up to attention??

Thanks, Govind

[Govind Rai]

Hi Thanasis,

How were you able to resolve your problem?

Govind

[David Charles Martinez]

Hello Govind,

Google Issue Tracker is the right place for these types of issues and with the provided details we can track it and troubleshoot it accordingly since Google Groups is meant for conceptual questions and discussions and this will require proper troubleshooting. In addition, I would like to inform you that this issue is currently being investigated.