Cloud Tasks getting executed hours beyond their ETA

[Jeff O'Neill]

I have a GAE app (opavote.appspot.com) and yesterday evening around 9PM ET (Boston time) I had hundreds of tasks in my task queue that were way beyond their ETA (some more than 2 hours overdue).  I clicked the "Run Now" button on the Cloud Tasks console hundreds of times to get these tasks executing.

The task queue then seemed to be operating normally.

But this morning, I again had hundreds of tasks that were beyond the ETA though a little better in that they were about 90 minutes overdue.  I'm again clicking the run now button to get them running.

This seems to be an issue on Google's end.

Is anyone else having this issue, and can Google please comment on what the problem is and when a resolution is expected?

My app depends on tasks being executed reasonably close to on time so 1-2 hour delays is ruining my customer experience.

[Jeff O'Neill]

This is still happening today.  

My task queue is again full of tasks past their execution date.

How do we get assistance when Google Cloud is not functioning correctly?

[yka...@google.com]

This could be due to App Engine instances being overloaded causing latency. Please see these best practices docs to help you manage Tasks better [1][2].

If you are still unable to diagnose the root cause, I suggest opening an issue on the Issue Tracker website [3], where we can investigate further and forward the issue to Engineering if needed.

[1] https://cloud.google.com/tasks/docs/common-pitfalls#latency_spikes_and_max_concurrent

[2] https://cloud.google.com/tasks/docs/manage-cloud-task-scaling

[3] https://cloud.google.com/tasks/docs/support#follow_the_issue_tracker

[Jeff O'Neill]

Thanks for the response.

I just noticed that Cloud Tasks is consistently executing only 6 tasks per minute which causes the queue to fill up.  Any idea why?

I've got autoscaling turned on and my app is not experiencing a traffic spike so don't believe it is an overload issue.

[Jeff O'Neill]

I just noticed that my task queue has an "Enforced Rate" of "0.1/s" which comes to 6/minute.

I suspect this happened because my app hit its daily spending limit a few days ago (I've now increased it).

I opened an issue to increase it and I hope they address this quickly...

[George (Cloud Platform Support)]

Hello Jeff, 

Has the quota increase solved your issue? If not, I fully support my Colleague's recommendation, and suggest you should open an issue of similar content in the Public Issue Tracker. 

[Jeff O'Neill]

No, the rate is still low.

I've opened an issue and hope they get to it soon.

Thanks for the feedback!