Tasks Queues / processing rate.

[Thanasis Delenikas]

Hello,

I have a Java application which happily runs in the free tier of Google App Engine. In the past I've encountered some spikes - from that moment I've enabled the paid tier with a small daily budget. So, although I have allocated some $$$ for daily spikes, my application runs in the free-quote range for most of its life.

My problem is task queues.

For example, I have a queue with bucket size "20", rate "1/s" and max concurrency "1".

When this queue gets something like 10-20 tasks (each task takes some seconds to execute), I see an actual processing rate of "1 / minute" - yes, per minute. In the same time, the running instance does no work at all and GAE makes no attempt to start a second instance - to "maybe" boost the processing rate in order to try and align with the "1/s" configured rate.

Here is a log of the workload and the processing ratio - check the hours / minutes:

I 2015-12-16 11:55:50.488  200       0 B  3.19 s /tasks/EventProcessor?eventId=W475967

I 2015-12-16 11:54:06.255  200       0 B 14.67 s /tasks/EventProcessor?eventId=W476837

I 2015-12-16 11:53:10.281  200       0 B  4.65 s /tasks/EventProcessor?eventId=W476829

I 2015-12-16 11:52:11.583  200       0 B  9.21 s /tasks/EventProcessor?eventId=W476800

I 2015-12-16 11:51:11.480  200       0 B  5.25 s /tasks/EventProcessor?eventId=W476697

I 2015-12-16 11:50:11.377  200       0 B  5.47 s /tasks/EventProcessor?eventId=W476678

I 2015-12-16 11:49:11.311  200       0 B  4.68 s /tasks/EventProcessor?eventId=W476660

I 2015-12-16 11:48:11.216  200       0 B 10.97 s /tasks/EventProcessor?eventId=W476523

I 2015-12-16 11:47:11.143  200       0 B  2.76 s /tasks/EventProcessor?eventId=W476522

I 2015-12-16 11:46:11.048  200       0 B   6.3 s /tasks/EventProcessor?eventId=W476394

I 2015-12-16 11:45:10.945  200       0 B  3.41 s /tasks/EventProcessor?eventId=W476391

I 2015-12-16 11:44:10.921  200       0 B   6.2 s /tasks/EventProcessor?eventId=W476361

I 2015-12-16 11:43:10.817  200       0 B  2.68 s /tasks/EventProcessor?eventId=W476359

I 2015-12-16 11:42:10.713  200       0 B  6.21 s /tasks/EventProcessor?eventId=W476345

I 2015-12-16 11:41:10.696  200       0 B  438 ms /tasks/EventProcessor?eventId=W476252

I 2015-12-16 11:40:10.593  200       0 B  6.41 s /tasks/EventProcessor?eventId=W476130

F 2015-12-16 11:40:01.720  500       0 B   8.8 s /tasks/EventProcessor?eventId=W475967

Why doesn't GAE respect the defined ratio? Am I doing something very wrong? Have I missed anything?

Does this happen because the actual processing takes seconds instead of ms?

Should I leave automatic scaling behind and move to custom scaling?

Thanks!

[Nick (Cloud Platform Support)]

If this isn't an example of mis-configuration (somehow having pushed a "1/m" queue definition in the past), then it probably warrants a post to the Public Issue Tracker. The information you might include would be the code behind /tasks/EventProcessor, along with the app id of the affected app and the queue name, along with other potentially-related information, such as the QPS on the instances which are handling the requests.

Posting there will quickly lead to a reply from someone in Community Support doing triage of Public Issue Tracker issues. Whether this has always been this way, or started recently, would also be some useful information. It might have been hinted by your post, but I can't determine for sure.

Best of luck!