Why Google arbitrarily change my task queue setting!!!

[gks]

I just found task queue processing rate of my apps was changed to 5qps last night without any deployment from me! It caused a major service interruption to my customers since a lot of tasks were delayed.

There is no way this change came from my team since everyone was sleeping when the change happened. And there is no corresponding deployment activity from admin log. Can Google provide an explanation on why it happened, and what can be done to prevent it from happening again?

My AppID are: aos-prod-1, aos-prod-2

[gks]

After I changed the processing rate to 100QPS to drain the queue, it was changed back to 5QPS just now. Can anyone from Google start looking into this issue and stop changing my task queue setting?

[gks]

And the task queue of my app aos-prod-1 is lying about the # of tasks in queue since it always says 100, which is impossible given my customers usage pattern. 

[nischalshetty]

We are facing problems with the task queue as well. The tasks shot up to millions and everything stopped working for us!

[Nicholas Verne]

Could you please give us some more details:

Which rate did you see go down? The Maximum Rate or the Enforced Rate?

Were your tasks running on backends or frontends?

Do your admin logs indicate that you were the only developer uploading changes?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/SIo8pCc0-WYJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Nicholas Verne]

nischalshetty,

We can't investigate anything without an app id.

On Fri, Oct 19, 2012 at 12:59 PM, nischalshetty <nischal...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[gks]

Following are the answers to your questions:

1. Rate: Both maximum rate and enforced rate were changed to 5 QPS. I was only using default queue and set 20 QPS on it.

2. Task: All my tasks run on frontends

3. Admin Log: Yes, I am the only developer uploading changes and there is no activity in admin log when the setting was changed.

In addition, when I tried to change the setting to 100 QPS this morning in order to drain the queue, it was always changed back to 5 QPS about 5 minutes later after the deployment. Finally I found a workaround to create another queue with the same setting which stopped the madness of automatically configuration change.

Actually task queue has been causing service interruption to me in the past 2 days. Yesterday, the default queue suddenly stopped accepting any task with name, which I had to workaround in order to bring my service back to normal state. I was too tired to write a post here about the issue after I finished live site support yesterday. Since you are you investigating, would you please also look into this issue as well? I really need the feature to add task with name so that I can avoid duplicated task processing.

[nischalshetty]

My app id is justunfollow-h

Please look into this as now it's just filling up our

[nischalshetty]

For us the rate did not go down. There were millions of tasks created in the task queue (it should have been 40,000). The stored task byte count shot to 0.5GB

I then purged the queue and increased the stored task byte to 1.5GB a couple of hours ago. Everything seemed fine and now the stored task byte count has reached 1.5GB.

No code changes have been done by us, this started happen 10 hours ago. Please look into this.

App Id: justunfollow-h

[nischalshetty]

I can now see the total task created count to be stuck at 100. 

[gks]

Again, the rate setting of default queue is changed to 5 QPS for both of my apps. This kind of random behavior is not acceptable if GAE would like to be a serious cloud platform.