Upcoming changes to Task Queues

3025 views
Skip to first unread message

Ikai Lan (Google)

unread,
Oct 13, 2010, 5:42:25 PM10/13/10
to google-a...@googlegroups.com
Hey everyone,

We're working on moving Task Queues out of labs/experimental for an upcoming release slated for November. One of the requirements for doing so is to better enforce the Task Queues storage quota. The documentation describes this quota as storage for data for tasks that have not yet executed:


Currently, this quota is not enforced, and it is possible for applications with backed up task queues to exceed this quota without running into quota denials. Check your admin console's quota page for your application to see if you are reaching or exceeding this quota. If so, you'll have the following options:

1. Allocate more storage quota to taskqueue storage by updating:
         - total_storage_limit in queue.yaml for Python applications
         - TotalStorageLimit in queue.xml for Java applications. 
        Note: you may need to buy more disk quota if there's not enough quota for datastore, blobstore and taskqueue storage altogether.

2. Fix backed up queues using one of the options below. You will recognize these as queues with high numbers of tasks (typically >2000).
        - Purge backed up queues (an easy button click in admin console's Queue Details page for the queue).
        - Increase the execution rate and let your app work through the backlog. To do this, edit queue.yaml (or queue.xml for Java apps) and alter the rates on the queues. You can set a higher bucket size than the default (currently 5) to maximize throughput.
        - Reduce the rate at which new tasks are added to queues. This will involve some editing of app code.

It can take a few hours to update the admin console after the quotas have been updated.

We'll post a reminders to the App Engine blog as well as to this thread as the release nears.

--
Ikai Lan 
Developer Programs Engineer, Google App Engine

Eli Jones

unread,
Oct 13, 2010, 11:51:12 PM10/13/10
to google-a...@googlegroups.com
Ikai,

On occasion I execute thousands of tasks.  Works great too.

But, it seems like Task Queue Stored Task Bytes does not reset very quickly.

Currently my Task Queue shows (and I have 0 Tasks in any Queues.. and haven't for over 10 minutes):

Task Queue Stored Task Count    136,603 of 200,000,000
Task Queue Stored Task Bytes    124,768,866 of 104,857,600

Recently I just ran under 1,000 tasks, and a few hours ago I was running many other tasks.. but they all run and finish within 3 minutes.. so it seems strange that the Stored Task values keep increasing..

How often are these values reset?  Are they not really reset as soon as the Tasks in a queue are finished or purged?

Thanks for any information.


Eli

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

Ikai Lan (Google)

unread,
Oct 14, 2010, 1:49:06 PM10/14/10
to google-a...@googlegroups.com
I believe this are garbage collected. This'll be a gating issue for us before we release - it would be pretty bad if this caused quota denials.


--
Ikai Lan 
Developer Programs Engineer, Google App Engine



vtscout

unread,
Oct 14, 2010, 9:29:06 PM10/14/10
to Google App Engine
Since we are on this topic. Task Quota has 3 items:

Task API Calls: 100,000
Task Stored Task Count: 1,000,000
Task Stored Task Bytes: 104M // This one is understood

Since it does not appear possible to enqueue a new task without
invoking a Task Queue API, it seems like API Call Count is the choking
point. In other words 100K of API calls is not enough to hit Stored
task Count of 1M ever. The only situation in my mind Stored Task Count
can come into play is if Tasks remain pending from before Quotas were
refilled. Could you please explain?

Thanks


On Oct 14, 10:49 am, "Ikai Lan (Google)" <ikai.l+gro...@google.com>
wrote:
> I believe this are garbage collected. This'll be a gating issue for us
> before we release - it would be pretty bad if this caused quota denials.
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> Blogger:http://googleappengine.blogspot.com
> Reddit:http://www.reddit.com/r/appengine
> Twitter:http://twitter.com/app_engine
>
> >> google-appengi...@googlegroups.com<google-appengine%2Bunsu...@googlegroups.com>
> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengi...@googlegroups.com<google-appengine%2Bunsu...@googlegroups.com>
> > .

Eli Jones

unread,
Oct 14, 2010, 9:35:07 PM10/14/10
to google-a...@googlegroups.com
100,000 is just the max for the Free Default Quota.. Billing Enabled gets you Max 20,000,000 Task API Calls.


So, if you use a lot of tasks, you can enable billing with a low daily max (on what you can be billed) and get the 20,000,000 Task API Call limit.

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

Jamie H

unread,
Oct 14, 2010, 10:41:40 PM10/14/10
to Google App Engine
Also you can add tasks in bulk, I believe 100.. so with 1 api call you
could add 100 tasks.

Cameron

unread,
Oct 15, 2010, 11:03:00 AM10/15/10
to Google App Engine
Task Queues coming out of labs is great news! Will performance and
stability improve at this time as well? Looking at the Taskqueue
Status page, every single day for the last 3 months there have been
huge spikes in execution latency. Every single day (I clicked through
them all). So much yellow and red in the graphs. The last time there
was a solid blue graph was July 11, 2010.

http://code.google.com/status/appengine/detail/taskqueue/2010/10/15#ae-trust-detail-taskqueue-execution-latency

I realize Task Queues are for background processing which is how I use
them in my apps. However, when I set the rate at 50 tasks per second
I expect actual throughput to be somewhat close to that rate. When
execution latency jumps up to 25 seconds per task, then essentially
all the benefits of using task queues is lost.

Can we expect improvements in this area when the "experimental" tag is
removed?

-Cameron

On Oct 13, 4:42 pm, "Ikai Lan (Google)" <ikai.l+gro...@google.com>
wrote:

Ikai Lan (Google)

unread,
Oct 15, 2010, 5:41:38 PM10/15/10
to google-a...@googlegroups.com
It's the other way around - it's only when performance has improved that we can take it out of labs. We're investigating issues now regarding backed up tasks.

To answer an earlier question about task queue space: this is swept asynchronously. The expectation should be that space used for tasks will be swept within 12 hours. If you run a lot of tasks, you'll want to be sure you raise the amount of space you can allocate for tasks in your configuration files.


--
Ikai Lan 
Developer Programs Engineer, Google App Engine



--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

风笑雪

unread,
Oct 16, 2010, 12:12:30 AM10/16/10
to google-a...@googlegroups.com
Hi, Ikai

I would like to know whether Google will let us be able to get tasks
in the specified queue, then edit or delete them in the future.

----------
keakon

hawkett

unread,
Oct 17, 2010, 1:58:12 PM10/17/10
to Google App Engine
Hi Keakon,

I raised an issue for editing tasks in the queue around 15 months
ago - http://code.google.com/p/googleappengine/issues/detail?id=1772 -
please star this issue if you would like to see this feature.

The main use case is a task that is generating an error - i.e. it is
stuck in a cycle of 'execute->error->return to queue'. At the moment
the fixes for this are

1. Fix the code that raised the bad task and accept that the broken
job will never complete properly (delete the stuck task)
2. hack the code that receives the task to modify the broken value(s)
- this is pretty hard/risky work

In complex process environment, both options can be very difficult,
especially if the task is part of a long chain - it can be tough to
manually sort out the problem in many situations.

So I agree the use case for this is really only when the app code
has a bug in it, but bugs happen, and this would help in plenty of
situations.

One other way to consider this problem is the implementation of a
dead letter queue - a task that continually fails is moved to the dead
letter queue, to avoid using up too much CPU/API quota if it goes
unnoticed. Editing might be a function of the dead letter queue (along
with relaunching). It is possible to implement this in user space -
i.e. if you receive a task with > 10 retries (for example), then store
it in the datastore, return 200 and implement an admin API and
relaunch capability. Would be great if this was a feature of the task
queue administration screens rather than each app doing a roll-your-
own solution though.

Cheers,

Colin

Ikai Lan (Google)

unread,
Nov 22, 2010, 4:26:11 PM11/22/10
to google-a...@googlegroups.com
Hey everyone,

I just wanted to post some updates since this is coming soon in the
1.4.0 release:

1. Again, we will be enforcing task queue stored bytes. This quota
will now be rolled up into the Stored Data quota and can cause your
app to exceed the Stored Data quota. If you do not adjust your quota
settings, as described in the last message, your app may stop working.

2. Java developers: "TotalStorageLimit" in queue.xml should be
corrected to "total-storage-limit". We've changed this for consistency
(see my earlier post)

3. The quota details for the Task Queue Stored Task Bytes are
available on the Task Queues page (and not the Quota Details page as
mentioned in the last message).

Let us know if you have any questions.

Will

unread,
Nov 22, 2010, 8:47:51 PM11/22/10
to google-a...@googlegroups.com
Will the 'google.appengine.api.labs' still be valid for a while? What is the name of the new package?

Thanks,

Will

Ikai Lan (Google)

unread,
Nov 22, 2010, 9:04:19 PM11/22/10
to google-a...@googlegroups.com
That package should still be valid for a few releases, though we should be logging deprecation warnings. The new package will drop the "labs" namespace.


--
Ikai Lan 
Developer Programs Engineer, Google App Engine



GAE Developer Wanted

unread,
Jan 14, 2011, 5:07:49 PM1/14/11
to google-a...@googlegroups.com
I am looking for a programmer to develop a dating site running on Google Apps Engine. Interested, please contact me at chinesesociopolitics at ............. gmail com

devon....@yahoo.com

unread,
Feb 13, 2011, 5:41:13 PM2/13/11
to google-a...@googlegroups.com
i have bought an app from you (onslaught arena) and it shows up with this error what does it mean!??!

Error: Server Error

The server encountered an error and could not complete your request.

If the problem persists, please report your problem and mention this error message and the query that caused it. 

DokiTops

unread,
Feb 14, 2011, 2:32:10 AM2/14/11
to google-a...@googlegroups.com
I also get this error enetering this app: 

Error: Server Error

The server encountered an error and could not complete your request.

If the problem persists, please report your problem and mention this error message and the query that caused it.

Then I end up here.

Message has been deleted

Darien Caldwell

unread,
Feb 14, 2011, 11:37:37 AM2/14/11
to Google App Engine
It's kind of humorous you guys ended up here with that question. The
error message is related to the service that 'app' was written on, and
not an error with the App itself. It should clear up on it's own.

The person who wrote the 'app' shouldn't be passing that message on to
end users, as it's obviously confusing. You should be directing your
questions to the creator of the App, not Google. They didn't make the
app. Obviously the error says to come here though.

Geoffrey Spear

unread,
Feb 14, 2011, 12:41:52 PM2/14/11
to google-a...@googlegroups.com
The app developer doesn't do anything to pass those messages on; that's the default 500 error page for all apps. It is, of course, almost certainly never helpful to direct end users here for help.

kha

unread,
Feb 21, 2011, 3:50:21 AM2/21/11
to google-a...@googlegroups.com
yes, I programming a Data about Google apps Engine...........!!why? Do you want my Skills in Programming!!!!!!!!!!!,,,,,,,,,,,,,,????????????

kha

unread,
Feb 21, 2011, 3:51:06 AM2/21/11
to google-a...@googlegroups.com
hi!!!!!!!!!

kha

unread,
Feb 21, 2011, 3:51:54 AM2/21/11
to google-a...@googlegroups.com
where cames you from!???????????????

alarm

unread,
Mar 8, 2011, 6:19:13 AM3/8/11
to google-a...@googlegroups.com
Well great job!

Now I can see that I've been warned here, but my live application is broken and I'm loosing customers until I will be able to find my app sources and  do the testing.

Dear Google, please think about backward compatibility next time you pull the plug for some api in production system!

Thanks!

alfher

unread,
Mar 8, 2011, 7:53:09 AM3/8/11
to google-a...@googlegroups.com
my appalication gods war was error pls..build it


alfher

unread,
Mar 8, 2011, 7:54:03 AM3/8/11
to google-a...@googlegroups.com
ahhm

raka prasetyo

unread,
Mar 8, 2011, 7:57:26 AM3/8/11
to google-a...@googlegroups.com
my godz war game error. please fix it soon as possible

Ikai Lan (Google)

unread,
Mar 8, 2011, 1:06:12 PM3/8/11
to google-a...@googlegroups.com
All,

The error is not related to moving task queues out of labs. Please follow this thread for details:


We should be publishing a post-mortem.

That being said, we are moving task queues out of labs. Labs features are subject to API changes. Please prepare your applications for the package naming changes and quota changes described in this thread.

Ikai Lan 
Developer Programs Engineer, Google App Engine



On Tue, Mar 8, 2011 at 4:57 AM, raka prasetyo <raka...@gmail.com> wrote:
my godz war game error. please fix it soon as possible

--

Brandon Donnelson

unread,
Mar 21, 2011, 11:07:44 AM3/21/11
to google-a...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages