AppEngine Timeout with Task Queues

[Nathaniel Payne via StackOverflow]

Might be a little too general, but here are some thoughts that might help close the loop. There are 2 kinds of task queues, push queues and pull queues. Push queue tasks execute automatically, and they are only available to your App Engine app. On the other hand, pull queue tasks wait to be leased, are available to workers outside the app, and can be batched.

If you want to configure your queue, you can do it in the queue config file. In Java, that happens in the queue.xml file, and in Python that happens in the queue.yaml file. In terms of push queues specifically, push queue tasks are processed by handlers (URLs) as POST requests. They:

1. Are executed ASAP
2. May cause new instances (Frontend or Backend)
3. Have a task duration limit of 10 minutes
4. But, they have an unlimited duration if the tasks are run on the backend

Here is a quick Python code example showing how you can add tasks to a named push queue. Have a look at the Google developers page for Task Queues if you need more information: https://developers.google.com/appengine/docs/python/taskqueue/

Adding Tasks to a Named Push Queue:

queue = taskqueue.Queue("Qname")
task = taskqueue.Task(url='/handler', params=args)
queue.add(task)

On the other hand, let's say that you wanted to use a pull queue. You could add tasks in Python to a pull queue using the following:

queue = taskqueue.Queue("Qname")
task = taskqueue.Task(payload=load, method='PULL')
queue.add(task)

You can then lease these tasks out using the following approach in Python:

queue = taskqueue.Queue("Qname")
tasks = queue.lease_tasks(how-long, how-many)

Remember that, for pull queues, if a task fails, App Engine retries it until it succeeds.

Hope that helps in terms of providing a general perspective!

Please DO NOT REPLY directly to this email but go to StackOverflow:
http://stackoverflow.com/questions/19981918/appengine-timeout-with-task-queues/22877823#22877823