Task Queue feature

[Mahron]

Hi,

I need a way for certain tasks from the queue to not execute in parallel to avoid contention. I figured out the best way to do that was with a pull queue and a backend filtering tasks to avoid that.

I was wondering if it would be possible to add a feature to do such a think automatically, with tasks having a specific group id, preventing them from executing in parallel but one after the other. I think it would be a great feature for avoiding contention issues.

[Nicholas Verne]

You can set a queue's max_concurrent_requests in queue.yaml
(max-concurrent-requests in queue.xml). You could try adding these
tasks to their own queue where max_concurrent_requests is 1.

Nick Verne

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/I7zg4SezX_4J.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Mahron]

Hi Nick,

that is what I am currently doing. The problem is it doesn't scale.

[Nicholas Verne]

In a pull queue, you can use the task tagging feature

https://developers.google.com/appengine/docs/python/taskqueue/overview-pull#Leasing_Tasks

See the section marked Experimental. This sounds like it could fit
your use case well.

Nick Verne

On Mon, Jun 18, 2012 at 10:43 AM, Mahron <gan...@xehon.com> wrote:
> Hi Nick,
>
> that is what I am currently doing. The problem is it doesn't scale.
>

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[Mahron]

No, unfortunately that doesn't help either.

Basically, there will be one or even two tasks for every user request
but only a small amount of them are not allowed to run in parallel. So
tasks will be coming at a faster rate than user requests. I need a
system to threat them in batches and filter those which should not run
at the same time and run the batch in parallel for speed.

Example :

queue of 25 tasks. 2 of those tasks of a specific type(group) should
not run in parallel. and another 3 tasks of an other type should not
run in parallel. (The group types are two many to make individual
queues for them)

So what needs to be done is to kick out one of the tasks from the
first group and 2 from the second groups so that the batch has no two
tasks of the same group. then execute that batch in threads and move
on to the next.

The only way I see to do that in a clean way is a pull queue and a
backend that is always on which is costly. It could be done much more
efficiently if it were a queue feature. Such a thing would eliminate
all risks of reading/modifying/writing at the same time a creating
inconsistent data.

[Nicholas Verne]

If you have enough work to get through, a resident backend using a
pull queue won't be more costly in instance hours than the push
scenario you describe.

[Mahron]

Yes, it will most probably be cheaper under high and continuous work
loads. I am building a web service so I don't know when, or even if,
it will reach those numbers but I still need to be prepared for it.
And beyond the starting price, implementing this directly in the task
queue removes additional moving parts from the app, the chances of
something going wrong are strongly reduced.

Not sure how complicated that would be.
Filed an issue anyway : http://code.google.com/p/googleappengine/issues/detail?id=7709
hope you will look into it.