scheduler: task concurrency

[Paolo Valleri]

Dear all,

I've been using the scheduler for a long time, and it worked very well.

Now I need to instantiate several workers to run several tasks, few of them are  light, other will take several minutes to complete.

The problem I have is that: few of them can be executed in concurrency, other no.

For example: 

given two workers (A and B) and the task 'test': in the scenario where A is executing the task 'test', B is free and it is the time to execute the task 'test' again:

if test has been defined to be executed in parallel: B will start to run the task 'test'

if test has been defined to avoid parallel executions: B will never start to run 'test'.

That behavior can be reached by adding a new option that defines if a task can be executed in concurrency or not.

Thoughts?

Paolo

[Niphlod]

let's say that "parallel" is the default behaviour....

when you need to run a task to avoid parallel (for speech sake, let's turn to "concurrent" ) executions..... in respect to what ? any other task? one task only ? all tasks named as 'foo' or 'bar' ?

The possibilities for creating a workflow for tasks are endless, but it's not going to be as easy as adding a "boolean" to the task options...

[Paolo Valleri]

I don't think about arbitrary workflow, I would only avoid concurrent execution of the same task.

 Paolo

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/BGwg4ky6QB4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Niphlod]

how do you identify "the same task" ? the same function name ?

[Paolo Valleri]

yes, the function name identifies the same task

 Paolo

[Niphlod]

What's the "general" use case ?
There are a bunch of simple different things you can do right in the current scheduler to achieve that goal....
- schedule the same with different start_time(s)
- queue the first with enabled=True and the following with enabled=False with a callback that flips the enabled bit for the next one
- use an external "lock" record to coordinate
- use a task that coordinates "subtasks" (always flipping the enabled bit)
- etc etc etc

TBH, I see other things as useful for a lot of cases (i.e. tasks dependencies, e.g. task A needs to run before task B)....right now this feels like a corner case. It's also a bit difficult to implement without external tables, but that's another story....if it's useful we'll make it work ^_^

[Paolo Valleri]

I knew there are bunch of different approaches to implement what I need, the one I prefer is to use enabled=False/True but by doing that we are moving that feature (in my view it belongs to the scheduler) in the function code. In that case, I have to reinvent the wheel, actually I have also to hard core it, for each function I have to run in that manner. Moreover, I don't have much control. 

I can create a wrapper to flip the enable=True/False, but at that point, the wrapper could be placed in the scheduler, by setting enable=False before executing tasks defined as 'concurrent=False'? isn't?

I agree with you, this is not a corner case, and to be honest, I also need tasks dependencies for other elaborations :P 

 Paolo

--

[Niphlod]

Sorry, I missed the general usecase scenario ... and come on, it's not that difficult to implement a wrapper with a "flags" table, even for a general approach... or an "update the first queued task named 'foo'" right at the end of the 'foo' function...it takes maybe 10 lines of code at best.
Actually, it would be a lot better (and safer) to code a repeating task that fetches args/vars from a queue stored in a different table (because if you need to coordinate the same task to not run more than 1 at a time, I figure that the only thing that changes is args/vars?! ).