scheduler and long tasks

[Pierre]

Hi,

right now I have a single task that runs once a day and am wondering if it would be a good idea to split it into "atoms"/ smaller bits soas to take advantage of work division ( use several workers instead of just one). I can't figure out how to do this since all the work to do depends on a single db(db.....).select
so actually the task behaves like so :
(1) select a bunch of rows
(2) aggregate rows into atoms/groups
(3) process every atom

It's not clear to me wether it's a good thing  to have more than one worker to  gain performances and in such case how to  distribute work related to a single select accross workers ?

[Niphlod]

You can have a task that selects and groups and queues for each group the task that processes the atom.

[Pierre]

so far works :)

is there any restriction to queueing_task inside of a task ?
is every task independant from the other tasks ?  (any hierarchical relation between "inner tasks" and "outer tasks" ?)

[Niphlod]

nope. Technically, tasks are queued as long as the "outer" thingy (be it whatever) get all of the pending transactions committed (a task IS queued when the corresponding "insert" is committed to the database).

Every task gets processed according by the usual rules, but if you want coordination among them (i.e. task 5 needs to be executed after task 7), you can code a "coordinator" task (i.e. that waits for 5 to be completed before queuing task 7) or use the included JobGraph factory.

[Pierre]

thanks Niphlod

right now coordination isn't a problem.......
I have 2 workers and they seem to cooperate well.....