Sorry to intrude,
Do not know much about this automation - workflow tool but have couple of questions, hopefully someone can help...
I have pretty complex application stack that consist of templates, execution harness, decoupled drivers, tons of inputs.
driver -. statemachine (automata) -> collect info -> setup jobs (tasks)
task consist of python objects and arguments for execution of these objects
tasks should be queued to some "bus" or storage option...
I can have from tens to thousands of these tasks.
worker infrastructure is passive until started. When started workers will take one task from the queue, execute it return success or failure and dump payloads to NFS.
Parallelization is achieved on the worker side, essentially horizontal scaling of worker nodes (servers)
if the worker node is done, or has metered resources, worker node will pick more than one task and process them in parallel.
on worker nodes, each process is its own python process. Task are completely independent of each other.
When whole queue is drained, driver will collect stuff and process it in single shot.
I rad documentation briefly. Framework looks great. I did this in past with fireworks framework and mongoDB.
if someone can point me to documentation for parallelization and ability to do above i would much appreciate it ...