How to controll the order of WF to be excuted

[ling...@gmail.com]

If I have a work flow with multiple fireworks, and I would like to run this WF for multiple times. 

some thing like this:

launchpad.add_wf(wf)

launchpad.add_wf(wf)

launchpad.add_wf(wf)

rapidfire(launchpad)

But I noticed that it would run the first firework in each WF first, even though it runs one firework at a time, instead of finishing one WF then starting next one.

What should I do to make it finish WF one by one, instead of starting all the WFs at one time?

Thanks,

Ling

[Anubhav Jain]

https://pythonhosted.org/FireWorks/priority_tutorial.html

--
You received this message because you are subscribed to the Google Groups "fireworkflows" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fireworkflow...@googlegroups.com.
To post to this group, send email to firewo...@googlegroups.com.
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/f1ad8032-6b59-4f37-9970-85483d1c3009%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ling...@gmail.com]

Sorry, I didn't metion that I know the job priority can do that.

But our situation is different. We can't set that up in the WF. We have a service to run the WF, so it is quite ofter user send mutiple requests to run the WF at similar time. The request would add the WFs to lpad, then rlaunch is starting the WFs. If you see several requests come in at the similar time, Fireworks runs the first task in multiple WFs one at time, instead of finishing one WF then starting next one. 

Do you have any suggessiont to deal with this? Thanks, 

Ling

[Anubhav Jain]

Hi Ling

Sorry, but I do not understand your question or use case.

There are only a couple of ways that FireWorks can set job priority - either by setting the priority within each Firework (which can be done either at the time the workflow is created, or can be added at anytime using the lpad set_priority command), or by setting the order of workflow pulling as FIFO vs FILO in the FWConfig.

If users are submitting the workflows themselves, you can write a function that finds jobs with no priority set and assign priorities to each job such that "root" Fireworks have lower priority and "children" Fireworks have higher priority. Or you can write a function that takes in any workflow and adds priorities to each Firework, and ask users to run their workflows through that function before submitting them. e.g. 

def add_priority(original_wf, root_priority, child_priority=None):
    """
    Adds priority to a workflow

    Args:
        original_wf (Workflow): original WF
        root_priority (int): priority of first (root) job(s)
        child_priority(int): priority of all child jobs. Defaults to
                            root_priority

    Returns:
       (Workflow) priority-decorated workflow
    """

    child_priority = child_priority or root_priority
    root_fw_ids = original_wf.root_fw_ids
    for fw in original_wf.fws:
        if fw.fw_id in root_fw_ids:
            fw.spec["_priority"] = root_priority
        else:
            fw.spec["_priority"] = child_priority
    return original_wf

​

If you run a workflow through that function, and set the child priority higher than the root priority, it will finish that workflow before starting another workflow with the same root priority.

Best,

Anubhav

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/6d79cc86-60a6-4b9e-9d66-cfdec60197e8%40googlegroups.com.

[ling...@gmail.com]

Thanks a lot, Aunbhav. 

I tried to set up FIFO in the FWConfig, and set the first Firework priority as 1 and the rest of Firework as 10 in my WF, then add a few the WF to lpad, the do rapidfire. It still runs the first Firework in each WF first, then finish the rest in the WF one by one.

Any more thoughts to fix this. 

Really appreciate your help.

Ling

[Anubhav Jain]

Hi Ling

>> set the first Firework priority as 1 and the rest of Firework as 10 in my WF, then add a few the WF to lpad, the do rapidfire. It still runs the first Firework in each WF first, then finish the rest in the WF one by one.

Any more thoughts to fix this. 

If the priority is set correctly (e.g., as the "_priority" key in the spec - don't forget the underscore) - then this should not happen. Can you attach your code that you are using to test this?

Best,

Anubhav

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/d83f8daf-15e0-47b4-b1d0-c769c290dd64%40googlegroups.com.

[ling...@gmail.com]

from fireworks import LaunchPad, PyTask, ScriptTask, Workflow

from fireworks.core.firework import FWAction, Firework, FireTaskBase, Workflow

from fireworks.core.rocket_launcher import rapidfire

from fireworks.utilities.fw_utilities import explicit_serialize

import time

@explicit_serialize

class exceptionTask(FireTaskBase):

    def run_task(self, fw_spec):

        raise NameError("just to throw an exception")

@explicit_serialize

class testTask(FireTaskBase):

    def run_task(self, fw_spec):

        fireworks = []

        name1 = "sleep 1"

        fw1 = Firework(ScriptTask.from_str("sleep 15"), spec={"_priority": 10},

                       name=name1)

        fireworks.append(fw1)

        name1 = "sleep 2"

        fw2 = Firework(ScriptTask.from_str("sleep 15"), spec={"_priority": 10},

                       name=name1,

                       parents=[fw1])

        fireworks.append(fw2)

        name1 = "sleep 3"

        fw3 = Firework(ScriptTask.from_str("sleep 15"), spec={"_priority": 10},

                       name=name1,

                       parents=[fw2])

        fireworks.append(fw3)

        links = {}

        for i in range(0, len(fireworks) - 1):

            links[fireworks[i]] = fireworks[i + 1]

        workflow = Workflow(fireworks, links_dict=links)

        return FWAction(detours=workflow)

@explicit_serialize

class endTask(FireTaskBase):

    def run_task(self, fw_spec):

        time.sleep(30)

launchpad = LaunchPad()

fireworks = []

name = "start"

sfw = Firework(ScriptTask.from_str("sleep 15"), name=name, spec={"_priority": 1})

fireworks.append(sfw)

name = "detours"

dfw = Firework(testTask(), name=name, parents=[sfw])

fireworks.append(dfw)

name = "end"

efw = Firework(ScriptTask.from_str("sleep 15"), name=name, spec={"_priority": 10},

               parents=[dfw]

               )

fireworks.append(efw)

wf = Workflow(fireworks, name="test detours")

launchpad.add_wf(wf)

launchpad.add_wf(wf)

launchpad.add_wf(wf)

launchpad.add_wf(wf)

launchpad.add_wf(wf)

rapidfire(launchpad)

[Anubhav Jain]

Hi Ling

You did not set a priority for the "detours" Firework. As stated in the docs, Fireworks that have no priority set will be run last. So your "detours" Firework is waiting for all the other Fireworks in all the other workflows to finish first since there is no priority at all set for this Firework.

You can fix your code just by changing this line to add a priority to dfw:

dfw = Firework(testTask(), name=name, parents=[sfw], spec={"_priority": 10})

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/ff70b048-68df-44de-923b-7fd8921605fd%40googlegroups.com.