How to specify a number of workers?

[Roman M]

Stumbled upon Fireworks today, and have a newbie question.  Thanks in advance.

I have eight cores on my machine, my workflow has a diamond shape.  I first split work into 1000 tasks, calculate 1000 in parallel and then join the results together.  I don't quite understand, where I specify the number of concurrent workers, I want to set it to 8. 

If someone can point me to an example in python, that would be great.

-Roman

[Anubhav Jain]

Hi Roman,

If you are running on a single machine, typically you want to just use a single Worker unless you have different job types and want the same machine to handle two different categories of jobs differently. In your case, I think you want to keep a single Worker but use the multi-launch:

https://pythonhosted.org/FireWorks/multi_job.html

Let me know if it doesn't solve your problem

Best,

Anubhav

--
You received this message because you are subscribed to the Google Groups "fireworkflows" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fireworkflow...@googlegroups.com.
To post to this group, send email to firewo...@googlegroups.com.
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/11f9fa81-6d95-4273-b1ab-e8661aa72cd8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Anubhav Jain]

Hi Roman,

With the help of Xiaohui Qu, we noticed one important followup to this conversation. The multi-launch script works by spawning multiple parallel workers (e.g., one on each core when parallelizing over a single processor). Each worker functions as normal, but sometimes this causes behavior that looks strange.

For example, it you have a diamond shaped workflow:

1 -> 2,3 -> 4

and you run multi-launch with num_jobs = 2, you would expect Fireworks 2 and 3 to run in parallel over the 2 workers. However, what happens in practice is:

* 2 parallel workers start in the beginning

* Worker A starts Firework 1

* Worker B sees nothing to run (since Firework 1 is not yet finished, and other Fireworks depend on Firework 1)

* With the nlaunches setting set to 0 (default), Worker B quits since it sees no jobs available to run and nlaunches=0 means a worker stops when there is nothing left to run.

* Now, only Worker A is left, and things do not run in parallel since Worker B has already quit.

The easiest fix is to set nlaunches to "infinity" or equal to a large number. In the future it might be nice to have other options, e.g. to have Worker B quit only if no jobs can be found for N minutes and not to immediately quit if there are no jobs (or better, if there is nothing left waiting to run within the constraints of Worker B).

I hope this helps address some of the issues we were seeing in a private conversation.

Best,

Anubhav

On Wednesday, March 30, 2016 at 10:52:28 PM UTC-7, ajain wrote:

Hi Roman,

If you are running on a single machine, typically you want to just use a single Worker unless you have different job types and want the same machine to handle two different categories of jobs differently. In your case, I think you want to keep a single Worker but use the multi-launch:

https://pythonhosted.org/FireWorks/multi_job.html

Let me know if it doesn't solve your problem

Best,

Anubhav

On Wed, Mar 30, 2016 at 9:44 PM, Roman M <roma...@gmail.com> wrote:

Stumbled upon Fireworks today, and have a newbie question.  Thanks in advance.

I have eight cores on my machine, my workflow has a diamond shape.  I first split work into 1000 tasks, calculate 1000 in parallel and then join the results together.  I don't quite understand, where I specify the number of concurrent workers, I want to set it to 8. 

If someone can point me to an example in python, that would be great.

-Roman

--
You received this message because you are subscribed to the Google Groups "fireworkflows" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fireworkflows+unsubscribe@googlegroups.com.

[Nick Vandewiele]

Hi,

I am a bit confused about what can be parallelized and what not through the mlaunch application:

The documentation mentions that independent workflows can be parallelized, whereas the reply by Anubhav suggests that individual Fireworks are run in parallel, when the structure of the workflow permits it.

Should I interpret the term workflow and firework as interchangeable here?

Thanks!

[Anubhav Jain]

Hi Nick,

Sorry for the confusion. Usually we are careful in the docs to properly distinguish Firework vs workflow. In this case, the docs were written a bit sloppily.

The parallelization is over Fireworks, which can be either within the same Workflow or across several Workflows. Each parallel worker just pulls available (=READY) Fireworks and runs them regardless of what Workflow they are in.

I will update the docs soon to be more clear.

Best,

Anubhav

--
You received this message because you are subscribed to the Google Groups "fireworkflows" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fireworkflow...@googlegroups.com.

To post to this group, send email to firewo...@googlegroups.com.
Visit this group at https://groups.google.com/group/fireworkflows.

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/45421449-9be9-4c60-9ad3-176847844136%40googlegroups.com.

[Anubhav Jain]

Ok I took a stab at updating the docs:

http://pythonhosted.org/FireWorks/multi_job.html

Let me know if anything is still unclear.

Best,

Anubhav

On Friday, April 15, 2016 at 9:09:31 AM UTC-7, ajain wrote:

Hi Nick,

Sorry for the confusion. Usually we are careful in the docs to properly distinguish Firework vs workflow. In this case, the docs were written a bit sloppily.

The parallelization is over Fireworks, which can be either within the same Workflow or across several Workflows. Each parallel worker just pulls available (=READY) Fireworks and runs them regardless of what Workflow they are in.

I will update the docs soon to be more clear.

Best,

Anubhav

On Fri, Apr 15, 2016 at 9:04 AM, Nick Vandewiele <nickvan...@gmail.com> wrote:

Hi,

I am a bit confused about what can be parallelized and what not through the mlaunch application:

The documentation mentions that independent workflows can be parallelized, whereas the reply by Anubhav suggests that individual Fireworks are run in parallel, when the structure of the workflow permits it.

Should I interpret the term workflow and firework as interchangeable here?

Thanks!

On Thursday, March 31, 2016 at 12:44:01 AM UTC-4, Roman M wrote:

Stumbled upon Fireworks today, and have a newbie question.  Thanks in advance.

I have eight cores on my machine, my workflow has a diamond shape.  I first split work into 1000 tasks, calculate 1000 in parallel and then join the results together.  I don't quite understand, where I specify the number of concurrent workers, I want to set it to 8. 

If someone can point me to an example in python, that would be great.

-Roman

--
You received this message because you are subscribed to the Google Groups "fireworkflows" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fireworkflows+unsubscribe@googlegroups.com.

To post to this group, send email to firewo...@googlegroups.com.
Visit this group at https://groups.google.com/group/fireworkflows.