how to make the firetasks via the same firetask class but created by multiple executions of my python script unique

[Alex Li]

Dear Fireworks support,

With Anubhav's and my teammate's help, I am able to run subfireworks in parallel. Thanks!

Now I have a different question:

I have a python script that creates a few subworkflows and processes the input data and writes the output files into the unique directories each time it runs.

It runs fine alone (ie no concurrent multiple executions of the same script). However, if I run the same script twice with the different parameters at the same time (or their run times overlap), one would interfere another, eg: the second run would try to use the output files from the first run, which should not as it should use its own unique files.

I played a few tricks, eg: by giving each subworkflow a unique fw_id, name, and by setting the unique values (input/output files) in the "spec" section, but none worked. I also try to set "_dupefinder" in "spec" to "DupFinderExact" in hoping that different "spec" will somehow make the fireworks unique, but then I got this error:

  File "/opt/common/compbio/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap

    self.run()

  File "/opt/common/compbio/lib/python2.7/multiprocessing/process.py", line 114, in run

    self._target(*self._args, **self._kwargs)

  File "/opt/common/compbio/lib/python2.7/site-packages/fireworks/features/multi_launcher.py", line 70, in rapidfire_process

    rapidfire(launchpad, fworker, None, nlaunches, -1, sleep, loglvl)

  File "/opt/common/compbio/lib/python2.7/site-packages/fireworks/core/rocket_launcher.py", line 68, in rapidfire

    while skip_check or launchpad.run_exists(fworker):

  File "<string>", line 2, in run_exists

  File "/opt/common/compbio/lib/python2.7/multiprocessing/managers.py", line 773, in _callmethod

    raise convert_to_error(kind, result)

AttributeError: 'unicode' object has no attribute 'query'

So could you please help again? Basically I want to make the fireworks think that the firetasks via the same firetask class created by multiple executions of my python script unique.

Hope that my question is not too confusing.

Thanks in advance!

Alex.

 

[Anubhav Jain]

Hi Alex,

Sorry, but I am not able to understand your question.

Note that duplicate checking is *off* by default. So in the default case, you can add identical workflows 100 times and FWS will run it 100 times. It will not try to do anything clever unless you explicitly turn duplicate checking on via the "_dupefinder" spec key

Looking at the error, it looks like it is unrelated to your description. Instead, it looks like "fworker" object being passed into launchpad.run_exists() is a string/unicode object instead of a proper FWorker object. Are you sure you are passing an FWorker object into your launch, e.g. launch_multiprocess()? If not, please do this and refer to the comment strings to see the type of object needed to be passed.

Best,

Anubhav

--
You received this message because you are subscribed to the Google Groups "fireworkflows" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fireworkflow...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/dae8a09f-a5be-4217-a43c-d894bb826f36%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Li]

Anubhav,

Thanks for clarifying the default behavior of dupe checking! 

There is no internal collision inside a script (ie: calling the same firetask module in two subworkflows in one python script is fine). However, there is somehow a cross-collision between the same subworkflow in two concurrent executions of the same script (using the different parameters and writing to different output files). I will try to narrow down the problem and come up with a simplest script that can reproduce the problem.

Regards,

Alex.

[Anubhav Jain]

>>  I will try to narrow down the problem and come up with a simplest script that can reproduce the problem

Ok, I think that will be needed in order to debug

Best,

Anubhav

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/8362d66b-c3e2-4923-a97c-cb084f3c8cc1%40googlegroups.com.

[Alex Li]

Anubhav,

I think that we have solved the problem.

Problem recap:

When two python scripts (can be the same script with the different input/output data) run at the same time, one script tries to run the fw_ids created by anther script, and vice versa.

How we solved the problem:

When creating subfirework objects, pass a session unique string to fireworks objects

Pass this session string to launchpad when creating the launchpad object

Modified Method _get_a_fw_to_run() in launchpad class to make it optionally use the session string to narrow down the mongodb results to the session's fw_ids.

Suggestion for future development:

To optionally allow a python script to run the jobs for the fw_ids it has created ONLY, and not run the jobs for the fw_ids created by another python script.

Thank you very much anyway.

Alex

[Anubhav Jain]

Hi Alex

One of the points of having a database is so that multiple scripts can enter jobs and multiple scripts can execute jobs. To be clear, this is not a "problem" in the code. If you want to maintain separate lists of jobs, you can either have two separate databases or use the FWorker features to pull only the appropriate jobs. No "hacks" are needed. For details on the second strategy see the second half of this documentation for setting categories of jobs:

https://pythonhosted.org/FireWorks/controlworker.html

I'm happy that you found a solution but I also don't want to propagate a notion that such a change to the code is necessary. The built in tools let you easily control what jobs get pulled by what scripts.

Best,

Anubhav

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/39095136-a314-4f47-a6e1-134430d11bde%40googlegroups.com.

[Alex Dong Li]

Anubhav,

I tried setting the "_category" to runId and in "spec" and use FWorker(category=runId) and it worked!

I guess that I did not understand how the categroy is used when I was reading the documentation before.

Thanks for the information!

Regards,

Alex.