Change DOIT_CONFIG while running

[Harry Jubb]

Hello,

My name's Harry, I'm a bioinformatics scientist using doit for 2-3 years for analysis pipelines and workflows. Thank you for making it, it's been very helpful.

I've looked through the docs, code and Google but haven't found an answer to this, but apologies if I've missed something.

I'm interested in being able to change the DOIT_CONFIG while doit is running. Specifically, I'd like to be able to change DOIT_CONFIG['num_process'] without stopping my pipeline, so I can reduce the load of my pipeline on the server if other users need to run jobs.

In GNU parallel I would do this by having the number of jobs set in a file, then if the number changes, parallel catches this and doesn't spawn new jobs until it's below the limit. However, I'm trying to move to using doit subtasks for parallelisation rather than switching out to parallel.

Is this currently possible in doit?

Many thanks in advance for any answers,

Harry

[Eduardo Schettino]

Hi Harry,

Currently not possible. The parallel running of doit is quite limited in terms of control.
There are already some issues related to better manage it.
And interactive control as you suggested would also be a nice addition.

I dont use parallel very much myself, and dont have time to work on it.

But if you are more than welcome to contribute.

cheers,

  Eduardo

--
You received this message because you are subscribed to the Google Groups "python-doit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-doit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Harry Jubb]

Hi Eduardo,

Thanks for your response, that's good to know, if I can contribute I will :)

Best,

Harry

To unsubscribe from this group and stop receiving emails from it, send an email to python-doit...@googlegroups.com.