doit auto with other tasks running in multiple processes

[jan.vl...@gmail.com]

I have couple of tasks, one being long running process. I would like to use "auto" option to rerun the processing if something change, but due to long running process I need to run the processing in multiple processes otherwise all processig gets stuck with my long term task.

Question: How to run doit with auto and at the same time instruct it to run with multiple processes?

More details:

My current dodo.py file is fetching files from mulitple urls, merging them and posting to AWS S3.
The transformation involves couple of processes, some being expensive, so I am happy, that doit detects, when input files do not change and expensive process does not have to run again.

To keep the things moving on on regular periods, I added file denendency on now.txt file. At this moment I am changing the file manually and keep doit running with auto. This way all works nice.

I want to add longrunning task into dodo.py, which would modify the file now.txt automatically once uppon a time.

Unfortunatelly I did not find a way to start doit with auto ($ doit auto publish) and at the same time ask to use multiple processes.

Is that possible?

With best regards

Jan Vlčinský

[Eduardo Schettino]

On Mon, Mar 7, 2016 at 9:01 AM, <jan.vl...@gmail.com> wrote:

I have couple of tasks, one being long running process. I would like to use "auto" option to rerun the processing if something change, but due to long running process I need to run the processing in multiple processes otherwise all processig gets stuck with my long term task.

Question: How to run doit with auto and at the same time instruct it to run with multiple processes?

It is not supported. But even if was implemented it would probably not work well for your use case.

Using `auto` it keeps watch for file modifications but process those only after the current iteration is over.

More details:

My current dodo.py file is fetching files from mulitple urls, merging them and posting to AWS S3.
The transformation involves couple of processes, some being expensive, so I am happy, that doit detects, when input files do not change and expensive process does not have to run again.

To keep the things moving on on regular periods, I added file denendency on now.txt file. At this moment I am changing the file manually and keep doit running with auto. This way all works nice.

I want to add longrunning task into dodo.py, which would modify the file now.txt automatically once uppon a time.

Have you seen: http://pydoit.org/uptodate.html#timeout

 

Unfortunatelly I did not find a way to start doit with auto ($ doit auto publish) and at the same time ask to use multiple processes.

Is that possible?

By your description it seems the long running task has no direct relation with other tasks...

So can you just split the long running as a separate "instance". I mean have 2 different `doit` processes running.

cheers

[Jan Vlčinský (TamTam Research)]

Eduardo

Thanks for quick response (and really nice tool).

As you proposed, I ended up with two processes, one being based on doit, the other simply changing content of now.txt file once in 60 seconds.

The timeout uptodate function sounds interesting, but I guess, it would take effect only after I would start the doit jobs later on. My current solution (changing now.txt file by external process) has an advantage, that it "kicks" the jobs via "watch" feature (I know, it will start processing not sooner then previous jobs will be completed, this is fine).

Regarding long running task relationship with other ones - logically there is relationship (via now.txt), but technically from doit point of view it is not. There is the contradiction of the task being long running (in other words never finished) and the fact, doit shall at some moment evaluate results of the task (which is probably when it finishes - thus never).

Remaining question: is there an option to use "auto" and let doit use 2 or more processes? In my case the "auto" would schedule processing and "-n 4" would allow to do the processing faster when the time comes.

Alternate solution for me would be to run doit (with "-n 4") in a loop, or within a process being driven by watchdog or similar solution.

cheers

Jan Vlčinský
TamTam Research s.r.o.
Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic
tel: +420-5588 49 736; mob: +420-608 979 040
http://www.tamtamresearch.com

--
You received this message because you are subscribed to the Google Groups "python-doit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-doit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eduardo Schettino]

On Tue, Mar 8, 2016 at 1:55 AM, Jan Vlčinský (TamTam Research) <jan.vl...@tamtamresearch.com> wrote:

Eduardo

Thanks for quick response (and really nice tool).

Glad you like. I am running a donation campaign in case you feel generous :D
https://www.indiegogo.com/projects/doit-python-automation-tool-release-0-30/x/2922409#/

 

Remaining question: is there an option to use "auto" and let doit use 2 or more processes? In my case the "auto" would schedule processing and "-n 4" would allow to do the processing faster when the time comes.

Currently there is no option. But I guess this should be just a matter of actually add the command argument as

the internal API already supports this. *Someone* need to give it a try to double check.

Regards,

  Eduardo