How to let doit not redo tasks that have already been executed but the .doit.db was not updated

[soumi...@gmail.com]

Hi,

I implemented a pipeline using doit. It does execute a large number of tasks and takes more than a week to execute.

Last week I started the pipeline and it was almost done. However, there was a recent reboot at the server and doit lost track of what has been already executed.

Today when I started executing the pipeline again by restarting doit, it restarted from the beginning. It seems that doit will re-execute the whole pipeline just because the .doit.db was not updated. In fact most of the targets are already built and not needed to be rebuilt.

Is there a way to let doit know skip these already executed tasks?

Thank you,

Soumitra 

[Eduardo Schettino]

Hi,

doit command line was NOT designed to run week-long pipelines...

It was my intention to develop a web-based UI. But that requires a considerable effort and I never got to that.

I suggest you use another task runner to control doit execution.

Regards

--
You received this message because you are subscribed to the Google Groups "python-doit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-doit...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python-doit/09bb08db-b2c0-4ae3-b75b-ab22195d00b1n%40googlegroups.com.

[Eduardo Schettino]

On Sat, Aug 1, 2020 at 6:09 AM soumi...@gmail.com <soumi...@gmail.com> wrote:

 In fact most of the targets are already built and not needed to be rebuilt.

Is there a way to let doit know skip these already executed tasks?

Check the command "doit reset-dep", that might work in this case.

[Soumitra Pal]

Hi Eduardo,

The command "doit reset-dep" did exactly what I was looking for. Thank you so much for saving me a week.

Best regards,

Soumitra

--
You received this message because you are subscribed to the Google Groups "python-doit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-doit...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python-doit/CAHSrCY1q9u%2B%2BDnmot7FHaFk4rHkNNs2rLW4J-Ghmou2kB4ZPZQ%40mail.gmail.com.

[Soumitra Pal]

Hi Eduardo,

Thank you for your email. I understand your situation about the long-term goal and not being able to actively working on it.

The pipeline I mentioned is in fact has only one task. However, it has many subtasks under different parameter settings. I create these subtasks using the trick mentioned at your blog. In fact, I have created a new workflow manager for pipelines under different parameter settings based upon this: please have a look at this paper and this manual.

Each of the subtasks takes less than an hour. However, the number of tasks is huge.

Thus, it would be good, if .doit.db is updated more often.

Thank you,

Soumitra

You received this message because you are subscribed to a topic in the Google Groups "python-doit" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-doit/P4qzwScfW5s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-doit...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python-doit/CAHSrCY0hjrs5DJ12eDs9kqMqs7Q2xkVb3xm8cPsx%2B-6X3PpucQ%40mail.gmail.com.