Is there a way to preview or preflight what will happen?

[Andrew Mellinger]

Greetings!

TL;DR: Is there a way to have doit just indicate/report which tasks would be executed based on current dependency checks but not actually execute these things?

I am a researcher working on a machine learning platform to automate the training and comparison of multiple models with data sets, architectures and hyperparameters. Our focus is on reproducibility and predictability of machine learning experiments.  We are currently switching our workflow execution system (simple homegrown python script) to doit so we can have extensible workflows.

So far, doit seems to be the sweet spot for dependency management, scalability, expressiveness, and compactness.

With a large multi-day preprocess/train/evaluate going on, it would nice to see what the system will do before it begins to execute, especially if I expect certain tasks to need to be done and other ones that shouldn't. 

Thanks for your great tool!

-Andrew

[Eduardo Schettino]

Hi Andrew,

On Fri, May 7, 2021 at 9:08 PM Andrew Mellinger <andrew.m...@gmail.com> wrote:

Greetings!

TL;DR: Is there a way to have doit just indicate/report which tasks would be executed based on current dependency checks but not actually execute these things?

Both commands `doit list` and `doit info`, have a `--status` flag that allows you to introspect a task state.

But a "dry-run" command has never been implemented, See issue #48 (https://github.com/pydoit/doit/issues/48).

 

I am a researcher working on a machine learning platform to automate the training and comparison of multiple models with data sets, architectures and hyperparameters. Our focus is on reproducibility and predictability of machine learning experiments.  We are currently switching our workflow execution system (simple homegrown python script) to doit so we can have extensible workflows.

So far, doit seems to be the sweet spot for dependency management, scalability, expressiveness, and compactness.

With a large multi-day preprocess/train/evaluate going on, it would nice to see what the system will do before it begins to execute, especially if I expect certain tasks to need to be done and other ones that shouldn't. 

Nice. But to tell you the truth, not sure how well suited doit will be for your use-case.

While I believe doit core is good enough there 2 important pieces missing:

 - parallel distributed (cluster) execution

 - real time management & monitoring UI. The command line interface works well for interactive, relatively short execution. But not for long running tasks.

Of course these could be built on top of doit...

I would love to work on those but these are too big tasks for me to do on "open-source" time.

Regards,

  Eduardo

[Andrew Mellinger]

Eduardo,

The function "list -s" is exactly what I was looking for!  I was searching for preflight, dryrun, etc.

Regarding the clustering/scaling we use the platform's back (pytorch distributed data parallel) for the gpu/cluster support. Our base environment is pretty static so we don't need too many smarts. Meaning, we can use the same node for the "controller' (rank 0) consistently and can make a ton of assumptions. Doit is really just supporting the dependency graph determination part. 

My one remaining question: For the parallel part (when the jobs are small), the one piece I am trying to figure out is which "slot" in the parallel pool is being used. For example, if I want to run 16 jobs across 4 gpus is there an easy way to know which "slot" is being used, so I know which GPU to assign? So is some sort of environment variable or argument available with "worker id" or something?

A little more background if you're bored:

What we really is generate an intermediate json structure similar to doit which contains stanza of inputs, outputs and actions.  Then we have scripts that recapitulate that into dodo.py files, or linear python.  When we want to scale to larger platforms we'll make a converter to snakemake or something more heavyweight.  For now, doit fits the sweet spot of size, ease of use and easy installation.

-Andrew