Can we rely on subset-of-subtasks behavior

[Oliver Dain]

We have a group of subtasks but we don't want all of them to run when the main task is run. That is, if you run "doit foo" we want to run "foo:bar" and "foo:baz" but not "foo:bap". Experimentally this can be achieved by using "task_dep" or "calc_dep" on the top-level task (the one with name set to None) as follows:

def task_foo():
    yield {
        'name': 'bar',
        'actions': ['echo bar'],
    }

    yield {
        'name': 'baz',
        'actions': ['echo baz'],
    }

    yield {
        'name': 'bap',
        'actions': ['echo bap'],
    }

    yield {
        'name': None,
        'task_dep': ['foo:bar', 'foo:baz'],
    }

The following seems to work:

$ doit list --all
foo        
foo:bap    
foo:bar    
foo:baz   

$ doit -v2 foo
.  foo:bar
bar
.  foo:baz

However, I don't see this behavior documented. The documentation seems to imply that running `doit foo` would always run all sub-tasks. Is the behavior shown above intended and will it remain supported in the future?

For the curious, our use case is various build steps to generate disk images for a hardware device. There's a few different image types and many build steps and we don't want to clutter our "doit list" too much so grouping the tasks into subtasks is appealing. But we need a "calc_dep" to determine which steps are actually required (e.g. pull a remote Docker image if it exists with the right version, if not build it and push it).

Thanks,

Oliver

[Oliver Dain]

Hmmm.. it seems this behavior is a bit finicky. I get different results for what is run and the output of `doit list --all foo` depending on where I put the definition of the "name": None task.

If I define the "name": None task after the sub-tasks are defined the "task_dep" works but "doit list --all foo" shows only the depended upon sub-tasks, not all sub-tasks:

$ doit list --all foo
foo        

foo:bar    
foo:baz   

If I yield the name: None task before I define the other tasks the "task_dep" is ignored, the `doit list --all foo" shows all sub-tasks but it shows some more than once:

def task_foo():

    yield {
        'name': None,
        'task_dep': ['foo:bar', 'foo:baz'],
    }

    yield {
        'name': 'bar',
        'actions': ['echo bar'],
    }

    yield {
        'name': 'baz',
        'actions': ['echo baz'],
    }

    yield {
        'name': 'bap',
        'actions': ['echo bap'],
    }

results in:

$ doit list --all foo
foo        
foo:bap    
foo:bar    

foo:bar    
foo:baz    
foo:baz    

$ doit -v2 foo
.  foo:bar
bar
.  foo:baz

baz
.  foo:bap
bap

So there seems to be 3 sets of behaviors:

1. If you don't have a name: None task doit list --all main_task lists all the sub-tasks.
2. If you have a name: None task and it is yielded before the subtasks task_dep is ignored and list --all shows all tasks but shows those in task_dep twice.
3. If you have a name: None task and it is yielded after the subtasks are defined the task_dep works as expected but list -all only shows a subset of the subtasks (those that are in task_dep).
   

[Eduardo Schettino]

Hi Oliver,

What you are trying to do is not supported. You can check the code in loader.py:_generate_task_from_yield()

> I get different results for what is run and the output of `doit list --all foo` depending on where I put the definition of the "name": None task.

The name=None should be the first yielded task, otherwise it will be implicitly created...

And being implicit or not all other foo:XXX tasks will be added as a task_dep.

When you put name=None, you just overwrote the implicitly created group task. but...

> $ doit list --all foo

You will not be able to get a list of tasks here that is different from what is executed (actual task_dep).

For your use case I guess you will need to write a custom "list" command.

I recently added a "meta" attribute for tasks, maybe you could use that and integrate with the "list" command somehow.

We could even have some standard that would be integrated into doit core implementation.

ie. "doit list --tag main" would display only a subset of tasks with that tag.

And of course you would be able to put this "--tag main" in your project.toml file so by default you get a less verbose output.

Or just accept that "list --all foo" will list the tasks to be executed.

And to really get all tasks you have to do: "list --all | grep foo".

Regards

--
You received this message because you are subscribed to the Google Groups "python-doit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-doit...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python-doit/6c2df352-42d5-4e11-ab42-68952413331fn%40googlegroups.com.

[Oliver Dain]

Thanks Eduardo. I'll stop trying to make it do something it doesn't want to do :)

You received this message because you are subscribed to a topic in the Google Groups "python-doit" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-doit/MmkOsL3iKKY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-doit...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python-doit/CAHSrCY1qHi6DoZbE1puBDVD%3DpjXzEM8S04gZq8ED2_ycg1Nmaw%40mail.gmail.com.