Intentionally having multiple tasks with the same targets

[Michael Milton]

I have a build script that has two distinct code paths; if the user has access to our online data store, doit downloads the required assets as one big bundle. If not, doit sources them from various different places. The problem with this, is that I have a number of tasks that share a target; they both are intended to make the same file, but in a different way.

Of course, doit complains about this.

I can solve this error by doing a conditional import of tasks so that both conflicting tasks can't see each other, but I wonder if you could add a configuration option that stops doit from even raising an error if this happens? I'm happy to write a PR, but I'd just like to discuss my reasoning first.

[Eduardo Schettino]

Hi Michael,

This is a good use case but simply stop raising an error is not a solution...

The reason for raising errors is that if you specify a target from command line doit executes the task,

if there are 2 tasks with same target what doit should do? The same problem if another task has a file_dep

generated by 2 different tasks.

And also if 2 tasks have the same target you will probably want to mark both as completed successfully.

I guess it is possible to have a sensible solution that solve the problems above.

But we need to think about it more carefully.
No need to look the code.. could you specify from a user perspective what should be implemented?

Please also provide a sample.

Regards,

  Eduardo

--
You received this message because you are subscribed to the Google Groups "python-doit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-doit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Milton]

That's a good point, I'm sure the solution to this problem can be generalised to be more useful. The key problem, as you've said, is what we should do if the user tries to build a file that is the target for multiple tasks; the problem is ambiguity.

The first, obvious suggestion, is for the user to disambiguate the tasks in their code, so doit knows from the beginning which task should run. For example, the user could provide a priority for each target, indicating which task should be run if there is a conflict:

def task_a():
  return {
     'targets': [{'path': 'file.txt', 'priority': 1}]
  }

def task_b():
  return {
     'targets': [{'path': 'file.txt', 'priority': 2}]
  }

doit file.txt
 > . b

This idea could be expanded into the user giving a priority function etc.

However, what I think is a better solution, although it would be a more drastic change, is for the 'multiple tasks have the same target' error to only occur at execution time, not while doit is performing its first pass to analyse the tasks. This means, in my use case, that `doit install` would have no issue, as I've designed, because in one task generator it checks to see if the user has the right credentials, and if so, sets the task_dep to 'download_asset_bundle', whereas if they don't, it sets it to 'download_assets_manually'. In the case that the user should write `doit asset.tar.gz`, i.e. they ask for a specific target to be built, doit should then throw an exception, with the message 'ambigious build task for target asset.tar.gz: please specify either the task "download_assets_manually" or "download_asset_bundle"'.

The problem with this solution is that errors by the user where they accidentally set two tasks to have the same target will be harder to catch, for example if they copy a task and forget to edit the target, they made a mistake and they should be told. So perhaps the solution is to make doit complain about ambigious targets at load time, unless the user sets a configuration option to complain at runtime?

[Jonas Wagner]

Hi,

There's something I don't understand in this discussion: Why do you need to generate both tasks? What about something like this:

if credentials is not None:

    def task_download_from_store():

        return {...}

else:

    def task_download_anonymous():

        return {...}

Note that Python is a dynamic language, and it's perfectly legal to define a function conditionally.

Or something like this, with a single task that returns a different task definition depending on a condition:

def task_download():

    if credentials is not None:

        return generate_download_from_store_task()

    else:

        return generate_download_anonymous_task()

Best,

Jonas

--
You received this message because you are subscribed to the Google Groups "python-doit" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python-doit...@googlegroups.com.

[Michael Milton]

You're correct, I don't need to define both tasks. Indeed what I've done currently is define each task in a different module and do a conditional import:

if has_auth():
  from tasks.bundle_assets import *
else:
  from tasks.manual_assets import *

My motivation for this change is that, even if I have authentication, I might want to manually download the assets, either for testing purposes, or because I want to upgrade to a version not available in my data store. In other words, I want the user to be able to run any doit task, no matter the situation. This way, if the user does a `doit install`, my code will run either one or the other build path based on the authentication, but if they want, they also have the flexibility to run any individual task from either build path.

[Michael Milton]

To expand upon my previous idea, I'm suggesting we add a global config option such that having

DOIT_CONFIG = {'conflict_check': 'runtime'}

would result in no error being thrown when two tasks are declared with the same target.txt, but will throw an error if the user tries to run any command that would invoke both tasks at the same time (e.g. running `doit target.txt` or by running a task that has both conflicting tasks as dependencies.

Having

DOIT_CONFIG = {'conflict_check': 'inititalize'}

would result in the default behaviour, and this could be the default value for this option.

[Jonas Wagner]

Hi,

My motivation for this change is that, even if I have authentication, I might want to manually download the assets, either for testing purposes, or because I want to upgrade to a version not available in my data store.

OK, I understand better what you would like to do now.

Here's how I would solve this problem in a way that is supported by the current version of Doit:

- Create a single task to generate your file.
  From a user's perspective, this is nice because the user really shouldn't care too much about how the file is generated.

- The task by default checks whether it has authentication credentials, and uses them if available.

- Add a command-line parameter to your doit task (something like --force-download=true) that will cause the task to ignore credentials.

Life just seems much simpler to me if the dependencies of all tasks form a directed acyclic graph. Once you introduce tasks that depend on A or B, or targets that can be built by multiple tasks, things get more complex. I'm not sure that the complexity is justified in this case.

What do you think?

Jonas

[Michael Milton]

The way the application is designed, if the user has authentication, the the assets are all downloaded in one go. Otherwise, each is downloaded separately. So I have one download_from_store function that has a large number of targets, and lots of smaller tasks that each have a single target.

Your suggestion is good from a user's perspective, and I could refactor my code to work like that, but I guess I also want to solve the more theoretical problem of "what if I want to have multiple tasks that share targets", even if this use case isn't a perfect example of that.