Tasks that modify other task targets in-place: best practice?

[Nikolaus Waxweiler]

Hi,

I currently have a build script that wants to modify the output of task in-place if a certain other task is called. The part reads roughly like this:

RELEASE_TASKS = ["a"]


def task_a():
    for i in range(10):
        yield {
            "name": f"{i}",
            "targets": [f"{i}.txt"],
            "actions": ["touch %(targets)s"],
        }


def task_sign():
    target_paths = [target for subtask in task_a() for target in subtask["targets"]]
    return {
        "file_dep": target_paths,
        # "targets": target_paths,
        "actions": [f"echo sign {target_paths}"],
        "verbosity": 2,
    }


def task_release():
    task_dep = RELEASE_TASKS
    if "a" in task_dep:
        task_dep.append("sign")

    return {
        "task_dep": task_dep,
        "actions": ["echo release"],
        "verbosity": 2,
    }

task_a generates some files and if the user does "doit release", we want to sign the generated files cryptographically, which the release task then copies somewhere (it uses the original task's name to determine what to copy, hence the sneaky in-place modification). This in-place modifcation (same targets) is naturally not allowed in doit, so we are currently using the workaround of explicitly modifying the release task's task dependencies. The reason this signing is not part of task_a with maybe a check if we want to release is that the utility asks for a password in each run, so making it part of task_a would result in 10 password prompts. With the above approach, it only asks for a password once.

I am wondering if there is a better appraoch to in-place modification (multiple same targets) from within doit or if we should instead tackle the signing utility or the way the release task works?

[Nikolaus Waxweiler]

Oops, task_release should more realistically read:

def task_release():
    task_dep = [t for t in RELEASE_TASKS]

    if "a" in task_dep:
        task_dep.append("sign")

    return {
        "task_dep": task_dep,

        "actions": [f"echo release {t}" for t in RELEASE_TASKS],
        "verbosity": 2,
    }

expected output:

$ doit release
. a:0
. a:1
. a:2
. a:3
. a:4
. a:5
. a:6
. a:7
. a:8
. a:9
. sign
sign [0.txt, 1.txt, 2.txt, 3.txt, 4.txt, 5.txt, 6.txt, 7.txt, 8.txt, 9.txt]
. release
release a

[Jan Vlčinský]

Nikolaus.

To me the in-place target modification seems like breaking doit house rules which reads: "result of any task is represented by single artifact (mostly a file)". Nice thing is that if the task dependencies are considered complete and existing task result is the same as the run before, the task can be considered complete and there is no need to run it at the moment.

Solution for your scenario is very simple:

• created folder "candidates", where you store the things to sign
• create folder "signed", where you store the candidates with a signature

I use this approach for years.

One could say, there is inefficiency in keeping almost identical copies of the same file, but in most cases, this is not real problem.

Jan

Jan Vlčinský +420-608979040 https://jan-vlcinsky-ttr.mojeid.cz/

--

You received this message because you are subscribed to the Google Groups "python-doit" group.

To unsubscribe from this group and stop receiving emails from it, send an email to python-doit...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python-doit/74688f3d-d2e0-4feb-9a25-1a459443490f%40googlegroups.com.

[Nikolaus Waxweiler]

Hi Jan,

thanks for weighting in. I came to the same conclusion after thinking about it some more, as never doing in-place modification simplifies the logic. Will tackle the release task then :)