Handling code generator without knowing its output in advance

[Joe]

Hi,

I am wondering how is it possible to solve the following situation in pydoit.

There is e.g. a C++ project which main target is an executable e.g. "demo.bin". It has got a static source file e.g. "src/Main.cpp". It has got a code generator e.g. "tools/generator.bin" and a configuration file e.g "cfg/sample.cfg".
The generator generates C++ sources and headers, the name and number of the generated files are depending on the content of the configuration file, it would be too expensive to determine the generator output at task creation time.

The build should do the following:
1. execute the generator
2. scan for .cpp files
3. build objects from Main.cpp and the generated .cpp files
4. link demo.bin from Main.o and the other .o files

I have tried to solve it by "calculated dependencies" and "delayed task creation". I didn't manage.
Has anyone idea for solving this?

I guess this would be solvable if a task action could make new tasks and could modify the metadata of other tasks.

[Eduardo Schettino]

A task action can modify it's own metadata through a custom `uptodate` function
 (I guess no example in the docs but I am sure once I posted an example in this list).

But I am not sure you even need that...

This combined with "delayed task creation" should be enough to solve your problem.

It would be useful if you post a simple example of what you got so far...

cheers,

  Eduardo

[Joe]

Hi,

Here is what currently I have: http://pastebin.com/VCTNVWSm
It is not so simple and clean and yep it may contain many mistakes, sorry.

I guess you should read it from bottom to top.

task_main() creates target "All" which depends on 2 executables: "release.bin" and "debug.bin".
task_link() creates tasks for "release.bin" and "debug.bin" targets, these have calculated dependencies: "generatedDep:release" and "generatedDep:debug".
"generatedDep:release" and "generatedDep:debug" tasks are made by task_generatedDep(). Both "generatedDep:" tasks are depending on the "generator" task which is created by task_generator().
There is a delayed task creator: task_generatedObjs(), this should be executed just after the execution of the "generator" task. This task generator creates task for building .o files from the generated .cpp files.

When I run this, the output looks like:
$ ./build.py
-- obj/debug/Main.o
-- obj/release/Main.o
.  generator
generator running
.  generatedDep:debug
getDep(debug)
.  generatedDep:release
getDep(release)
.  debug.bin
obj/debug/Main.o: In function `main':
/home/endre/Prg/Repos/doit/src/Main.cpp:16: undefined reference to `str2'
/home/endre/Prg/Repos/doit/src/Main.cpp:16: undefined reference to `str1'
/home/endre/Prg/Repos/doit/src/Main.cpp:16: undefined reference to `str0'
collect2: error: ld returned 1 exit status

I'd expect to see the "task_generatedObjs() started" message just after the "generator running" message, since task_generatedObjs() is a delayed task creator, which should be executed when the "generator" task has been finished.

[Joe]

I have found some conceptual and python coding bug in the cited code. I have managed to go forward but it is still not good.
Now as I see the "file_dep"s, which are made by the calculated dependency task, are considered to be up-to-date. "task_generatedObjs()" creates tasks for producing those files, but those tasks are not executed.

[Joe]

Here is the fixed code: http://pastebin.com/8hpXtEAA

Here is its output:

$ ./build.py 

-- obj/debug/Main.o

-- obj/release/Main.o

.  generator

generator running

.  generatedDep:debug

getDep(debug)

fileDep=['obj/debug/res0.o', 'obj/debug/res1.o', 'obj/debug/res2.o']

.  generatedDep:release

getDep(release)

fileDep=['obj/release/res0.o', 'obj/release/res1.o', 'obj/release/res2.o']

task_generatedObjs() started

obj/debug/res0.o obj/release/res0.o

obj/debug/res1.o obj/release/res1.o

obj/debug/res2.o obj/release/res2.o

.  release.bin

linkAction(targets:['release.bin'], file_dep:set(['obj/release/res2.o', 'obj/release/res0.o', 'obj/release/res1.o', 'obj/release/Main.o'])

g++: error: obj/release/res2.o: No such file or directory

g++: error: obj/release/res0.o: No such file or directory

g++: error: obj/release/res1.o: No such file or directory

The linker task's dependency calculator task is called. The generator task runs as the setup task of the dependency calculator task, that generates .cpp sources.  Than the delayed task (task_generatedObjs()) creates tasks to build objects from the generated sources. The linker task calls linkAction() with the right object file list. But those object files are not exist.

Why don't doit executes the tasks which would create the object files from the generated sources?

[Eduardo Schettino]

On Sat, Mar 5, 2016 at 5:03 AM, Joe <eba...@gmail.com> wrote:

Why don't doit executes the tasks which would create the object files from the generated sources?

I guess you hit a bug: https://github.com/pydoit/doit/blob/master/doit/control.py#L113

If I remember correctly I did not implement it because there was not way to implement in a efficient way.

And because I though this situation would not happen... it is odd that you create a task that you know the file_dep

before you create the task that create those targets.
The fact that you mix calc_dep with delayed-tasks also make the code very confusing.

Can you open an issue on github? This limitation should at least be documented.

I would recommend you implement everything in terms of delayed-tasks.

Regards,

  Eduardo

[Joe]

"it is odd that you create a task that you know the file_dep
before you create the task that create those targets."

I am afraid I don't understand what you write. Could you describe it in more detail? E.g. what is the name of the task what I create, what file_dep do I know, etc.

"I would recommend you implement everything in terms of delayed-tasks."

I cannot imagine currently how it could be done for this case. Could you provide an example?

I am currently looking for a build tool which can handle well situations like this. "pydoit" is very promising for me, I like its concepts, I like that it just tries to provide a nice core and I am free to add my decorators.