Looking at alternatives: tup
Mildred Ki'Lya
Looking around, i found another build system that i find quite interesting. It is called tup <http://gittup.org/tup/> and features:
- fastest builds possible
- dependency detection (through fuse filesyste)
- exact builds, you can't get wrong
- if you miss a dependency, it will tell you, and will add it anyway
- if you miss an output file, it will tell you as well
- features a nice looking progress bar
Unfortunately:
- too strict about output files. Sometimes the program you try to run does write files all around the place, and tup doesn't allow that
- introduces a specific syntax (I quite like the no syntax approach of redo) that uses system(3), so it doesn't support file names with spaces
- build things, do not run them. Build everything. I don't think you can use it to run a test suite for example.
I find the dependency detection quite interesting. Note: it only works on relative paths, if the path is absolute, it doesn't work. It works by having a fuse repository mounted in its .tup directory and monitoring what job does what with which files.
I remember a while ago having a very hard to debug problem caused by the same library being built twice, with different compilation options, in the same file. If the build system prevented from overriting a file, or at least told me that the file was the output of two dofferent commands, it would have helped a bit.
I think it would be possible tohave a similar detection mechanism. If we want to keep a minimal/do implementation, we'd still have to use redo-ifchange, but it would at least provide interesting debug messages "you forgot to list file.xyz as a dependency". We could also detect the script overriting files that may cause problems, and issue a warning.
As for a nice progression bar, I know cmake does it as well, and I think if we can find a way to include it, it would be a nice feature to show off. But it would require a global view of the build, which don't fit with the recursive design of redo.
What do you think ?
--
Mildred Ki'Lya
http://mildred.fr
Miles Gould
problem, which is obviously a huge win. But:
(1) it completely gives up on atomicity
(2) it introduces YET ANOTHER half-arsed language based on textual
macroexpansion
(3) AFAICT, its reliance on FUSE and inotify means that it's closely
tied to Linux.
For me, (2) is a dealbreaker. I've dealt with far too much lossage
resulting from multiple levels of string-expansion to ever put myself
through that voluntarily. Though I'll note in tup's favour that it
allows for fine-grained build-rule tracking: each target depends
implicitly on the precise string that's executed.
As for the pretty output bar, I rather like redo's existing
tree-structured output :-)
Miles
Alan DeKok
> For me, (2) is a dealbreaker. I've dealt with far too much lossage
> resulting from multiple levels of string-expansion to ever put myself
> through that voluntarily. Though I'll note in tup's favour that it
> allows for fine-grained build-rule tracking: each target depends
> implicitly on the precise string that's executed.
I've never understood why tup didn't just track the commands to
execute. "command foo reads files A, B, C, and writes file X, Y".
How to rebuild X? run command foo!
The build system then becomes a simple shell script which runs the C
compiler. The magic dependency tracking then takes care of runnning
only the *minimal* commands the second time around.
Alan DeKok.
Prakhar Goel
> I've never understood why tup didn't just track the commands to
> execute. "command foo reads files A, B, C, and writes file X, Y".
I assume because it's not that simple. How do you know what random
command "foo" does?
Also, how do you know what the arguments are?
--
________________________
Warm Regards
Prakhar Goel
LinkedIn Profile: http://www.linkedin.com/in/newt0311
Alan DeKok
> I assume because it's not that simple. How do you know what random
> command "foo" does?
See "tup". It uses LD_PRELOAD to catch open/unlink/execve, etc.
> Also, how do you know what the arguments are?
It catches calls to execve and friends. Then, writes the arguments
out to a permanent store.
It's actually pretty easy to do on modern OS's. See the "tup"
documentation and paper for how it's done. It's very nice.
Alan DeKok.
Miles Gould
> I've never understood why tup didn't just track the commands to
> execute. "command foo reads files A, B, C, and writes file X, Y".
>
> How to rebuild X? run command foo!
Like fabricate, you mean?
> The build system then becomes a simple shell script which runs the C
> compiler. The magic dependency tracking then takes care of runnning
> only the *minimal* commands the second time around.
I *think* that makes it harder to make global changes to the build
process: things like changing CFLAGS. Now *all* your cached cc
invocations are wrong. I suppose you could work around this by wrapping
the compiler in a shell script which sourced its configuration variables
from a file, but that's starting to get messy.
But yes, I don't see why tup insists on bottom-up builds. It should be
simple enough to make it do top-down builds as well.
Miles
Miles Gould
> Tup's petri-net based design completely obviates the "multiple output"
> problem, which is obviously a huge win. But:
>
> (1) it completely gives up on atomicity
> (2) it introduces YET ANOTHER half-arsed language based on textual
> macroexpansion
> (3) AFAICT, its reliance on FUSE and inotify means that it's closely
> tied to Linux.
Mildred has contacted me off-list to point out that tup has been ported
to Windows; I see it can also be installed on OS X via homebrew. Note to
self: next time I should actually check my facts before spreading FUD
based on my hazy memories. Sorry, everyone!
My point (2) is still valid, and I can't see any equivalent to redo's
atomicity guarantees - though as we've seen, it's tricky to get
atomicity and multiple outputs per rule. I'm becoming less and less sure
about the CFLAGS thing, though.
Miles