Another take on build systems

[Guillaume Melquiond]

I was greatly impressed by the work behind apenwarr/redo, so I wanted to share the following on this list.

I was looking for a build system with the following features:

1. supports a centralized rule file (as does make),

2. supports dynamic dependencies (as does redo),

3. supports rules that create several targets (neither make nor redo),

4. supports parallel builds (both make and redo),

5. fits into a single source file written in a widely supported language, so that I can ship it along my other projects.

In the end, I decided to code it myself. The end result is attached to this post. The system has far less features than make and redo, and it is a bit naive in its use of inter-process communications. But it already works well enough for my purpose. You can compile it with "g++ -pthread remake.cpp". (Only linux-like systems are supported for now.)

The program reads rules from a file called Remakefile. The rules have the same syntax as Makefile rules, except that, if there are several targets before a colon, the rule is understood as producing several files at once. The shell script of a rule can recursively call the program, which then behaves like redo-ifchange. As with redo, dynamic dependencies are remembered from one run to the other, and they only serve to check whether a target is obsolete. There is a lot more documentation at the top of the source file, for those interested.

Best regards,

Guillaume

[Diggory Hardy]

Hi

Looks pretty good. I might give it a try next time I write some new build rules!

One question: does it support dynamic determination of targets? Javac, for instance, outputs several object files for a source depending on the classes defined within — would this list have to be updated by hand?

Cheers,
Diggory

[Guillaume Melquiond]

Le mercredi 14 novembre 2012 14:29:07 UTC+1, Diggory Hardy a écrit :

Looks pretty good. I might give it a try next time I write some new build rules!

One question: does it support dynamic determination of targets? Javac, for instance, outputs several object files for a source depending on the classes defined within — would this list have to be updated by hand?

It would have to be updated by hand. From the point of view of the system, if you don't know the name of a target beforehand, being able to build it through its name is useless; you presumably have some other way to refer to it. For instance, you could have a pseudo-target that would point at the source file. Am I missing some use case where you really need its name as a target of a rule?

That said, the system allows the rule file itself to appear as a rule target. (I needed the ability to write "Remakefile : Remakefile.in config.status" so as to support the common autoconf idiom.) So, while the system does not directly support dynamic targets, if you have a script that is able to output these targets, it should somehow work.

Best regards,

Guillaume

[Avery Pennarun]

On Wed, Nov 14, 2012 at 11:34 AM, Guillaume Melquiond
<guillaume...@gmail.com> wrote:
> Le mercredi 14 novembre 2012 14:29:07 UTC+1, Diggory Hardy a écrit :
>> Looks pretty good. I might give it a try next time I write some new build
>> rules!
>>
>> One question: does it support dynamic determination of targets? Javac, for
>> instance, outputs several object files for a source depending on the classes
>> defined within — would this list have to be updated by hand?
>
> It would have to be updated by hand. From the point of view of the system,
> if you don't know the name of a target beforehand, being able to build it
> through its name is useless; you presumably have some other way to refer to
> it. For instance, you could have a pseudo-target that would point at the
> source file. Am I missing some use case where you really need its name as a
> target of a rule?

Well, fundamentally, this is the requirement that makes it hard to
implement the feature in redo :) Otherwise we'd just have a file
named fileA_and_fileB.do to generate both fileA and fileB, and we
could do some caching thing to make sure lookups of that .do file are
efficient. The ability to produce multiple targets from the same rule
seems to be the only claimed advantage of your system over redo, so it
needs to work right. :) (Putting all your rules in one file is
entirely possible in redo; just put a case statement in default.do.)

pseudo-targets work okay, but aren't great. You then have to remember
to refer to that pseudo-target every time you refer to the dependency,
rather than referring to a file by name. So if a 'configured' target
runs ./configure and produces config.h, you have to remember, in every
C file that depends on config.h, to actually depend on 'configured'
instead. That's messy.

(I realize that with config.h you could just use a multi-target rule
with defined outputs that include config.h. But imagine instead that
./configure produces a variable set of config_*.h files, for example.
You don't seem to have solved the general case, and redo has been
leaving out that feature until we can solve the general case.)

Have fun,

Avery

[Guillaume Melquiond]

Le mercredi 14 novembre 2012 18:46:21 UTC+1, apenwarr a écrit :

> It would have to be updated by hand. From the point of view of the system,
> if you don't know the name of a target beforehand, being able to build it
> through its name is useless; you presumably have some other way to refer to
> it. For instance, you could have a pseudo-target that would point at the
> source file. Am I missing some use case where you really need its name as a
> target of a rule?

Well, fundamentally, this is the requirement that makes it hard to
implement the feature in redo :)  Otherwise we'd just have a file
named fileA_and_fileB.do to generate both fileA and fileB, and we
could do some caching thing to make sure lookups of that .do file are
efficient.  The ability to produce multiple targets from the same rule
seems to be the only claimed advantage of your system over redo, so it
needs to work right. :)  (Putting all your rules in one file is
entirely possible in redo; just put a case statement in default.do.)

As far as I know, it does work right. Let us see.

My personal use case is the compilation of OCaml files without interface files. A file foo.ml gets compiled into two files foo.cmo and foo.cmi. So a rule with generic targets "%.cmo %cmi" is exactly what is needed. Note that the compilation of other files depend on .cmi files, and that their name is known. So there is no difficulty in writing rules. With redo, that would be default.cmo_and_cmi.do, that is, a pseudo-target; the build system system is not being helpful.

Now let us go to Diggory's use case. (It has been a long time since I last did some Java, so my memory might be fuzzy.) When you compile a file foo.java, you get a file foo.class (assuming the file contains a public class) and a lot of other .class files (which names could be random for all we know, e.g. bar$1$int_.class). The only .class file you might ever need to explicitly have as a target is the one containing the public class, thus the one with the known name. Unless I'm mistaken, you never need to explicitly build a .class file which name was generated by the compiler, so everything is fine again (even for redo).

So I'm asking again. Does anyone have a use case, where the compiled files have unknown names (and by unknown, I mean a mangled name, not just a change of extension), yet one still needs to target them explicitly?

pseudo-targets work okay, but aren't great.  You then have to remember
to refer to that pseudo-target every time you refer to the dependency,
rather than referring to a file by name.  So if a 'configured' target
runs ./configure and produces config.h, you have to remember, in every
C file that depends on config.h, to actually depend on 'configured'
instead.  That's messy.

Right, that's why I wanted the build system to support multiple targets, so that I never have to introduce pseudo-targets.

 

(I realize that with config.h you could just use a multi-target rule
with defined outputs that include config.h.  But imagine instead that
./configure produces a variable set of config_*.h files, for example.
You don't seem to have solved the general case, and redo has been
leaving out that feature until we can solve the general case.)

Sorry, I don't understand your remark. The files produced by config.status are well-known, these are the ones passed on its command line and/or the ones the user explicitly typed in the configure.in file. I don't know of a single case where the user writing the build rules would have no idea what the name of these files are.

Best regards,

Guillaume