[dev] Automatic C header dependency tracking for the redo build-system
2 views
Skip to first unread message
Thomas Oltmann
Sep 7, 2021, 1:52:24 PM9/7/21
to dev mail list
Hi everybody!
redo is a pretty well-designed family of build-systems
that enjoys a certain popularity among people on this mailing list.
Its recursive nature makes it well-suited to projects comprising a
large amount of files.
However, unlike comparable build-systems like make or tup, it lacks provisions
to automatically track C #include dependencies, which is sorely needed
for projects
with many source files.
To overcome this, I wrote a simple companion tool that integrates
fairly seamlessly with redo.
This tool is able to parse the dependency-only Makefiles that modern C
compilers can
produce during compilation, and feed these dependencies into redo (via
'redo-ifchange').
It should work with pretty much any redo implementation
(other than maybe apenwarr's *minimal do*, because that one doesn't implement
'redo-ifchange' as a separate executable).
You can find it here: https://github.com/tomolt/redo-depfile
Any feedback is appreciated.
Cheers,
Thomas Oltmann
P.S.: My sincere apologies if this post if considered off-topic, as
redo is not under the suckless banner.
I considered posting on the redo mailing-list instead, but that one
seems very obscure and inactive,
plus I recognize some of the posters there as being regulars on the
suckless mailing lists as well.
redo is a pretty well-designed family of build-systems
that enjoys a certain popularity among people on this mailing list.
Its recursive nature makes it well-suited to projects comprising a
large amount of files.
However, unlike comparable build-systems like make or tup, it lacks provisions
to automatically track C #include dependencies, which is sorely needed
for projects
with many source files.
To overcome this, I wrote a simple companion tool that integrates
fairly seamlessly with redo.
This tool is able to parse the dependency-only Makefiles that modern C
compilers can
produce during compilation, and feed these dependencies into redo (via
'redo-ifchange').
It should work with pretty much any redo implementation
(other than maybe apenwarr's *minimal do*, because that one doesn't implement
'redo-ifchange' as a separate executable).
You can find it here: https://github.com/tomolt/redo-depfile
Any feedback is appreciated.
Cheers,
Thomas Oltmann
P.S.: My sincere apologies if this post if considered off-topic, as
redo is not under the suckless banner.
I considered posting on the redo mailing-list instead, but that one
seems very obscure and inactive,
plus I recognize some of the posters there as being regulars on the
suckless mailing lists as well.
Sergey Matveev
Sep 7, 2021, 3:39:33 PM9/7/21
to d...@suckless.org
Greetings!
*** Thomas Oltmann [2021-09-07 19:50]:
read D < "$2".d
redo-ifchange ${D#* }
POSIX "read" out-of-box understands \-newlines that can appear in those
.d-Makefiles and read the whole .d file as a single line, where you just
have to strip the first word ("target:").
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
*** Thomas Oltmann [2021-09-07 19:50]:
>This tool is able to parse the dependency-only Makefiles that modern C
>compilers can
>produce during compilation, and feed these dependencies into redo (via
>'redo-ifchange').
Just out of curiosity, why POSIX shell abilities are not enough for that task?
>compilers can
>produce during compilation, and feed these dependencies into redo (via
>'redo-ifchange').
read D < "$2".d
redo-ifchange ${D#* }
POSIX "read" out-of-box understands \-newlines that can appear in those
.d-Makefiles and read the whole .d file as a single line, where you just
have to strip the first word ("target:").
--
Sergey Matveev (http://www.stargrave.org/)
OpenPGP: CF60 E89A 5923 1E76 E263 6422 AE1A 8109 E498 57EF
b...@0x1bi.net
Sep 7, 2021, 4:23:47 PM9/7/21
to d...@suckless.org
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
What's wrong with plain old make? I don't think there's a need to write
more build tools when one is already enough; if we keep writing build
tools we'll end up with tools like autoconf.
Ben Raskin
-----BEGIN PGP SIGNATURE-----
iHUEARYIAB0WIQSVj5ObtUZn7L8NuN+O1YyzbQMTOgUCYTe8GQAKCRCO1YyzbQMT
OhvfAP9zvrBVQ6kZZLeiZsczahSp8tQsiNF/R7Mc8WM+iNx/zAD/bQ3PhFGRxvVQ
cN5Zu82/SPbI0DijmtGH9/2DSCa+6QI=
=fSVp
-----END PGP SIGNATURE-----
Hash: SHA256
What's wrong with plain old make? I don't think there's a need to write
more build tools when one is already enough; if we keep writing build
tools we'll end up with tools like autoconf.
Ben Raskin
-----BEGIN PGP SIGNATURE-----
iHUEARYIAB0WIQSVj5ObtUZn7L8NuN+O1YyzbQMTOgUCYTe8GQAKCRCO1YyzbQMT
OhvfAP9zvrBVQ6kZZLeiZsczahSp8tQsiNF/R7Mc8WM+iNx/zAD/bQ3PhFGRxvVQ
cN5Zu82/SPbI0DijmtGH9/2DSCa+6QI=
=fSVp
-----END PGP SIGNATURE-----
Thomas Oltmann
Sep 7, 2021, 4:58:41 PM9/7/21
to dev mail list
Dear Sergey,
Thank you for your reply!
Am Di., 7. Sept. 2021 um 21:39 Uhr schrieb Sergey Matveev
<star...@stargrave.org>:
understand backslash-escaped newlines.
That does indeed make using only POSIX shell features a viable option.
Still, I'm personally a bit wary of such an approximate solution since
it only superficially aligns with Makefile syntax.
My implementation keeps relatively true to the way real make
implementations parse Makefiles, and as such
there are much less chances for things to break.
For example, it is also able to properly recognize:
- comments
- spaces in filenames, escaped by preceding backslash (common
extension, but not part of POSIX make)
- dollar signs in filenames, escaped by preceding dollar sign
I have to admit, these concerns are mostly esoteric.
But I do think there are situations were these things are useful, for
example when users have weird symbols, like spaces,
in their build paths, because their Unix user name contains them too
(sadly happens pretty often).
Cheers,
Thomas Oltmann
Thank you for your reply!
Am Di., 7. Sept. 2021 um 21:39 Uhr schrieb Sergey Matveev
<star...@stargrave.org>:
> Just out of curiosity, why POSIX shell abilities are not enough for that task?
>
> read D < "$2".d
> redo-ifchange ${D#* }
>
> POSIX "read" out-of-box understands \-newlines that can appear in those
> .d-Makefiles and read the whole .d file as a single line, where you just
> have to strip the first word ("target:").
This is quite embarrassing. I was not aware that 'read' was able to
>
> read D < "$2".d
> redo-ifchange ${D#* }
>
> POSIX "read" out-of-box understands \-newlines that can appear in those
> .d-Makefiles and read the whole .d file as a single line, where you just
> have to strip the first word ("target:").
understand backslash-escaped newlines.
That does indeed make using only POSIX shell features a viable option.
Still, I'm personally a bit wary of such an approximate solution since
it only superficially aligns with Makefile syntax.
My implementation keeps relatively true to the way real make
implementations parse Makefiles, and as such
there are much less chances for things to break.
For example, it is also able to properly recognize:
- comments
- spaces in filenames, escaped by preceding backslash (common
extension, but not part of POSIX make)
- dollar signs in filenames, escaped by preceding dollar sign
I have to admit, these concerns are mostly esoteric.
But I do think there are situations were these things are useful, for
example when users have weird symbols, like spaces,
in their build paths, because their Unix user name contains them too
(sadly happens pretty often).
Cheers,
Thomas Oltmann
Thomas Oltmann
Sep 7, 2021, 5:17:22 PM9/7/21
to dev mail list
Dear Ben,
I believe Sergey Matveev here is much more of an expert on this matter
than I am,
seeing how he has even developed his own high-performance implementation.
Nevertheless, these are some of the reasons I care about redo, in no
particular order:
- It does not suffer from the same obvious oversights as make, like
the awkward handling of spaces in filenames.
- Out-of-source builds are easy, even in complicated source hierarchies.
- make is bad at modularity, especially recursion (subprojects). See
'Recursive make considered harmful'.
redo, on the other hand, relies on well-designed recursion as a
fundamental building block of the entire build system.
- It is actually much simpler to implement than traditional make, not harder
- IMO it handles generated dependencies / dynamic dependency discovery
much better than make
- Purely POSIX-compatible Makefiles are painful. All make
implementations therefore provide ample extensions,
and they are all incompatible. Ever compared an ideomatic BSD
Makefile to a GNU one?
redo, on the other hand, is fairly universal, even though it is not
as well-standardized.
This is simply because it does not need much in terms of extensions
to be usable.
Cheers,
Thomas Oltmann
I believe Sergey Matveev here is much more of an expert on this matter
than I am,
seeing how he has even developed his own high-performance implementation.
Nevertheless, these are some of the reasons I care about redo, in no
particular order:
- It does not suffer from the same obvious oversights as make, like
the awkward handling of spaces in filenames.
- Out-of-source builds are easy, even in complicated source hierarchies.
- make is bad at modularity, especially recursion (subprojects). See
'Recursive make considered harmful'.
redo, on the other hand, relies on well-designed recursion as a
fundamental building block of the entire build system.
- It is actually much simpler to implement than traditional make, not harder
- IMO it handles generated dependencies / dynamic dependency discovery
much better than make
- Purely POSIX-compatible Makefiles are painful. All make
implementations therefore provide ample extensions,
and they are all incompatible. Ever compared an ideomatic BSD
Makefile to a GNU one?
redo, on the other hand, is fairly universal, even though it is not
as well-standardized.
This is simply because it does not need much in terms of extensions
to be usable.
Cheers,
Thomas Oltmann
Sergey Matveev
Sep 8, 2021, 4:31:51 AM9/8/21
to d...@suckless.org
*** Thomas Oltmann [2021-09-07 22:57]:
>For example, it is also able to properly recognize: [...]
I see, thanks!
However personally in C-projects I use https://include-what-you-use.org/
and h-extract.pl script to make depends only on local headers:
whatever.do:
[...]
redo-ifchange `./h-extract.pl $src`
h-extract.pl:
#!/usr/bin/env perl
# Extract all locally included header files (not <>-ones, but "")
# hack, to badly exit if there is unexistent file
$SIG{__WARN__} = sub { die @_ };
map { $inc{$_} = 1 } @ARGV;
while (<>) {
/^#include "([^\/]+)"$/ and ($1 !~ /\.in$/) and $inc{$1} = 1;
};
print join " ", sort keys %inc;
>For example, it is also able to properly recognize: [...]
I see, thanks!
However personally in C-projects I use https://include-what-you-use.org/
and h-extract.pl script to make depends only on local headers:
whatever.do:
[...]
redo-ifchange `./h-extract.pl $src`
h-extract.pl:
#!/usr/bin/env perl
# Extract all locally included header files (not <>-ones, but "")
# hack, to badly exit if there is unexistent file
$SIG{__WARN__} = sub { die @_ };
map { $inc{$_} = 1 } @ARGV;
while (<>) {
/^#include "([^\/]+)"$/ and ($1 !~ /\.in$/) and $inc{$1} = 1;
};
print join " ", sort keys %inc;
Sergey Matveev
Sep 8, 2021, 4:33:01 AM9/8/21
to d...@suckless.org
*** Thomas Oltmann [2021-09-07 23:15]:
Good points I support too! Also:
* The whole redo build system description/rules fits on single screen,
like here: http://www.goredo.cypherpunks.ru/Usage-rules.html
Compare it to NetBSD bmake (used in FreeBSD) or GNU Make, having an
unbelievable size. redo does not even force you to use shell in
anyway -- you can use whatever language you wish for
* mtime, used in Make, won't work in practice for many possible reasons:
https://apenwarr.ca/log/20181113. Make does not give any atomicity in
builds. It hard to parallelize and all of us know how often many
software can be built only sequentially (however it is related
directly to "recursive make considered harfmful"
http://www.stargrave.org/recursive_make.pdf).
Basically, Make just does not work (reliably)
>Nevertheless, these are some of the reasons I care about redo, in no
>particular order: [...]
Good points I support too! Also:
* The whole redo build system description/rules fits on single screen,
like here: http://www.goredo.cypherpunks.ru/Usage-rules.html
Compare it to NetBSD bmake (used in FreeBSD) or GNU Make, having an
unbelievable size. redo does not even force you to use shell in
anyway -- you can use whatever language you wish for
* mtime, used in Make, won't work in practice for many possible reasons:
https://apenwarr.ca/log/20181113. Make does not give any atomicity in
builds. It hard to parallelize and all of us know how often many
software can be built only sequentially (however it is related
directly to "recursive make considered harfmful"
http://www.stargrave.org/recursive_make.pdf).
Basically, Make just does not work (reliably)
Georg Lehner
Jun 6, 2022, 9:13:46 AM6/6/22
to d...@suckless.org
Hi,
The topic of header dependency tracking is already addressed since the
inception of redo by DJB.
The Appenwarr documentation offers a fairly simple answer in the form of
an "implicit" .do file for object files.
---
cat > default.o.do <<EOF
redo-ifchange $2.c
gcc -MD -MF $2.d -c -o $3 $2.c
read DEPS <$2.d
redo-ifchange ${DEPS#*:}
---
1. check the dependency on the source file $2.c.
2. build the object file and generate a make-style one-line include
dependency on all header files with the compiler itself.
3. read the rule into the variable DEPS.
4. check the dependency on all header files.
The second line and the fourth line are one of the special things about
redo: *after* building the target you can check *again* if it needs to
be rebuilt.
Best Regards,
Georg
The topic of header dependency tracking is already addressed since the
inception of redo by DJB.
The Appenwarr documentation offers a fairly simple answer in the form of
an "implicit" .do file for object files.
---
cat > default.o.do <<EOF
redo-ifchange $2.c
gcc -MD -MF $2.d -c -o $3 $2.c
read DEPS <$2.d
redo-ifchange ${DEPS#*:}
---
1. check the dependency on the source file $2.c.
2. build the object file and generate a make-style one-line include
dependency on all header files with the compiler itself.
3. read the rule into the variable DEPS.
4. check the dependency on all header files.
The second line and the fourth line are one of the special things about
redo: *after* building the target you can check *again* if it needs to
be rebuilt.
Best Regards,
Georg
David Demelier
Jun 7, 2022, 9:25:54 AM6/7/22
to d...@suckless.org
On Mon, 2022-06-06 at 15:13 +0200, Georg Lehner wrote:
> Hi,
>
> The topic of header dependency tracking is already addressed since
> the
> inception of redo by DJB.
About C header dependencies, you can actually do it using POSIX Make
> Hi,
>
> The topic of header dependency tracking is already addressed since
> the
> inception of redo by DJB.
(in the next version, the -include statement is not available yet but
widely supported).
SRCS= foo.c bar.c baz.c
OBJS= ${SRCS:.c=.o}
DEPS= ${SRCS:.c=.d}
.c.o:
${CC} -o $@ -MMD -c $< ${CFLAGS}
all: hello
-include ${DEPS}
hello: ${OBJS}
${CC} -o $@ ${OBJS}
And you're done. On a fresh directory the -include ${DEPS} will
silently fail and rebuild everything. Then .d files will be generated
and touching any file will rebuild exactly what should be.
The only non portable thing is the -MMD option (gcc and clang support
though).
HTH
--
David
Roberto E. Vargas Caballero
Jun 9, 2022, 4:43:49 AM6/9/22
to dev mail list
Hi,
On Tue, Jun 07, 2022 at 03:21:49PM +0200, David Demelier wrote:
> -include ${DEPS}
...
but in general is better to just define OBJ instead of defining SRC,
because you as a object generator are interested in the objects, no
in the sources. It can drive to problems because the standard does not
define how the expansion is done in inference rules and things like:
$(OBJ): xxxx
don't work in nmake if OBJ is a expansion of SRC.
Regards,
On Tue, Jun 07, 2022 at 03:21:49PM +0200, David Demelier wrote:
> -include ${DEPS}
...
> And you're done. On a fresh directory the -include ${DEPS} will
> silently fail and rebuild everything. Then .d files will be generated
> and touching any file will rebuild exactly what should be.
>
> The only non portable thing is the -MMD option (gcc and clang support
> though).
-include is not portable. Also, not related to the original topic,
> silently fail and rebuild everything. Then .d files will be generated
> and touching any file will rebuild exactly what should be.
>
> The only non portable thing is the -MMD option (gcc and clang support
> though).
but in general is better to just define OBJ instead of defining SRC,
because you as a object generator are interested in the objects, no
in the sources. It can drive to problems because the standard does not
define how the expansion is done in inference rules and things like:
$(OBJ): xxxx
don't work in nmake if OBJ is a expansion of SRC.
Regards,
Reply all
Reply to author
Forward
0 new messages