makefile rules when multiple targets are generated by a single command

[James Harris]

Basic query: Is there a way to tell make to run a command if either of
two targets needs to be generated and yet both are produced by the same
command?

I need to run an old assembler, as86, as follows.

as86 program.as86 -o program.o -s program.o.symbols

That command line is simplified to remove various switches but, as you
can see, it produces both an object file AND a symbol file in one go. I
know I can write a makefile recipe along the lines of

A B: C
as86 $< -o A -s B

but as far as I understand it that is meant to tell make to consider
running the same recipe twice, once to produce A and once to produce B.
What I am looking for is a way to tell make to run the recipe once if
either of A and B need to be updated, and, ideally, to let it know that
one command will produce both of them.

Maybe someone's come across this before.

Any suggestions?


--
James Harris

[Jorgen Grahn]

This looks fine to me, with GNU make:

foo bar: baz
touch foo bar

But I may have misunderstood what you want. Can you point out a
concrete scenario that the makefile above doesn't support?

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[Kaz Kylheku]

Yes, someone's come across this before: for instance, "yacc" producing
y.tab.h an y.tab.c, the one being involved in dependencies, and the
other needed for y.tab.o.

In the Makefile for the TXR language, I do this (stripped of extraneous
details):

y.tab.c: parser.y # y.tab.h not expressed here
... logic to update y.tab.c

y.tab.h: y.tab.c
if ! [ -e y.tab.h ] ; then \
echo "Someone removed y.tab.h but left y.tab.c" ; \
echo "Remove y.tab.c and re-run make" ; \
exit 1 ; \
fi

So we are expressing that y.tab.h depends on y.tab.c (and therefore on
parser.y indirectly!)

y.tab.h always exists if y.tab.c does, unless foul play is going on:
someone removed it, but left behind y.tab.c. We diagnose the foul play.

Otherwise, we supply no command to rebuild y.tab.h; it's a null action.
If it exists, we are good (we don't care if it's newer or older than
y.tab.c).

This takes care of the requirement that if we touch parser.y and rebuild
y.tab.c, then y.tab.h is considered out of date, and all of its
dependencies get rebuilt.

[Kaz Kylheku]

Unfortunately:

4.10 Multiple Targets in a Rule
===============================

A rule with multiple targets is equivalent to writing many rules, each
with one target, and all identical aside from that. The same commands
apply to all the targets, but their effects may vary because you can
substitute the actual target name into the command using `$@'. The
rule contributes the same prerequisites to all the targets also.

So the above is equivalent to:

foo: baz
touch foo bar

bar: baz
touch foo bar

Not what we want, which is for the command to be run once.

That's not necessarily just an optimization. Even if the command is
idempotent (multiple serial executions of it have the same effect as
just one), there may be incorrect results somehow if the command is
duplicated in parallel.

Something depending on "foo" is not aware that a "bar:" rule is
executing at the same time, overwriting foo.

Of course, for such ailments, you can just take a .NOTPARALLEL and
call me in the morning.

[Jorgen Grahn]

Ok, so in concrete terms:

% touch baz; make -j2 foo bar
touch foo bar
touch foo bar

I didn't see that coming. Surprising, and disappointing.

> That's not necessarily just an optimization. Even if the command is
> idempotent (multiple serial executions of it have the same effect as
> just one), there may be incorrect results somehow if the command is
> duplicated in parallel.
>
> Something depending on "foo" is not aware that a "bar:" rule is
> executing at the same time, overwriting foo.
>
> Of course, for such ailments, you can just take a .NOTPARALLEL and
> call me in the morning.

[James Harris]

Thanks, that's exactly the issue.

Can anyone think of a good way to change make to have it deal with this
issue in a better way?


--
James Harris

[Rainer Weikusat]

James Harris <james.h...@gmail.com> writes:
> Basic query: Is there a way to tell make to run a command if either of
> two targets needs to be generated and yet both are produced by the
> same command?

May not be useful but assuming there's an assembly source file a.s and
an object code file a.o and a listing file a.l are to be generated from
that, the following (GNU) Makefile seems to work as intended (minus
printing a warning):

--------
ALL: a.o a.l

a.%: a.s
as -o a.o -a a.s >a.l

[Kaz Kylheku]

On 2019-03-11, James Harris <james.h...@gmail.com> wrote:
> Can anyone think of a good way to change make to have it deal with this
> issue in a better way?

Found this in the GNU Make manual:

This pattern rule has two targets:

%.tab.c %.tab.h: %.y
bison -d $<

This tells `make' that the command `bison -d X.y' will make both
`X.tab.c' and `X.tab.h'. If the file `foo' depends on the files
`parse.tab.o' and `scan.o' and the file `scan.o' depends on the file
`parse.tab.h', when `parse.y' is changed, the command `bison -d parse.y'
will be executed only once, and the prerequisites of both `parse.tab.o'
and `scan.o' will be satisfied.

This is gratuitously inconsistent with how GNU Make treats non-pattern targets.

It prohibits us from using a simple instantiation model to explain how pattern
rules work. I.e. it cannot just be the case that a match is found for the %
stem, and then a direct rule is instantiated with that stem and executed,
because that direct rule will treat x.tab.c and x.tab.h independently.

I would propose parenthesized rules:

(y.tab.c y.tab.h): parser.y ...

if the target list is parenthesized, it is treated as a group.

I think, mixtures would not be allowed: either all the targets are
parenthesized, or else none are.

A single parenthesized target is equivalent to an unparenthesized one:

(x): y <==> x: y

Then an "backsplanation" can be introduced that pattern rules behave as if
their targets have parentheses around them.

In any case, the good news is that GNU Make must already have the logic already
to treat multiple targets as one unit to be supporting the instantiation of
these multi-target pattern rules. Introducing this (...) syntax could just be
as simple as just introducing that syntax, and connecting it with existing
semantics.

--
TXR Programming Lanuage: http://nongnu.org/txr
Music DIY Mailing List: http://www.kylheku.com/diy
ADA MP-1 Mailing List: http://www.kylheku.com/mp1

[Kaz Kylheku]

You're very close; turns out GNU Make has a special behavior in pattern
rules, and thus this can be expected to work if the documentation is
telling the truth:

%.o %.l: %.s
as -o %.o -a %.s > %.l

But if this is manually expanded to

a.o a.l: a.s

as -o a.o -a a.s > a.l

the semantics changes.

[Rainer Weikusat]

Kaz Kylheku <157-07...@kylheku.com> writes:
> On 2019-03-11, Rainer Weikusat <rwei...@talktalk.net> wrote:
>> James Harris <james.h...@gmail.com> writes:
>>> Basic query: Is there a way to tell make to run a command if either of
>>> two targets needs to be generated and yet both are produced by the
>>> same command?
>>
>> May not be useful but assuming there's an assembly source file a.s and
>> an object code file a.o and a listing file a.l are to be generated from
>> that, the following (GNU) Makefile seems to work as intended (minus
>> printing a warning):
>>
>> --------
>> ALL: a.o a.l
>>
>> a.%: a.s
>> as -o a.o -a a.s >a.l
>
> You're very close; turns out GNU Make has a special behavior in pattern
> rules, and thus this can be expected to work if the documentation is
> telling the truth:
>
> %.o %.l: %.s
> as -o %.o -a %.s > %.l

This needs to use the 'stem' automatic variable, ie,

-------
ALL: a.o a.l

%.o %.l: %.s
as -o $*.o -a $*.s >$*.l
-------

But the example I gave works as well (for the problem it's supposed to
solve).

[Siri Cruise]

In article <q63mmf$c31$1...@dont-email.me>,

James Harris <james.h...@gmail.com> wrote:

> A B: C
> as86 $< -o A -s B
>
> but as far as I understand it that is meant to tell make to consider
> running the same recipe twice, once to produce A and once to produce B.
> What I am looking for is a way to tell make to run the recipe once if
> either of A and B need to be updated, and, ideally, to let it know that
> one command will produce both of them.

I do this on MacOSX just fine. I think it is gnu make underneath.

--
:-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted. @
'I desire mercy, not sacrifice.' /|\
The first law of discordiamism: The more energy This post / \
to make order is nore energy made into entropy. insults Islam. Mohammed

[James K. Lowden]

On Mon, 11 Mar 2019 17:13:25 +0000 (UTC)
Kaz Kylheku <157-07...@kylheku.com> wrote:

> I would propose parenthesized rules:
>
> (y.tab.c y.tab.h): parser.y ...

+1

I've run into this problem more than once, at least once a year. As
someone else mentioned, yacc is one use case. Many code generators
have the property of producing defintions and logic in two files from
one source.

--jkl

[Kaz Kylheku]

It seems I have it working.

Alas, the proposed (...) syntax for this was not meant to be. GNU Make reads
line by line and performs ad hoc tokenizing, with cruft in it like the
foo.a(bar.o xyzzy.o) archive syntax and of course variable/macro expansions.
Working parentheses around the target list in that grotty code would be a lot
of work, and create risk.

So, syntax-wise, I have it like this:


+ y.tab.c y.tab.h: parser.y ...
recipe ...

The + symbol is already involved in += syntax. If it's not followed by
=, then it's not recognized; I added a new token type for it when it's
followed by whitspace or EOL.

When the + symbol is present, then every file is added to the
"also_make" list of every other file, which groups them all together
as one lump.

How it works is that the rules are still replicated exactly as before
for each target; it's this "also_make" linkage which suppresses the
duplicate recipe invocations; Make understands that if the recipe is run
for a target, then all of its "also_make" targets are considered
updated.

Patch against GNU Make 4.2.1:

8-<-------8-<-------8-<-------8-<-------8-<-------8-<-------8-<-------8-<---

From: Kaz Kylheku <k...@kylheku.com>
Date: Tue, 12 Mar 2019 19:35:02 -0700
Subject: [PATCH] Implement "grouped targets" in ordinary rules.

This patch adds the + (plus) syntax:

+ tgt1 tgt2 ... tgtn : pre1 pre2 ...
recipe

When the + is present, then the targets are understood to be built
together by one invocation of the recipe: each one lists the others in
its "also_make" list, similarly to what multiple-target pattern rules
already do.

* doc/make.texi: Section on Multiple Targets updated.

* read.c (enum make_word_type): New enum member, w_plus.
(record_files): New argument, are_also_makes flag which indicates
that the targets are all linked together as also-makes.
If this is true, then we build an also_make list which contains
each target represented as a dep. When all targets are registered,
we then make a second pass, installing a copy of this also_make
list into each target, possibly catenating it with an existing
also_make list.
(record_waiting_files): Pass new argument to record_files.
(eval): Check for the + token and set a new local variable
also_make_targets if that is so. This is then passed to
record_files by the record_waiting_files macro.
(get_next_mword): If a + is followed by whitespace, or end of string,
rathe than =, then instead of falling through, recognize it as the new
w_plus token.
---
doc/make.texi | 41 ++++++++++++++++++++++++++++-----
read.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++----
2 files changed, 95 insertions(+), 10 deletions(-)

diff --git a/doc/make.texi b/doc/make.texi
index 01bcec7..033c72d 100644
--- a/doc/make.texi
+++ b/doc/make.texi
@@ -3012,13 +3012,27 @@ both pieces to the suffix list. In practice, suffixes normally begin with
@cindex targets, multiple
@cindex rule, with multiple targets

-A rule with multiple targets is equivalent to writing many rules, each with
-one target, and all identical aside from that. The same recipe applies to
-all the targets, but its effect may vary because you can substitute the
-actual target name into the recipe using @samp{$@@}. The rule contributes
-the same prerequisites to all the targets also.
+When a rule has multiple targets, the semantics depends on what kind
+of rule it is, and whether the "grouped targets" feature is invoked.

-This is useful in two cases.
+There are two possible semantics. A rule with multiple targets may be
+equivalent to writing many rules, each with one target, and all identical aside
+from that. Under this semantics, the same recipe applies to all the targets,
+but its effect may vary because you can substitute the actual target name into
+the recipe using @samp{$@@}. The rule contributes the same prerequisites to
+all the targets also. This is known as "replicated recipe semantics".
+The other semantics is "grouped target" semantics. Under this semantics,
+the files are all understood to be updated or created together by a single
+invocation of the recipe.
+
+Pattern rules implicitly follow "grouped target" semantics.
+
+Ordinary rules use "replicated recipe" semantics by default. If
+the symbol @samp{+} is placed before the first target, separated
+from it by whitespace, then the rule follows "grouped target"
+semantics.
+
+"Replicated recipe semantics" is useful in two cases.

@itemize @bullet
@item
@@ -3064,6 +3078,21 @@ types of output, one if given @samp{-big} and one if given
for an explanation of the @code{subst} function.
@end itemize

+"Grouped target" semantics is useful when a recipe builds multiple
+files, which are further involved in dependencies. For instance,
+it can express a rule rule for building a parser using Yacc, whose
+recipe generates @samp{y.tab.c} and @samp{y.tab.h} at the same time.
+Both are made to be dependent on @samp{parser.y}. If neither one
+exists, or if both are out of date with respect to @samp{parser.y},
+the recipe is invoked only once. Furthermore, if for some reason
+just one of these two files is missing or out of date, GNU Make
+knows that running the recipe also updates the other.
+
+Note that under "grouped target" semantics, the @samp{$@@} variable
+still expands to a single target: it refers to that target which
+triggered the execution of the recipe. The recipe cannot rely on
+@samp{$@@} being a stable reference to a particular one of the targets.
+
Suppose you would like to vary the prerequisites according to the
target, much as the variable @samp{$@@} allows you to vary the recipe.
You cannot do this with multiple targets in an ordinary rule, but you
diff --git a/read.c b/read.c
index b870aa8..983bdc5 100644
--- a/read.c
+++ b/read.c
@@ -71,7 +71,7 @@ struct vmodifiers
enum make_word_type
{
w_bogus, w_eol, w_static, w_variable, w_colon, w_dcolon, w_semicolon,
- w_varassign
+ w_varassign, w_plus
};


@@ -141,7 +141,8 @@ static void do_undefine (char *name, enum variable_origin origin,
static struct variable *do_define (char *name, enum variable_origin origin,
struct ebuffer *ebuf);
static int conditional_line (char *line, int len, const floc *flocp);
-static void record_files (struct nameseq *filenames, const char *pattern,
+static void record_files (struct nameseq *filenames, int are_also_makes,
+ const char *pattern,
const char *pattern_percent, char *depstr,
unsigned int cmds_started, char *commands,
unsigned int commands_idx, int two_colon,
@@ -576,6 +577,7 @@ eval (struct ebuffer *ebuf, int set_default)
unsigned int cmds_started, tgts_started;
int ignoring = 0, in_ignored_define = 0;
int no_targets = 0; /* Set when reading a rule without targets. */
+ int also_make_targets = 0; /* Set when reading grouped targets. */
struct nameseq *filenames = 0;
char *depstr = 0;
long nlines = 0;
@@ -593,7 +595,8 @@ eval (struct ebuffer *ebuf, int set_default)
{ \
fi.lineno = tgts_started; \
fi.offset = 0; \
- record_files (filenames, pattern, pattern_percent, depstr, \
+ record_files (filenames, also_make_targets, pattern, \
+ pattern_percent, depstr, \
cmds_started, commands, commands_idx, two_colon, \
prefix, &fi); \
filenames = 0; \
@@ -601,6 +604,7 @@ eval (struct ebuffer *ebuf, int set_default)
commands_idx = 0; \
no_targets = 0; \
pattern = 0; \
+ also_make_targets = 0; \
} while (0)

pattern_percent = 0;
@@ -1027,6 +1031,20 @@ eval (struct ebuffer *ebuf, int set_default)
beginning, expanding as we go, and looking for "interesting"
chars. The first word is always expandable. */
wtype = get_next_mword (line, NULL, &lb_next, &wlen);
+
+ /* If the line starts with + followed by whitespace, then
+ it specifies that the targets are understood to be updated/created
+ together by a single invocation of the recipe. In this case,
+ we record a single entry for the first target which follows the +,
+ and install the remaining targets as the also_make list of deps
+ for that file, rather than iterating over the targets and replicating
+ the rule for each one. */
+ if (wtype == w_plus) {
+ also_make_targets = 1;
+ lb_next += wlen;
+ wtype = get_next_mword (lb_next, NULL, &lb_next, &wlen);
+ }
+
switch (wtype)
{
case w_eol:
@@ -1931,7 +1949,8 @@ record_target_var (struct nameseq *filenames, char *defn,
that are not incorporated into other data structures. */

static void
-record_files (struct nameseq *filenames, const char *pattern,
+record_files (struct nameseq *filenames, int are_also_makes,
+ const char *pattern,
const char *pattern_percent, char *depstr,
unsigned int cmds_started, char *commands,
unsigned int commands_idx, int two_colon,
@@ -1939,6 +1958,7 @@ record_files (struct nameseq *filenames, const char *pattern,
{
struct commands *cmds;
struct dep *deps;
+ struct dep *also_make = 0;
const char *implicit_percent;
const char *name;

@@ -2158,6 +2178,15 @@ record_files (struct nameseq *filenames, const char *pattern,
f->cmds = cmds;
}

+ if (are_also_makes)
+ {
+ struct dep *also = alloc_dep();
+ also->name = f->name;
+ also->file = f;
+ also->next = also_make;
+ also_make = also;
+ }
+
f->is_target = 1;

/* If this is a static pattern rule, set the stem to the part of its
@@ -2222,6 +2251,27 @@ record_files (struct nameseq *filenames, const char *pattern,
O (error, flocp,
_("*** mixed implicit and normal rules: deprecated syntax"));
}
+
+ /* If the files are also-makes, then populate a copy of the also-make list
+ into each one. */
+
+ if (are_also_makes && also_make)
+ {
+ struct dep *i;
+
+ for (i = also_make; i != 0; i = i->next)
+ {
+ struct file *f = i->file;
+ struct dep *also_copy = copy_dep_chain(also_make);
+ struct dep *j;
+
+ for (j = also_copy; j->next != 0; j = j->next)
+ ;
+
+ j->next = f->also_make;
+ f->also_make = also_copy;
+ }
+ }
}

/* Search STRING for an unquoted STOPCHAR or blank (if BLANK is nonzero).
@@ -2676,6 +2726,12 @@ get_next_mword (char *buffer, char *delim, char **startp, unsigned int *length)
break;

case '+':
+ if (isspace(*p) || *p == 0)
+ {
+ wtype = w_plus;
+ break;
+ }
+ /* fallthrough */
case '?':
case '!':
if (*p == '=')

[Kaz Kylheku]

On 2019-03-10, James Harris <james.h...@gmail.com> wrote:

> Basic query: Is there a way to tell make to run a command if either of
> two targets needs to be generated and yet both are produced by the same
> command?

There is now, in GNU Make. Prompted by this discussion, I implemented
it a couple of months ago:

http://git.savannah.gnu.org/cgit/make.git/commit/?id=8c888d95f61814dd698bf76126a9079234080d77

You must use the &: separator instead of the colon.

[Jorgen Grahn]

On Mon, 2019-05-27, Kaz Kylheku wrote:
> On 2019-03-10, James Harris <james.h...@gmail.com> wrote:
>> Basic query: Is there a way to tell make to run a command if either of
>> two targets needs to be generated and yet both are produced by the same
>> command?
>
> There is now, in GNU Make. Prompted by this discussion, I implemented
> it a couple of months ago:
>
> http://git.savannah.gnu.org/cgit/make.git/commit/?id=8c888d95
>

> You must use the &: separator instead of the colon.

Cool! Thanks! I especially like the large diff to the documentation.

I stupidly assumed the maintainers wouldn't accept additions to the
core Make functionality.

[Ben Bacarisse]

Kaz Kylheku <847-11...@kylheku.com> writes:

> On 2019-03-10, James Harris <james.h...@gmail.com> wrote:
>> Basic query: Is there a way to tell make to run a command if either of
>> two targets needs to be generated and yet both are produced by the same
>> command?
>
> There is now, in GNU Make. Prompted by this discussion, I implemented
> it a couple of months ago:
>
> http://git.savannah.gnu.org/cgit/make.git/commit/?id=8c888d95f61814dd698bf76126a9079234080d77
>
> You must use the &: separator instead of the colon.

Nice. I remember hitting this problem a long time ago (with another make).

--
Ben.

[William Ahern]

Kaz Kylheku <847-11...@kylheku.com> wrote:
> On 2019-03-10, James Harris <james.h...@gmail.com> wrote:
>> Basic query: Is there a way to tell make to run a command if either of
>> two targets needs to be generated and yet both are produced by the same
>> command?
>
> There is now, in GNU Make. Prompted by this discussion, I implemented
> it a couple of months ago:
>
> http://git.savannah.gnu.org/cgit/make.git/commit/?id=8c888d95f61814dd698bf76126a9079234080d77
>
> You must use the &: separator instead of the colon.

Excellent.

I wish Apple could be presuaded to officially fork their ancient version of
GNU Make and backport this and other features. While GNU Make's LISPy
extension syntax is regrettable (thankfully not applicable to this latest
extension), from a pragmatic perspective it's always been an easy decision
to rely on GNU Make to solve portability headaches.[1] But Apple's refusal
to update has become increasingly problematic as GNU Make and other makes
(e.g. NetBSD Make, OpenBSD Make) have evolved.

[1] Notwithstanding differences of opinion on the threshold for switching to
GNU Make.

[john...@nowhere.co.uk]

Apple don't bother to ship make any more (along with a load of other useful
standard unix stuff), you have to install it manually using brew and when you
do you get gnu make.

fenris$ uname -a
Darwin fenris 18.6.0 Darwin Kernel Version 18.6.0: Thu Apr 25 23:16:27 PDT 2019;
root:xnu-4903.261.4~2/RELEASE_X86_64 x86_64
fenris$ make -v
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0