Re: makefile thoughts
Nicholas Clark
> It's a race condition. It's present in trunk too. The problem is the
> makefile rule:
>
> $(INC_DIR)/oplib/ops.h lib/Parrot/OpLib/core.pm : [etc.]
> $(PERL) $(BUILD_TOOLS_DIR)/ops2pm.pl @no_lines_flag@ $(OPS_FILES)
>
> The single tool, ops2pm.pl, generates two files, ops.h and core.pm.
> However, make doesn't know that. It sees two separate targets and can
> consider running ops2pm.pl for each. Accordingly, ops2pm.pl can get
> run twice by a parallel make. Normally, that's not a problem unless
> both instances happen to be running at the same time. In that case,
> the output files can get messed up. That's what happened to me above.
Ah. Interesting. I wasn't aware of that. I had assumed that a rule of the
form
target1 target2: ...
action with parameters
meant that targets 1 and 2 were built together by 1 invocation of action,
not that action was capable of building either, and hence had to be run
twice
(and TFM isn't that easy to R, and my book on make was lost when boo.com went
bust, and was never replaced)
That would mean that this rule in perl 5 is wrong:
lib/Config_git.pl git_version.h: $(MINIPERL_EXE) make_patchnum.pl
$(MINIPERL) make_patchnum.pl
http://perl5.git.perl.org/perl.git/blob/HEAD:/Makefile.SH#l562
and likely several others.
Nicholas Clark
Ben Morrow
Presumably the correct solution would be
lib/Config_git.pl: $(MINIPERL_EXE) make_patchnum.pl
$(MINIPERL) make_patchnum.pl
git_version.h: lib/Config_git.pl
$(NOOP)
(do we have a $(NOOP) in the Makefile?)
Ben
H.Merijn Brand
git_version.h: lib/Config_git.pl
@true
or
git_version.h: lib/Config_git.pl
-@echo ""
--
H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, OpenSuSE 10.3, 11.0, and 11.1, AIX 5.2 and 5.3.
http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/
http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
Andy Dougherty
> On Mon, Jan 04, 2010 at 09:55:50AM -0500, Andy Dougherty wrote:
[on the parrot-dev mailing list about multiple targets and parallel make]
> That would mean that this rule in perl 5 is wrong:
>
> lib/Config_git.pl git_version.h: $(MINIPERL_EXE) make_patchnum.pl
> $(MINIPERL) make_patchnum.pl
>
Yes, that rule is wrong.
[Long response not addressed to Nick but to p5p, which hasn't seen any of
the background.]
A rule of the form
target1 target2: ...
action that updates both target1 and target2
is equivalent to two rules
target1: ...
action
target2: ...
action
A parallel make could, in principle, run them both simultaneously.
This results in a race condition, since two separate instances of 'action'
could be simultaneously trying to update the target files.
You can verify this with perl 5 with GNU make by running
sh Configure -Dusedevel -des
make -j 2 perl 2>&1 | tee make.log
grep make_patchnum.pl make.log
You should find two entries for make_patchnum.pl.
If the writes are short and quick (as they are in this perl 5 example)
then a collision is highly unlikely. If the writes are longer and
slower (as they are in the parrot example that spawned this thread)
then a collision becomes more likely.
There are a variety of non-portable ways to tell make not to run these
in parallel. With Sun's dmake, you could write the rule as
target1 + target2: ...
action that updates both target1 and target2
where the '+' sign indicates that the multiple targets are built by a
single invocation of the rule. GNU make doesn't support that notation.
There are also various special targets, such as .WAIT, .NOTPARALLEL, and
.NO_PARALLEL, but the names and meanings vary among different versions of
make, so there's no simple portable invocation.
The workaround I proposed for parrot (assuming target1 is written first by
'action' and then target2) is to simply write
target1: [... target 1 dependencies . . . ]
action that updates both target1 and target2
target2: target1 [... target 1 dependencies . . . ]
This isn't strictly correct, since if you manually update target1, target2
won't get fixed at all. In the case in parrot, target1 has a big 'Do not
edit!!!' header, so I'm not too worried about it.
Another alternative is to rewrite 'action' to use lockfiles or other
race-condition-avoiding techniques.
A third alternative is to rewrite 'action' into two separate actions. For
perl 5, generating lib/Config_git.pl looks to be rather complicated.
However, I'd think that if you can assume lib/Config_git.pl has correctly
been built, then writing git_version.h based on it probably isn't too
hard. Perhaps someone who understands those two files could help split up
the make_patchnum.pl accordingly.
Meanwhile, perhaps I'll try to split up the target. (It'll probably
actually take me longer to remember how to do anything in git than it will
to actually make the actual patch!)
Disclaimer: I am in no way a "parallel make" expert. I just read the
fine manuals and experiment with variants of make debugging flags.
--
Andy Dougherty doug...@lafayette.edu
Aristotle Pagaltzis
> I had assumed that a rule of the form
>
> target1 target2: ...
> action with parameters
>
> meant that targets 1 and 2 were built together by 1 invocation
> of action, not that action was capable of building either, and
Actually, writing targets and dependencies on a single line is
pure syntactic sugar in make. The following rule:
target1 target2: dep1 dep2 dep3
something
is completely equivalent to the following:
target1: dep1
target1: dep2
target1: dep3
target2: dep1
target2: dep2
target2: dep3
target1:
something
target2:
something
This is not pseudocode. You can actually write it this way.
Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>
(John P. Linderman)
> Disclaimer: I am in no way a "parallel make" expert.
> I just read the fine manuals and experiment with variants
> of make debugging flags.
I'm no expert either, but after buying myself a shiny new
quadcore PC, I've been parallelizing (or sometimes paralyzing)
some makefiles so I can move more than 1 CPU bar on the
system monitor. I often use the construct
@ARGV = qw( file1 file2 file6 ) if ((@ARGV == 0) && -t);
in scripts, so I don't have to remember the files that should
be used by default, but I can pipe a small sample input in
for testing.
One nasty surprise everyone should know about is (with gnu make):
Another problem is that two processes cannot both take
input from the same device; so to make sure that only one
command tries to take input from the terminal at once,
make will invalidate the standard input streams of all but
one running command. This means that attempting to read from
standard input will usually be a fatal error (a `Broken pipe'
signal) for most child processes if there are several. It is
unpredictable which command will have a valid standard input
stream (which will come from the terminal, or wherever you
redirect the standard input of make). The first command run
will always get it first, and the first command started after
that one finishes will get it next, and so on.
This can render -t false in parallel makes, so ARGV remains empty,
and (the empty) standard in is read, not the default files.
Makes that work just fine without -j can break mysteriously
(and not always repeatably) with it. In fact, running "nohupped"
can produce similar results (since the terminal is "disconnected"),
but at least the steps *always* run with no input, not the timing-
dependent behavior that parallel make can induce.
My solution has been to plug in the default file names in the
makefiles. That's a bit unfortunate, because the defaults are
now exposed both in the command and in the makefile, but it
does shake the dust off a couple more cores. -- jpl