makefile thoughts
Will Coleda
some cleanup on our makefiles to make them easier to
process/validate/debug.
* tend to avoid separate top level makefiles, going with a single, root makefile
* have that makefile actually do includes of smaller makefile functionality
* rework tools/dev/checkdepend.pl to actually generate the
dependencies at Config time instead of just validating them after a
full build. (and not just for C, but for perl & PIR also.)
Splitting work out amongst that already exists in separate makefiles
will be converted to 'include'-able makefiles. (We can always do an
inline include instead of using a directive if necessary). We can also
split up the existing makefile even further: testing, common variable
defs, pmc dependencies; whatever chunks make sense.
This will remove the ability to remake an individual 'module' (e.g. cd
compilers/tge && make), but will conversely improve the dependencies
across the project and mean that a top level make will only rebuild
those things that need to anyway, and it will include things that
might depend on your module.
This work can be done in stages in trunk with minimal risk.
Questions:
* Where to keep the generated (and potentially static) 'include'-able
.mak files?
* Do we currently support any versions of 'make' that don't support an
include directive of some kind? (gmake and nmake both seem to)
--
Will "Coke" Coleda
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Andy Dougherty
> Because I am tired of fixing bugs with makefile deps, I'd like to do
> some cleanup on our makefiles to make them easier to
> process/validate/debug.
Good luck with that! It is rather complex now -- some directories
have their own makefile while others do not. There are also a number
of little quirks which I've never been able to understand (e.g. MAKE vs
MAKE_C) and for which the commit logs are singularly unhelpful.
> * Do we currently support any versions of 'make' that don't support an
> include directive of some kind? (gmake and nmake both seem to)
Solaris make and dmake (distributed make) and OpenBSD make all support an
include directive.
--
Andy Dougherty doug...@lafayette.edu
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Will Coleda
> On Tue, 17 Nov 2009, Will Coleda wrote:
>
>> Because I am tired of fixing bugs with makefile deps, I'd like to do
>> some cleanup on our makefiles to make them easier to
>> process/validate/debug.
>
> Good luck with that! It is rather complex now -- some directories
> have their own makefile while others do not. There are also a number
> of little quirks which I've never been able to understand (e.g. MAKE vs
> MAKE_C) and for which the commit logs are singularly unhelpful.
>
>> * Do we currently support any versions of 'make' that don't support an
>> include directive of some kind? (gmake and nmake both seem to)
>
> Solaris make and dmake (distributed make) and OpenBSD make all support an
> include directive.
>
> --
> Andy Dougherty doug...@lafayette.edu
>
I've created a branch (one_make) to start cleaning up the makefile.
I've just now eliminated the compiler.dummy target there, and moved
several generated independent makefiles into include'd ones.
I've also broken out a few chunks of static deps into their own files.
Can I get some verification from some non-gmake builds?
Any suggestions on better names than Makefile.mak ?
Andy Dougherty
> I've created a branch (one_make) to start cleaning up the makefile.
> I've just now eliminated the compiler.dummy target there, and moved
> several generated independent makefiles into include'd ones.
>
> I've also broken out a few chunks of static deps into their own files.
>
> Can I get some verification from some non-gmake builds?
OpenSolaris make doesn't like the ':=' style of variables. Something like
the appended patch is needed.
There seem to be some missing dependencies somewhere: A parallel build
with OpenSolaris distributed make (dmake) gave this error:
perl tools/build/ops2c.pl CGP --core
Parrot/OpLib/core.pm did not return a true value at
/export/home/doughera/src/parrot/parrot-svn/tools/build/../../lib/Parrot/Ops2c/Utils.pm line 7.
BEGIN failed--compilation aborted at
/export/home/doughera/src/parrot/parrot-svn/tools/build/../../lib/Parrot/Ops2c/Utils.pm line 7.
Compilation failed in require at tools/build/ops2c.pl line 11.
BEGIN failed--compilation aborted at tools/build/ops2c.pl line 11.
*** Error code 255
dmake: Fatal error: Command failed for target `src/ops/core_ops_cgp.c'
I might have a chance to look into it early next week, but probably not
before then.
> Any suggestions on better names than Makefile.mak ?
Nothing better comes to mind.
Index: ext/nqp-rx/Makefile.mak
===================================================================
--- ext/nqp-rx/Makefile.mak (revision 43325)
+++ ext/nqp-rx/Makefile.mak (working copy)
@@ -1,6 +1,6 @@
## XXX does not cover .includes of core .pasm files
-NQPRX_LIB_PBCS := \
+NQPRX_LIB_PBCS = \
$(LIBRARY_DIR)/Regex.pbc \
$(LIBRARY_DIR)/HLL.pbc \
$(LIBRARY_DIR)/P6Regex.pbc \
Index: compilers/tge/Makefile.mak
===================================================================
--- compilers/tge/Makefile.mak (revision 43325)
+++ compilers/tge/Makefile.mak (working copy)
@@ -1,5 +1,5 @@
-TGE_LIB_PBCS := $(LIBRARY_DIR)/TGE.pbc compilers/tge/tgc.pbc
+TGE_LIB_PBCS = $(LIBRARY_DIR)/TGE.pbc compilers/tge/tgc.pbc
$(LIBRARY_DIR)/TGE.pbc: \
$(LIBRARY_DIR)/PGE.pbc \
Index: compilers/data_json/Makefile.mak
===================================================================
--- compilers/data_json/Makefile.mak (revision 43325)
+++ compilers/data_json/Makefile.mak (working copy)
@@ -1,4 +1,4 @@
-DATA_JSON_LIB_PBCS := compilers/data_json/data_json.pbc
+DATA_JSON_LIB_PBCS = compilers/data_json/data_json.pbc
compilers/data_json/data_json.pbc : $(PARROT) \
compilers/data_json/data_json/grammar.pbc \
Index: compilers/pct/Makefile.mak
===================================================================
--- compilers/pct/Makefile.mak (revision 43325)
+++ compilers/pct/Makefile.mak (working copy)
@@ -1,6 +1,6 @@
## XXX does not cover .includes of core .pasm files
-PCT_LIB_PBCS := \
+PCT_LIB_PBCS = \
$(LIBRARY_DIR)/PCT.pbc \
$(LIBRARY_DIR)/PCT/PAST.pbc \
$(LIBRARY_DIR)/PCT/Grammar.pbc \
Index: compilers/imcc/Makefile.mak
===================================================================
--- compilers/imcc/Makefile.mak (revision 43325)
+++ compilers/imcc/Makefile.mak (working copy)
@@ -1,5 +1,5 @@
# these are private to the IMCC subsystem
-IMCC_O_FILES := \
+IMCC_O_FILES = \
compilers/imcc/imcparser$(O) \
compilers/imcc/imclexer$(O) \
compilers/imcc/imc$(O) \
Index: compilers/pge/Makefile.mak
===================================================================
--- compilers/pge/Makefile.mak (revision 43325)
+++ compilers/pge/Makefile.mak (working copy)
@@ -1,6 +1,6 @@
# the default target
-PGE_LIB_PBCS := $(LIBRARY_DIR)/PGE.pbc
+PGE_LIB_PBCS = $(LIBRARY_DIR)/PGE.pbc
## Two-stage build that reuses builtins_gen.pir ; use the Perl6Grammar.pir
## of stage one to avoid a circular dependency.
Index: compilers/nqp/Makefile.mak
===================================================================
--- compilers/nqp/Makefile.mak (revision 43325)
+++ compilers/nqp/Makefile.mak (working copy)
@@ -1,6 +1,6 @@
-NQP_LIB_PBCS := compilers/nqp/nqp.pbc
+NQP_LIB_PBCS = compilers/nqp/nqp.pbc
-NQP_SOURCES := \
+NQP_SOURCES = \
compilers/nqp/nqp.pir \
compilers/nqp/src/Grammar.pg \
compilers/nqp/src/Grammar/Actions.pir \
@@ -27,7 +27,7 @@
compilers/nqp/bootstrap/nqp.pbc : $(LIBRARY)/PCT.pbc
-NQP_CLEANUPS := \
+NQP_CLEANUPS = \
nqp.pbc \
src/Grammar_gen.pir \
bootstrap/gen_actions.pir \
Will Coleda
> On Wed, 30 Dec 2009, Will Coleda wrote:
>
>
>> I've created a branch (one_make) to start cleaning up the makefile.
>> I've just now eliminated the compiler.dummy target there, and moved
>> several generated independent makefiles into include'd ones.
>>
>> I've also broken out a few chunks of static deps into their own files.
>>
>> Can I get some verification from some non-gmake builds?
>
> OpenSolaris make doesn't like the ':=' style of variables. Something like
> the appended patch is needed.
Something like it applied, thanks. I have dealing with the := issue on
my list, but had blindly copied its usage in the meantime.
Andy Dougherty
> On Wed, 30 Dec 2009, Will Coleda wrote:
>
>
> > I've created a branch (one_make) to start cleaning up the makefile.
> > I've just now eliminated the compiler.dummy target there, and moved
> > several generated independent makefiles into include'd ones.
> >
> There seem to be some missing dependencies somewhere: A parallel build
> with OpenSolaris distributed make (dmake) gave this error:
>
> perl tools/build/ops2c.pl CGP --core
> Parrot/OpLib/core.pm did not return a true value at
> /export/home/doughera/src/parrot/parrot-svn/tools/build/../../lib/Parrot/Ops2c/Utils.pm line 7.
> BEGIN failed--compilation aborted at
> /export/home/doughera/src/parrot/parrot-svn/tools/build/../../lib/Parrot/Ops2c/Utils.pm line 7.
> Compilation failed in require at tools/build/ops2c.pl line 11.
> BEGIN failed--compilation aborted at tools/build/ops2c.pl line 11.
> *** Error code 255
> dmake: Fatal error: Command failed for target `src/ops/core_ops_cgp.c'
It's a race condition. It's present in trunk too. The problem is the
makefile rule:
$(INC_DIR)/oplib/ops.h lib/Parrot/OpLib/core.pm : [etc.]
$(PERL) $(BUILD_TOOLS_DIR)/ops2pm.pl @no_lines_flag@ $(OPS_FILES)
The single tool, ops2pm.pl, generates two files, ops.h and core.pm.
However, make doesn't know that. It sees two separate targets and can
consider running ops2pm.pl for each. Accordingly, ops2pm.pl can get
run twice by a parallel make. Normally, that's not a problem unless
both instances happen to be running at the same time. In that case,
the output files can get messed up. That's what happened to me above.
Here's a simple fix. It does have one minor problem: If the user
ignores the instructions at the top of core.pm and edits it anyway,
then the Makefile will not rebuild ops.h, even if it should.
diff -r -u one_make/config/gen/makefiles/root.in andy/config/gen/makefiles/root.in
--- one_make/config/gen/makefiles/root.in 2010-01-04 09:38:11.578364739 -0500
+++ parrot-andy/config/gen/makefiles/root.in 2010-01-04 08:21:34.000000000 -0500
@@ -1016,7 +1016,10 @@
#
###############################################################################
-$(INC_DIR)/oplib/ops.h lib/Parrot/OpLib/core.pm : $(OPS_FILES) $(BUILD_TOOLS_DIR)/ops2pm.pl \
+# ops.h is built by ops2pm.pl after it builds core.pm
+$(INC_DIR)/oplib/ops.h: lib/Parrot/OpLib/core.pm
+
+lib/Parrot/OpLib/core.pm : $(OPS_FILES) $(BUILD_TOOLS_DIR)/ops2pm.pl \
lib/Parrot/OpsFile.pm lib/Parrot/Op.pm src/ops/ops.num src/ops/ops.skip
$(PERL) $(BUILD_TOOLS_DIR)/ops2pm.pl @no_lines_flag@ $(OPS_FILES)
Nicholas Clark
> It's a race condition. It's present in trunk too. The problem is the
> makefile rule:
>
> $(INC_DIR)/oplib/ops.h lib/Parrot/OpLib/core.pm : [etc.]
> $(PERL) $(BUILD_TOOLS_DIR)/ops2pm.pl @no_lines_flag@ $(OPS_FILES)
>
> The single tool, ops2pm.pl, generates two files, ops.h and core.pm.
> However, make doesn't know that. It sees two separate targets and can
> consider running ops2pm.pl for each. Accordingly, ops2pm.pl can get
> run twice by a parallel make. Normally, that's not a problem unless
> both instances happen to be running at the same time. In that case,
> the output files can get messed up. That's what happened to me above.
Ah. Interesting. I wasn't aware of that. I had assumed that a rule of the
form
target1 target2: ...
action with parameters
meant that targets 1 and 2 were built together by 1 invocation of action,
not that action was capable of building either, and hence had to be run
twice
(and TFM isn't that easy to R, and my book on make was lost when boo.com went
bust, and was never replaced)
That would mean that this rule in perl 5 is wrong:
lib/Config_git.pl git_version.h: $(MINIPERL_EXE) make_patchnum.pl
$(MINIPERL) make_patchnum.pl
http://perl5.git.perl.org/perl.git/blob/HEAD:/Makefile.SH#l562
and likely several others.
Nicholas Clark
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Andy Dougherty
> On Mon, Jan 04, 2010 at 09:55:50AM -0500, Andy Dougherty wrote:
[on the parrot-dev mailing list about multiple targets and parallel make]
> That would mean that this rule in perl 5 is wrong:
>
> lib/Config_git.pl git_version.h: $(MINIPERL_EXE) make_patchnum.pl
> $(MINIPERL) make_patchnum.pl
>
Yes, that rule is wrong.
[Long response not addressed to Nick but to p5p, which hasn't seen any of
the background.]
A rule of the form
target1 target2: ...
action that updates both target1 and target2
is equivalent to two rules
target1: ...
action
target2: ...
action
A parallel make could, in principle, run them both simultaneously.
This results in a race condition, since two separate instances of 'action'
could be simultaneously trying to update the target files.
You can verify this with perl 5 with GNU make by running
sh Configure -Dusedevel -des
make -j 2 perl 2>&1 | tee make.log
grep make_patchnum.pl make.log
You should find two entries for make_patchnum.pl.
If the writes are short and quick (as they are in this perl 5 example)
then a collision is highly unlikely. If the writes are longer and
slower (as they are in the parrot example that spawned this thread)
then a collision becomes more likely.
There are a variety of non-portable ways to tell make not to run these
in parallel. With Sun's dmake, you could write the rule as
target1 + target2: ...
action that updates both target1 and target2
where the '+' sign indicates that the multiple targets are built by a
single invocation of the rule. GNU make doesn't support that notation.
There are also various special targets, such as .WAIT, .NOTPARALLEL, and
.NO_PARALLEL, but the names and meanings vary among different versions of
make, so there's no simple portable invocation.
The workaround I proposed for parrot (assuming target1 is written first by
'action' and then target2) is to simply write
target1: [... target 1 dependencies . . . ]
action that updates both target1 and target2
target2: target1 [... target 1 dependencies . . . ]
This isn't strictly correct, since if you manually update target1, target2
won't get fixed at all. In the case in parrot, target1 has a big 'Do not
edit!!!' header, so I'm not too worried about it.
Another alternative is to rewrite 'action' to use lockfiles or other
race-condition-avoiding techniques.
A third alternative is to rewrite 'action' into two separate actions. For
perl 5, generating lib/Config_git.pl looks to be rather complicated.
However, I'd think that if you can assume lib/Config_git.pl has correctly
been built, then writing git_version.h based on it probably isn't too
hard. Perhaps someone who understands those two files could help split up
the make_patchnum.pl accordingly.
Meanwhile, perhaps I'll try to split up the target. (It'll probably
actually take me longer to remember how to do anything in git than it will
to actually make the actual patch!)
Disclaimer: I am in no way a "parallel make" expert. I just read the
fine manuals and experiment with variants of make debugging flags.
Aristotle Pagaltzis
> I had assumed that a rule of the form
>
> target1 target2: ...
> action with parameters
>
> meant that targets 1 and 2 were built together by 1 invocation
> of action, not that action was capable of building either, and
Actually, writing targets and dependencies on a single line is
pure syntactic sugar in make. The following rule:
target1 target2: dep1 dep2 dep3
something
is completely equivalent to the following:
target1: dep1
target1: dep2
target1: dep3
target2: dep1
target2: dep2
target2: dep3
target1:
something
target2:
something
This is not pseudocode. You can actually write it this way.
Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
James E Keenan
>
> (It'll probably
> actually take me longer to remember how to do anything in git than it will
> to actually make the actual patch!)
>
I know the feeling well. ;-)
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
John P. Linderman
> Disclaimer: I am in no way a "parallel make" expert.
> I just read the fine manuals and experiment with variants
> of make debugging flags.
I'm no expert either, but after buying myself a shiny new
quadcore PC, I've been parallelizing (or sometimes paralyzing)
some makefiles so I can move more than 1 CPU bar on the
system monitor. I often use the construct
@ARGV = qw( file1 file2 file6 ) if ((@ARGV == 0) && -t);
in scripts, so I don't have to remember the files that should
be used by default, but I can pipe a small sample input in
for testing.
One nasty surprise everyone should know about is (with gnu make):
Another problem is that two processes cannot both take
input from the same device; so to make sure that only one
command tries to take input from the terminal at once,
make will invalidate the standard input streams of all but
one running command. This means that attempting to read from
standard input will usually be a fatal error (a `Broken pipe'
signal) for most child processes if there are several. It is
unpredictable which command will have a valid standard input
stream (which will come from the terminal, or wherever you
redirect the standard input of make). The first command run
will always get it first, and the first command started after
that one finishes will get it next, and so on.
This can render -t false in parallel makes, so ARGV remains empty,
and (the empty) standard in is read, not the default files.
Makes that work just fine without -j can break mysteriously
(and not always repeatably) with it. In fact, running "nohupped"
can produce similar results (since the terminal is "disconnected"),
but at least the steps *always* run with no input, not the timing-
dependent behavior that parallel make can induce.
My solution has been to plug in the default file names in the
makefiles. That's a bit unfortunate, because the defaults are
now exposed both in the command and in the makefile, but it
does shake the dust off a couple more cores. -- jpl
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev