Bug#532805: dh_auto_build: prevents make jobs from being run simultaneously
Jakub Wilk
Version: 7.2.15
Severity: normal
Using dh_auto_build prevents make jobs from being run simultaneously:
$ make -j1
AFTER
BEFORE
$ make -j2
BEFORE
AFTER
$ dpkg-buildpackage -j2
[snip]
dh_auto_build
make[1]: Entering directory `/home/jwilk/trunk/bugs/makejobs-1'
make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule.
AFTER
BEFORE
make[1]: Leaving directory `/home/jwilk/trunk/bugs/makejobs-1'
[snip]
(Perhaps some other debhelper parts which call make are affected with
the same issue.)
--
Jakub Wilk
Joey Hess
> $ make -j2
> BEFORE
> AFTER
>
> $ dpkg-buildpackage -j2
> [snip]
> dh_auto_build
> make[1]: Entering directory `/home/jwilk/trunk/bugs/makejobs-1'
> make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule.
make relies on submakes being called via $(MAKE) in the make file in
order to support parallel builds. Since the rules file runs a submake
via a debhelper command, even though that command then runs $MAKE,
make will not do its parallel magic (specifically, it won't leave
the jobserver FDs open when running the debhelper command). You can
see the same result with this simple case:
build: foo bar
foo:
echo foo
bar:
echo bar
FAKE=$(MAKE)
debian_rules_build:
$(FAKE)
If you run 'make -j2 debian_rules_build', it will not parallelize.
If the last line is changed to use $(MAKE), it will.
Or, you can change it to +$(FAKE); this causes make to treat the line
as a sub-make invocation.
So, there's nothing debhelper can do to help here; it's up to your rules
file. If you want to work around this make limitation, you will have to
ensure your rules file calls $(MAKE) directly, with something like this:
override_dh_auto_build:
$(MAKE)
Or, modify the call to dh_auto_build or dh to be prefixed with a +:
%:
+dh $@
--
see shy jo
Jakub Wilk
>make relies on submakes being called via $(MAKE) in the make file in
>order to support parallel builds. Since the rules file runs a submake
>via a debhelper command, even though that command then runs $MAKE,
>make will not do its parallel magic (specifically, it won't leave
>the jobserver FDs open when running the debhelper command). You can
>see the same result with this simple case:
>
>build: foo bar
>
>foo:
> echo foo
>bar:
> echo bar
>
>FAKE=$(MAKE)
>debian_rules_build:
> $(FAKE)
>
>If you run 'make -j2 debian_rules_build', it will not parallelize.
>If the last line is changed to use $(MAKE), it will.
>Or, you can change it to +$(FAKE); this causes make to treat the line
>as a sub-make invocation.
>
>So, there's nothing debhelper can do to help here; it's up to your rules
>file.
>If you want to work around this make limitation, you will have to
>ensure your rules file calls $(MAKE) directly, with something like this:
>
>override_dh_auto_build:
> $(MAKE)
Well, that looks promising but does not actually fix anything.
>Or, modify the call to dh_auto_build or dh to be prefixed with a +:
>
>%:
> +dh $@
That one is fine, thanks.
Maybe examples in /usr/share/doc/debhelper/examples/ and dh(1) should be
fixed to take this issue into consideration?
--
Jakub Wilk
--
To UNSUBSCRIBE, email to debian-bugs-...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Joey Hess
>> override_dh_auto_build:
>> $(MAKE)
> Well, that looks promising but does not actually fix anything.
In my tests it worked just as well as the other approach.
>> %:
>> +dh $@
> That one is fine, thanks.
>
> Maybe examples in /usr/share/doc/debhelper/examples/ and dh(1) should be
> fixed to take this issue into consideration?
Possibly.. it does add even more crazy punctuation and probably many
packages cannot parallel build anyway.
--
see shy jo
Modestas Vainius
thanks
Hello,
On sekmadienis 14 Birželis 2009 03:43:25 Joey Hess wrote:
> Jakub Wilk wrote:
> > $ make -j2
> > BEFORE
> > AFTER
[ snip ]
>
> So, there's nothing debhelper can do to help here; it's up to your rules
> file. If you want to work around this make limitation, you will have to
> ensure your rules file calls $(MAKE) directly, with something like this:
>
> override_dh_auto_build:
> $(MAKE)
>
> Or, modify the call to dh_auto_build or dh to be prefixed with a +:
>
> %:
> +dh $@
Actually, it turns out makefile.pm can support parallel building without any
additional trickery needed from package maintainer. It just needs to handle
DEB_BUILD_OPTIONS parallel option and clean up MAKEFLAGS a bit in order to
avoid confusing users with make warnings or prevent it from enforcing -j1.
The patch is attached. More details in the patch header. Please consider
applying it.
--
Modestas Vainius <mode...@vainius.eu>
Joey Hess
> Actually, it turns out makefile.pm can support parallel building without any
> additional trickery needed from package maintainer. It just needs to handle
> DEB_BUILD_OPTIONS parallel option and clean up MAKEFLAGS a bit in order to
> avoid confusing users with make warnings or prevent it from enforcing -j1.
> + if (exists $ENV{DEB_BUILD_OPTIONS} && $ENV{DEB_BUILD_OPTIONS} =~ /\bparallel=(\d+)\b/) {
> + unshift @_, "-j$1";
> + # Make sure a new jobserver is started.
> + cleanup_makeflags("force");
I need to be careful here. This change makes *all* packages that use
a make-based buildsystem start honoring DEB_BUILD_OPTIONS=parallel.
But, turning on -j is known to break many crummy Makefiles. That's why
DEB_BUILD_OPTIONS=parallel is documented as something that should only
be enabled if the package's build system is known to support it.
So, if we make this change, someone who currently always builds with
dpkg-buildpackage -j2 may suddently see packages failing to build, or
even misbuilding somehow. This seems very risky.
Maybe instead dh needs to get a -j option; then if a rules file wants
to support DEB_BUILD_OPTIONS it could set the option. Or something along
those lines. However, would that really be any better than my earlier
suggestion for adding an override target?
--
see shy jo
Jakub Wilk
>Maybe instead dh needs to get a -j option; then if a rules file wants
>to support DEB_BUILD_OPTIONS it could set the option. Or something along
>those lines. However, would that really be any better than my earlier
>suggestion for adding an override target?
$ printf 'override_dh_auto_build:\n\t$(MAKE)\n' >> debian/rules
$ dpkg-buildpackage -j2
[snip]
debian/rules override_dh_auto_build
make[1]: Entering directory `/home/jwilk/vault/download/1/makejobs-1'
make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule.
/usr/bin/make
make[2]: Entering directory `/home/jwilk/vault/download/1/makejobs-1'
AFTER
BEFORE
make[2]: Leaving directory `/home/jwilk/vault/download/1/makejobs-1'
[snip]
In fact, when you are using overrides, you are in an even worse
situation, as you cannot fix anything with the "+" hack.
Modestas Vainius
On šeštadienis 08 Rugpjūtis 2009 22:52:22 Joey Hess wrote:
> I need to be careful here. This change makes *all* packages that use
> a make-based buildsystem start honoring DEB_BUILD_OPTIONS=parallel.
>
> But, turning on -j is known to break many crummy Makefiles. That's why
> DEB_BUILD_OPTIONS=parallel is documented as something that should only
> be enabled if the package's build system is known to support it.
>
> So, if we make this change, someone who currently always builds with
> dpkg-buildpackage -j2 may suddently see packages failing to build, or
> even misbuilding somehow. This seems very risky.
>
> Maybe instead dh needs to get a -j option; then if a rules file wants
> to support DEB_BUILD_OPTIONS it could set the option. Or something along
> those lines. However, would that really be any better than my earlier
> suggestion for adding an override target?
We have a couple of issues here:
1) dpkg-buildpackage -jX sets both parallel=X AND MAKEFLAGS=-jX. Therefore,
regardless if the package declares support for parallel=X or not, but uses
$(MAKE) to build the package, it will build with -jX. Currently, dh "wall"
prevents this behaviour, but it is possible to fix this in a clean way
(without '+' hacks or overriding). But do we want to fix this? Btw, I'm
proposing a clean implementation of your '+' hack here.
Obviously, if only DEB_BUILD_OPTIONS has parallel=X without MAKEFLAGS set,
parallel should not be automatically enabled.
2) Supporting overriding via parallel is non-optimal since parallel is an
optional but official feature of Debian Policy. debhelper/dh should provide
support for standardized things if possible (and it is possible).
3) The way to solve this is indeed dh -j/--parallel option to declare that the
package supports parallel. What is more, some build systems (e.g. cmake)
always generates parallel-safe Makefiles so it makes much sense to default to
--parallel for them. However, the fact that makefile.pm is auto-detected at
"build" step for cmake is not helping this case and unfortunately, I don't see
another way how how to to solve this but some temporary file hacks (urgh!) or
reordering of @BUILDSYSTEMS (moving cmake above makefile).
So waiting for your answer. This bug IS a show stopper for huge source
packages which could be built much faster on SMP systems (which are commodity
these days).
--
Modestas Vainius <mode...@vainius.eu>
Joey Hess
> 1) dpkg-buildpackage -jX sets both parallel=X AND MAKEFLAGS=-jX.
My opnion of dpkg-buildpackage's recent spate of interfering with the build
environment just went down another notch. They seem not to grok the
intent of DPKG_BUILD_OPTIONS=parallel, which is to allow the package
maintainer to *decide* if parallel builds are supported.
> Obviously, if only DEB_BUILD_OPTIONS has parallel=X without MAKEFLAGS set,
> parallel should not be automatically enabled.
Not obvious to me. Policy only talks about DEB_BUILD_OPTIONS=parallel, not
MAKEFLAGS. MAKEFLAGS may have nothing to do with the actual build
system. Packages may be built without use of dpkg-buildpackage.
> 2) Supporting overriding via parallel is non-optimal since parallel is an
> optional but official feature of Debian Policy. debhelper/dh should provide
> support for standardized things if possible (and it is possible).
Not sure what you're saying here. Supporting *what* overriding parallel
is non-optimal?
> 3) The way to solve this is indeed dh -j/--parallel option to declare that the
> package supports parallel. What is more, some build systems (e.g. cmake)
> always generates parallel-safe Makefiles so it makes much sense to default to
> --parallel for them. However, the fact that makefile.pm is auto-detected at
> "build" step for cmake is not helping this case and unfortunately, I don't see
> another way how how to to solve this but some temporary file hacks (urgh!) or
> reordering of @BUILDSYSTEMS (moving cmake above makefile).
Are you really sure cmake always supports parallel builds? It's
easy to take an innocuous feature, like the ability to run an arbitrary
command during the build, and use that to make a makefile that breaks
parallelism. Not being familiar with cmake, I don't know if it has
such a feature, but that would be a significant feature to leave out.
--
see shy jo
Modestas Vainius
On ketvirtadienis 27 Rugpjūtis 2009 00:09:45 you wrote:
> Modestas Vainius wrote:
> > 1) dpkg-buildpackage -jX sets both parallel=X AND MAKEFLAGS=-jX.
>
> My opnion of dpkg-buildpackage's recent spate of interfering with the build
> environment just went down another notch. They seem not to grok the
> intent of DPKG_BUILD_OPTIONS=parallel, which is to allow the package
> maintainer to *decide* if parallel builds are supported.
But maybe it is not such a bad idea after all? If you want to *safely* build
in parallel mode, export DEB_BUILD_OPTIONS=parallel=X manually (dpkg-
buildpackage messes with MAKEFLAGS if and only if -jX option is passed to it
regardless of DEB_BUILD_OPTIONS contents). So dpkg-buildpackage does not
violate policy. `dpkg-buildpackage -jX` != DEB_BUILD_OPTIONS=parallel=X, it is
more.
However, if you want to *try* to build in a parallel mode even if the package
does not declare support for it (packaging might be out of date, maintainer
might not care or something), you use `dpkg-buildpackage -jX`. There is a
rather high probability the package may build. Policy does not talk about
`dpkg-buildpackage -jX` while dpkg-buildpackage manual page clearly documents
that it sets MAKEFLAGS only if -jX is passed to it. So I don't see why we
should not try to support this use case. That's dh which puts a sort of
artificial wall here.
So should I include `dpkg-buildpackage -jX` detection in the patch or not?
Either will be fine for me.
> > Obviously, if only DEB_BUILD_OPTIONS has parallel=X without MAKEFLAGS
> > set, parallel should not be automatically enabled.
>
> Not obvious to me. Policy only talks about DEB_BUILD_OPTIONS=parallel, not
> MAKEFLAGS. MAKEFLAGS may have nothing to do with the actual build
> system. Packages may be built without use of dpkg-buildpackage.
Do not forget that dh_auto_* makefile support has always (so far) relied upon
dpkg-buildpackage to set CFLAGS/CXXFLAGS. When a package is built with plain
`debian/rules binary`, it will frequently end up being compiled without any
optimizations and/or -g option (unless upstream is clever enough to enable
properly configured release builds by default). What is more, makefile.pm does
not handle DEB_BUILD_OPTIONS="noopt" either. So another bug?
> > 2) Supporting overriding via parallel is non-optimal since parallel is an
> > optional but official feature of Debian Policy. debhelper/dh should
> > provide support for standardized things if possible (and it is possible).
>
> Not sure what you're saying here. Supporting *what* overriding parallel
> is non-optimal?
I meant debhelper should have support for parallel without requiring the use
of override_dh_auto_build. Parallel is too important these days to require
such non-obvious and tricky workarounds from maintainers...
> > 3) The way to solve this is indeed dh -j/--parallel option to declare
> > that the package supports parallel. What is more, some build systems
> > (e.g. cmake) always generates parallel-safe Makefiles so it makes much
> > sense to default to --parallel for them. However, the fact that
> > makefile.pm is auto-detected at "build" step for cmake is not helping
> > this case and unfortunately, I don't see another way how how to to solve
> > this but some temporary file hacks (urgh!) or reordering of @BUILDSYSTEMS
> > (moving cmake above makefile).
>
> Are you really sure cmake always supports parallel builds? It's
> easy to take an innocuous feature, like the ability to run an arbitrary
> command during the build, and use that to make a makefile that breaks
> parallelism. Not being familiar with cmake, I don't know if it has
> such a feature, but that would be a significant feature to leave out.
Well, cmake has hooks to add custom targets and dependencies which may mess up
its good support for parallelism. However, those are rather advanced stuff,
not many people use. So basically yeah, it would be safe to enable --parallel
for cmake. In those rare cases when it breaks, maintainers could disable it.
However, since this is complicated to implement, let's ignore this part of the
issue for now. It is far more important for dh_auto_* to get a manually
specifiable --parallel option, imho.
--
Modestas Vainius <mode...@vainius.eu>
Joey Hess
> But maybe it is not such a bad idea after all? If you want to *safely* build
> in parallel mode, export DEB_BUILD_OPTIONS=parallel=X manually (dpkg-
> buildpackage messes with MAKEFLAGS if and only if -jX option is passed to it
> regardless of DEB_BUILD_OPTIONS contents). So dpkg-buildpackage does not
> violate policy. `dpkg-buildpackage -jX` != DEB_BUILD_OPTIONS=parallel=X, it is
> more.
>
> However, if you want to *try* to build in a parallel mode even if the package
> does not declare support for it (packaging might be out of date, maintainer
> might not care or something), you use `dpkg-buildpackage -jX`. There is a
> rather high probability the package may build.
It'd make more sense to me if dpkg-buildpackage's interface for
parallel building used the safe mode and the unsafe mode was available
for users who want to set MAKEFLAGS on their own. Since
dpkg-buildpackage has the option, and the option is not marked as unsafe,
it's encouraging people to use it.
> Do not forget that dh_auto_* makefile support has always (so far) relied upon
> dpkg-buildpackage to set CFLAGS/CXXFLAGS. When a package is built with plain
> `debian/rules binary`, it will frequently end up being compiled without any
> optimizations and/or -g option (unless upstream is clever enough to enable
> properly configured release builds by default). What is more, makefile.pm does
> not handle DEB_BUILD_OPTIONS="noopt" either. So another bug?
There have already been threads pointing out that this is a misfeature
in dpkg-buildpackage. (That the concerns were dismissed is why my opinion
of it was low to begin with.) Anyway, packages can set CFLAGS in
debian/rules while still using dh, just as they might if running make by
hand. So dh doesn't prevent packages from doing the right thing. I
*hope* it doesn't obfuscate things to the point that maintainers forget
to do the right thing.
But, I pessimistically expect that things are going to rot to the point
that use of dpkg-buildpackage becomes mandatory, despite policy not
requiring it, and policy will probably then be changed to require it.
Which may be their actual intent. I have no opinion whether that's a
good result; just don't like the way we're getting there.
Anyway, what did this have to do with debhelper? I guess I feel that
making debhelper work around flags set by dpkg-buildpackage is not a
productive use of time. I also feel that it is worthwhile for debhelper
to not block legitimate, informed use of eg, MAKEFLAGS. Hope that
somehow answers your question?
--
see shy jo
Modestas Vainius
On penktadienis 28 Rugpjūtis 2009 00:11:10 Joey Hess wrote:
> Anyway, what did this have to do with debhelper? I guess I feel that
> making debhelper work around flags set by dpkg-buildpackage is not a
> productive use of time.
What I meant is that your dh examples for makefile are not 100% correct
then... They need to suggest CFLAGS/CXXFLAGS handling or we are assuming dh is
relying on dpkg-buildpackage to set those. So either:
1) Fix makefile.pm to set those.
2) Fix docs that maintainer should set those (absolute majority) and manually
handle noopt deb build flags. But then debian/rules will not be minimal ...
Anyway, I think that current assumption that debian/rules is run under dpkg-
buildpackage and it sets those flags is fine for me...
--
Modestas Vainius <mode...@vainius.eu>
Modestas Vainius
On šeštadienis 24 Spalis 2009 03:48:58 Modestas Vainius wrote:
> the attached patch is my second attempt to address parallel building as
> transparently as possible:
> 1) without the need to use cryptic punctuation in the debian/rules or
> similar hacks
> 2) without pointless make job control warnings.
>
> Test suite is included. More information can be found in the patch header
> and the patch itself.
This new improved version has a more conservative behaviour when --parallel is
not passed. In such a case it used to assume the package did not support
parallel and prevented it by cleaning MAKEFLAGS extensively (like --parallel=1
in this version). Now if --parallel is not specified, makefile.pm won't mess
with MAKEFLAGS expect clean it from stale unreachable --jobserver-fds options
to prevent pointless make warnings about job control from appearing.
What is more, dh won't implicitly pass --parallel to dh_auto_* anymore if
jobserver is reachable (i.e. dh was run with "+" prefix). This is more
compatible with legacy behaviour when "+" hack was necessary to enable
parallel building.
--
Modestas Vainius <mode...@vainius.eu>
Modestas Vainius
the attached patch is my second attempt to address parallel building as
transparently as possible:
1) without the need to use cryptic punctuation in the debian/rules or similar
hacks
2) without pointless make job control warnings.
Test suite is included. More information can be found in the patch header and
the patch itself.
--
Modestas Vainius <mode...@vainius.eu>
Joey Hess
and refactors and simplifies get_make_jobserver_status.
I'd appreciate your review; it does pass the test suite but
I could have missed a corner.
---
But, I think that the handling of --parallel is confusingly
inconsistent. If dh_auto_build is run directly from the rules file,
--parallel must be specified to enable parallel building. If it is run
by dh, --parallel does not need to be specified.
My inclination is to remove the implicit --parallel setting from dh,
thus commit 3c3ebf1ba6a7e93f7fb66d375ca8c1106d99af2c. I welcome your
feedback on that, too..
--
see shy jo
Modestas Vainius
On ketvirtadienis 29 Spalis 2009 00:31:10 Joey Hess wrote:
> I've pushed a 'parallel' branch in git that takes this patch
> and refactors and simplifies get_make_jobserver_status.
>
> I'd appreciate your review; it does pass the test suite but
> I could have missed a corner.
Looks fine.
> ---
>
> But, I think that the handling of --parallel is confusingly
> inconsistent. If dh_auto_build is run directly from the rules file,
> --parallel must be specified to enable parallel building. If it is run
> by dh, --parallel does not need to be specified.
>
> My inclination is to remove the implicit --parallel setting from dh,
> thus commit 3c3ebf1ba6a7e93f7fb66d375ca8c1106d99af2c. I welcome your
> feedback on that, too..
I still feel that this feature-hack is important. Why?
1) Multicore systems are everywhere. I think it is important to have a unified
way to force the package to build in parallel even a maintainer of the package
didn't bother to read debhelper manual and add pass --parallel to dh.
2) It is expected. dpkg-buildpackage -jX enables parallel for standard dh-make
templates, cdbs and all packages using $(MAKE) (majority since it is
recommend) in their debian/rules even without maintainer adding support for
it.
3) Due to the fact that debhelper is written in perl (rather than makefiles),
the -jX is not passed along. This is somewhat an artificial barrier.
Why in dh rather than dh_auto_*?
1) It is very likely that maintainers who use dh_auto_* directly in
debian/rules (not via overrides) want more control over build process. So they
are given it.
2) Hackish nature of the feature. In the first versions of the patch, I tried
to implement this in dh_auto_* but it felt very out-of-place for me. In my
opinion, dh_* commands themselves should keep a very clean interface. On the
contrary, dh is already doing some black magic so adding a bit more does not
hurt too much. dh feels more like debian/rules replacement and build process
manager while dh_* commands are tools to get the specific job done.
3) This black magic in dh_auto_* would be completely hidden while dh still
prints `dh_auto_* --parallel` in its output so it is clear what might have
gone wrong if build fails.
4) More complicated implementation is needed. If we clean makeflags in dh to
suppress warnings, dh_auto_* will not be able to auto-detect dpkg-buildpackage
-jX using "unavailable jobserver" method. So if want to enable this for dh and
dh_auto_* (to keep consistency as you say), we need to do this detection in
both dh and dh_auto_*. I think it is too complicated having in mind the value
of this feature.
Actually, I disagree that packages using dh do not need to specify
--parallel with this feature present. If the package actually *builds* in
parallel, it still needs to specify --parallel and that's still a valid bug if
it doesn't. --parallel is an indication that a maintainer is *sure* the
package builds in parallel. So DEB_BUILD_OPTIONS=parallel=n will work on them
without any messing with MAKEFLAGS. When (if) dpkg-buildpackage -jX stops
messing with MAKEFLAGS, this dh hack will stop working but so will standard
dh-make templates without any explicit parallel handling and most other
packages (not sure about cdbs though)...
Having all this said, I won't feel very disappointed if you omit this feature
(then you need to remove two test cases that currently fail and were written
for it). Personally, I will add --parallel to all my packages (since they all
support it) in any case. But other maintainers may not by following a well
known mini dh template and expecting this parallel thing to work transparently
like everywhere else... Unfortunately, this is my time that I'm saving by
rebuilding faster. But thinking more about it I rarely ever need to rebuild
packages of others...
--
Modestas Vainius <mode...@vainius.eu>
Joey Hess
> Why in dh rather than dh_auto_*?
>
> 1) It is very likely that maintainers who use dh_auto_* directly in
> debian/rules (not via overrides) want more control over build process. So they
> are given it.
You can set --parallel=1 at either level. The inconsistency of defaults
does not lead to more control. It does however, mean that adding:
build:
dh_auto_build
to a rules file suddenly, and unexpecedly, makes parallel builds stop
working when they would have worked before.
> 4) More complicated implementation is needed. If we clean makeflags in dh to
> suppress warnings, dh_auto_* will not be able to auto-detect dpkg-buildpackage
> -jX using "unavailable jobserver" method.
Why does the -jX method need to be used? dpkg-buildpackage -j also sets
DEB_BUILD_OPTIONS.
> Actually, I disagree that packages using dh do not need to specify
> --parallel with this feature present. If the package actually *builds* in
> parallel, it still needs to specify --parallel and that's still a valid bug if
> it doesn't.
I saw that you said that in the man page, but even with that warning I
would expect 9/10th of debhelper users to stop once dpkg-buildpackage -j
worked, and never explicitly add --parallel.
> When (if) dpkg-buildpackage -jX stops
> messing with MAKEFLAGS, this dh hack will stop working but so will standard
> dh-make templates without any explicit parallel handling and most other
> packages (not sure about cdbs though)...
Which is another reason to dislike the hack, since it will be another reason
for the dpkg developers to continue touching MAKEFLAGS.
Currently there seem to me to be two viable choices:
1. Require --parallel explicitly be used before looking at
DEB_BUILD_OPTIONS. This tries to avoid misbuilds of unparallelizable
packages by hoping that maintainers will test when they add it. It
doesn't really avoid all breakage, because
a) Some maintainers will cargo-cult it from a template w/o testing.
b) Nobody will test with every possible -j number.
c) When a build system has parallel support newly added to it,
the flag will start to take effect, and expose previously hidden
problems.
(Implemented in parallel branch)
2. Always honor DEB_BUILD_OPTIONS unless --parallel=1 (or =2 etc)
overrides it. This maximises the set of packages that can be built
in parallel and assumes that guarding against misbuilds is not
something debhelper should worry about, while still allowing maintainers
who know of a problem with parallel builds to disable it.
(Implemented in parallel-implicit branch)
3. Actually, honorable mention would be to move the data about
what build flags a package supports into the control file, as I
suggested in #489771. Then dpkg-buildpackage could avoid enabling
a parallel build mode if a package's control file indicated it did not
support it. Anyway, the implementation of this in debhelper is the
same as #2, except without needing the --parallel flag.
--
see shy jo
Modestas Vainius
On ketvirtadienis 29 Spalis 2009 22:36:32 Joey Hess wrote:
> > 1) It is very likely that maintainers who use dh_auto_* directly in
> > debian/rules (not via overrides) want more control over build process. So
> > they are given it.
>
> You can set --parallel=1 at either level. The inconsistency of defaults
> does not lead to more control. It does however, mean that adding:
>
> build:
> dh_auto_build
>
> to a rules file suddenly, and unexpecedly, makes parallel builds stop
> working when they would have worked before.
Well, yeah.
> > 4) More complicated implementation is needed. If we clean makeflags in dh
> > to suppress warnings, dh_auto_* will not be able to auto-detect
> > dpkg-buildpackage -jX using "unavailable jobserver" method.
>
> Why does the -jX method need to be used? dpkg-buildpackage -j also sets
> DEB_BUILD_OPTIONS.
Policy defines parallel=n. n is NOT optional. `dpkg-buildpackage -j` sets some
bogus value -1 (aka unlimited?) which is not supported by policy... Yeah, it
seems that thing is really a bit broken.
My vote surely goes for this. But since it seemed like you had already
rejected this idea earlier [1], I resorted to some `dpkg-buildpackage -j`
detection hacks :-) I fully support this idea including rename to --max-
parallel. Consistency and clean interface is kept and parallel support is ON
by default.
However, I'm a bit concerned about the fact that there is no way to tell
dh_auto_* (-> makefile.pm) not to pass -jX to make at all (i.e. defined $this-
>get_parallel() in makefile.pm:do_make() will never be false). Some packages
might need this. In parallel branch, this used to be the case when --parallel
was not specified at all (hence set_parallel() never got called and
$opt_parallel defaulted to undef). Now set_parallel() is always called and
always defines $opt_parallel to some value. Maybe --max-parallel=-1 for this
(since --max-parallel/--max-parallel=0 is already an alias for "unlimited")?
1. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=532805#32
>
> 3. Actually, honorable mention would be to move the data about
> what build flags a package supports into the control file, as I
> suggested in #489771. Then dpkg-buildpackage could avoid enabling
> a parallel build mode if a package's control file indicated it did not
> support it. Anyway, the implementation of this in debhelper is the
> same as #2, except without needing the --parallel flag.
Yeah, I hope, dpkg people work something out for Squeeze. I'm also concerned
about #544844 since I do not export C(XX)FLAGS in debian/rules anywhere and
rely on dpkg-buildpackage to do this.
--
Modestas Vainius <mode...@vainius.eu>
Joey Hess
> Policy defines parallel=n. n is NOT optional. `dpkg-buildpackage -j` sets some
> bogus value -1 (aka unlimited?) which is not supported by policy... Yeah, it
> seems that thing is really a bit broken.
Since policy has no way to specify unlimited, I assume this behavior is
intentional. I suppose debhelper can support it.
> However, I'm a bit concerned about the fact that there is no way to tell
> dh_auto_* (-> makefile.pm) not to pass -jX to make at all (i.e. defined $this-
> >get_parallel() in makefile.pm:do_make() will never be false). Some packages
> might need this. In parallel branch, this used to be the case when --parallel
> was not specified at all (hence set_parallel() never got called and
> $opt_parallel defaulted to undef). Now set_parallel() is always called and
> always defines $opt_parallel to some value. Maybe --max-parallel=-1 for this
> (since --max-parallel/--max-parallel=0 is already an alias for "unlimited")?
Perhaps we can wait to see if there is any use case for this before
worrying about it? Only use case I can think of is a package that
somehow builds using two dh_auto_builds, and can do it in parallel.
Something like:
build: build-foo build-bar
build-foo:
cd foo && dh_auto_build
build-bar:
cd bar && dh_auto_build
Here parallel=4 would cause up to 8 concurrent jobs to run,
which is too many. So you might do:
build: build-foo build-bar
export MAKEFLAGS=-j4
build-foo:
+cd foo && dh_auto_build --max-parallel=-1
build-bar:
+cd bar && dh_auto_build --max-parallel=-1
Now the two builds would run, sharing a jobserver.
But isn't this a clearer way to do the same thing?
build: build-foo build-bar
export MAKEFLAGS=-j4 # really, calculate from DEB_BUILD_OPTIONS
build-foo:
$(MAKE) -C foo
build-bar:
$(MAKE) -C bar
(It also has the bonus of working, since dh_auto_build
will fail if foo/debian/control doesn't exist.)
--
see shy jo
Modestas Vainius
On penktadienis 30 Spalis 2009 02:05:53 Joey Hess wrote:
> > However, I'm a bit concerned about the fact that there is no way to tell
> > dh_auto_* (-> makefile.pm) not to pass -jX to make at all (i.e. defined
> > $this-
> >
> > >get_parallel() in makefile.pm:do_make() will never be false). Some
> > > packages
> >
> > might need this. In parallel branch, this used to be the case when
> > --parallel was not specified at all (hence set_parallel() never got
> > called and $opt_parallel defaulted to undef). Now set_parallel() is
> > always called and always defines $opt_parallel to some value. Maybe
> > --max-parallel=-1 for this (since --max-parallel/--max-parallel=0 is
> > already an alias for "unlimited")?
>
> Perhaps we can wait to see if there is any use case for this before
> worrying about it?
I like to define strict interface in advance. If we can support this without
too much effort and in a clean way now, why wait?
So --max-parallel=0 becomes "free" since --max-parallel=-1 is now an alias for
unlimited. I attach a patch which turns --max-parallel=0 into "do nothing" as
I suggested previously. As a consequence, an argument for --max-parallel is no
longer optional (which does make much sense since it has lost its "enable
parallel" meaning). So -1 = unlimited, 0 - do nothing, 1 - no parallel, n > 1
- n max jobs. This looks very logical for me.
> Only use case I can think of is a package that
> somehow builds using two dh_auto_builds, and can do it in parallel.
> Something like:
Probably you are right, but sometimes people need to do some hackish things.
In my opinion, having an interface complete (if it does not end up convoluted)
is always a good thing.
So as far as I'm concerned, I like everything else in "parallel-implicit"
branch and I think it is ready to be merged to master.
--
Modestas Vainius <mode...@vainius.eu>