Re: Environment variables, debian/rules and dpkg-buildpackage
Manoj Srivastava
> [ Bcced to -devel and -dpkg, discussion should happen on -policy ]
>
> Hello,
>
> we have an unfortunate situation where the practice in dpkg-buildpackage
> and the policy do not match fully.
>
> I tried to summarize the problem here:
> http://wiki.debian.org/Teams/Dpkg/DebianRules
>
> I want to resolve this problem. I see two solutions:
>
> * either we modify policy to mandate the set of environment variables that
> dpkg-buildpackage sets
>
> * or we roll-back changes to dpkg-buildpackage and design something else
> that provide the same feature in terms of distribution-wide control of
> default build options. (I ignore the DEB_VENDOR feature, it can be easily
> replaced with dpkg-vendor but the build options case is much more tricky
> to get right)
>
> In terms of efforts, the first solution is the easiest. But we aim at the
> _right_ solution so feel free to design something that makes the second
> solution viable.
There are several things I dislike about the first option.
1. I do not like that policy would turn around 15 years of convention
and suddenly outlaw $(MAKE) -f debian/rules foo. This will break
many of my packages; and I do not think that that is
justified. There are better solutions than the ones proposed.
2. As a packager, it is easy enough to over ride the setting
externally, by setting my own variables; but this makes it hard for
the end user to set the variables. I can certainly see cases where
one-size-does-not-fit all cases; so making it easy to change the
flags on a per package, per site, and a per compilation case can be,
and should be addressed.
3. dpkg-buildpackage is probably the wrong place to put this solution
in.
As a simple, off the cuff design, I can see hewing closer to the
Gentoo use flags solution, and putting in /etc/debian-build.mk makefile
snippet, which contains the default setting of the flags.
A) Project wide flags can be set by making that file part of an
essential package (perhaps dpkg).
B) The site admin can make any changes they want, like adding SELinux
or hardening flags in that file. Regular conffile handlig will allow
site admins to be aware of any project wide changes. This allows
each site to set variables for that site.
C) Each package can include that file (-include, so no errors if file
does not exist), and then override whatever variables they want in a
package specific variation.
D) The user can set the variable on the command line, which will
override the Makefile versions of the variable, for a one off per
compilation change.
The fact that dpkg-buildpackage's setting the variables is not
easily configurable, and presents to make as though it was set
on the commandline, and thus making it hard for the *USER* to set the
flag variables via env variables, is, in my opinion, a major bug.
Of course, this requires that every package rules file will need
the -include /etc/debian-build.mk at top, which makes it less than
ideal, but I am sure we can come up with a better solution.
I consider the issue of being able to set these flags on these
levels fairly important; if I have missed some use case of where one
might want to set defaults otherwise (buildd's can set arch specific
flags just like site admins can), please feel free to point it out.
manoj
--
A penny saved is a penny to squander. Ambrose Bierce
Manoj Srivastava <sriv...@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Mike Hommey
> ifeq (,$(findstring nostrip,$(DEB_BUILD_OPTIONS)))
> STRIP += -s
> LDFLAGS += -s
> INT_INSTALL_TARGET = install
> else
> INT_INSTALL_TARGET = install
> endif
I doubt dpkg-buildpackage sets LDFLAGS to -s. That would mean all
sources building -dbg binary packages would need to override this, which
is not the case.
Mike
Manoj Srivastava
The external file is nice in the fact that it decouples the
installation of the essential package and the rules files; this might
be useful for backported packages as well. In the rules file, one may
check for the presence of the file to be included, and take different
action based on whether it is present or not.
Packages may continue to move over to supporting the common
included file on their own timetable, by just not including the snippet
until the ya re ready. And the snippet may take care of tsome of the
DEB_BUILD_OPTIONS stuff.
So. Here is a little snippet that sets some of the flags (this
is a demo only, I am not setting all the flags here that dpkg sets)
--8<---------------cut here---------------start------------->8---
# set CC to $(DEB_HOST_GNU_TYPE)-gcc only if a cross-build is detected
ifneq ($(DEB_HOST_GNU_TYPE),$(DEB_BUILD_GNU_TYPE))
CC=$(DEB_HOST_GNU_TYPE)-gcc
else
CC = cc
endif
# Policy 10.1 says to make this the default
CFLAGS = -Wall -g
ifneq (,$(filter noopt,$(DEB_BUILD_OPTIONS)))
CFLAGS += -O0
else
CFLAGS += -O2
endif
## ifneq (,$(findstring debug,$(DEB_BUILD_OPTIONS)))
## endif
ifeq (,$(findstring nostrip,$(DEB_BUILD_OPTIONS)))
STRIP += -s
LDFLAGS += -s
INT_INSTALL_TARGET = install
else
INT_INSTALL_TARGET = install
endif
ifneq (,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
NUMJOBS = $(patsubst parallel=%,%,$(filter parallel=%,$(DEB_BUILD_OPTIONS)))
MAKEFLAGS += -j$(NUMJOBS)
endif
--8<---------------cut here---------------end--------------->8---
manoj
--
The computing field is always in need of new cliches. Alan Perlis
Stephen Gran
> [ Bcced to -devel and -dpkg, discussion should happen on -policy ]
>
> Hello,
>
> we have an unfortunate situation where the practice in dpkg-buildpackage
> and the policy do not match fully.
>
> I tried to summarize the problem here:
> http://wiki.debian.org/Teams/Dpkg/DebianRules
>
> I want to resolve this problem. I see two solutions:
>
> * either we modify policy to mandate the set of environment variables that
> dpkg-buildpackage sets
>
> * or we roll-back changes to dpkg-buildpackage and design something else
> that provide the same feature in terms of distribution-wide control of
> default build options. (I ignore the DEB_VENDOR feature, it can be easily
> replaced with dpkg-vendor but the build options case is much more tricky
> to get right)
>
>
> In terms of efforts, the first solution is the easiest. But we aim at the
> _right_ solution so feel free to design something that makes the second
> solution viable.
I'm more than slightly uncomfortable with mandating a particular tool as
the only way to build packages (and mandating something that's different
than current policy at that). I'd be much happier with a Makefile
snippet and a conversion period. I know it's more work and slower to
migrate to, but it seems to me to have the advantage of being the most
flexible.
Cheers,
--
-----------------------------------------------------------------
| ,''`. Stephen Gran |
| : :' : sg...@debian.org |
| `. `' Debian user, admin, and developer |
| `- http://www.debian.org |
-----------------------------------------------------------------
Stephen Gran
> Note that I'm not asking to mandate the tool. I would like to mandate the
> fact that packages can rely on some environment variables being set to
> some values.
>
> > I'd be much happier with a Makefile snippet and a conversion period. I
> > know it's more work and slower to migrate to, but it seems to me to have
> > the advantage of being the most flexible.
>
I don't. I'm imagining that some of our downstreams may.
Raphael Hertzog
> There is no documented semantic for CFLAGS et. al. in Debian policy. While
> some Makefile handle it in a certain way, this is not mandatory in
> any way. For example some configure scripts append options to CFLAGS while
> other will not change it if it is defined.
This is precisely what I want to change. We should provide a reasonable
way for the user/admin to give default flags and for the packager to
override/extend the default set of flags.
> You are conflating breakage that were reported with breakage that appeared
> but were not reported.
[…]
> Changing CFLAGS behind the back of the maintainers is potentially changing
> how the package is build in a undefined way. A package built in such way should
> be assumed broken.
So you should assume that most Lenny packages are broken. I know the
difference but while I don't deny that some unused packages might be
affected by this, it's not my concern right now (those packages are
probably candidate for removal if nobody used them to detect that they are
not working anymore).
Cheers,
--
Raphaël Hertzog
Contribuez à Debian et gagnez un cahier de l'admin Debian Lenny :
http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/
Raphael Hertzog
> This one time, at band camp, Raphael Hertzog said:
> > Care to elaborate what kind of flexibility you need in this specific case ?
>
> I don't. I'm imagining that some of our downstreams may.
It's precisely one of our downstreams that pushed the
dpkg-buildpackage change (Ubuntu). I don't think most downstreams
care a lot about the exact implementation, but they clearly want
a way to alter defauld build flags at the distribution level.
This all started with:
https://wiki.ubuntu.com/DistCompilerFlags
Felipe Sateler
> Note that I'm not asking to mandate the tool. I would like to mandate the
> fact that packages can rely on some environment variables being set to
> some values.
Note that packages will not necessarily pickup the environment variables.
autoconf using packages will probably do automatically, but other build-systems
still require the maintainer to pick up the changes from the environment and
passing them to the build command (eg, scons doesn't have any standard way for
that, its up to the package to define a variable). So you don't actually
guarantee that all packages use the default flags.
--
Felipe Sateler
Stefano Zacchiroli
> * either we modify policy to mandate the set of environment variables that
> dpkg-buildpackage sets
> In terms of efforts, the first solution is the easiest. But we aim
> at the _right_ solution so feel free to design something that makes
> the second solution viable.
FWIW I'd love to see the first solution implemented. My main reason,
as a maintainer of packages which mostly escape the autotools / C
language pattern, is to see written somewhere the intended semantics
in Debian of the various environment variable. That would offer an
explanation to maintainers about how to handle such variables even if
their upstreams are not using "standard" makefiles.
Regarding the objection of mandating dpkg-buildpackage as *the* tool
to build, it looks moot to me. The main point would be defining the
API and the intended meaning of variables. Then if it will happen that
dpkg-buildpackage sets them properly, let's be happy about that or
implement alternative tools.
Same goes for the configuration file objection: dpkg-buildpackage can
have one.
... and pretty please, do not choose a solution that will require
adding an "-include" to 15'000 thousands debian/rules; we will finish
doing that by Lenny+50, the earliest.
Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -<>- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..| . |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...........| ..: |.... Je dis tu à tous ceux que j'aime
Manoj Srivastava
> On Sun, 15 Mar 2009, Bill Allombert wrote:
>> There is no documented semantic for CFLAGS et. al. in Debian policy. While
>> some Makefile handle it in a certain way, this is not mandatory in
>> any way. For example some configure scripts append options to CFLAGS while
>> other will not change it if it is defined.
>
> This is precisely what I want to change. We should provide a reasonable
> way for the user/admin to give default flags and for the packager to
> override/extend the default set of flags.
So here are what I think we need, based on the most common use
cases:
A) Provide a way to specify project wide defaults for env variables
B) Allow a site admin to selectively override these, and provide site
wide defaults
C) Allow a package to set some flags that would otherwise break the
package, overriding site and/or project defaults as needed
D) Allow the user to specify and override any of the above.
In the non-snippet method proposed, there is no way for the
package maintainer to override project defaults yet cater user set
variable settings, since the information is lost.
Also, I think that just CFLAGS is probably too broad, I might
want to override only some of the categories of flags (optimization,
machine related, warning, debugging, etc) and not others; pulling
warning and optimization out as separate subflags might be a good
compromise.
All of this can be done with Makefile snippets, and do not allow
you to mandate any set of tools, without which the environment set
would be different. You can use dpkg-vuildpackage, $MAKE) -f
deban/rules, write a wrapper Makefile that include ./debian/rules -- it
all would just work.
I do not see any reason to limit ourselves to one tool to set
the env variables, especially since it prevents the package maintainer
from distinguishing between the project wide defaults anduser set
values, and has no easy way for a site to set site wide values. Hey, I
might like hardening flags the project does not cater to when I
rebuild the archive on my buildd -- and the snippet approach allows for
it.
manoj
--
s = (char*)(long)retval; /* ouch */ Larry Wall in doio.c from the perl
source code
Manoj Srivastava <sriv...@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Manoj Srivastava
> On Mon, 16 Mar 2009, Bdale Garbee wrote:
>>
>> > I think he ment that you can not know wether the setting comes from
>> > dpkg-buildpackage or the user. If it comes from dpkg-buildpackage then
>> > debian/rules should be free to override it as needed. If it comes from
>> > the user then that is another story. At least that is my take on it.
>>
>> This is a great point. It must be possible to craft a rules file that overrides
>> system or distribution wide defaults and which can be over-ridden by an individual
>> building the package.
>
> I don't see why it's needed, it's the job of the rules file to adjust the
> flags if there's a reason to deviate (like using -O0 instead of -O2 for
> some arch with compiler problems) but in general the package should not
> override completely the default flags but just complete them and/or
> adjust/fix them if needed.
But while overriding the project wide default makes sense,
overriding the human doing hte building does not.
>
> However, if the caller really wish that his build options prevail in
> all cases, he can use "make -e" (and dpkg-buildpackage has the -R
> option that let him call "debian/rules" as "make -e -f debian/rules"
> instead).
We do not want to override *EVERYTHING* in the Makefile, just
the CFLAGS. This blunt instrument of the -e flag is not a good
solution.
> If necessary we can add a new option --force-flags to dpkg-bp or similar
> to make this even easier.
Making a bad solution easy is not a good thing either.
>> That seems to require the ability for the code in a rules file to
>> distinguish between things in the environment because they're defaults
>> and things in the environment that are attempting to override defaults.
>
> Apparently there's no way to know from where the variable value come in
> make. That's true for environment variables like for command line
> variables.
> (at least according to my lookup of info make)
Err, I think you need to get more conversant with Make. Yes,
there is no way to know whether or not the environment variables were
set by dpkg or the user; but it is itrivial to make it so that the
project wide defaults can be overridden by the site wide stuff, and
each being overridden by the package at will. We can even make it so
that the package can tell which one set the value, and override only
the project wide one and not the site variable.
> So if we really want the rules files to be able to know, we have to
> add yet another requirement to create this information. This is not
> desirable IMO.
Given how trivial it is, I think it is eminently desirable. It
is just that dpkg is not the right place for such subtleties.
manoj
--
Though many hands make light work, too many cooks spoil the broth.
Raphael Hertzog
> In the non-snippet method proposed, there is no way for the
> package maintainer to override project defaults yet cater user set
> variable settings, since the information is lost.
You're not trying very hard to look from both sides: whether the default
value comes from the environment or from an included Makefile, in both
cases the user can override it with command-line arguments.
Granted it means that dpkg-buildpackage would have to pass user-overriden
flags on the command line instead of using the environment, but that can
be done if people really want this possibility.
So the policy would mandate a set of variables that could be communicated
either via the environment or via the command-line depending on how
authoritative the user wants to be.
That would be fine for me too.
Cheers,
--
Raphaël Hertzog
Contribuez à Debian et gagnez un cahier de l'admin Debian Lenny :
http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/
Raphael Hertzog
> > However, if the caller really wish that his build options prevail in
> > all cases, he can use "make -e" (and dpkg-buildpackage has the -R
> > option that let him call "debian/rules" as "make -e -f debian/rules"
> > instead).
>
> We do not want to override *EVERYTHING* in the Makefile, just
> the CFLAGS. This blunt instrument of the -e flag is not a good
> solution.
Then he can add the required parameter on the command line instead
of using the environment.
make -f debian/rules CFLAGS="..." <target>
You don't seem to grasp the fact that mandating some variables to be
preset doesn't forbid us to rely on Makefile features if we chose to.
> > Apparently there's no way to know from where the variable value come in
> > make. That's true for environment variables like for command line
> > variables.
> > (at least according to my lookup of info make)
>
> Err, I think you need to get more conversant with Make. Yes,
> there is no way to know whether or not the environment variables were
> set by dpkg or the user; but it is itrivial to make it so that the
> project wide defaults can be overridden by the site wide stuff, and
> each being overridden by the package at will. We can even make it so
> that the package can tell which one set the value, and override only
> the project wide one and not the site variable.
It is trivial to do with whatever solution we retain. The problematic part
that I pointed out is "how to know in the rules file if the CFLAGS value
comes from the command-line, from the environment or from the makefile
itself".
Do you have a solution to that ? If no, then there was no point in
suggesting me to get more conversant with Make. If yes, show it to
me.
Suppose you have "FOO ?= bar" in the Makefile, write me the rest of the
Makefile so that I have this:
$ FOO=foo make
FOO was set in the environment
$ make FOO=foo
FOO was set on the command-line
$ make
FOO was set in the Makefile
Cheers,
--
Raphaël Hertzog
Contribuez à Debian et gagnez un cahier de l'admin Debian Lenny :
http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/
Manoj Srivastava
> On Mon, 16 Mar 2009, Manoj Srivastava wrote:
>> In the non-snippet method proposed, there is no way for the
>> package maintainer to override project defaults yet cater user set
>> variable settings, since the information is lost.
>
> You're not trying very hard to look from both sides: whether the default
> value comes from the environment or from an included Makefile, in both
> cases the user can override it with command-line arguments.
>
> Granted it means that dpkg-buildpackage would have to pass user-overriden
> flags on the command line instead of using the environment, but that can
> be done if people really want this possibility.
>
> So the policy would mandate a set of variables that could be communicated
> either via the environment or via the command-line depending on how
> authoritative the user wants to be.
>
> That would be fine for me too.
That still fails on two points:
a) Not using dpkg would mean different build environments
b) there is no way for a site or buildd admin to provide site wide
defaults that override dpkg ones but the user can override
Also, a dding new project wide builkd defaults is tied to dpkg
development and releases (which is not really a great idea either
manoj
--
"...a most excellent barbarian ... Genghis Kahn!" _Bill And Ted's
Excellent Adventure_
Manoj Srivastava <sriv...@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
James Vega
> Suppose you have "FOO ?= bar" in the Makefile, write me the rest of the
> Makefile so that I have this:
> $ FOO=foo make
> FOO was set in the environment
> $ make FOO=foo
> FOO was set on the command-line
> $ make
> FOO was set in the Makefile
$ cat Makefile
FOO ?= bar
all:
ifeq "$(origin FOO)" "command line"
$(info "FOO was set on the command-line")
endif
ifeq "$(origin FOO)" "environment"
$(info "FOO was set in the environment")
endif
ifeq "$(origin FOO)" "file"
$(info "FOO was set in the Makefile")
endif
--
James
GPG Key: 1024D/61326D40 2003-09-02 James Vega <jame...@debian.org>
Russ Allbery
> You're not trying very hard to look from both sides: whether the default
> value comes from the environment or from an included Makefile, in both
> cases the user can override it with command-line arguments.
>
> Granted it means that dpkg-buildpackage would have to pass
> user-overriden flags on the command line instead of using the
> environment, but that can be done if people really want this
> possibility.
I think it's clearly mandatory to implement a hierarchy of settings:
* Debian defaults
* Local distribution overrides
* Local package overrides
* User settings
where each overrides the previous ones.
Personally, I'd rather see this done via an included make fragment. I
think the logic in debian/rules is simpler in that case, and I don't agree
with the argument that it's that much harder to implement than relying on
environment variables. In practice, huge numbers of Debian packages
already unconditionally set CFLAGS in debian/rules and hence override any
environment variable settings. All those packages are going to have to be
modified anyway. I don't see a way of meaningfully deploying this change
without modifying most of the archive.
--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>
Manoj Srivastava
> On Mon, 16 Mar 2009, Manoj Srivastava wrote:
>> > However, if the caller really wish that his build options prevail in
>> > all cases, he can use "make -e" (and dpkg-buildpackage has the -R
>> > option that let him call "debian/rules" as "make -e -f debian/rules"
>> > instead).
>>
>> We do not want to override *EVERYTHING* in the Makefile, just
>> the CFLAGS. This blunt instrument of the -e flag is not a good
>> solution.
>
> Then he can add the required parameter on the command line instead
> of using the environment.
>
> make -f debian/rules CFLAGS="..." <target>
>
> You don't seem to grasp the fact that mandating some variables to be
> preset doesn't forbid us to rely on Makefile features if we chose to.
Heh. I never thought I would be lectured on the use of make. Cute.
And you are missing the point that making people type stuff on
the command line for site specific stuff looses out to being able to
edit a conffile instead.
dpkg might suffice if we were not wanting the ability to set
site wide build defaults a la gentoo use flags -- but it seems silly to
give up on that i we are designing a solution now.
>> > Apparently there's no way to know from where the variable value come in
>> > make. That's true for environment variables like for command line
>> > variables.
>> > (at least according to my lookup of info make)
>>
>> Err, I think you need to get more conversant with Make. Yes,
>> there is no way to know whether or not the environment variables were
>> set by dpkg or the user; but it is itrivial to make it so that the
>> project wide defaults can be overridden by the site wide stuff, and
>> each being overridden by the package at will. We can even make it so
>> that the package can tell which one set the value, and override only
>> the project wide one and not the site variable.
>
> It is trivial to do with whatever solution we retain. The problematic part
> that I pointed out is "how to know in the rules file if the CFLAGS value
> comes from the command-line, from the environment or from the makefile
> itself".
> Do you have a solution to that ? If no, then there was no point in
Of course I do.
> suggesting me to get more conversant with Make. If yes, show it to
> me.
*Sigh*.
--8<---------------cut here---------------start------------->8---
File: make.info, Node: Origin Function, Next: Flavor Function, Prev:
Eval Function, Up: Functions
--8<---------------cut here---------------end--------------->8---
This is precisely what the origin function is for.
Also, you might not need it with the snippet example:
--8<---------------cut here---------------start------------->8---
default_foo := BAR
site_foo := baz # by default, this is empty -- set by site admin
ifndef (,$(strip $(site_foo)))
foo = $(site_foo)
else
foo = $(default_foo)
endif
--8<---------------cut here---------------end--------------->8---
Now the variable foo is set, but we can tell it was set in a
makefile, and set by default or the site admin. And make a decision in
the rules file based on that. (Note the use of := versus = above; the
difference is significant, really).
We can also tell if a variable was set in the environment, or
the command line -- but the whole include makefile snippet means we do
not have to replicate what dpkg does on the command line if we do not
want to use that single point of entry.
> Suppose you have "FOO ?= bar" in the Makefile, write me the rest of the
> Makefile so that I have this:
> $ FOO=foo make
> FOO was set in the environment
> $ make FOO=foo
> FOO was set on the command-line
> $ make
> FOO was set in the Makefile
Goodness me.
manoj
--
The world is coming to an end ... SAVE YOUR BUFFERS!!!
Manoj Srivastava <sriv...@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Manoj Srivastava
> Raphael Hertzog <her...@debian.org> writes:
>
>> You're not trying very hard to look from both sides: whether the default
>> value comes from the environment or from an included Makefile, in both
>> cases the user can override it with command-line arguments.
>>
>> Granted it means that dpkg-buildpackage would have to pass
>> user-overriden flags on the command line instead of using the
>> environment, but that can be done if people really want this
>> possibility.
>
> I think it's clearly mandatory to implement a hierarchy of settings:
>
> * Debian defaults
> * Local distribution overrides
> * Local package overrides
> * User settings
>
> where each overrides the previous ones.
We can do better than that with the Make snippets.
Say, as an user, I think my admin is nuts, and I want o
override his setting, and revert to the project wide default, for all
packages I build on this machine. Easy (just set site_foo in the env,
call debian/rules with -e).
Or, as an admin, I want to always override the project wide
settings (hey, I like hardening flags), but on some machines I have
taken time out to set up site wide defaults -- and I do not want to
over ride those. Again, easy to do with the example snippet I posted.
All kinds of flexibility become ours once we embrace toe
sour^H^H^Hsnippets, I mean.
> Personally, I'd rather see this done via an included make fragment. I
> think the logic in debian/rules is simpler in that case, and I don't
> agree with the argument that it's that much harder to implement than
> relying on environment variables. In practice, huge numbers of Debian
> packages already unconditionally set CFLAGS in debian/rules and hence
> override any environment variable settings. All those packages are
> going to have to be modified anyway. I don't see a way of
> meaningfully deploying this change without modifying most of the
> archive.
The advantage of the snippet approach is where packages might
break if the builtin defaults are changed will not be broken by the
snippet approach, since the change is opt-in.
manoj
--
It is a city built of bones, and daubed with flesh and blood, in which
old age and death, pride and hypocrisy are the inhabitants. 150
Manoj Srivastava <sriv...@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Raphael Hertzog
>
> * Debian defaults
> * Local distribution overrides
> * Local package overrides
> * User settings
>
> where each overrides the previous ones.
I think we all mostly agree on that. I see only two remarks:
- the package can either fully override the default settings or
filter the provided build options: i.e. add/remove/replace build
options. (and I think that the "filter" option should be the recommended
approach)
- the user should also be able to provide a default build option that the
package can override in case the user wants to trust the
maintainer's opinion on what build options are sane or not for this
specific package (it might allow him to have a better success rate if you
want to recompile lots of packages with some options that are known to
not work in some cases)
Do we want that hierarchy set in stone in policy or do we simply
want to define how the rules file is supposed to retrieve the relevant
build options ?
What would the policy change look like if we select the Makefile snippet
approach ?
> In practice, huge numbers of Debian packages already unconditionally set
> CFLAGS in debian/rules and hence override any environment variable
> settings. All those packages are going to have to be modified anyway.
> I don't see a way of meaningfully deploying this change without
> modifying most of the archive.
It would be nice to have figures here.
Note that we have packages switching to debhelper 7 rules files as
simple as /usr/share/doc/debhelper/examples/rules.tiny so the number of
packages explicitely setting CFLAGS might drop.
Cheers,
--
Raphaël Hertzog
Contribuez à Debian et gagnez un cahier de l'admin Debian Lenny :
http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/
Russ Allbery
> I think we all mostly agree on that. I see only two remarks:
> - the package can either fully override the default settings or
> filter the provided build options: i.e. add/remove/replace build
> options. (and I think that the "filter" option should be the recommended
> approach)
Yes.
> - the user should also be able to provide a default build option that the
> package can override in case the user wants to trust the
> maintainer's opinion on what build options are sane or not for this
> specific package (it might allow him to have a better success rate if you
> want to recompile lots of packages with some options that are known to
> not work in some cases)
I don't think this is particularly important at a level of granularity
less than the system configuration level (in other words, having the user
do this by changing a conffile would be fine with me). I base this on my
experience as a user overriding compilation flags long before I'd ever
used Linux, where if need be I'll just cut and paste what the package
currently uses and change it slightly when building regular packages.
This is pretty much par for the course for overriding configuration like
that.
> Do we want that hierarchy set in stone in policy or do we simply want to
> define how the rules file is supposed to retrieve the relevant build
> options ?
I think we should say something about the hierarchy of options in Policy
at least as non-normative text as an explanation for what on earth we're
trying to accomplish, so that ten years from now we can figure out what we
were thinking.
> What would the policy change look like if we select the Makefile snippet
> approach ?
Roughly:
All packages should add:
include /etc/dpkg-dev/build-options.mk
to their debian/rules file. This file will set the following options:
CFLAGS
LDFLAGS
...
Packages should use those settings by default (with appropriate changes or
propagation into other variables as needed by the build system of the
package -- insert explanation of what each of them means here), but the
package may override or change those settings if they're not appropriate
for this particular package.
Users when building a package may override both the defaults and
debian/rules by setting the appropriate variable on the make command line
with, for example:
make -f debian/rules CFLAGS=blah
Not in Policy: dpkg-buildpackage could certainly have a useful option for
doing the last, and I think that would be a worthwhile feature.
> It would be nice to have figures here.
I don't have time to write the code myself, but it should be fairly
trivial to find most packages that set CFLAGS explicitly by looking at the
rules file in the Lintian lab on gluck. You'll miss more complex makefile
include systems, but you'll get most of the cases.
> Note that we have packages switching to debhelper 7 rules files as
> simple as /usr/share/doc/debhelper/examples/rules.tiny so the number of
> packages explicitely setting CFLAGS might drop.
Yes, agreed.
Note that all CDBS packages can easily switch to whatever we want,
including makefile snippets, with a rev to CDBS. debhelper 7 with minimal
rules will probably need to add the include file directly, since debhelper
isn't in as good of a position to force the makefile inclusion. (There
are ways in which it could, but they all seem very ugly to me.)
--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>
Raphael Geissert
>
> ... and pretty please, do not choose a solution that will require
> adding an "-include" to 15'000 thousands debian/rules; we will finish
> doing that by Lenny+50, the earliest.
It would take some time, yes; but packages using cdbs would only require a
binNMU once cdbs includes that file on its own.
IMO this is what looks like The Right Solution; are we willing to ignore it
just because it would take some time?
Anyway, any package that was last updated before dpkg-buildpackage started
to set the environment vars could easily be discarded, because it was never
affected by the ENV vars mess.
What other approach (which fulfils all the pros already mentioned by Manoj)
do you suggest?
Cheers,
Raphael Geissert
Raphael Hertzog
> Stefano Zacchiroli wrote:
> >
> > ... and pretty please, do not choose a solution that will require
> > adding an "-include" to 15'000 thousands debian/rules; we will finish
> > doing that by Lenny+50, the earliest.
>
> It would take some time, yes; but packages using cdbs would only require a
> binNMU once cdbs includes that file on its own.
BinNMU are not needed. The default CFLAGS won't have changed. A rebuild to
verify that none are FTBFS would be enough.
> IMO this is what looks like The Right Solution; are we willing to ignore it
> just because it would take some time?
I don't know your criteria for "The Right Solution", so this opinion
doesn't buy me much.
> What other approach (which fulfils all the pros already mentioned by Manoj)
> do you suggest?
What are the pros mentioned by Manoj that are specific to the Makefile
snippet approach except the fact that we can continue to call debian/rules
directly on all packages ?
At this point, I have the feeling it's a matter of individual taste
mainly. Every time we have accepted that we have ended up with 2 different
approachs (python-support vs python-central) and we have not won much.
Is there a way to reconcile both approaches ?
Maybe this one: we let the caller pass the required build options as
environment variables or command line options but the package maintainer
has the possibility to include a Makefile snippet that would set
any missing build-option to the default value chosen by the distribution
so that he can continue to call debian/rules directly without having to
take care of setting the environment variables.
Such a Makefile snippet could then be like this:
CFLAGS ?= -Wall -g
ifeq "$(origin CFLAGS)" "file"
ifneq (,$(filter noopt,$(DEB_BUILD_OPTIONS)))
CFLAGS += -O0
else
CFLAGS += -O2
endif
endif
How does that sound ?
Cheers,
--
Raphaël Hertzog
Contribuez à Debian et gagnez un cahier de l'admin Debian Lenny :
http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/
Manoj Srivastava
> What are the pros mentioned by Manoj that are specific to the Makefile
> snippet approach except the fact that we can continue to call debian/rules
> directly on all packages ?
That by itself is reason enough, I think.
Secondly, I have never seen the dpkg approach address the
site-wide configuration: How is that to be done?
> At this point, I have the feeling it's a matter of individual taste
> mainly. Every time we have accepted that we have ended up with 2 different
> approachs (python-support vs python-central) and we have not won much.
If you address these issues, I can accept your staatement:
a) The site admin can set, and optionally override, the project wide
defaults
b) The package can set, and optionally override the site defaults, or
the global defaults, or either, or none
c) The user may set defaults, ot override the global, site, or package
defaults, in any order.
> Is there a way to reconcile both approaches ?
If you thik the dpkg approach can also provide the above
functionality of the Makefile snippetr approach, then yes, the only
advandate of the Makefile approach is that one is not dependent on dpkg
to build packages (which, in my eyes, is a critical advantage by
itself).
manoj
--
grep me no patterns and I'll tell you no lines.
Manoj Srivastava <sriv...@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Stefano Zacchiroli
> # in debian/rules
> -include /etc/buildpackage.mk
It seems to me that you are indeed close, but with the exception of
this required include in all our debian/rules, which will be a PITA to
achieve. AFAIU Raphael's proposal, the site-specific configuration
will be achieved via dpkg-buildpackage config's file, without needing
to include anything explicitly in debian/rules, while you are
insisting in the explicit include.
While I understand (though I disagree) with your reasons for having
that include, that looks like a major differences in the two positions
still.
Manoj Srivastava
> On Tue, Mar 17, 2009 at 11:36:07AM -0500, Manoj Srivastava wrote:
>> # in debian/rules
>> -include /etc/buildpackage.mk
>
> It seems to me that you are indeed close, but with the exception of
> this required include in all our debian/rules, which will be a PITA to
> achieve.
Modulo debhelper, cdbs, et.al can help.
> AFAIU Raphael's proposal, the site-specific configuration will be
> achieved via dpkg-buildpackage config's file, without needing to
> include anything explicitly in debian/rules, while you are insisting
> in the explicit include.
And will be mostly useless. dpkg can set these variables until
it is blue in the face, and it has zero effect on my packages until I
change my rules file. Or change upstream Makefiles to pay attention to
the cflags set in the env. This is a major change for some of my
packages, and without them the dpkg proposal does nothing.
See, I think we need have package opt-in in any case.
Also, the dpkg proposal blurs the distinction between the site
wide policy and project defaults, as far as the user or the package
maintainer is concerned. This is a deficiency.
> While I understand (though I disagree) with your reasons for having
> that include, that looks like a major differences in the two positions
> still.
Yup. The major differences are:
With the dpkg proposal:
Bad:
o) people must always use dpkg to build packages, or the packages come
out different
o) The user or the package maintainer can no longer tell the difference
between site policy and project policy
o) The environment variables are set even if the package is not ready
for them,
o) rules file still need to be modified to take advantage of the
variables set -- none of my rules files will be affected by any
settings in dpkg right now.
Good:
o) There is no transition period involved -- except that rules files
still need to be modified, so there is a transition period anyway.
With the Make snippets proposal:
Bad:
o) Every package has to change (but,every package has to change anyway,
since currently dpkg can set the variables till it is blue in the
face and nothing will take effect in my packages)
Good:
o) the person building the package is not constrained to using dpkg;
o) The site admin can add on to the variables set on that site, and is
not constrained by what dpkg handles
o) the process is opt in, and packages opt in to using the new
mechanism when they are ready.
o) The end user can override project, site, or package policies,
individually, or in any combination
If I am biased (likely), please tel me where I have gone wrong
above.
manoj
--
Unfair competition: Selling cheaper than we do. Kelvin Throop III, "The
Management Dictionary"
Raphael Hertzog
> On Tue, Mar 17 2009, Stefano Zacchiroli wrote:
> > It seems to me that you are indeed close, but with the exception of
> > this required include in all our debian/rules, which will be a PITA to
> > achieve.
>
> Modulo debhelper, cdbs, et.al can help.
Debhelper can't help. Only CDBS can (as its design is precisely based on
included Makefiles).
> And will be mostly useless. dpkg can set these variables until
> it is blue in the face, and it has zero effect on my packages until I
> change my rules file. Or change upstream Makefiles to pay attention to
> the cflags set in the env. This is a major change for some of my
> packages, and without them the dpkg proposal does nothing.
That's true for a lot of packages but not all packages. It also means that
there is some usage of this approach in the archive (otherwise if all
packages ignored the environment variables, there would never have been
packages that FTBFS when this change was introduced in Lenny).
So according to your rule that policy should standardize "common practice"
and not mandate something completely new, the env variable proposal is in
more widespread usage.
(Note that I don't find this rule particularly useful and that's why I
didn't mention it before but since you ask about points where you are
biased I thought it could be worth telling)
> See, I think we need have package opt-in in any case.
Not in all cases. CDBS could also be updated to cope with the env var
and debhelper 7 packages using rules.tiny could also make use of it
without a modification.
> Also, the dpkg proposal blurs the distinction between the site
> wide policy and project defaults, as far as the user or the package
> maintainer is concerned. This is a deficiency.
While I agree that the distinction is blurred because in the rules files
you don't know where the env var comes from, I don't agree that it's a
deficiency.
The rules files receives a value and should use it. It doesn't need to
know whether it's the site-wide default or the distribution default.
> Yup. The major differences are:
> With the dpkg proposal:
> Bad:
> o) people must always use dpkg to build packages, or the packages come
> out different
If the policy document the fact that the rules files need some input
variables, then it's not a big deal: if you really care about being able
to call debian/rules directly you can always adapt your rules files
accordingly like I suggested at the end of this mail:
http://lists.debian.org/debian-devel/2009/03/msg00920.html
We could even write such a Makefile that mimics more closely what
dpkg-buildpackage does for people that care about it. But I don't
really want to impose a Makefile snippet to everybody.
> o) The user or the package maintainer can no longer tell the difference
> between site policy and project policy
The user is intelligent, he can read the output of dpkg-buildpackage that
tells him where the data comes from (or he can read the doc and the
configuration file). In any case, it's comparable to having to
read a Makefile snippet.
For the package maintainer, why should he care ? The variable is provided
in a manner where he can or can't override it, but that's all that
matters.
> o) The environment variables are set even if the package is not ready
> for them,
We already took that hit. It's not an issue.
> o) rules file still need to be modified to take advantage of the
> variables set -- none of my rules files will be affected by any
> settings in dpkg right now.
The fact that your rules files need change doesn't meant that all need it.
> With the Make snippets proposal:
> Bad:
> o) Every package has to change (but,every package has to change anyway,
> since currently dpkg can set the variables till it is blue in the
> face and nothing will take effect in my packages)
> Good:
> o) the person building the package is not constrained to using dpkg;
> o) The site admin can add on to the variables set on that site, and is
> not constrained by what dpkg handles
> o) the process is opt in, and packages opt in to using the new
> mechanism when they are ready.
> o) The end user can override project, site, or package policies,
> individually, or in any combination
>
> If I am biased (likely), please tel me where I have gone wrong
> above.
If the policy documents the environment that needs to be setup, nobody is
forced to use dpkg-buildpackage to do it. But it's probably more
convenient to use a helper compared to manually setting the expected
variable.
dpkg-buildpackages should be considered like a helper and I'm available to
fix any problem that makes it painful to use (if you find its usage too
much constraining).
I would like to point out that we speak mainly of CFLAGS and al. but that
I also included DEB_VENDOR in the set of variables. This variable is not
used currently (as I just introduced it with dpkg 1.15.0) but I expect it
to become more used in the future for things like this:
- enable additional patch/features depending on the vendor (while still
sharing a common source package for better cooperation)
- special purpose derivative could change the behaviour of helpers like
debhelper/CDBS based on this value (for example to strip doc for
Emdebian, to extract debug infos in .ddeb, ...)
Most packages would not have to be modified to benefit from this variable
because the derivatives would have designed their solution with this goal
in mind. And it also concerns arch: all packages whereas CFLAGS only
concerns arch: any.
Cheers,
--
Raphaël Hertzog
Contribuez à Debian et gagnez un cahier de l'admin Debian Lenny :
http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/
Goswin von Brederlow
> On Tue, 17 Mar 2009, Manoj Srivastava wrote:
>> On Tue, Mar 17 2009, Stefano Zacchiroli wrote:
>> > It seems to me that you are indeed close, but with the exception of
>> > this required include in all our debian/rules, which will be a PITA to
>> > achieve.
>>
>> Modulo debhelper, cdbs, et.al can help.
>
> Debhelper can't help. Only CDBS can (as its design is precisely based on
> included Makefiles).
With the magic
%:
dh $@
debhelper could help.
MfG
Goswin
Loïc Minier
> Of course an Ubuntu derivative could be surprised if they get
> a Debian-variant of a package instead of an Ubuntu-variant…
Yes, that's exactly the issue I'm raising
> I see how we can solve it (add new fields in /etc/dpkg/origins/* to
> describe parent relationship, and create a new tool to query those
> meta-information) but I wonder what impact you expect it would have
> on the decision of exporting DEB_VENDOR in the build environment.
Well I just want this issue to be considered near the initial
introduction of DEB_VENDOR.
> Would you like a DEB_VENDORS="Gobuntu Ubuntu Debian" or similar
> so that no external tool is required ?
>
> ifneq (,$(filter Ubuntu,$(DEB_VENDORS)))
> # Ubuntu or derivative
> endif
That would be a possible implementation which I wouldn't mind using;
albeit the name of the field isn't too obvious but I don't have any
good idea. I think I'd be fine with either an external tool or
environment vars, but in any case the handling of specific vendor and
inherited vendors should be similar IMO.
Thanks!
--
Loïc Minier
Loïc Minier
> There is a good use case for this that doesn't require a conditional, which is
> passing --with-package-name=$(DEB_VENDOR) to configure for packages that have
> this option (e.g. the GStreamer stack, that right now checks
> lsb_release -si output.
Good point; that's similar to when the "exactly this vendor" check
makes sense
> It doesn't solve your concerns but perhaps you could do some common
> action or apply the Debian logic for anything else (which would be the
> same as not looking at DEB_VENDOR at all).
Applying the Debian logic is not good enough for e.g. derivatives of
Ubuntu as they want to fork the Ubuntu behavior and patches etc.
--
Loďc Minier
Loïc Minier
> > if vendor is Ubuntu:
> > foo
> > elif vendor is Debian:
> > bar
> > you face a problem when you meet:
> > else:
> >
> > What behavior should one use here?
>
> Debian should always be the "else" because it's the default case. It
> ensures that the lack of DEB_VENDOR results in correct package and
> also that any unknown derivatives get the Debian behaviour.
[ There are two difference things here: I agree that if no vendor is
set, Debian should be assumed but I don't think the choice of Debian's
behavior for unknown vendors is the right one, I think I'd personally
prefer erroring out to raise the issue, but that's only if there's a
sane way to handle that properly, which could be your DEB_VENDORS
proposal. ]
--
Loďc Minier
Manoj Srivastava
> On Tue, 17 Mar 2009, Manoj Srivastava wrote:
>> On Tue, Mar 17 2009, Stefano Zacchiroli wrote:
>> > It seems to me that you are indeed close, but with the exception of
>> > this required include in all our debian/rules, which will be a PITA to
>> > achieve.
>>
>> Modulo debhelper, cdbs, et.al can help.
>
> Debhelper can't help. Only CDBS can (as its design is precisely based on
> included Makefiles).
>
>> And will be mostly useless. dpkg can set these variables until
>> it is blue in the face, and it has zero effect on my packages until I
>> change my rules file. Or change upstream Makefiles to pay attention to
>> the cflags set in the env. This is a major change for some of my
>> packages, and without them the dpkg proposal does nothing.
>
> That's true for a lot of packages but not all packages. It also means that
> there is some usage of this approach in the archive (otherwise if all
> packages ignored the environment variables, there would never have been
> packages that FTBFS when this change was introduced in Lenny).
Any package that follows policy §4.9 examples to set CFLAGS
based on DEB_BUILD_OPTIONS will be immune to CFLAGS et al set in the
environment.
Also, any upstream Makefile that sets CFLAGS (don't most ones
that use automake do that?) will also be not affected, unless even more
hackery is done. At this point, what dpkg does to these variables not
enough to justify any such effort.
> So according to your rule that policy should standardize "common
> practice" and not mandate something completely new, the env variable
> proposal is in more widespread usage.
That was not my rule, just a misunderstanding of what I
said. The rules is that policy
a) standardizes practices needed for integration, and no more.
b) when it must add stuff, policy tries to add the best, and minimal,
solution.
c) Often the viability of s solution is not clear until it has been put
into practice, and the kinks ironed out.
This does not mean shoddy practices will be enshrined in policy.
> (Note that I don't find this rule particularly useful and that's why I
> didn't mention it before but since you ask about points where you are
> biased I thought it could be worth telling)
An incorrect rendering of the rule does not help, sorry.
>> Also, the dpkg proposal blurs the distinction between the site
>> wide policy and project defaults, as far as the user or the package
>> maintainer is concerned. This is a deficiency.
>
> While I agree that the distinction is blurred because in the rules files
> you don't know where the env var comes from, I don't agree that it's a
> deficiency.
A missing feature is a deficiency. How critical a deficiency it
is is a matter of opinion, and we apparently differ.
> The rules files receives a value and should use it. It doesn't need to
> know whether it's the site-wide default or the distribution default.
>
>> Yup. The major differences are:
>> With the dpkg proposal:
>> Bad:
>> o) people must always use dpkg to build packages, or the packages come
>> out different
>
> If the policy document the fact that the rules files need some input
> variables, then it's not a big deal: if you really care about being able
> to call debian/rules directly you can always adapt your rules files
> accordingly like I suggested at the end of this mail:
> http://lists.debian.org/debian-devel/2009/03/msg00920.html
> We could even write such a Makefile that mimics more closely what
> dpkg-buildpackage does for people that care about it. But I don't
> really want to impose a Makefile snippet to everybody.
Frankly, dpkg does very little: All it does is to discard the
policy suggestion (which I think is a bad idea) and only sets FLAGS to
-g -O0 or -g -O2
I do not think that this is much of an assistance, if this is as
far as dpkg goes.
Setting CPPFLAGS, COPTS, CWARNS, CFLAGS, CMACHINE (and things
for other languages) might be better, but trivially just setting CFLAGS
to select between -O0 and -O2 is not enough to justify mangling build
systems to
>> o) The user or the package maintainer can no longer tell the difference
>> between site policy and project policy
>
> The user is intelligent, he can read the output of dpkg-buildpackage that
> tells him where the data comes from (or he can read the doc and the
> configuration file). In any case, it's comparable to having to
> read a Makefile snippet.
With the makefile snippet, the Makefile can tell the difference,
and there is a standard method to tell what variables are set in the
site config.
> For the package maintainer, why should he care ? The variable is
> provided in a manner where he can or can't override it, but that's all
> that matters.
Because it gives the option to override one or the other -- a
generic default that is project wide might be less applicable, but site
policies can still be followed.
I think it is a useful feature.
>
>> o) The environment variables are set even if the package is not ready
>> for them,
>
> We already took that hit. It's not an issue.
>> o) rules file still need to be modified to take advantage of the
>> variables set -- none of my rules files will be affected by any
>> settings in dpkg right now.
>
> The fact that your rules files need change doesn't meant that all need it.
Anything that follows policy §4.9 needs it.
>> With the Make snippets proposal:
>> Bad:
>> o) Every package has to change (but,every package has to change anyway,
>> since currently dpkg can set the variables till it is blue in the
>> face and nothing will take effect in my packages)
>> Good:
>> o) the person building the package is not constrained to using dpkg;
>> o) The site admin can add on to the variables set on that site, and is
>> not constrained by what dpkg handles
>> o) the process is opt in, and packages opt in to using the new
>> mechanism when they are ready.
>> o) The end user can override project, site, or package policies,
>> individually, or in any combination
>>
>> If I am biased (likely), please tel me where I have gone wrong
>> above.
>
> If the policy documents the environment that needs to be setup, nobody is
> forced to use dpkg-buildpackage to do it. But it's probably more
> convenient to use a helper compared to manually setting the expected
> variable.
I think that there has to be more of a justification of why the
rules files need to be hacked, upstream Makefiles need to be hacked,
and for what?
All packages already need to select between -O0 and -O2 or they
fail policy §4.9. What does adding this bit to policy buy us?
manoj
--
ignorance, n.: When you don't know anything, and someone else finds out.
Manoj Srivastava <sriv...@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Bill Allombert
> So according to your rule that policy should standardize "common practice"
> and not mandate something completely new, the env variable proposal is in
> more widespread usage.
For ten years, the "common practice" was that dpkg-buildpackage did not set
any variable.
We cannot standardize on the "env variable proposal" because such proposal has
never be made. Instead dpkg-buildpackage was broken in Lenny, and should be
fixed ASAP. Now we have packages that do not build correctly with
dpkg-buildpackage, others that do not build correctly with debian/rules binary,
and all handle env var differently.
Cheers,
--
Bill. <ball...@debian.org>
Imagine a large red swirl here.
Clint Adams
> For ten years, the "common practice" was that dpkg-buildpackage did not set
> any variable.
>
> We cannot standardize on the "env variable proposal" because such proposal has
> never be made. Instead dpkg-buildpackage was broken in Lenny, and should be
> fixed ASAP. Now we have packages that do not build correctly with
> dpkg-buildpackage, others that do not build correctly with debian/rules binary,
> and all handle env var differently.
I concur.
Roger Leigh
> > So according to your rule that policy should standardize "common practice"
> > and not mandate something completely new, the env variable proposal is in
> > more widespread usage.
>
> For ten years, the "common practice" was that dpkg-buildpackage did not set
> any variable.
>
> We cannot standardize on the "env variable proposal" because such proposal
> has never be made. Instead dpkg-buildpackage was broken in Lenny, and should
> be fixed ASAP. Now we have packages that do not build correctly with
> dpkg-buildpackage, others that do not build correctly with debian/rules
> binary, and all handle env var differently.
Exactly.
dpkg-buildpackage setting environment variables is broken. Using make
to set site/distro/user-specific options makes more sense. Not only is
it more flexible and extensible, it will also work no matter how one
builds the package since make will handle it.
Regards,
Roger
--
.''`. Roger Leigh
: :' : Debian GNU/Linux http://people.debian.org/~rleigh/
`. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/
`- GPG Public Key: 0x25BFB848 Please GPG sign your mail.
Russ Allbery
> Also, any upstream Makefile that sets CFLAGS (don't most ones
> that use automake do that?) will also be not affected, unless even more
> hackery is done. At this point, what dpkg does to these variables not
> enough to justify any such effort.
Most packages that use Automake also use Autoconf, and Autoconf picks up
CFLAGS from the environment at the time configure is run, so most Automake
packages would pick up an environmental default. However, I expect many
packages are shielded at present by use of the default Policy-recommended
explicit setting of CFLAGS. I know nearly all of mine (that care about
CFLAGS at all) are. The main exception are Perl modules, for which I've
mostly switched to debhelper v7 rule minimization.
When this change was originally made in dpkg-buildpackage, it broke
several packages that didn't have this shield. I'm not sure if there was
a common way in which those packages are fixed or how many of them went
back to setting CFLAGS explicitly.
> On Wed, Mar 18 2009, Raphael Hertzog wrote:
>> While I agree that the distinction is blurred because in the rules files
>> you don't know where the env var comes from, I don't agree that it's a
>> deficiency.
> A missing feature is a deficiency. How critical a deficiency it
> is is a matter of opinion, and we apparently differ.
And for what it's worth, I agree with Manoj here.
--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>
Matthias Klose
> On Mon, Mar 16 2009, Raphael Hertzog wrote:
>
>> On Sun, 15 Mar 2009, Bill Allombert wrote:
>>> There is no documented semantic for CFLAGS et. al. in Debian policy. While
>>> some Makefile handle it in a certain way, this is not mandatory in
>>> any way. For example some configure scripts append options to CFLAGS while
>>> other will not change it if it is defined.
>> This is precisely what I want to change. We should provide a reasonable
>> way for the user/admin to give default flags and for the packager to
>> override/extend the default set of flags.
>
> So here are what I think we need, based on the most common use
> cases:
>
> A) Provide a way to specify project wide defaults for env variables
> B) Allow a site admin to selectively override these, and provide site
> wide defaults
> C) Allow a package to set some flags that would otherwise break the
> package, overriding site and/or project defaults as needed
> D) Allow the user to specify and override any of the above.
E) Provide a way to specify project wide defaults and override any of the above.
This is much needed to for rebuild checks ignoring package specific work
arounds. Currently workarounds tend to live forever, and there is no way to
automatically override those.
> I do not see any reason to limit ourselves to one tool to set
> the env variables, especially since it prevents the package maintainer
> from distinguishing between the project wide defaults anduser set
> values, and has no easy way for a site to set site wide values. Hey, I
> might like hardening flags the project does not cater to when I
> rebuild the archive on my buildd -- and the snippet approach allows for
> it.
maybe, but for E) a package should honour these overwrite flags.
Matthias
Russ Allbery
> Manoj Srivastava schrieb:
>> A) Provide a way to specify project wide defaults for env variables
>> B) Allow a site admin to selectively override these, and provide site
>> wide defaults
>> C) Allow a package to set some flags that would otherwise break the
>> package, overriding site and/or project defaults as needed
>> D) Allow the user to specify and override any of the above.
> E) Provide a way to specify project wide defaults and override any of
> the above. This is much needed to for rebuild checks ignoring package
> specific work arounds. Currently workarounds tend to live forever, and
> there is no way to automatically override those.
Wouldn't the user override work for you in this case? If you're doing
archive-wide test builds overriding any package settings, that seems to me
to be exactly the same case as a user overriding all package settings to
force a particular set of build flags and you could do this with the same
mechanism.
--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>
Raphael Hertzog
> > I see how we can solve it (add new fields in /etc/dpkg/origins/* to
> > describe parent relationship, and create a new tool to query those
> > meta-information) but I wonder what impact you expect it would have
> > on the decision of exporting DEB_VENDOR in the build environment.
> >
> > so that no external tool is required ?
> >
> > ifneq (,$(filter Ubuntu,$(DEB_VENDORS)))
> > # Ubuntu or derivative
> > endif
>
> This wouldn't hard-code any specific vendor names.
I don't see your point. In the example above $(DEB_VENDORS) is created by
dpkg-buildpackage (so not hardcoded) and Ubuntu is the parameter of the
"is_derivative_of()" logical function.
Guillem Jover
Hi!
On Fri, 2009-03-13 at 21:04:30 +0100, Raphael Hertzog wrote:
> we have an unfortunate situation where the practice in dpkg-buildpackage
> and the policy do not match fully.
Ok, had finally the time to read and think about this. I've to say first
of all that I've never quite liked dpkg-buildpackage as a way to set
distro-wide flags, and less so by doing it by default and kind of
deprecating debian/rules in the way. But at the time this got implemented
I didn't have the time to ponder about better alternatives or discuss
about it (although the makefile include crossed my mind at some point,
and we talked about it with Rapha�l before he started this thread).
In retrospect I guess I should have chimed in on the initial discussion
or the subsequent complaints, but nothing that cannot be repaired now
anyway.
I'll try to summarize the discussion (althought it might be biased to
my preferred option) and some facts, so that we can get to a conclusion:
* We'd like to have at least this overriding hierarchy (the
implementation could be a bit more complex, but I think that's
not important now, as this one already exposes problems in the
dpkg-buildpackage proposal):
- Distro defaults.
- Site overrides over the above (can be partial filters).
- Package overrides over the above (can be partial filters).
- User overrides over the above (total overrides).
* dpkg-buildpackage currently sets the build flags through env vars.
* Lots of packages (roughly around 4k) do not honour env vars for build
flags, as they follow the examples from policy (or dh-make and similar)
and thus are setting them unconditionally, which env vars do not
override.
,-- chk-flags --
#!/bin/sh
set -eu
regex="$@"
cd /srv/lintian.debian.org/laboratory/source/
ls | xargs -I% grep -H -l "$regex" %/debfiles/rules | wc -l
`--
./chk-flags '^[^#]*\(CXX\|F\|CPP\|LD\|C\)FLAGS[[:space:]]*:\?='
* Thus lots of packages in the archive have to be modified anyway
regardless of either solution (centralization through dpkg-buildpackage
or makefile include). If we have to modify them, we can make them
include a makefile. Adding such include line at the top of debian/rules
should certainly imply less changes than the proposed changes in
<http://lists.debian.org/debian-devel/2009/03/msg00920.html>.
* dpkg-buildpackage has been setting env vars for dpkg-architecture
output for a long time (2001), but those flags are a bit different
than the other build flags. First the example code in dh-make uses ?=
which makes the env vars take preference, and they are (or should be)
always initialized in the debian/rules file as recommended by
dpkg-architecture(1). And second the *_BUILD_* ones should alway match
the current system, and the *_HOST_* ones should be changed by
changing the toolchain, so there's no reason to override them in the
distro or packaging (if there's a need then dpkg should be fixed
instead).
* Externalizing the setup of the build flags means that debian/rules
stops being a working standalone interface for building packages, as
mandated by policy, and imposes an additional layer to rely on for
building packages
* Setting system (distro and site) build vars through command line (or
forced environment 'make -e') does not allow the pkg to override them,
nor do partial filtering.
* Using 'make -e' is not really a good idea as it's not fine grained. So
the correct solution when total override is desired is to set the
vars via the command line.
* Setting system (distro and site) build vars through command line,
implies we can only use limited make syntax for those vars.
* We can set the architecture and default flags (from policy) on the
makefile to be included, and packagers will be able to do the change
and fix any possible problems (progressive opt-in), but once it's
included by all packages, then we can do system-wide default changes
in the same we change toolchains (mass rebuild, bug filing, change
when bug count goes down). The makefile has the advantage that the
distro default can be temporarily changed for the mass build w/o
needing to totally override the build flags.
So I think for next dpkg upload we should make dpkg-buildpackage stop
setting any flags by default, and switch the setting to go through the
command line arguments to override the package options instead of the
environment, so this would become the user override part. The DEB_VENDOR
env var should not be set either, and we should work on getting a
dpkg-vendor instead, before people try to start using the variable.
Then if we get consensus that this is the righ path, agree on the
makefile names (as once decided it will be a pain to change later on in
all packages), we'll need to ship the distro defaults file somewhere and
start fixing packages to include that makefile. The files could look
something like:
,-- /usr/share/dpkg/build-options.mk
# distro defaults
FOO := distro
-include /etc/dpkg/build-options.mk
`--
,-- /etc/dpkg/build-options.mk
# site overrides
#FOO := site
`--
,-- debian/rules
-include /usr/share/dpkg/build-options.mk
# package overrides
FOO := pkg
`--
,-- command line
# user overrides
$ make -f debian/rules FOO=cmd
`--
regards,
guillem
Manoj Srivastava
> I'll try to summarize the discussion (althought it might be biased to
> my preferred option) and some facts, so that we can get to a conclusion:
+1
I agree with almost everything said in this email.
manoj
--
What use is your matted hair, you fool? What use is your antelope skin?
You are tangled inside, and you are just making the outside pretty. 394
Manoj Srivastava <sriv...@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Andrew McMillan
>
> On Fri, 2009-03-13 at 21:04:30 +0100, Raphael Hertzog wrote:
> > we have an unfortunate situation where the practice in dpkg-buildpackage
> > and the policy do not match fully.
...
That was really a very well-spent two months!
+1 from me :-)
Regards,
Andrew.
------------------------------------------------------------------------
andrew (AT) morphoss (DOT) com +64(272)DEBIAN
You will be advanced socially, without any special effort on your part.
------------------------------------------------------------------------
Bernd Zeimetz
> [ BCCed debian-dpkg for the proposed dpkg changes. ]
> * Lots of packages (roughly around 4k) do not honour env vars for build
> flags, as they follow the examples from policy (or dh-make and similar)
> and thus are setting them unconditionally, which env vars do not
> override.
Not to forget these packages where flags in the environment made the package
FTBFS as they were overriding flags set by makefiles or other build tools by
some default values which did not include all necessary options.
> * We can set the architecture and default flags (from policy) on the
> makefile to be included, and packagers will be able to do the change
> and fix any possible problems (progressive opt-in), but once it's
> included by all packages, then we can do system-wide default changes
> in the same we change toolchains (mass rebuild, bug filing, change
> when bug count goes down). The makefile has the advantage that the
> distro default can be temporarily changed for the mass build w/o
> needing to totally override the build flags.
You need to take care here to use names which will not clash with variable names
upstream may be using, otherwise you'd add new problems while adding new vars.
>
>
> So I think for next dpkg upload we should make dpkg-buildpackage stop
> setting any flags by default, and switch the setting to go through the
Please do so!
+1 from me, too.
Thanks for the good work!
Bernd
--
Bernd Zeimetz Debian GNU/Linux Developer
GPG Fingerprint: 06C8 C9A2 EAAD E37E 5B2C BE93 067A AD04 C93B FF79
Raphael Hertzog
> env var should not be set either, and we should work on getting a
> dpkg-vendor instead, before people try to start using the variable.
FYI, I already started this.
> start fixing packages to include that makefile. The files could look
> something like:
>
> ,-- /usr/share/dpkg/build-options.mk
> # distro defaults
> FOO := distro
>
> -include /etc/dpkg/build-options.mk
> `--
Note that recent changes have aimed at avoiding fork of dpkg by
integrating upstream some distro-specific logic. We should do the same for
this file then... ie accept that external vendors provide us their
defaults and have the dpkg source package pick the right one based on the
current vendor.
> ,-- debian/rules
> -include /usr/share/dpkg/build-options.mk
>
> # package overrides
> FOO := pkg
> `--
Probably without the leading "-", otherwise we have a risk to not have any
sane default at all. And we should add a build-dependency instead.
Or maybe that choice can be left to the maintainer.
Cheers,
--
Raphaï¿œl Hertzog
Contribuez ￜ Debian et gagnez un cahier de l'admin Debian Lenny :
http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/
Manoj Srivastava
> On Mon, 04 May 2009, Guillem Jover wrote:
>> ,-- debian/rules
>> -include /usr/share/dpkg/build-options.mk
>>
>> # package overrides
>> FOO := pkg
>> `--
>
> Probably without the leading "-", otherwise we have a risk to not have
> any sane default at all.
I disagree. Without the leading -, it would make it impossible
to build on a machine with older dpkg -- and why be unnecessarily
unfriendly to backporters?
If you want sane defaults, by all means provide sane defaults
yourself.
> And we should add a build-dependency instead.
And restrict the machines where the package can be built? I
think not.
> Or maybe that choice can be left to the maintainer.
This is better than forcing people into restricting the
platforms where their sources can be built, but still a cop
out. Guillem's proposal is technically better.
manoj
--
You k'n hide de fier, but w'at you gwine do wid de smoke? Joel Chandler
Harris, proverbs of Uncle Remus
Manoj Srivastava <sriv...@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Peter Eisentraut
I like this proposal. A small nit:
> ,-- /usr/share/dpkg/build-options.mk
> # distro defaults
> FOO := distro
Please be sure to use
FOO = bar
instead of ":=", unless you have determined that you really wanted ":=". In
most cases it won't make a difference, but in some it does, and then it would
behave contrary to how make usually treats variables.
Manoj Srivastava
> On Monday 04 May 2009 08:35:18 Guillem Jover wrote:
>
> I like this proposal. A small nit:
>
>> ,-- /usr/share/dpkg/build-options.mk
>> # distro defaults
>> FOO := distro
>
> Please be sure to use
>
> FOO = bar
>
> instead of ":=", unless you have determined that you really wanted ":=". In
> most cases it won't make a difference, but in some it does, and then it would
> behave contrary to how make usually treats variables.
Why, in your opinion, would we want _not_ to use :=? What does
delayed evaluation buy us?
manoj
puzzled
--
When you live in a sick society, just about everything you do is wrong.
Manoj Srivastava <sriv...@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Peter Eisentraut
> On Mon, May 04 2009, Peter Eisentraut wrote:
> > Please be sure to use
> >
> > FOO = bar
> >
> > instead of ":=", unless you have determined that you really wanted ":=".
> > In most cases it won't make a difference, but in some it does, and then
> > it would behave contrary to how make usually treats variables.
>
> Why, in your opinion, would we want _not_ to use :=? What does
> delayed evaluation buy us?
That is up for debate, to some degree, I guess. I just want to make sure that
a conscious decision is made either way. (In my experience, many uses of :=
are made without knowledge about what it does.)
I think delayed evaluation is sort of the default way in which make treats
variables, and so to avoid surprises and confusion, we should go with that one
unless there is a specific reason otherwise.
In practical terms, using delayed evaluation makes the outcome mostly
independent of the order of the assignments. Which might become quite
relevant when you design an include-a-bit-here, override-a-bit-there scheme
that is supposed to be robust against all the nonsense that 10000 source
packages might do with it afterwards.
Also note that someone who wants to be careful not to overwrite values
supplied elsewhere might use ?=, which creates a delayed expansion type
variable.
In any case, we should be careful to define and document it one way or the
other. Otherwise stuff like
CPPFLAGS += -DFOO=$(BAR)
has unclear behavior, depending on how or whether CPPFLAGS was previously set
up. (And note that it will change if initially you don't define CPPFLAGS at
all and in a later version make a := definition for it -- delayed variables
being the default.)
Manoj Srivastava
> On Monday 04 May 2009 23:53:15 Manoj Srivastava wrote:
>> On Mon, May 04 2009, Peter Eisentraut wrote:
>> > Please be sure to use
>> >
>> > FOO = bar
>> >
>> > instead of ":=", unless you have determined that you really wanted ":=".
>> > In most cases it won't make a difference, but in some it does, and then
>> > it would behave contrary to how make usually treats variables.
>>
>> Why, in your opinion, would we want _not_ to use :=? What does
>> delayed evaluation buy us?
>
> That is up for debate, to some degree, I guess. I just want to make
> sure that a conscious decision is made either way. (In my experience,
> many uses of := are made without knowledge about what it does.)
That is surprising (and somewhat disheartening), it is not a
complex concept. Now double-colon rules, that's something else.
> I think delayed evaluation is sort of the default way in which make
> treats variables, and so to avoid surprises and confusion, we should
> go with that one unless there is a specific reason otherwise.
Err, no. If you use =, there is one behaviour, if you use :=,
there is another. I would not call either one "default", apart from the
fact that non-gnu makes do not have :=.
> In practical terms, using delayed evaluation makes the outcome mostly
> independent of the order of the assignments. Which might become quite
> relevant when you design an include-a-bit-here, override-a-bit-there
> scheme that is supposed to be robust against all the nonsense that
> 10000 source packages might do with it afterwards.
We do *NOT* want settings from the distribution, from the site,
the package, and the user happen in a manner mostly independent of the
order -- making it harder for the user to see wtf is going on.
Indeed, this is one case where we *must* have as little
ambiguity as possible, where we are trying to provide _default_ values
in a hierarchical fashion. Simply expanded variables generally make
complicated makefile programming more predictable because they work
like variables in most programming languages. They allow you to
redefine a variable using its own value (or its value processed in some
way by one of the expansion functions) and to use the expansion
functions much more efficiently.
> Also note that someone who wants to be careful not to overwrite values
> supplied elsewhere might use ?=, which creates a delayed expansion type
> variable.
Have you been reading at all? Has it not been stated repeatedly
that we _want_ a specific order of overrides for the variable values?
We _want_ the site set values to override the distribution defaults --
and then we want the package to override that, and the command line
trumps all. There is no "careful not to override" meme here -- we want
emphatically to overwrite what the outer layer has set.
All the dilly dallying with delayed evaluation just confuses the
issue, and makes make work harder, for no other reason than a vague
"don't use := cause I don't remember what it does" fear.
> In any case, we should be careful to define and document it one way or the
> other. Otherwise stuff like
>
> CPPFLAGS += -DFOO=$(BAR)
>
> has unclear behavior, depending on how or whether CPPFLAGS was
> previously set up. (And note that it will change if initially you
> don't define CPPFLAGS at all and in a later version make a :=
> definition for it -- delayed variables being the default.)
There is nothing unclear about it. You are appending to the value
of the preprocessor flags in that line -- at the time of use (that is,
when the flags are "used" or evaluated).
However, whenever you use a :=, the variable is cleared, and set
to the value on the right hand side. So the := setting does what it is
supposed to: ignore anything we knew about the variable before that
point in the Makefile. If you do not know what := does, I suppose that
can be confusing, and/or terrifying.
You know, I get it that you have trouble understanding what =
and := do in make. But this knee jerk "don't use :=" is not
helping. There is nothing mystical about :=
manoj
--
The penalty for laughing in a courtroom is six months in jail; if it
were not for this penalty, the jury would never hear the evidence. --
H. L. Mencken
Manoj Srivastava <sriv...@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Loïc Minier
> I'm really surprised to see this approach getting traction. To me, this
> seems like a significant, unprecedented departure from the kinds of
> interfaces we've mandated in Policy in the past (i.e., environment
> variables, executables and command-line options). While one build helper or
> another may mandate Makefile includes, there's never been anything of the
> sort in Policy, and I don't think it's good to add such a thing now. I
> thought it was generally recognized that it's a Bad Idea to implement config
> files using your interpreter's 'include' functionality, but that's basically
> what we have here.
>
> If there's any intention at all that Policy eventually mandate use of these
> Makefile includes, then at a minimum I think Policy needs to *very* tightly
> constrain what dpkg is allowed to put in those files, to avoid future
> incompatibilities. But unfortunately, if we're going to support site files,
> we're in no position to enforce such requirements there; so packages are
> still subject to breakage from admins populating their site file with random
> settings (or syntax errors?).
Full ack, thanks for putting it so clearly
--
Lo�c Minier
Holger Levsen
On Sonntag, 10. Mai 2009, Raphael Hertzog wrote:
> With the include approach, we lack this feature and bad/broken local
> overrides can't be detected if we only have the build log at hand.
which reminds me that we dont have build logs for probably a lot more than 25%
(*) of the most used packages in the archive. The ones build manually by the
developer...
I would really like to see that binaries for all archs are (re-)build on
builds and that those logs are kept archived and linked from the PTS. (And
arch all too of course).
regards,
Holger
(*) wild guess, asuming that most packages are upload on amd64/i386/powerpc
and mostly used on i386.
Steve Langasek
> On Sunday 10 May 2009 13:56:04 Steve Langasek wrote:
> > I thought it was generally recognized that it's a Bad Idea to implement
> > config files using your interpreter's 'include' functionality, but that's
> > basically what we have here.
> Guillem pointed out one problem: Either you do it via a make include (which
> you have issues with), or you stop supporting calling debian/rules directly
> (inconvenient, probably prone to break things)
I don't agree that "dpkg-buildpackage sets additional environment variables
to implement a distro/site policy for builds" implies "calling debian/rules
directly is unsupported". Or maybe I've misunderstood, and there are Debian
developers who are building official packages for *upload* by calling
debian/rules by hand, and that's what people are concerned about preserving
while still getting the benefits of these distro build flags?
I hadn't considered that possibility, because I can't imagine anyone wanting
to build packages that way instead of using dpkg-buildpackage, which does it
all in a single command. So I really don't consider that an important use
case, weighed against the concerns I outlined.
> For example, you possibly get something different depending on whether you
> call debian/rules, dpkg-buildpackage, debuild, or pbuilder. And the
> difference is hard to explain or analyze.
Er, both debuild and pbuilder invoke dpkg-buildpackage. So it seems clear
to me that the only difference would be when calling debian/rules directly,
and at that point you're opting out of lots of other conveniences, not just
distro build policy.
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slan...@ubuntu.com vor...@debian.org
Goswin von Brederlow
> On Sun, May 10, 2009 at 11:37:46PM +0300, Peter Eisentraut wrote:
>> On Sunday 10 May 2009 13:56:04 Steve Langasek wrote:
>> > I thought it was generally recognized that it's a Bad Idea to implement
>> > config files using your interpreter's 'include' functionality, but that's
>> > basically what we have here.
>
>> Guillem pointed out one problem: Either you do it via a make include (which
>> you have issues with), or you stop supporting calling debian/rules directly
>> (inconvenient, probably prone to break things)
>
> I don't agree that "dpkg-buildpackage sets additional environment variables
> to implement a distro/site policy for builds" implies "calling debian/rules
> directly is unsupported". Or maybe I've misunderstood, and there are Debian
> developers who are building official packages for *upload* by calling
> debian/rules by hand, and that's what people are concerned about preserving
> while still getting the benefits of these distro build flags?
>
> I hadn't considered that possibility, because I can't imagine anyone wanting
> to build packages that way instead of using dpkg-buildpackage, which does it
> all in a single command. So I really don't consider that an important use
> case, weighed against the concerns I outlined.
And yet people do.
Also don't forget that many people call debuild, get an error, edit
some file, run debian/rules foo to see if it got fixed. Now suddenly
that quick check if it got fixed behaves differently.
>> For example, you possibly get something different depending on whether you
>> call debian/rules, dpkg-buildpackage, debuild, or pbuilder. And the
>> difference is hard to explain or analyze.
>
> Er, both debuild and pbuilder invoke dpkg-buildpackage. So it seems clear
> to me that the only difference would be when calling debian/rules directly,
> and at that point you're opting out of lots of other conveniences, not just
> distro build policy.
MfG
Goswin
Goswin von Brederlow
Debuild already creates a build log. I think it would be nice to
include that file in the changes file and have DAK forward it to
buildd.debian.org for archival. git-buildpackage, svn-buildpackage,
... or even dpkg-buildpackage could do this too.
Roger Leigh
> Debuild already creates a build log. I think it would be nice to
> include that file in the changes file and have DAK forward it to
> buildd.debian.org for archival. git-buildpackage, svn-buildpackage,
> ... or even dpkg-buildpackage could do this too.
Full ACK. So long as the build logs don't get held up by dak for
processing with the rest of the uploaded files and just get
forwarded to buildd.debian.org, this could make the buildd code
a bit simpler.
Regards,
Roger
--
.''`. Roger Leigh
: :' : Debian GNU/Linux http://people.debian.org/~rleigh/
`. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/
`- GPG Public Key: 0x25BFB848 Please GPG sign your mail.
Paul Wise
> Debuild already creates a build log. I think it would be nice to
> include that file in the changes file and have DAK forward it to
> buildd.debian.org for archival. git-buildpackage, svn-buildpackage,
> ... or even dpkg-buildpackage could do this too.
It would be much nicer to discard maintainer-built packages and build
everything on the buildds. Then we get build logs as well as the
opportunity to replicate Ubuntu's automatically created debug debs.
--
bye,
pabs
http://wiki.debian.org/PaulWise
Manoj Srivastava
> On Sun, May 10, 2009 at 11:37:46PM +0300, Peter Eisentraut wrote:
>> On Sunday 10 May 2009 13:56:04 Steve Langasek wrote:
>> > I thought it was generally recognized that it's a Bad Idea to implement
>> > config files using your interpreter's 'include' functionality, but that's
>> > basically what we have here.
>
>> Guillem pointed out one problem: Either you do it via a make include (which
>> you have issues with), or you stop supporting calling debian/rules directly
>> (inconvenient, probably prone to break things)
>
> I don't agree that "dpkg-buildpackage sets additional environment
> variables to implement a distro/site policy for builds" implies
> "calling debian/rules directly is unsupported". Or maybe I've
> misunderstood, and there are Debian developers who are building
> official packages for *upload* by calling debian/rules by hand, and
> that's what people are concerned about preserving while still getting
> the benefits of these distro build flags?
The only builds Debian supports are not just the buildd ones. As
members of the free software community, we should also cater to end
users building, tweaking, and rebuilding our software.
If the behaviour of the package is substantively different when
we use or not use dpkg-buildpackage, it would be a clear violation of
the principle of least surprise.
> I hadn't considered that possibility, because I can't imagine anyone
> wanting to build packages that way instead of using dpkg-buildpackage,
> which does it all in a single command. So I really don't consider
> that an important use case, weighed against the concerns I outlined.
If you take the self centered approach that Debian cares only
about the binary packages, and does not cater to end users also
building packages, sure, it is not an important use case.
I would prefer Debian to remain a full fledged member of the free
software community, and continue to not let build behaviour diverge
whether or not dpkg-buildpackage was used -- which can be a substancial
resource hog for multiple binary source packages.
manoj
--
When you were born, a big chance was taken for you.
Manoj Srivastava <sriv...@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Raphael Hertzog
> I would prefer Debian to remain a full fledged member of the free
> software community, and continue to not let build behaviour diverge
> whether or not dpkg-buildpackage was used -- which can be a substancial
> resource hog for multiple binary source packages.
resource hog??
If you refer to the fact, that dpkg-buildpackage cleans and rebuilds
everything and that it can take a lot of time, then please stop using
arguments that do not hold at all. you can call arbitrary debian/rules
targets with dpkg-buildpackage (without calling the clean rule
before-hand) and I pointed that to you multiple times already.
It's in dpkg 1.15.x (in experimental right now, in sid this week) and
it's the --target option.
Cheers,
--
Raphaï¿œl Hertzog
Contribuez ￜ Debian et gagnez un cahier de l'admin Debian Lenny :
http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/
Peter Eisentraut
> Or maybe I've misunderstood, and there are
> Debian developers who are building official packages for *upload* by
> calling debian/rules by hand, and that's what people are concerned about
> preserving while still getting the benefits of these distro build flags?
>
> I hadn't considered that possibility, because I can't imagine anyone
> wanting to build packages that way instead of using dpkg-buildpackage,
> which does it all in a single command. So I really don't consider that an
> important use case, weighed against the concerns I outlined.
I don't either, but it would probably be better to spell that out explicitly
somewhere.
> > For example, you possibly get something different depending on whether
> > you call debian/rules, dpkg-buildpackage, debuild, or pbuilder. And the
> > difference is hard to explain or analyze.
>
> Er, both debuild and pbuilder invoke dpkg-buildpackage. So it seems clear
> to me that the only difference would be when calling debian/rules directly,
> and at that point you're opting out of lots of other conveniences, not just
> distro build policy.
Well, debuild calls dpkg-buildpackage most of the time, unless you give a
specific target (which would again possibly be of interest to those who are
interested in calling debian/rules by hand for some reason). And that is also
something newish. Plus, you can set separate environment variables for
debuild. And probably also for pbuilder. And you can set environment
variables or possibly site files within the pbuilder chroot. And there is also
the option of pbuilder calling debuild -- who knows what that really does.
So again some of this would become clearer if the actually supported build
methods are more clearly spelled out. And then if someone could fold all of
the functionality of debuild into dpkg-buildpackage, there would be even less
distraction and variation.
Raphael Hertzog
> > these Makefile includes, then at a minimum I think Policy needs to
> > *very* tightly constrain what dpkg is allowed to put in those files,
> > to avoid future incompatibilities.
>
> policy should not be a stick to hit the dpkg developers on the head
> with. I say we let an implementation blossom in the wild, see what
> works, and then codif that in policy.
BTW, just to make things clear. It's likely that those Makefile snippet
(if we decide to go that way) become quite more elaborated as we try
integrating support for things like hardening-wrapper (see #489771).
Expect stuff like "if debian/control has Build-Options-Supported:
hardening" then use that set of flags, otherwise that one.
Cheers,
--
Raphaï¿œl Hertzog
Contribuez ￜ Debian et gagnez un cahier de l'admin Debian Lenny :
http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/
Raphael Hertzog
> Well, debuild calls dpkg-buildpackage most of the time, unless you give a
> specific target (which would again possibly be of interest to those who are
> interested in calling debian/rules by hand for some reason). And that is also
> something newish.
Any pointer to this feature?
> So again some of this would become clearer if the actually supported build
> methods are more clearly spelled out. And then if someone could fold all of
> the functionality of debuild into dpkg-buildpackage, there would be even less
> distraction and variation.
It's planned, see #476221.
Cheers,
--
Raphaï¿œl Hertzog
Contribuez ￜ Debian et gagnez un cahier de l'admin Debian Lenny :
http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/
Peter Eisentraut
> On Mon, 11 May 2009, Peter Eisentraut wrote:
> > Well, debuild calls dpkg-buildpackage most of the time, unless you give a
> > specific target (which would again possibly be of interest to those who
> > are interested in calling debian/rules by hand for some reason). And that
> > is also something newish.
>
> Any pointer to this feature?
Um, didn't you yourself orginally report this? (bug #476100) Anyway, the
current man pages of debuild says this:
"""
An alternative way of using debuild is to use one or more of the parameters
binary, binary-arch, binary-indep and clean, in which case debuild will
attempt to gain root privileges and then run debian/rules with the given
parameters.
"""
> > So again some of this would become clearer if the actually supported
> > build methods are more clearly spelled out. And then if someone could
> > fold all of the functionality of debuild into dpkg-buildpackage, there
> > would be even less distraction and variation.
>
> It's planned, see #476221.
That sounds like the ticket.
Giacomo A. Catenazzi
> Holger Levsen <hol...@layer-acht.org> writes:
>
>> Hi,
>>
>> On Sonntag, 10. Mai 2009, Raphael Hertzog wrote:
>>> With the include approach, we lack this feature and bad/broken local
>>> overrides can't be detected if we only have the build log at hand.
>> which reminds me that we dont have build logs for probably a lot more than 25%
>> (*) of the most used packages in the archive. The ones build manually by the
>> developer...
>>
>> I would really like to see that binaries for all archs are (re-)build on
>> builds and that those logs are kept archived and linked from the PTS. (And
>> arch all too of course).
>>
>>
>> regards,
>> Holger
>>
>> (*) wild guess, asuming that most packages are upload on amd64/i386/powerpc
>> and mostly used on i386.
>
> Debuild already creates a build log. I think it would be nice to
> include that file in the changes file and have DAK forward it to
> buildd.debian.org for archival. git-buildpackage, svn-buildpackage,
> ... or even dpkg-buildpackage could do this too.
but I would like to have more :-)
Currently I prefer to build package with --quiet flag, so that I see
easily problems on building package:
I see that a lot of my sponsoree forgot to check *carefully* logs,
to look for deprecated features, or important but non fatal warnings.
So I would like to have a short log (e.g. what I put in stdout/stderr,
with "./configure --quiet"), so that people will have no excuses for
not be carefukll, but also to have access to configure.log (or/and
other build time log), on build failure.
Is it to much to ask?
ciao
cate
Giacomo A. Catenazzi
> On Sun, May 10 2009, Steve Langasek wrote:
>
>> On Sun, May 10, 2009 at 11:37:46PM +0300, Peter Eisentraut wrote:
>>> On Sunday 10 May 2009 13:56:04 Steve Langasek wrote:
>>>> I thought it was generally recognized that it's a Bad Idea to implement
>>>> config files using your interpreter's 'include' functionality, but that's
>>>> basically what we have here.
>>> Guillem pointed out one problem: Either you do it via a make include (which
>>> you have issues with), or you stop supporting calling debian/rules directly
>>> (inconvenient, probably prone to break things)
>> I don't agree that "dpkg-buildpackage sets additional environment
>> variables to implement a distro/site policy for builds" implies
>> "calling debian/rules directly is unsupported". Or maybe I've
>> misunderstood, and there are Debian developers who are building
>> official packages for *upload* by calling debian/rules by hand, and
>> that's what people are concerned about preserving while still getting
>> the benefits of these distro build flags?
>
> The only builds Debian supports are not just the buildd ones. As
> members of the free software community, we should also cater to end
> users building, tweaking, and rebuilding our software.
You are a very special case: a developer since very long time, with a
enormous knowledge of debian policy (and dpkg internal).
But I really think that most people outside DD use dpkg-buildpackage
because it is the easier way (without need to remember a lot of
details). I think also that most of DDs use dpkg-buildpackage.
Note: it is also used to do "make clean" before *any* build, because
the non traced dependencies. E.g. on old time, some kernel configurations
needed a "make clean". I don't think I want to remember when I need
to run "make clean", so any changes on compiler, on configuration or
other heavy changes I run "make clean". I think for the same reason
people (but people very deep in debian packaging) will use
dpkg-buildpackage: not the optimal way, but it is safer, and with
less "surprises".
A new point, still neglected on this thread:
there are other nasty cases: as we saw in an other thread on debian-policy,
the current locale changes the way wide-characters (and wide strings) are
encoded in binaries (UTF-16 if current charset in current locale could
be encoded in 16-bit; UTF-32 on other cases, and fortunately when no charset
is specified on current locale, thus also on "C" locale).
Wide characters are not common, but they are specified not only in POSIX but
also in standard C (also in C89/C90 version IIRC).
I don't see a good way that:
- developers will do the right things (how many of you doesn know such
strangeness?)
[ add a flag to CFLAG in every debian/rules is IMHO not the right thing ]
- it is documented (e.g. on logs), actually we need to parse binaries to
find what type of encoding did the original uploader (not so complex
because we have only currently two encoding, but anyway...]
- security team and MNU doesn't break package because of nasty interaction
because of different locale.
other than using dpkg-buildpackage (and letting dpkg-buildpackage
to set this (and probably other nearly-unknow paramenters)
ciao
cate
Manoj Srivastava
> On Sun, 10 May 2009, Manoj Srivastava wrote:
>> I would prefer Debian to remain a full fledged member of the free
>> software community, and continue to not let build behaviour diverge
>> whether or not dpkg-buildpackage was used -- which can be a substancial
>> resource hog for multiple binary source packages.
>
> resource hog??
That is indeed what I said.
> If you refer to the fact, that dpkg-buildpackage cleans and rebuilds
> everything and that it can take a lot of time, then please stop using
> arguments that do not hold at all. you can call arbitrary debian/rules
> targets with dpkg-buildpackage (without calling the clean rule
> before-hand) and I pointed that to you multiple times already.
> It's in dpkg 1.15.x (in experimental right now, in sid this week) and
> it's the --target option.
Nice rant about a facility that is not yet in proper
distribution (ustable, testing, or stable).
So, until a date in the future at this point, using
dpkg-buildpackageis a resource hog, and no, you have not told me
multiple times.
manoj
--
"And remember: Evil will always prevail, because Good is dumb."
Spaceballs
Manoj Srivastava <sriv...@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Russ Allbery
> BTW, just to make things clear. It's likely that those Makefile
> snippet (if we decide to go that way) become quite more elaborated as
> we try integrating support for things like hardening-wrapper (see
> #489771). Expect stuff like "if debian/control has
> Build-Options-Supported: hardening" then use that set of flags,
> otherwise that one.
I still think Build-Options-Supported is fundamentally the wrong way to
implement that. You have to modify every package to add it anyway, in
which case you can just as easily support it in the package's
debian/rules the way that we already support various other
DEB_BUILD_OPTIONS settings.
--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>
Russ Allbery
> Manoj Srivastava wrote:
>> The only builds Debian supports are not just the buildd ones. As
>> members of the free software community, we should also cater to end
>> users building, tweaking, and rebuilding our software.
> You are a very special case: a developer since very long time, with a
> enormous knowledge of debian policy (and dpkg internal). But I really
> think that most people outside DD use dpkg-buildpackage because it is
> the easier way (without need to remember a lot of details). I think
> also that most of DDs use dpkg-buildpackage.
The number of people using different methods is fairly irrelevant to
Manoj's point. The question is more fundamental: are packages built by
a makefile that we call debian/rules, or are they built by the program
dpkg-buildpackage using debian/rules as a configuration file for that
program?
If they're built by the program, then anyone who wants to properly build
the software, even if they don't want to go all the way to the package,
will need to use the program, since people will write debian/rules such
that it assumes the program is in use. They'll assume default CFLAGS
are set and so forth.
I don't think this is the right direction to go, but I'm not going to
stomp off in a huff if we go that direction or anything. :) But I do
want to be sure that we're all clear on what we're saying if we do take
that approach and make dpkg-buildpackage the only supported way to build
packages. I think it's likely that if we go that route, with it
providing the defaults, we'll find over time that some packages will
either not build or will mis-build with debian/rules build and no one
will notice or be particularly interested in fixing it.
--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>
Gabor Gombas
> You are a very special case: a developer since very long time, with a
> enormous knowledge of debian policy (and dpkg internal).
> But I really think that most people outside DD use dpkg-buildpackage
> because it is the easier way (without need to remember a lot of
> details). I think also that most of DDs use dpkg-buildpackage.
My experiences are quite the contrary: people who are not deeply
involved with Debian tend to run debian/rules directly, and the only
Debian-specific command they know is "dpkg -i --force-depends"...
Gabor
--
---------------------------------------------------------
MTA SZTAKI Computer and Automation Research Institute
Hungarian Academy of Sciences
---------------------------------------------------------
Raphael Hertzog
> > If you refer to the fact, that dpkg-buildpackage cleans and rebuilds
> > everything and that it can take a lot of time, then please stop using
> > arguments that do not hold at all. you can call arbitrary debian/rules
> > targets with dpkg-buildpackage (without calling the clean rule
> > before-hand) and I pointed that to you multiple times already.
> > It's in dpkg 1.15.x (in experimental right now, in sid this week) and
> > it's the --target option.
>
> Nice rant about a facility that is not yet in proper
> distribution (ustable, testing, or stable).
You have however "-nc" since a long time and if you have proper Make
dependencies it should be pretty quick by not redoing unnecessary work.
Furthermore, we're designing stuff for the future so you have no reason
to dismiss that fact even if it's not yet in sid.
> So, until a date in the future at this point, using
> dpkg-buildpackageis a resource hog, and no, you have not told me
> multiple times.
Right I have not repeated it multiple times in the discussion because
I try to avoid that but it was in the initial wiki page that started this
discussion: http://wiki.debian.org/Teams/Dpkg/DebianRules
Cheers,
--
Raphaï¿œl Hertzog
Contribuez ￜ Debian et gagnez un cahier de l'admin Debian Lenny :
http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/
Raphael Hertzog
> Raphael Hertzog <her...@debian.org> writes:
>
> > BTW, just to make things clear. It's likely that those Makefile
> > snippet (if we decide to go that way) become quite more elaborated as
> > we try integrating support for things like hardening-wrapper (see
> > #489771). Expect stuff like "if debian/control has
> > Build-Options-Supported: hardening" then use that set of flags,
> > otherwise that one.
>
> I still think Build-Options-Supported is fundamentally the wrong way to
> implement that. You have to modify every package to add it anyway, in
> which case you can just as easily support it in the package's
> debian/rules the way that we already support various other
> DEB_BUILD_OPTIONS settings.
Except that with the centralized approach we can have an opt-out policy
after some time (ie use hardening options by default except for packages
that have set no-hardening).
My interest in centralizing those is simplifying any transition in
default flags that Debian would want to do. User customization is nice
but it's not what motivates me to push this forward.
Cheers,
--
Raphaï¿œl Hertzog
Contribuez ￜ Debian et gagnez un cahier de l'admin Debian Lenny :
http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/
Russ Allbery
> On Mon, 11 May 2009, Russ Allbery wrote:
>> I still think Build-Options-Supported is fundamentally the wrong way
>> to implement that. You have to modify every package to add it
>> anyway, in which case you can just as easily support it in the
>> package's debian/rules the way that we already support various other
>> DEB_BUILD_OPTIONS settings.
> Except that with the centralized approach we can have an opt-out
> policy after some time (ie use hardening options by default except for
> packages that have set no-hardening).
That seems orthogonal. Either way, you have to get most package
maintainers to modify their packages and test to be sure that you can
change the default build flags. Either way, the results of that change
will produce artifacts that you can look for to see how many packages
are currently building with the new flags. Either way, there is a way
for maintainers to opt out of default flags.
I understand conceptually how Build-Options-Supported could be useful,
but I've yet to see a specific application for it that doesn't seem like
it would be better done some other way. For example, saying that you
support parallel builds there seems inferior to just enabling parallel
builds if the DEB_BUILD_OPTIONS flag is set and you support it. Opting
in to specific features similarly makes more sense as DEB_BUILD_OPTIONS
to me. Opting out of default flags can be done with make's filter
support. build-arch/build-indep should just be *done* rather than
asking packages to say whether they support it, IMO.
> My interest in centralizing those is simplifying any transition in
> default flags that Debian would want to do. User customization is nice
> but it's not what motivates me to push this forward.
Sure, that's the point of the whole discussion: how best to do that
across the archive and whether we should break debian/rules build in
order to do so. I think there's general agreement on the goal.
--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>
Raphael Hertzog
> Raphael Hertzog <her...@debian.org> writes:
> > On Mon, 11 May 2009, Russ Allbery wrote:
>
> >> I still think Build-Options-Supported is fundamentally the wrong way
> >> to implement that. You have to modify every package to add it
> >> anyway, in which case you can just as easily support it in the
> >> package's debian/rules the way that we already support various other
> >> DEB_BUILD_OPTIONS settings.
>
> > Except that with the centralized approach we can have an opt-out
> > policy after some time (ie use hardening options by default except for
> > packages that have set no-hardening).
>
> That seems orthogonal. Either way, you have to get most package
> maintainers to modify their packages and test to be sure that you can
> change the default build flags. Either way, the results of that change
> will produce artifacts that you can look for to see how many packages
> are currently building with the new flags. Either way, there is a way
> for maintainers to opt out of default flags.
The fact that we can filter out some default flags doesn't make it
a better approach IMO. If you just want to disable hardening for your
package, it would be a pain to have to filter out a possibly evolving
list of default flags.
And yes, it's best when all package maitainers test their package
for the change, but quite a few are not as pro-active and you should not
assume that we will modify all packages. To complete any migration, we
must have the possibility to just do the change and manually fix up
packages where nothing has been stated.
(The approach might vary depending on the risks, etc but you get the idea)
> support. build-arch/build-indep should just be *done* rather than
> asking packages to say whether they support it, IMO.
Which means that policy must state "MUST" for those targets to exist.
In which case the Build-Options-Supported is implicit and derived from the
Standards-Version.
Cheers,
--
Raphaï¿œl Hertzog
Contribuez ￜ Debian et gagnez un cahier de l'admin Debian Lenny :
http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/
Russ Allbery
> On Mon, 11 May 2009, Russ Allbery wrote:
>> That seems orthogonal. Either way, you have to get most package
>> maintainers to modify their packages and test to be sure that you can
>> change the default build flags. Either way, the results of that
>> change will produce artifacts that you can look for to see how many
>> packages are currently building with the new flags. Either way,
>> there is a way for maintainers to opt out of default flags.
> The fact that we can filter out some default flags doesn't make it a
> better approach IMO. If you just want to disable hardening for your
> package, it would be a pain to have to filter out a possibly evolving
> list of default flags.
Why would you want to disable all hardening instead of filtering out the
flag that breaks the package?
> And yes, it's best when all package maitainers test their package for
> the change, but quite a few are not as pro-active and you should not
> assume that we will modify all packages. To complete any migration, we
> must have the possibility to just do the change and manually fix up
> packages where nothing has been stated.
Which again I agree with but it's orthogonal to what you're talking
about. You can do that either way.
> Which means that policy must state "MUST" for those targets to exist.
> In which case the Build-Options-Supported is implicit and derived from
> the Standards-Version.
In which case again you're not using Build-Options-Supported.
Besides, I'm skeptical that's the right way to handle that transition.
It seems to me like it would be a lot more effective to add a Lintian
test first, warn everyone, do a mass bug filing, and otherwise follow
the same practice we always use for global archive changes. That's the
procedure that we follow when we're trying to do something that changes
large numbers of packages. Basing it off Standards-Version to my mind
just creates FTBFS landmines in the archive that will be exploding for
years to come as people update old packages and don't think to add
build-arch. I suppose that has the advantage of spreading the pain out
across many years, but wouldn't it be better to get it over with?
--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>
Giacomo A. Catenazzi
> "Giacomo A. Catenazzi" <ca...@debian.org> writes:
>> Manoj Srivastava wrote:
>
>>> The only builds Debian supports are not just the buildd ones. As
>>> members of the free software community, we should also cater to end
>>> users building, tweaking, and rebuilding our software.
>
>> You are a very special case: a developer since very long time, with a
>> enormous knowledge of debian policy (and dpkg internal). But I really
>> think that most people outside DD use dpkg-buildpackage because it is
>> the easier way (without need to remember a lot of details). I think
>> also that most of DDs use dpkg-buildpackage.
>
> The number of people using different methods is fairly irrelevant to
> Manoj's point. The question is more fundamental: are packages built by
> a makefile that we call debian/rules, or are they built by the program
> dpkg-buildpackage using debian/rules as a configuration file for that
> program?
I see.
Anyway I don't see it as configuration file, but as environment for
debian/rules.
Note: currently we use fakeroot in order to setup other things
of "environment". It is not mandatory (we can build deb as superuser),
but also dpkg-buildpackage will not be mandatory (if user set
some environment variable).
Could maintainer do a sensible choice of CFLAGS? I don't think so.
The flags are too much dependent on architecture then the
theoretical meaning.
- Support of some flags on some architecture is broken
(according to documentation)
- optimization is not safe on some architectures (broken
compiler) or it worse the code (broken default choices)
- debug symbol format changes in different architectures:
we need to use -g, -ggdb or what?
For these reasons I see CFLAGS more as an architectural choice
than a maintainer choice. But I'm not an expert on the field,
and I've no packages on core Debian, where flags could be
more interesting.
And in the thread, it seems that there are such cases. I'm
currious to see what are such cases (maintainer MUST override
flags) and the reason.
In every case, I think the .dsc should include some build
information, to block earlier some build errors
(e.g. include with custom setting, or dpkg-buildpackage
non Debian-standard settings). We will do errors, so we need
also a way to find out such errors.
ciao
cate
Steve Langasek
> If they're built by the program, then anyone who wants to properly build
> the software, even if they don't want to go all the way to the package,
> will need to use the program, since people will write debian/rules such
> that it assumes the program is in use. They'll assume default CFLAGS
> are set and so forth.
> I don't think this is the right direction to go, but I'm not going to
> stomp off in a huff if we go that direction or anything. :) But I do
> want to be sure that we're all clear on what we're saying if we do take
> that approach and make dpkg-buildpackage the only supported way to build
> packages. I think it's likely that if we go that route, with it
> providing the defaults, we'll find over time that some packages will
> either not build or will mis-build with debian/rules build and no one
> will notice or be particularly interested in fixing it.
That's a fair point, and if preserving the behavior of debian/rules as a
standalone is important to others, then I'm happy for us to find a solution
that meets this requirement *as long as* it also avoids the pain of letting
mandated "config" files arbitrarily modify the behavior of debian/rules.
Robert Collins' suggestion in <1241989292.8026.20.camel@lifeless-64> seems
like a good approach for this, then (modulo the syntax bits, and putting the
tool in the dpkg-* namespace instead of debhelper's namespace).
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slan...@ubuntu.com vor...@debian.org
Raphael Hertzog
> > The fact that we can filter out some default flags doesn't make it a
> > better approach IMO. If you just want to disable hardening for your
> > package, it would be a pain to have to filter out a possibly evolving
> > list of default flags.
>
> Why would you want to disable all hardening instead of filtering out the
> flag that breaks the package?
Because no-one has identified the precise flag that breaks the package?
Or because you really want no hardening at all because you don't want to
have to deal with subtle breakages caused by a supplementary
hardening flags added later?
I know that we always have many different opinions when it comes to
implement some new stuff and I'd rather have some supplementary
flexibility that is not needed rather than finding atferwards
that we need more of it because when it concerns the whole archive, it's
painful.
> > And yes, it's best when all package maitainers test their package for
> > the change, but quite a few are not as pro-active and you should not
> > assume that we will modify all packages. To complete any migration, we
> > must have the possibility to just do the change and manually fix up
> > packages where nothing has been stated.
>
> Which again I agree with but it's orthogonal to what you're talking
> about. You can do that either way.
We suppose all packages already have the right include that imports
the default cflags. So now, you decide it's time to use hardened flags
by default so you just change the default flags. Is that right?
How did we enable hardening flags on individual package before (in your
scenario)? (Do we have a copy of all hardened flags in each package? if
not where do they come from since the default flags don't contain them?)
What were maintainers supposed to do in advance to disable some
hardening flags that break their packages given that they
don't even know for sure what the default flags will look like.
Having stuff in place and just switching a test looks
more predictable than not having it and simply relying on
some d-d-a mail stating: in X months the default flags will be Y,
try it out and make sure your package don't break when we do
it.
(But maybe I guessed this all wrong and you had something else in mind)
> It seems to me like it would be a lot more effective to add a Lintian
> test first, warn everyone, do a mass bug filing, and otherwise follow
> the same practice we always use for global archive changes. That's the
> procedure that we follow when we're trying to do something that changes
> large numbers of packages. Basing it off Standards-Version to my mind
> just creates FTBFS landmines in the archive that will be exploding for
> years to come as people update old packages and don't think to add
> build-arch. I suppose that has the advantage of spreading the pain out
> across many years, but wouldn't it be better to get it over with?
Not sure at all. In part because only a small percentage of the packages
would benefit from it (those that build arch: all and arch: any and where
the work to compile doc for the arch: all is expensive). So targetting a
MUST in policy is overkill in my opinion.
Maybe the right thing to do is to make it mandatory only for packages
that have both arch-indep and arch-specific binary packages.
Cheers,
--
Raphaï¿œl Hertzog
Contribuez ￜ Debian et gagnez un cahier de l'admin Debian Lenny :
http://www.ouaza.com/wp/2009/03/02/contribuer-a-debian-gagner-un-livre/
Russ Allbery
> Robert Collins' suggestion in <1241989292.8026.20.camel@lifeless-64>
> seems like a good approach for this, then (modulo the syntax bits, and
> putting the tool in the dpkg-* namespace instead of debhelper's
> namespace).
I like this idea on first glance. It matches what we currently
recommend with dpkg-architecture.
--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>
Russ Allbery
> On Tue, 12 May 2009, Russ Allbery wrote:
>> Why would you want to disable all hardening instead of filtering out
>> the flag that breaks the package?
> Because no-one has identified the precise flag that breaks the package?
Then filter out the ones that might cause the problem. Last I heard,
we're only talking about two or three flags here and they weren't
changing particularly fast.
> Or because you really want no hardening at all because you don't want to
> have to deal with subtle breakages caused by a supplementary
> hardening flags added later?
It sounds like you're thinking that we could potentially change the
hardening flags down the road without going through the same checking
and care that we did when introducing them in the first place. I'm
fairly sure that's unrealistic.
> I know that we always have many different opinions when it comes to
> implement some new stuff and I'd rather have some supplementary
> flexibility that is not needed rather than finding atferwards that we
> need more of it because when it concerns the whole archive, it's
> painful.
I think that having a big switch that turns on (or off) some
unrestricted set of hardening flags that may change in the future is not
horribly useful flexibility.
> We suppose all packages already have the right include that imports
> the default cflags. So now, you decide it's time to use hardened flags
> by default so you just change the default flags. Is that right?
Correct. Obviously you'd have to do a lot of preparation before doing
that sort of global change.
> How did we enable hardening flags on individual package before (in
> your scenario)?
If DEB_BUILD_OPTIONS contains "hardening", the package would add the
contents of DEB_BUILD_HARDENING_CFLAGS to CFLAGS and
DEB_BUILD_HARDENING_LDFLAGS to LDFLAGS. (For example. There are other
ways it could be done, but that's the one that comes to mind.)
At some point we'd add a nohardening DEB_BUILD_OPTIONS setting that
would disable hardening flags for people who wanted to do a build
without hardening and then add the hardening flags to the default flags
iff that option isn't set. (Assuming we decide to make hardening the
default.)
> What were maintainers supposed to do in advance to disable some
> hardening flags that break their packages given that they don't even
> know for sure what the default flags will look like.
Surely that's not a situation anyone would be in. It supposes that
we'll be in a situation where we're about to make hardening the global
default and yet we don't know what flags will be in it. If we're in
that situation, we screwed up, and we need to go back and finalize the
flags before we talk about changing the defaults.
> Not sure at all. In part because only a small percentage of the
> packages would benefit from it (those that build arch: all and arch:
> any and where the work to compile doc for the arch: all is
> expensive). So targetting a MUST in policy is overkill in my opinion.
That would argue for not doing it for all packages and instead providing
some way to say it's supported, and yeah, that's the case where I can
see having Build-Options-Supported. Although I also think it's becoming
increasingly easy to add the build-arch and build-indep targets. If one
added them to debhelper's rule minimization and cdbs, you'd already
catch a large percentage of the archive now.
Either way, we shouldn't base it off Standards-Version; that bothers me
much more than Build-Options-Supported does. :)
> Maybe the right thing to do is to make it mandatory only for packages
> that have both arch-indep and arch-specific binary packages.
It's really fairly rare that this matters that much, so yeah, the more I
think about it, the more you're probably right and this is a case where
the package should be able to communicate back to the build system that
it supports doing something. I have a ton of packages with both
arch-independent and arch-specific binary packages, but only one of them
would gain anything from this split.
--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>
Manoj Srivastava
> On Mon, May 11, 2009 at 10:01:14AM -0700, Russ Allbery wrote:
>> If they're built by the program, then anyone who wants to properly build
>> the software, even if they don't want to go all the way to the package,
>> will need to use the program, since people will write debian/rules such
>> that it assumes the program is in use. They'll assume default CFLAGS
>> are set and so forth.
>
>> I don't think this is the right direction to go, but I'm not going to
>> stomp off in a huff if we go that direction or anything. :) But I do
>> want to be sure that we're all clear on what we're saying if we do take
>> that approach and make dpkg-buildpackage the only supported way to build
>> packages. I think it's likely that if we go that route, with it
>> providing the defaults, we'll find over time that some packages will
>> either not build or will mis-build with debian/rules build and no one
>> will notice or be particularly interested in fixing it.
>
> That's a fair point, and if preserving the behavior of debian/rules as a
> standalone is important to others, then I'm happy for us to find a solution
> that meets this requirement *as long as* it also avoids the pain of letting
> mandated "config" files arbitrarily modify the behavior of debian/rules.
I am not sure I understand this. In effect, using helper
packages can allow arbitrary programs to change the build behaviour of
the package, but we heartily recommend using the helper packages.
But that is different you say, we do not expect debhelper to
change incompatibly without changing compat levels, we trust joey
hess to never do that. (I do too). But this also implies that you do
not trust dpkg maintainers to not put in things in the configuration
files without due delibration, and that they will not actually set up
a mechanism somewhat like COMPAT_VERSION (very easy to do in make
snippets, BTW).
I think we ought to trust the dpkg folks.
The other use case is the site admin building packages for
themselves, and the buildd maintainer, to set up the environment for
their build daemon. In other mails, you have argued that the only
thing you are worried about is build daemons, in that case, there are
scads of things that a build daemon admin can do that changes how
packages are built (versions of build essential packages, stuff related
to compiling in /etc, ld.so being just one) -- so we do implicitly
trust the build daemon maintainers to do the right thing.
So, the only places we distrust people is either the maintainer
for not setting their site config right, and uploading. But if you
distrust the maintainer, you might as well give it up, or set up a
process to require but discard .debs created by ordinary mere mortal
DD's, and let only the works of those on high (the buildd folks) to go
into Debian. A losing battle, I would say.
The other person you are so worried about braeking things is the
end user, who might muck up their site config, and thus build bad
packages. That the end user building packages for their own may wreak
greater havoc with vim is overlooked, neh?
What is this distrust of the configuration files all about,
then? I find it strange, and I think it stems from a very Cathedral
like approach of not trusting the end users to even be able to manage
site config files flies in the face of the bazaar; we allow for free
software adherents and our users to bui9ld packages, and let them
tinker and make mistakes.
The supposed control over "people would not be able to afect
how my package builds" is already a mirage. A buildd configuration is
already beyond most peoples control. A hacked debhelper is also beyond
your control.
One of the major shortcomings of the way we build things, and
that Gentoo has a serious advantage, is the so called build flags --
with this include file approach, we can come close to the build flags
approach, and let specialized build daemons be set up with site config
different from the norm to build specialized binary archives.
Being tied to tradition does not help us here.
manoj
--
Remember, UNIX spelled backwards is XINU. Mt.
Manoj Srivastava <sriv...@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Emilio Pozuelo Monfort
> It would be much nicer to discard maintainer-built packages and build
> everything on the buildds. Then we get build logs as well as the
> opportunity to replicate Ubuntu's automatically created debug debs.
Yes! At least discard arch:any debs, so that we don't need to wait until
building arch:all debs in the autobuilders is possible.
Emilio
Michael Banck
> but I would like to have more :-)
> Currently I prefer to build package with --quiet flag, so that I see
> easily problems on building package:
Neither debuild nor dpkg-buildpackage has a a --quiet flag, so I guess
you weren't clear enough on what you are talking about.
> So I would like to have a short log (e.g. what I put in stdout/stderr,
> with "./configure --quiet"), so that people will have no excuses for
> not be carefukll, but also to have access to configure.log (or/and
> other build time log), on build failure.
OK, so I am not sure what this has to do with the thread. Are you
proposing a completely different way of how autoconf packages should get
configured in debian/rules? What does ./configure --quiet have to do
with compiler warnings or deprecated features? Does it print out
relevant parts of config.log on failure? If not, what is the point of
"configure.log", having the same information then than one has now in
general?
> Is it to much to ask?
You can ask anything, the question is whether it gets picked up. If
what you ask for required changes in most source packages, it is likely
that it will not get picked up.
What I do think might make sense is changing dh7 and/or CDBS to print
out config.log if configure fails; that might help debug problems on the
autobuilders more quickly.
Michael
Roger Leigh
It's possible right now. You just set $build_arch_all=1 in the sbuild
configuration for a single arch (e.g. i386 or amd64). You would need
some small changes in wanna-build to track arch-all packages and sources
AFAICT, but it's not a huge task.
Regards,
Roger
--
.''`. Roger Leigh
: :' : Debian GNU/Linux http://people.debian.org/~rleigh/
`. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/
`- GPG Public Key: 0x25BFB848 Please GPG sign your mail.
Giacomo A. Catenazzi
Ok, I was not so clear. I was replying to:
>> Goswin von Brederlow wrote:
>>> Debuild already creates a build log. I think it would be nice to
>>> include that file in the changes file and have DAK forward it to
>>> buildd.debian.org for archival.
Thus: my request was more about our infrastructure then the
building tools.
In my experience, the ./configure script are usually too verbose
(whose created with autoconf in particulary). Maintainer tends
not to parse carefully the output of such script, e.g. ask my
sponsorees ;-) , missing important parts.
So, if we centrally save the logs, I would like to allow maintainer to
run configure with --quiet, but also to save extra informations for
late processing.
It is not only build failures, but the log (on file, which is much more
verbose then normal output) contains also other useful informations:
e.g. the compiler and other tool versions, etc.
E.g. if we saw a bug (secutity or also miscompilation) in one package
in the toolchain, we can found what package need to be recompiled (and
in what architecture).
I think such logs are more useful to archive, than the simple stdout/stderr.
BTW I just discovered dh_buildinfo, which unfortunately has no manpage.
ciao
cate