Bug#552276: dh: print full valid command line of each executed dh_* command
Modestas Vainius
Version: 7.4.3
Severity: normal
Hello,
This is a follow up to my patch from [1]. Since the patch does not really deal
with argument parsing, just with consistent and accurate debhelper command and
its arguments printing, I'm opening a new bug to track this.
Since options to debhelper commands can be passed via DH_INTERNAL_OPTIONS,
DH_OPTIONS and command line, the first two are typically not visible in the
logs. Invisibility of DH_INTERNAL_OPTIONS is especially bad since dh puts
"black magic" in there. To better expose this, dh should print a final command
line of dh_* commands as they are executed.
My previous patch proposed dh_* to print their command line by themselves. Now
I attach a proof of concept implementation (sequencer, aka dh, and dh_command)
of the following scheme:
1) Sequencer creates a pipe, exports its FDs in DH_BATCH_MODE environment
variable and forks into two processes. The first process listens on the pipe
for commands from dh_* commands and handles dh_* command line printing. The
2nd process executes commands (like current dh does now).
2) As dh executes dh_command, the latter parses its command line and
determines which options are valid and which are not (code in the attached
dh_command script). Real dh command would process DH_INTERNAL_OPTIONS,
DH_OPTIONS and @ARGV and merge the results into single command line. Finally,
dh_command sends its full and *valid* command line to dh via open pipe.
3) dh prints that command line and sends back a signal to dh_command to start
doing real work.
4) dh_command receives a start signal and continues as usual.
This way, dh_command does not have to print its own command line which was an
ugly disadvantage of my previous patch. All command line printing is handled
by sequencer itself.
The attached proof of concept scripts are rather small, easy to edit and play
with, and good for demonstrating the behaviour of this scheme. I will
eventually port them to dh/Dh_Getopt if you agree on the concept and/or output
format.
So quick how-to:
$ tar zxf dh-output.tar.gz
$ cd dh-output
$ debian/rules
sequencer
--> dh_command
--> dh_command
--> dh_command
--> dh_command --try=4
--> dh_command
--> dh_command
--> dh_command
--> dh_command
--> make -f debian/rules override_dh_command
make[1]: Entering directory `/home/modax/test/dh-output'
dh_command --bar
--> dh_command --bar
make[1]: Leaving directory `/home/modax/test/dh-output'
--> dh_command --try=10 --foo
--> dh_command --try=11 --bar
--> dh_command --try=12
--> dh_command
--> dh_command
--> dh_command
--> dh_command
--> dh_command
--> make -f debian/rules override_dh_command
make[1]: Entering directory `/home/modax/test/dh-output'
dh_command --try=18 --bar
--> dh_command
make[1]: Leaving directory `/home/modax/test/dh-output'
--> dh_command
--> dh_command
--> dh_command
--> dh_command --foo
--> dh_command
--> dh_command --try=24
--> dh_command --bar
--> dh_command
--> make -f debian/rules override_dh_command
make[1]: Entering directory `/home/modax/test/dh-output'
dh_command --try=27
--> dh_command
make[1]: Leaving directory `/home/modax/test/dh-output'
--> dh_command
--> dh_command --try=29
--> dh_command
As you see, sequencer simulates dh overrides too to be as near real world
output as possible. Such format looks rather well to me...
1. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=541773#10
--
Modestas Vainius <mode...@vainius.eu>
Joey Hess
> Since options to debhelper commands can be passed via DH_INTERNAL_OPTIONS,
> DH_OPTIONS and command line, the first two are typically not visible in the
> logs. Invisibility of DH_INTERNAL_OPTIONS is especially bad since dh puts
> "black magic" in there. To better expose this, dh should print a final command
> line of dh_* commands as they are executed.
I don't think that DH_OPTIONS is a problem since the maintainer
determines it themselves.
> My previous patch proposed dh_* to print their command line by themselves. Now
> I attach a proof of concept implementation (sequencer, aka dh, and dh_command)
Wow, that's so complicated!
I was thinking about this problem today in the context of #541773,
and it came to me that all debhelper need was an -O option that
dh could use to pass user-supplied options to each command it runs. This
is a kind of optional option; if -O--foo is passed and a command does
not know about --foo, it just ignores it.
Result is:
dh build -B build --parallel 2 --foo="bar baz"
dh_auto_test -O-Bbuild -O--parallel=2 "-O--foo=bar baz"
dh_auto_build -O-Bbuild -O--parallel=2 "-O--foo=bar baz"
DH_INTERNAL_OPTIONS is still hidden. It might make sense to say:
make -f debian/rules override_dh_auto_build DH_INTERNAL_OPTIONS=-O-Bbuild...
But, since DH_INTERNAL_OPTIONS currently uses non-printable separator
characters, that's a small problem. Also, if the variable were exposed
this way it should probably be renamed.
--
see shy jo
Modestas Vainius
On antradienis 05 Sausis 2010 03:11:02 Joey Hess wrote:
> I was thinking about this problem today in the context of #541773,
> and it came to me that all debhelper need was an -O option that
> dh could use to pass user-supplied options to each command it runs. This
> is a kind of optional option; if -O--foo is passed and a command does
> not know about --foo, it just ignores it.
>
> Result is:
>
> dh build -B build --parallel 2 --foo="bar baz"
> dh_auto_test -O-Bbuild -O--parallel=2 "-O--foo=bar baz"
> dh_auto_build -O-Bbuild -O--parallel=2 "-O--foo=bar baz"
But so will:
dh_install -O-Bbuild -O--parallel=2 "-O--foo=bar baz"
dh_installdocs -O-Bbuild -O--parallel=2 "-O--foo=bar baz"
dh_installman -O-Bbuild -O--parallel=2 "-O--foo=bar baz"
dh_strip -O-Bbuild -O--parallel=2 "-O--foo=bar baz"
etc.
I mean, command line might be valid but output is not any more readable. On
the contrary, it is (significantly) less readable...
> DH_INTERNAL_OPTIONS is still hidden. It might make sense to say:
>
> make -f debian/rules override_dh_auto_build
> DH_INTERNAL_OPTIONS=-O-Bbuild...
iirc, DH_INTERNAL_OPTIONS is post processed in parseopts() by the program
itself. So dh can't know how command line will end up nor what debhelper
commands are run in overrides without some sort of feedback. I think we ask
too much of the user to figure out how to combine DH_INTERNAL_OPTIONS with
options passed in override.
> But, since DH_INTERNAL_OPTIONS currently uses non-printable separator
> characters, that's a small problem. Also, if the variable were exposed
> this way it should probably be renamed.
Non-printable separator is not a big problem. Options can be shell escaped and
non-printable character replaced with a whitespace in the output.
--
Modestas Vainius <mode...@vainius.eu>
Modestas Vainius
On antradienis 05 Sausis 2010 03:11:02 Joey Hess wrote:
> Modestas Vainius wrote:
> > Since options to debhelper commands can be passed via
> > DH_INTERNAL_OPTIONS, DH_OPTIONS and command line, the first two are
> > typically not visible in the logs. Invisibility of DH_INTERNAL_OPTIONS is
> > especially bad since dh puts "black magic" in there. To better expose
> > this, dh should print a final command line of dh_* commands as they are
> > executed.
>
> I don't think that DH_OPTIONS is a problem since the maintainer
> determines it themselves.
>
> > My previous patch proposed dh_* to print their command line by
> > themselves. Now I attach a proof of concept implementation (sequencer,
> > aka dh, and dh_command)
>
> Wow, that's so complicated!
What in particular do you dislike about my proposal? I guess you want to get
rid of pipe() and fork() complexity. Ok, I figure dh command can simply print
to dup()ed STDERR fd of dh rather than let fork of dh do it. What is more,
this way dh_command still does not pollute its own STDERR which is what I
wanted to achieve with pipe() in the first place.
Or do you dislike how detection of valid options is done? IMHO, that code is
not that complicated.
Anyway, I attach a modified version of dh-output (as explained above). Please
note that a good part of the example is generation of random options which is
redundant for real implementation.
Btw, have you tested your solution to -Bbuild (#541773) problem? I don't see
how -O solves it. And my test confirms it does not:
make[1]: Leaving directory `/home/modax/src/kde/attica-0.1.1-1/build'
dh_install -O-Bbuild
dh_installdocs -O-Bbuild
dh_installchangelogs -O-Bbuild
dh_installexamples -O-Bbuild
dh_installman -O-Bbuild
dh_installcatalogs -O-Bbuild
dh_installcron -O-Bbuild
dh_installdebconf -O-Bbuild
dh_installemacsen -O-Bbuild
dh_installifupdown -O-Bbuild
dh_installinfo -O-Bbuild
dh_pysupport -O-Bbuild
dh_installinit -O-Bbuild
dh_installmenu -O-Bbuild
dh_installmime -O-Bbuild
dh_installmodules -O-Bbuild
dh_installlogcheck -O-Bbuild
dh_installlogrotate -O-Bbuild
dh_installpam -O-Bbuild
dh_installppp -O-Bbuild
dh_installudev -O-Bbuild
dh_installwm -O-Bbuild
dh_installxfonts -O-Bbuild
dh_bugfiles -O-Bbuild
dh_lintian -O-Bbuild
dh_gconf -O-Bbuild
dh_icons -O-Bbuild
dh_perl -O-Bbuild
dh_usrlocal -O-Bbuild
dh_link -O-Bbuild
dh_compress -O-Bbuild
dh_fixperms -O-Bbuild
dh_strip -O-Bbuild
debian/rules override_dh_makeshlibs
make[1]: Entering directory `/home/modax/src/kde/attica-0.1.1-1'
dh_makeshlibs -V 'libattica0 (>=0.1.1)'
dpkg-gensymbols: unknown option `ild'
Usage: dpkg-gensymbols [<option> ...]
Options:
-p<package> generate symbols file for package.
-P<packagebuilddir> temporary build dir instead of debian/tmp.
-e<library> explicitely list libraries to scan.
-v<version> version of the packages (defaults to
version extracted from debian/changelog).
-c<level> compare generated symbols file with the
reference file in the debian directory.
Fails if difference are too important
(level goes from 0 for no check, to 4
for all checks). By default checks at
level 1.
-I<file> force usage of <file> as reference symbols
file instead of the default file.
-O<file> write to <file>, not .../DEBIAN/symbols.
-O write to stdout, not .../DEBIAN/symbols.
-t write in template mode (tags are not
processed and included in output).
-d display debug information during work.
-h, --help show this help message.
--version show the version.
dh_makeshlibs: dpkg-gensymbols -plibattica0 -Idebian/libattica0.symbols -
Pdebian/libattica0 ild returned exit code 2
make[1]: *** [override_dh_makeshlibs] Error 2
make[1]: Leaving directory `/home/modax/src/kde/attica-0.1.1-1'
make: *** [binary] Error 2
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status
2
The problem here is bundling. So the solution would be to disable bundling
while processing DH_INTERNAL_OPTIONS (assuming dh always passes through
command options via DH_INTERNAL_OPTIONS, not only for overrides. Why not?). I
think it's a good compromise that options passed through via dh cannot be
bundled. -O does not help much in this case and just clutters output.
--
Modestas Vainius <mode...@vainius.eu>
Joey Hess
> What in particular do you dislike about my proposal? I guess you want to get
> rid of pipe() and fork() complexity.
I see no benefit to having the command communicate back to dh which
options it liked, only for dh to print them. So it seems no better, only
more complex, than the idea of having the command itself print the options
it likes.
But since dh seqauences can include arbitrary commands, which might look
like debhelper commands but not know how to print their options, either
approach seems flawed on that grounds.
That's not why I viscerally disliked it though. I think my visceral
dislike is just that either approach violates least suprise, which
suggests that dh should not behave entirely unlike make WRT how it runs
commands and prints what it's doing.
> Btw, have you tested your solution to -Bbuild (#541773) problem? I don't see
> how -O solves it. And my test confirms it does not:
I've fixed that.
> The problem here is bundling. So the solution would be to disable bundling
> while processing DH_INTERNAL_OPTIONS (assuming dh always passes through
> command options via DH_INTERNAL_OPTIONS, not only for overrides. Why not?). I
> think it's a good compromise that options passed through via dh cannot be
> bundled. -O does not help much in this case and just clutters output.
Disabling bundling could probably only be accomplished in v8. -O allows
fixing the issue for almost all cases without breaking compatability.
(Aside from the insane case where someone bundles two options together,
and different commands understand different options of the pair, but not
both).
--
see shy jo
Modestas Vainius
On trečiadienis 06 Sausis 2010 20:43:36 Joey Hess wrote:
> Modestas Vainius wrote:
> > What in particular do you dislike about my proposal? I guess you want to
> > get rid of pipe() and fork() complexity.
>
> I see no benefit to having the command communicate back to dh which
> options it liked, only for dh to print them. So it seems no better, only
> more complex, than the idea of having the command itself print the options
> it likes.
It would be great if dh could do this without actually invoking debhelper
command. But with current design it is hardly possible.
> But since dh seqauences can include arbitrary commands, which might look
> like debhelper commands but not know how to print their options, either
> approach seems flawed on that grounds.
Possible but how likely?
if (basename($command) !~ /^dh_/) { print manually... } should be pretty safe.
> That's not why I viscerally disliked it though. I think my visceral
> dislike is just that either approach violates least suprise, which
> suggests that dh should not behave entirely unlike make WRT how it runs
> commands and prints what it's doing.
My main intentions with this bug were:
1) I don't like how dh pollutes output with options most debhelper commands do
not care about. Seeing --dbg-package near dh_install with -O or without it
looks plain confusing in my eyes. Both `dh_install -O--foo` and `dh_install -
O--bar` will do the same thing (because both options are not supported) while
output implies otherwise until you read dh_install(1) manpage (and now
debhelper(7) manpage to figure out what -O means). What is more, it is
absolutely not clear what commands actually process those options. *Sigh* to
my disappointment, it seems that's how it's going to stay so nevermind.
2) The worst part is how debhelper commands are executed via overrides. It is
still black magic because dh automagically sets some undocumented option
DH_INTERNAL_OPTIONS which debhelper commands react to. To make things worse,
output of the real make(1) simply *lies* about what options are actually
passed to debhelper programs. In order to close this bug, those hidden options
will have see a documented daylight somehow.
> > Btw, have you tested your solution to -Bbuild (#541773) problem? I don't
> > see how -O solves it. And my test confirms it does not:
>
> I've fixed that.
It seems so.
> > The problem here is bundling. So the solution would be to disable
> > bundling while processing DH_INTERNAL_OPTIONS (assuming dh always passes
> > through command options via DH_INTERNAL_OPTIONS, not only for overrides.
> > Why not?). I think it's a good compromise that options passed through via
> > dh cannot be bundled. -O does not help much in this case and just
> > clutters output.
>
> Disabling bundling could probably only be accomplished in v8. -O allows
> fixing the issue for almost all cases without breaking compatability.
> (Aside from the insane case where someone bundles two options together,
> and different commands understand different options of the pair, but not
> both).
It would not hurt for options passed to dh(1).
--
Modestas Vainius <mode...@vainius.eu>