This Week on perl5-porters (28 November-4 December 2005)
David Landgren
A rather hectic week on p5p, when it was revealed that signed/unsigned
comparisons and unchecked format strings to "printf" and "sprintf"
could cause serious problems in poorly written applications.
Format strings in "s?printf"
It turned out that a nasty "sprintf" format string could cause havoc
in the "webmin" application suite (a set of web scripts geared towards
systems administration). Not the kind of place you want havoc to
occur.
Rafael noted that this could lead to a buffer overrun in the
interpreter, by taking advantage of a signed/unsigned conversion bug
in "printf" (which is pretty much all hand-rolled and not the "printf"
of the underlying C standard library), and that the next major release
will apply taint checks to format strings. Andy questioned whether it
was really possible to create a buffer overrun, and Gisle Aas
responded with a tiny one-liner:
$ perl -e 'printf "%4294967295d"'
Segmentation fault (core dumped)
In a subsequent thread, Andy was rather dismayed to learn that
pretty-printing a variable through a %d format string makes it lose
its taintedness. In later developments, Jan Dubois pointed out that
Python does not have this flaw:
>>> print "%4294967295d" % 1
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ValueError: width too big
Andy subsequently decided not to post a rebuttal to the News.com
article, since, to paraphrase Nathan Torkington: "everybody fucked
up", and the best that can be done is to get the fix into 5.8.8, and
get 5.8.8 out the door. Nicholas Clark replied that it would take a
couple of weeks, which would take us right up to Christmas time. Not
the kind of time you want a Perl upgrade to occur.
Philippe M. Chiasson cooked up a patch that produced the following
behaviour:
$ perl -e 'printf("%04294967294d",1)'
panic: memory wrap at -e line 1.
That patch was applied by Rafael, but Gisle still managed to punch a
hole through it with "sprintf "%#.4294967295b"". But made up for it by
fixing it. Dave Mitchell supplied a patch to fix the signed/unsigned
mismatch in the "printf" code. Hugo van der Sanden had a minor quibble
with the change in behaviour, and Nicholas provided a clearer change.
Gisle thought about patching the code and documentation for
"Sys::Syslog", to prevent the possibility of using %n. Ronald Kimball
improved the patch with a better regular expression to strip out %n.
(Summariser's note: %n, in case you weren't aware (I had to go and
look it up in the documentation), takes the current number of
characters emitted so far by the format string, and stores that count
in the next variable appearing in the argument list; problems occur
when there is no variable to take the result).
Gisle then came back later with a patch for "sprintf", to prevent
constant folding from taking place. Hugo appreciated the patch, and
suggested a long-term plan. (Constant folding in this context meaning
something like):
perl -MO=Deparse -e '$a = sprintf "%g", 2/3'
$a = '0.666667';
Which stops bad things happening when %g is replaced by %99g (where 99
is a very large number). But in general, constant folding is a Good
Thing, and a concensus seems to be forming around the idea that it
should be possible to back out of a constant folding attempt during
compilation without killing the compile, and defer the resolution
until run-time.
Andy started to look at GCC's warnings of signed/unsigned comparisons,
and picked a bit of low-hanging fruit in pp_pack.c. He also heard back
from Jack Louis, who reported the the initial integer overflow
problem. Dave Mitchell noted that one of them had already been fixed
in "blead". Andy forwarded another message from Jack showing how the
exploit could be brought to bear on Webmin.
Joshua ben Jore pointed to a couple of threads he wrote on Perlmonks,
showing the results of the code he wrote to look for uses of "printf"
and "sprintf" with non-constant format parameters.
Executive summary: the problems will be fixed in 5.8.8, and a series
of patches will be made available for all the 5.8 releases.
The article on News.com
http://news.com.com/2100-1002_3-5975954.html
Andy Lester's call for input
http://xrl.us/i395
sprintf and tainting
http://xrl.us/i396
Andy's first approximation to a PR response
http://xrl.us/i397
Andy declines to respond
http://xrl.us/i398
Philippe's patch
http://xrl.us/i399
Dave's patch
http://xrl.us/i4aa
Gisle's patch
http://xrl.us/i4ab
Disabling constant folding of sprintf
http://xrl.us/i4ac
Andy's patch of pp_pack.c
http://xrl.us/i4ad
Word back from the original finder of the integer overflow
http://xrl.us/i4ae
Details of a possible exploit
http://xrl.us/i4af
The message sent to bugtraq
http://xrl.us/i4ag
Joshua's findings
http://xrl.us/i4ah
Debugging lib/archive/tar.t/02_methods.t
John E. Malmberg was having difficulty tracking down why this test
file was failing on VMS, and had to resort to inserting "print"
statements to trace what was happening. Rafael Garcia-Suarez explained
that it was hard to find, because in fact it is created in 00_setup.t.
Ronald J Kimball thought it rather dubious that two different test
files cannot be run independently of each other (this precludes,
amongst other things, being able to run tests in a massively parallel
manner).
Looking in the wrong place
http://xrl.us/i4ai
"my $var = undef" fails to set $var when re-run
Erland Sommarskog posted bug #37776 showing that a declaration and
assignment of a variable to "undef" doesn't work when the assignment
is run subsequently. It turns out that it was due to an optimisation
that was, well, wrong. This behaviour, according to Robin Houston, is
a side-effect of change #22520. Rafael fixed it with change #26226.
Can't go there again
http://xrl.us/i4aj
Cwd.pm scan of $ENV{PATH}
Nick Ing-Simmons ran into a problem with "Cwd"'s use of "grep" on the
list of directories in $ENV{PATH}. This usually works well, but if
your PATH happens to contain automounted directories that are not
there, bad things happen. Indeed, Nick's "Cwd" was taking *minutes* to
load. This can be construed as an abuse of "grep", because only the
first result is needed, but "grep", by design, will always scan the
entire list it is given. Nick proposed a number of ways out of the
problem.
Graham Barr suggested "first" from "List::Util". Ken Williams said
that "Cwd" contains lots of ancient voodoo, and because it is so low
on the CPAN dependency graphs, that a "foreach" is probably the only
wise path to take. Some patches were put forward.
Using "I32" for arrays on 64 platforms
Jan Dubois noticed that the internal structures for arrays use 32 bits
for index computations, thus limiting arrays on 64 bit architectures
to *only* 2**32 elements. An array that size would consume a
non-trivial amount of memory, but Jan felt that it should be fixed in
blead, even if it wouldn't start being hit by applications for some
time yet. Or otherwise, paraphrasing Bill Gates, that "2**32 array
elements will be big enough for everyone." The concensus seems to be
to use an "IV" instead.
Passing function parameters in registers
Last week in his quest to const, Andy Lester stumbled across some
redundant code that he was able to chop out. In response Yitzchak
Scott-Thoennes asked whether Andy was considering investigating the
"regparm" attribute of the gcc compiler, which indicates that the
parameters of the function are to be passed in registers, for a nice
speed boost. But implementing this would add considerable complexity
to the codebase.
The regparm attribute
http://xrl.us/i4an
Declaring attributes in gcc
http://gcc.gnu.org/onlinedocs/gcc-4.0.2/gcc/Function-Attributes.html
POD Encoding
Alberto Simões was writing POD assuming Latin-1, but noted that it
gets mangled on a system that uses UTF-8 by default, and wondered what
the correct fix was. Russ Allbery replied that the correct solution
was to use the "=encoding" directive. POD translators that are based
on "Pod::Simple" get this for free. Other translators including
"pod2man" and "pod2txt" may not.
Worse, "pod2man" has difficulty in dealing with non-ASCII characters
because of limitations in "nroff" implementations. Russ is hoping to
get around to adding a switch to "pod2man" to tell it to "assume
"groff"", which does not how to generate UTF-8 output.
Tels noted that the POD in "blead" does not contain any "=encoding"
directives, and that it probably should. Then Sadahiro Tomoyuki
started talking about EBCDIC and my head exploded.
The Archive of Perl Changes (APC)
Philippe M. Chiasson wrote to say that the Archive of Perl Changes is
now running on more powerful hardware (a shade less than ten times
more powerful, if you lend any credence to BogoMIPS).
The main change is that <rsync://ftp.linux.activestate.com/> became
<rsync://public.activestate.com/>. Abe Timmerman experienced a bit of
transient grief with his "Test::Smoke" kit, but everything was sorted
out in the end.
New Modules
John Peacock released "version-0.50". Much of the change involves
improvements to the documentation.
Andreas König released "CPAN-1.80". Lots of new goodies, including
support for "sudo" and new commands "recent" and "perldoc". Now runs
(again) under 5.005_04.
Perl5 Bug Summary
1512 as of Monday the 5th. All the tickets that were opened last week
were commented on, which made Robert Spier happy.
In Brief
"podlators" 2.00 released by Russ Allbery. The underlying POD parsing
is now handled by "Pod::Simple", rather than "Pod::Parser". Stever
Peters planned to add it the core. Tels was very happy, and showed how
this would let him write custom POD paragraphs.
Ulrich Windl filed bug report #37781 show how to make the debugger
crash. Richard Foley replied with a couple of message IDs showing what
the probable fix would be, and otherwise how to work around it.
Torsten Förtsch queried a strange split feature, wondering why the
trailing empty elements of the "split" are discarded. H.Merijn Brand
explained that it was operating according to spec, and showed a
snippet that let Torsten achieve the desired result.
Redundant "SvUTF8_on()" calls were removed from the codebase in a
couple of places, thanks to careful observation from Gisle.
Tk compatibility was reported broken on "blead" by Gisle on the 23rd
of November. Andreas König traced the fault back to change #26110. The
fix had already been unwound in "maint", and Rafael unwound it in
"blead". But the bug that the change tried to fix in the first place,
as Nicholas reminded us, is still there.
"arenas by SV-type" work continued. Jim Cromie smoked the latest
"blead" and more or less came up with a clean bill of health. There
were a couple of compiler squawks, and one test failure that Jim had
difficulty in deciding whether it was because "blead" was in a state
of flux, or whether it was because of his patch since "monkeying with
arenas affects everything."
and a patch to unify "PL_body_arenaroots[]" to a single variable:
Sadahiro Tomoyuki improved his XS-assisted SWASHGET patch.
About this summary
This summary was written by David Landgren. Adriano and I are moving
to a Monday night publishing schedule, rather than Sunday night, to
give us a bit more time.
One thing I keep failing to mention in these summaries is the tireless
effort that Steve Peters puts into delving into the bug queue and
closing out fixed bugs and reviving the lost, the forgotten and the
ignored. The number of open bugs for Perl5 has been pretty stable over
the last few months (and no doubt longer, but I never paid close
attention before), and this is in no small part due to Steve's
diligence.
Unfortunately, as most of this activity is just one-shot messages to
the list, it's nearly impossible to summarise, so casual readers of
this summary have no idea of the work Steve does. So, thank-you Steve.
Information concerning bugs referenced in this summary (as #nnnnn) may
be viewed at http://rt.perl.org/rt3/Ticket/Display.html?id=nnnnn
Information concerning patches to maint or blead referenced in this
summary (as #nnnnn) may be viewed at
http://public.activestate.com/cgi-bin/perlbrowse?patch=nnnnn
Weekly summaries are published on http://use.perl.org/ and posted on a
mailing list, (subscription: perl5-summa...@perl.org). The
archive is at http://dev.perl.org/perl5/list-summaries/. Corrections
and comments are welcome.
If you found this summary useful or enjoyable, please consider
contributing to the Perl Foundation to help support the development of
Perl.
--
"It's overkill of course, but you can never have too much overkill."
Ronald J Kimball
> Gisle thought about patching the code and documentation for
> "Sys::Syslog", to prevent the possibility of using %n. Ronald Kimball
> improved the patch with a better regular expression to strip out %n.
I had actually commented on a patch from Gisle that had to do with %m, not
%n. Sys::Syslog::syslog() replaces %m with the contents of $! before
calling sprintf. (I don't know whether or not my suggested change was
applied.)
Ronald
David Landgren
Oh duh, well that makes more sense. I was struggling to figure out the
relation between %n and $!, saw none, and assumed that it was some sort
of esoteric crufto-feature.
My eyes grow dim.
Thanks,
David