GCC bug breaking Perl_sv_catpvfn()?
Marcus Holland-Moritz
I tried building bleadperl with a recent gcc-3.4 snapshot (20031217) using
./Configure -des -Dusedevel -Duse64bitint -Dcc=gcc-3.4 -Doptimize='-g -O3'
Running 'make test', test 67 of op/sprintf failed:
not ok 67 >%hd< >1< >1< >104< # More extensive testing of
It's easily reproducible from the command line:
mhx@r2d2 perl-current-gcc-3.4 $ ./perl -e 'printf"%hd\n",1'
104
What turns the 1 into a 104 is the (short) cast
case 'h': iv = (short)iv; break;
in this part of sv.c (at line 9062):
else {
iv = SvIVx(argsv);
switch (intsize) {
case 'h': iv = (short)iv; break;
default: break;
case 'l': iv = (long)iv; break;
case 'V': break;
#ifdef HAS_QUAD
case 'q': iv = (Quad_t)iv; break;
#endif
}
}
Adding some debug output
else {
iv = SvIVx(argsv);
DEBUG_c(PerlIO_printf(Perl_debug_log, "+++ %"IVdf" (before)\n", iv));
switch (intsize) {
case 'h': iv = (short)iv; break;
default: break;
case 'l': iv = (long)iv; break;
case 'V': break;
#ifdef HAS_QUAD
case 'q': iv = (Quad_t)iv; break;
#endif
}
DEBUG_c(PerlIO_printf(Perl_debug_log, "+++ %"IVdf" (after)\n", iv));
}
gave me:
mhx@r2d2 perl-current-gcc-3.4 $ ./perl -Dc -e 'printf"%hd\n",1'
EXECUTING...
+++ 1 (before)
+++ 104 (after)
104
The following plain C program should be equivalent to the above:
int main(void)
{
long long a = 1;
printf("%lld\n", a);
a = (short) a;
printf("%lld\n", a);
return 0;
}
And indeed, when compiled with 'gcc-3.4 -O3' gives me:
mhx@r2d2 ~/src/c/test $ gcc-3.4 -O3 -o test test.c && ./test
1
2
Not really 104, but at least 2. But interestingly it doesn't matter which of
my various gcc's I'm using. The behaviour is the same for gcc-3.2.3, gcc-3.3.2
and gcc-3.4. It also doesn't matter if I choose -O1, -O2 or -O3. Only if I
choose -O0 or -g, I get the correct, expected result.
This is interesting, because I don't seem to be able to get perl to fail with
e.g. gcc-3.4 -O2 or gcc-3.3.2 -O3.
However, the above seems to me like a bug in GCC when trying to cast long longs
to shorts (or chars, which I've also checked).
BTW, it works when assigning to an int variable before casting to a short:
int main(void)
{
long long a = 1;
int b;
printf("%lld\n", a);
b = a;
a = (short) b;
printf("%lld\n", a);
return 0;
}
If this is a bug in GCC, should there be a workaround for it in perl?
With the attached (very experimental and most probably incomplete) patch,
I'm at least getting "All tests successful" with my gcc-3.4 configuration.
Is this problem real or am I missing something here?
If anyone can confirm that this is a bug, I'll enter a bug report in the
GCC bugzilla.
-- Marcus
Marcus Holland-Moritz
writing emails after debugging from 1am to 5am on x-mas...
I've been playing around with different versions of gcc today,
casting long long's into all kinds of shorter data types. There
definitely seems to be a bug in the GCC optimizer since at least
version 2.95.3. Interestingly, only gcc-3.4 (and only with -O3)
seems to break perl.
I've also entered a bug report:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=13488
-- Marcus
Robert Spier
Marcus Holland-Moritz wrote:
> Of course I meant Perl_sv_vcatpvfn(). That happens when you start
> writing emails after debugging from 1am to 5am on x-mas...
Did you see Santa?
-R
Marcus Holland-Moritz
> > writing emails after debugging from 1am to 5am on x-mas...
>
> Did you see Santa?
Yeah, but who were those guys next to him?
http://www-stud.fh-fulda.de/~fd9099/santa.jpg
-- Marcus
Enache Adrian
> else {
> iv = SvIVx(argsv);
> switch (intsize) {
> case 'h': iv = (short)iv; break;
> default: break;
> case 'l': iv = (long)iv; break;
> case 'V': break;
> #ifdef HAS_QUAD
> case 'q': iv = (Quad_t)iv; break;
> #endif
> }
> }
I'd go for rewriting that as:
else {
IV tiv = SvIVx(argsv);
switch (intsize) {
case 'h': iv = (short)tiv; break;
case 'l': iv = (long)tiv; break;
case 'V':
default: iv = tiv; break;
#ifdef HAS_QUAD
case 'q': iv = (Quad_t)tiv; break;
#endif
}
}
And the same for sv.c:9147 and below.
Better not to take any chances. gcc seems to be very broken in this
respect.
Objections ?
Regards,
Adi
Marcus Holland-Moritz
None. Your approach is more readable than mine.
The attached patch implements the change required to work around the
GCC bug (which has been confirmed in the meantime), and additionally
reorders the members of the preceding switches for consistency and
fixes the quad cast in the unsigned branch from (Quad_t) to (Uquad_t).
-- Marcus
Nicholas Clark
> The attached patch implements the change required to work around the
> GCC bug (which has been confirmed in the meantime), and additionally
> reorders the members of the preceding switches for consistency and
> fixes the quad cast in the unsigned branch from (Quad_t) to (Uquad_t).
Thanks, applied. (21962)
Nicholas Clark
Nicholas Clark
> mhx@r2d2 perl-current-gcc-3.4 $ ./perl -Dc -e 'printf"%hd\n",1'
>
> EXECUTING...
>
> +++ 1 (before)
> +++ 104 (after)
> 104
>
> The following plain C program should be equivalent to the above:
>
> int main(void)
> {
> long long a = 1;
> printf("%lld\n", a);
> a = (short) a;
> printf("%lld\n", a);
> return 0;
> }
>
> And indeed, when compiled with 'gcc-3.4 -O3' gives me:
>
> mhx@r2d2 ~/src/c/test $ gcc-3.4 -O3 -o test test.c && ./test
> 1
> 2
>
> Not really 104, but at least 2. But interestingly it doesn't matter which of
> my various gcc's I'm using. The behaviour is the same for gcc-3.2.3, gcc-3.3.2
> and gcc-3.4. It also doesn't matter if I choose -O1, -O2 or -O3. Only if I
> choose -O0 or -g, I get the correct, expected result.
104 == 'h', and 'h' is in a(nother) local variable in sv.c's switch
statement.
> This is interesting, because I don't seem to be able to get perl to fail with
> e.g. gcc-3.4 -O2 or gcc-3.3.2 -O3.
Curious. I did some experimenting a few days ago (which I failed to report on)
On a SuSE 8.2 system the supplied gcc (something like
"cc version 3.3 20030226 (prerelease) (SuSE Linux)") fails this test on -O3
Build with -O2 and the test passes. The gcc documentation says that -O3 is
exactly -O2 -frename-registers -finline-functions, so I try each -f option
in turn to see which causes it. Neither does. I try both with -O2. Still no
bug tickled. I compare the assembly output of
"-O2 -frename-registers -finline-functions" and "-O3". It differs - the
latter is about 33% larger. I then try on a real gcc 3.3 I compiled from
source. Tests pass, and "-O2 -frename-registers -finline-functions" and "-O3"
are identical. I put it all down to a prerelease gcc, curse vendors who
ship pre-releases, and decide that as it's not even a bug in gcc then I
don't have the time of day for it.
Curious that it shows up in gcc 3.4
> If this is a bug in GCC, should there be a workaround for it in perl?
As I'd not realised that it's a bug in stable gcc releases, yes, I think
that it should (hence thanks applied in the other message)
Nicholas Clark
Marcus Holland-Moritz
> > e.g. gcc-3.4 -O2 or gcc-3.3.2 -O3.
>
> Curious. I did some experimenting a few days ago (which I failed to report on)
> On a SuSE 8.2 system the supplied gcc (something like
> "cc version 3.3 20030226 (prerelease) (SuSE Linux)") fails this test on -O3
> Build with -O2 and the test passes. The gcc documentation says that -O3 is
> exactly -O2 -frename-registers -finline-functions, so I try each -f option
> in turn to see which causes it. Neither does. I try both with -O2. Still no
> bug tickled. I compare the assembly output of
> "-O2 -frename-registers -finline-functions" and "-O3". It differs - the
> latter is about 33% larger. I then try on a real gcc 3.3 I compiled from
> source. Tests pass, and "-O2 -frename-registers -finline-functions" and "-O3"
> are identical. I put it all down to a prerelease gcc, curse vendors who
> ship pre-releases, and decide that as it's not even a bug in gcc then I
> don't have the time of day for it.
I've been using SuSE (7.2) until about one year ago. I didn't get a lot
to compile from source without problems, let alone compiling a kernel...
> Curious that it shows up in gcc 3.4
From the gcc-3.4 manpage:
-O3 Optimize yet more. -O3 turns on all optimizations specified by -O2 and
also turns on the -finline-functions, -fweb and -frename-registers options.
^^^^^
[...]
-fweb
Constructs webs as commonly used for register allocation purposes and
assign each web individual pseudo register. This allows our register
allocation pass to operate on pseudos directly, but also strengthens
several other optimization passes, such as CSE, loop optimizer and trivial
dead code remover. It can, however, make debugging impossible, since
variables will no longer stay in a ``home register''.
The -fweb is new with gcc-3.4. The part "but also strengthens several other
optimization passes" is probably what makes the bug only appear with 3.4.
The bug is in gcc for ages (at least since 2.91.6), but it probably wasn't
triggered yet because the code was too complex for the buggy optimization
to actually be applied.
To verify that -fweb is really the culprit, I just built the unpatched perl
with gcc-3.4 and '-O2 -finline-functions -frename-registers'. No failures.
Adding -fweb makes it fail again. So that appears to be the reason why it
works fine with gcc < 3.4.
-- Marcus
Nicholas Clark
> > > This is interesting, because I don't seem to be able to get perl to fail with
> > > e.g. gcc-3.4 -O2 or gcc-3.3.2 -O3.
> >
> > Curious. I did some experimenting a few days ago (which I failed to report on)
> > On a SuSE 8.2 system the supplied gcc (something like
> > "cc version 3.3 20030226 (prerelease) (SuSE Linux)") fails this test on -O3
> > Build with -O2 and the test passes. The gcc documentation says that -O3 is
> > exactly -O2 -frename-registers -finline-functions, so I try each -f option
> > in turn to see which causes it. Neither does. I try both with -O2. Still no
> > bug tickled. I compare the assembly output of
> > "-O2 -frename-registers -finline-functions" and "-O3". It differs - the
> > latter is about 33% larger. I then try on a real gcc 3.3 I compiled from
> > source. Tests pass, and "-O2 -frename-registers -finline-functions" and "-O3"
> > are identical. I put it all down to a prerelease gcc, curse vendors who
> > ship pre-releases, and decide that as it's not even a bug in gcc then I
> > don't have the time of day for it.
>
> I've been using SuSE (7.2) until about one year ago. I didn't get a lot
> to compile from source without problems, let alone compiling a kernel...
Oh. We've been using SuSE at work, and, generally found we like it a lot.
> > Curious that it shows up in gcc 3.4
>
> >From the gcc-3.4 manpage:
>
> -O3 Optimize yet more. -O3 turns on all optimizations specified by -O2 and
> also turns on the -finline-functions, -fweb and -frename-registers options.
>
> The -fweb is new with gcc-3.4. The part "but also strengthens several other
> optimization passes" is probably what makes the bug only appear with 3.4.
> The bug is in gcc for ages (at least since 2.91.6), but it probably wasn't
> triggered yet because the code was too complex for the buggy optimization
> to actually be applied.
Aha thanks.
I'd been wondering how much -O3 really helps perl, given that (for different
reasons) it seems to have been the cause of both your version of this bug,
and the SuSE 8.2 prerelease gcc's version of this bug.
I was wondering whether we'd be better dropping maint back to -O2, to avoid
us being the people who get to diagnose gcc's optimizer's bugs.
[Give someone else that joy. Mozilla compiles at -O3 doesn't it? :-)]
Nicholas Clark
Marcus Holland-Moritz
> > > > This is interesting, because I don't seem to be able to get perl to fail with
> > > > e.g. gcc-3.4 -O2 or gcc-3.3.2 -O3.
> > >
> > > Curious. I did some experimenting a few days ago (which I failed to report on)
> > > On a SuSE 8.2 system the supplied gcc (something like
> > > "cc version 3.3 20030226 (prerelease) (SuSE Linux)") fails this test on -O3
> > > Build with -O2 and the test passes. The gcc documentation says that -O3 is
> > > exactly -O2 -frename-registers -finline-functions, so I try each -f option
> > > in turn to see which causes it. Neither does. I try both with -O2. Still no
> > > bug tickled. I compare the assembly output of
> > > "-O2 -frename-registers -finline-functions" and "-O3". It differs - the
> > > latter is about 33% larger. I then try on a real gcc 3.3 I compiled from
> > > source. Tests pass, and "-O2 -frename-registers -finline-functions" and "-O3"
> > > are identical. I put it all down to a prerelease gcc, curse vendors who
> > > ship pre-releases, and decide that as it's not even a bug in gcc then I
> > > don't have the time of day for it.
> >
> > I've been using SuSE (7.2) until about one year ago. I didn't get a lot
> > to compile from source without problems, let alone compiling a kernel...
>
> Oh. We've been using SuSE at work, and, generally found we like it a lot.
I didn't say it's bad. It's still my favourite distro for systems that I
just want to "install and go". But tell anyone you're having trouble with
their app not compiling with a prerelease version of gcc...
> > > Curious that it shows up in gcc 3.4
> >
> > >From the gcc-3.4 manpage:
> >
> > -O3 Optimize yet more. -O3 turns on all optimizations specified by -O2 and
> > also turns on the -finline-functions, -fweb and -frename-registers options.
>
> >
> > The -fweb is new with gcc-3.4. The part "but also strengthens several other
> > optimization passes" is probably what makes the bug only appear with 3.4.
> > The bug is in gcc for ages (at least since 2.91.6), but it probably wasn't
> > triggered yet because the code was too complex for the buggy optimization
> > to actually be applied.
>
> Aha thanks.
>
> I'd been wondering how much -O3 really helps perl, given that (for different
> reasons) it seems to have been the cause of both your version of this bug,
> and the SuSE 8.2 prerelease gcc's version of this bug.
>
> I was wondering whether we'd be better dropping maint back to -O2, to avoid
> us being the people who get to diagnose gcc's optimizer's bugs.
> [Give someone else that joy. Mozilla compiles at -O3 doesn't it? :-)]
I've built my whole system with -O3. I haven't experienced a lot of problems.
Not sure if it helps a lot; except for finding gcc bugs, of course. ;-)
-- Marcus
Chip Salzenberg
> IV tiv = SvIVx(argsv);
> switch (intsize) {
> case 'h': iv = (short)tiv; break;
> case 'l': iv = (long)tiv; break;
> case 'V':
> default: iv = tiv; break;
> #ifdef HAS_QUAD
> case 'q': iv = (Quad_t)tiv; break;
> #endif
Something odd is going on in the 'q' section.
Either IV is not allowed to be smaller than Quad_t when HAS_QUAD is
defined; or, it is.
If IV is *not* allowed to be smaller than Quad_t, then the cast is a
no-op and is at best misleading.
If IV *is* allowed to be smaller than Quad_t, then, Shirley, we don't
want to push that value through a conversion to IV on the way to
printing it. On platforms where IV has fewer significant digits than
NV, you'd want (Quad_t)SvNV(argsv) instead. And the decision whether
to generate the Quad_t from the SvIV() or SvNV() should probably be
encapsulated behind SvQV(argsv).
Anybody know what's going on there?
--
Chip Salzenberg - a.k.a. - <ch...@pobox.com>
"I wanted to play hopscotch with the impenetrable mystery of existence,
but he stepped in a wormhole and had to go in early." // MST3K
Marcus Holland-Moritz
> > IV tiv = SvIVx(argsv);
> > switch (intsize) {
> > case 'h': iv = (short)tiv; break;
> > case 'l': iv = (long)tiv; break;
> > case 'V':
> > default: iv = tiv; break;
> > #ifdef HAS_QUAD
> > case 'q': iv = (Quad_t)tiv; break;
> > #endif
>
> Something odd is going on in the 'q' section.
>
> Either IV is not allowed to be smaller than Quad_t when HAS_QUAD is
> defined; or, it is.
There's some code in perl.h taking care of that, which is essentially:
#if defined(USE_64_BIT_INT) && defined(HAS_QUAD)
[...]
#else
[...]
# if IVSIZE == 8
# define IV_IS_QUAD
# define UV_IS_QUAD
# ifndef HAS_QUAD
# define HAS_QUAD
# endif
# else
# undef IV_IS_QUAD
# undef UV_IS_QUAD
# undef HAS_QUAD
# endif
#endif
So if IV is not large enough to hold a quad, HAS_QUAD will be undef'd
regardless of config.h defining it or not.
> If IV is *not* allowed to be smaller than Quad_t, then the cast is a
> no-op and is at best misleading.
Yes, it's a no-op; unless you plan for IV/UV becoming larger than
Quad_t/Uquad_t. But then you'd also have to change the code in perl.h
and probably add:
# elif IVSIZE > 8
Most of the time the cast to long for the 'l' case is also a no-op.
-- Marcus
Dave Mitchell
> According to Enache Adrian:
> > IV tiv = SvIVx(argsv);
> > switch (intsize) {
> > case 'h': iv = (short)tiv; break;
> > case 'l': iv = (long)tiv; break;
> > case 'V':
> > default: iv = tiv; break;
> > #ifdef HAS_QUAD
> > case 'q': iv = (Quad_t)tiv; break;
> > #endif
>
> Something odd is going on in the 'q' section.
>
> Either IV is not allowed to be smaller than Quad_t when HAS_QUAD is
> defined; or, it is.
>
> If IV is *not* allowed to be smaller than Quad_t, then the cast is a
> no-op and is at best misleading.
Judging from this bit in Configure, it looks likes IV == Quad_t when
HAS_QUAD is defined:
case "$use64bitint:$d_quad:$quadtype" in
define:define:?*)
ivtype="$quadtype"
uvtype="$uquadtype"
ivsize=8
uvsize=8
;;
*) ivtype="long"
uvtype="unsigned long"
ivsize=$longsize
uvsize=$longsize
;;
esac
Conceivably at some future point there could be a type > Quad_t, in which
case the cast would then cease to be a noop???
Dave.
--
Nothing ventured, nothing lost.
Nick Ing-Simmons
>> > Curious. I did some experimenting a few days ago (which I failed to report on)
>> > On a SuSE 8.2 system the supplied gcc (something like
>> > "cc version 3.3 20030226 (prerelease) (SuSE Linux)") fails this test on -O3
>> > ship pre-releases, and decide that as it's not even a bug in gcc then I
>> > don't have the time of day for it.
>>
>> I've been using SuSE (7.2) until about one year ago. I didn't get a lot
>> to compile from source without problems, let alone compiling a kernel...
>
>Oh. We've been using SuSE at work, and, generally found we like it a lot.
8.2 was unusual. SuSE 9.0 seems fine.
gcc here is:
cc version 3.3.1 (SuSE Linux)
I also (after the 8.2 mess) have a stock gcc-3.3.1 seems to behave
exactly like the SuSE9 one.
>
>I'd been wondering how much -O3 really helps perl, given that (for different
>reasons) it seems to have been the cause of both your version of this bug,
>and the SuSE 8.2 prerelease gcc's version of this bug.
As -O3 is mainly about inlining I have doubts it really helps much,
the way perl sources are there are not many functions small enough to
inline that are known about at point of call.
Nicholas Clark
> Nicholas Clark <ni...@ccl4.org> writes:
> >I'd been wondering how much -O3 really helps perl, given that (for different
> >reasons) it seems to have been the cause of both your version of this bug,
> >and the SuSE 8.2 prerelease gcc's version of this bug.
>
> As -O3 is mainly about inlining I have doubts it really helps much,
> the way perl sources are there are not many functions small enough to
> inline that are known about at point of call.
I was wondering whether to experiment with how the different options
beyond -O2 really behave, and if it turns out that -finline-functions
doesn't help, whether we should change Configure (or the hints) to
(say) check whether -frename-functions and -fweb work (or whatever the new
one is called), and only turn those on.
Possibly only turn the optimiser above -O2 for -Dusedevel
Nicholas Clark
Rafael Garcia-Suarez
>
> I think I'll do some benchmarking. I noticed that perlbench is
> rather dated (1999). Are there any new tests for perlbench that
> have emerged over that last five years?
None that I know of.
Quick idea : you could try to run perlbench with various combinations
of locales and PERLIO variables (a bit like for the smokes).
Marcus Holland-Moritz
I think I'll do some benchmarking. I noticed that perlbench is
rather dated (1999). Are there any new tests for perlbench that
have emerged over that last five years?
-- Marcus
Marcus Holland-Moritz
Yes, that's an option. However, building and testing 22 different
versions of perl took long enough for today. ;-)
So, here are some benchmarking results. I used the latest gcc-3.4
snapshot (20031224) and bleadperl@22021 on a P3/1GHz.
I found the following gcc optimization options that were not included
in -O2:
-finline-functions
-frename-registers
-ftracer
-funit-at-a-time
-funroll-loops
-funswitch-loops
-fweb
In a first run, I tested each of these options alone (complete output
is attached):
A) perl-5.009 optimize = -O2
B) perl-5.009 optimize = -O2 -finline-functions
C) perl-5.009 optimize = -O2 -frename-registers
D) perl-5.009 optimize = -O2 -ftracer
E) perl-5.009 optimize = -O2 -funit-at-a-time
F) perl-5.009 optimize = -O2 -funroll-loops
G) perl-5.009 optimize = -O2 -funswitch-loops
H) perl-5.009 optimize = -O2 -fweb
A B C D E F G H
---- ---- ---- ---- ---- ---- ---- ----
MIN 100 88 93 91 98 72 96 96
MAX 100 108 106 108 106 107 106 125
AVG 100.0 100.0 99.7 100.4 100.2 99.1 100.3 102.1
It seems that none of the optimization options is able to deliver a
performance gain in all tests. The -fweb (new with gcc-3.4) seems to
be the most promising. So I tested a set of "selected" combinations
of different optimizations in the second run:
A) perl-5.009 optimize = -O2
B) perl-5.009 optimize = -O2 -frename-registers -funswitch-loops
C) perl-5.009 optimize = -O2 -finline-functions -frename-registers
D) perl-5.009 optimize = -O2 -finline-functions -frename-registers -ftracer -funit-at-a-time -funroll-loops -funswitch-loops
E) perl-5.009 optimize = -O2 -finline-functions -frename-registers -funroll-loops
F) perl-5.009 optimize = -O2 -finline-functions -frename-registers -funswitch-loops
G) perl-5.009 optimize = -O2 -finline-functions -frename-registers -funswitch-loops -funroll-loops
H) perl-5.009 optimize = -O2 -finline-functions -funswitch-loops
I) perl-5.009 optimize = -O2 -fweb -finline-functions -frename-registers
J) perl-5.009 optimize
= -O2 -fweb -finline-functions -frename-registers -ftracer -funit-at-a-time -funroll-loops -funswitch-loops
K) perl-5.009 optimize = -O2 -fweb -finline-functions -frename-registers -funroll-loops
L) perl-5.009 optimize = -O2 -fweb -finline-functions -frename-registers -funswitch-loops
M) perl-5.009 optimize = -O2 -fweb -finline-functions -frename-registers -funswitch-loops -funroll-loops
N) perl-5.009 optimize = -O2 -fweb -finline-functions -funswitch-loops
O) perl-5.009 optimize = -O2 -fweb -frename-registers -funswitch-loops
A B C D E F G H I J K L M N O
---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
MIN 100 95 88 43 78 87 78 87 87 85 75 88 75 87 95
MAX 100 105 109 107 127 106 126 106 130 127 127 133 125 132 122
AVG 100.0 100.0 99.0 97.5 99.2 98.9 99.7 99.0 101.7 99.6 100.9 101.5 100.9 100.9 101.3
Not very much better. The best results are [I], which is actually the same
combination of flags as with -O3. Also quite good is [L], which is just -O3
and -funswitch-loops.
At least from the perlbench test suite (slightly extended by a couple of
tests I just hacked in) I cannot see much benefit in using -O3. Given that
most users won't have gcc-3.4, the average performance of -O3 [C] is even
worse than the performance of -O2 [A].
So, as Nick already suggested, it'll probably be a good idea to go back
from -O3 to -O2, not only because of possible compiler issues.
-- Marcus