[perl #128843] printf %a mishandles subnormals

[Zefram]

# New Ticket Created by Zefram
# Please include the string: [perl #128843]
# in the subject line of all future correspondence about this issue.
# <URL: https://rt.perl.org/Ticket/Display.html?id=128843 >



This is a bug report for perl from zef...@fysh.org,
generated with the help of perlbug 1.40 running under perl 5.24.0.


-----------------------------------------------------------------
[Please describe your issue here]

$ perl -lwe 'printf "%a\n", 3e-320'
0x1.00000000017b8p-1062

This output is numerically incorrect. This happens for any subnormal
floating-point value. The output is a mixture of two ways of correctly
describing the value: it is 0x0.00000000017b8p-1022 (displaying
the physical structure of a subnormal) or 0x1.7b8p-1062 (consistent
scientific style).

$ perl -MData::Float=float_hex -lwe 'print float_hex(3e-320, {subnormal_strategy=>$_}) for qw(SUBNORMAL NORMAL)'
+0x0.00000000017b8p-1022
+0x1.7b80000000000p-1062

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
category=core
severity=low
---
Site configuration information for perl 5.24.0:

Configured by zefram at Mon May 9 19:42:55 BST 2016.

Summary of my perl5 (revision 5 version 24 subversion 0) configuration:

Platform:
osname=linux, osvers=3.16.0-4-amd64, archname=x86_64-linux-thread-multi
uname='linux barba.rous.org 3.16.0-4-amd64 #1 smp debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 gnulinux '
config_args='-des -Dprefix=/home/zefram/usr/perl/perl_install/perl-5.24.0-i64-f52 -Duselargefiles -Dusethreads -Uafs -Ud_csh -Uusesfio -Uusenm -Duseshrplib -Dusedevel -Uversiononly -Ui_db'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2',
optimize='-O2',
cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
ccversion='', gccversion='4.9.2', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
libs=-lpthread -lnsl -ldb -ldl -lm -lcrypt -lutil -lc
perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
libc=libc-2.19.so, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version='2.19'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/home/zefram/usr/perl/perl_install/perl-5.24.0-i64-f52/lib/5.24.0/x86_64-linux-thread-multi/CORE'
cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'


---
@INC for perl 5.24.0:
/home/zefram/usr/perl/perl_install/perl-5.24.0-i64-f52/lib/site_perl/5.24.0/x86_64-linux-thread-multi
/home/zefram/usr/perl/perl_install/perl-5.24.0-i64-f52/lib/site_perl/5.24.0
/home/zefram/usr/perl/perl_install/perl-5.24.0-i64-f52/lib/5.24.0/x86_64-linux-thread-multi
/home/zefram/usr/perl/perl_install/perl-5.24.0-i64-f52/lib/5.24.0
.

---
Environment for perl 5.24.0:
HOME=/home/zefram
LANG (unset)
LANGUAGE (unset)
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=/home/zefram/usr/perl/perl_install/perl-5.24.0-i64-f52/bin:/home/zefram/usr/perl/util:/home/zefram/pub/x86_64-unknown-linux-gnu/bin:/home/zefram/pub/common/bin:/usr/bin:/bin:/usr/local/bin:/usr/games
PERL_BADLANG (unset)
SHELL=/usr/bin/zsh

[Tony Cook via RT]

On Thu Aug 04 13:51:17 2016, zef...@fysh.org wrote:

> $ perl -lwe 'printf "%a\n", 3e-320'
> 0x1.00000000017b8p-1062
>
> This output is numerically incorrect. This happens for any subnormal
> floating-point value. The output is a mixture of two ways of
> correctly
> describing the value: it is 0x0.00000000017b8p-1022 (displaying
> the physical structure of a subnormal) or 0x1.7b8p-1062 (consistent
> scientific style).

The handling of the implicit one bit is incorrect too.

> $ perl -MData::Float=float_hex -lwe 'print float_hex(3e-320,
> {subnormal_strategy=>$_}) for qw(SUBNORMAL NORMAL)'
> +0x0.00000000017b8p-1022
> +0x1.7b80000000000p-1062

It looks like subnormals simply weren't implemented:

/* XXX Inf/NaN/denormal handling in the HEXTRACT_IMPLICIT_BIT,
* and elsewhere. */

Tony

---
via perlbug: queue: perl5 status: new
https://rt.perl.org/Ticket/Display.html?id=128843

[Jarkko Hietaniemi via RT]

http://perl5.git.perl.org/perl.git/commit/b6d9b423fab1963346eb79e83b356114396b1f2e

---
via perlbug: queue: perl5 status: open
https://rt.perl.org/Ticket/Display.html?id=128843

[Zefram]

Jarkko Hietaniemi via RT wrote:
>http://perl5.git.perl.org/perl.git/commit/b6d9b423fab1963346eb79e83b356114396b1f2e

When I first reviewed this, before Jarkko merged it to blead, I thought
it fixed the problem, and declared it good. Actually it's not correct:
it's changed the problem into a different form. Sorry for not spotting
it first time.

With IEEE double:

$ perl -lwe 'printf "%a\n", 0x1.fffffffffffffp-1022'
0x1.fffffffffffffp-1022
$ perl -lwe 'printf "%a\n", 0x0.fffffffffffffp-1022'
0x1.fffffffffffffp-1023
$ perl -lwe 'printf "%a\n", 0x0.7ffffffffffffp-1022'
0x1.7ffffffffffffp-1024
$ perl -lwe 'printf "%a\n", 0x0.3ffffffffffffp-1022'
0x1.3ffffffffffffp-1025
$ perl -lwe 'printf "%a\n", 0x0.1ffffffffffffp-1022'
0x1.1ffffffffffffp-1026
$ perl -lwe 'printf "%a\n", 0x0.0ffffffffffffp-1022'
0x1.fffffffffffep-1027

The first one here (which is normalised) is OK, but the subnormals
have gone awry. The second should be "0x1.ffffffffffffep-1023",
the third should be "0x1.ffffffffffffcp-1024", the fourth
should be "0x1.ffffffffffff8p-1025", and the fifth should be
"0x1.ffffffffffffp-1026". The sixth output, also subnormal, is correct.
Smaller subnormals of the same pattern are all correct, down to the
smallest positive value.

All of the incorrect ones have the pattern of showing a normalised-style
integer part and exponent but the fractional part from subnormal style.
That's precisely what was wrong originally for all subnormals.

As far as I can see, the problem arises with any magnitude in the range
[2**-1026, 2**-1022), and not for anything outside that range. I expect
that the reason for exactly four binary orders of magnitude being affected
is down to that being a single order of magnitude of the output radix.

With x86 80-bit long double, the behaviour is stranger:

$ perl -lwe 'printf "%a\n", 0x1.fffffffffffffffep0 * 2**-16382'
Hexadecimal float: mantissa overflow at -e line 1.
0xf.fffffffffffffffp-16385
$ perl -lwe 'printf "%a\n", 0x0.fffffffffffffffep0 * 2**-16382'
0x7.fffffffffffffffp-16386
$ perl -lwe 'printf "%a\n", 0x0.7ffffffffffffffep0 * 2**-16382'
0x3.fffffffffffffffp-16387
$ perl -lwe 'printf "%a\n", 0x0.3ffffffffffffffep0 * 2**-16382'
0x1.fffffffffffffffp-16388
$ perl -lwe 'printf "%a\n", 0x0.1ffffffffffffffep0 * 2**-16382'
0x0.fffffffffffffffp-16389
$ perl -lwe 'printf "%a\n", 0x0.0ffffffffffffffep0 * 2**-16382'
0x0.7ffffffffffffffp-16390
$ perl -lwe 'printf "%a\n", 0x0.07fffffffffffffep0 * 2**-16382'
0x0.3ffffffffffffffp-16391
$ perl -lwe 'printf "%a\n", 0x0.03fffffffffffffep0 * 2**-16382'
0x0.1ffffffffffffffp-16392
$ perl -lwe 'printf "%a\n", 0x0.01fffffffffffffep0 * 2**-16382'
0x0.0ffffffffffffffp-16393
$ perl -lwe 'printf "%a\n", 0x0.00fffffffffffffep0 * 2**-16382'
0x0.07fffffffffffffp-16394

Subnormals here are consistently showing the integer and fractional
part from subnormal style but the exponent of normalised style. This is
different from the manifestation that was seen with IEEE double. 5.24.0
shows the same behaviour as blead here. Thus the manifestation of the
subnormal problem was always different depending on the NV format, and
Jarkko's patch hasn't affected the behaviour on x86 80-bit long double.

Things get even weirder for certain subnormals in this format:

$ perl -lwe 'printf "%a\n", 0x0.1000000000000000p0 * 2**-16382'
0x08p-16389
$ perl -lwe 'printf "%a\n", 0x0.1e00000000000000p0 * 2**-16382'
0x0fp-16389
$ perl -lwe 'printf "%a\n", 0x0.0000000000000040p0 * 2**-16382'
0x000000000000002p-16443
$ perl -lwe 'printf "%a\n", 0x0.0000000000000020p0 * 2**-16382'
0x000000000000001p-16444
$ perl -lwe 'printf "%a\n", 0x0.000000000000001ep0 * 2**-16382'
0x000000000000000fp-16445

Where only one output hex digit is non-zero, in addition to the initial
subnormal bug the output is missing its radix point. This too hasn't
changed between 5.24.0 and blead. This is presumably because if the
values were output correctly normalised these ones would not require a
radix point.

-zefram

[James E Keenan via RT]

On Sat Aug 13 16:19:16 2016, zef...@fysh.org wrote:
> Jarkko Hietaniemi via RT wrote:
> > http://perl5.git.perl.org/perl.git/commit/b6d9b423fab1963346eb79e83b356114396b1f2e
>
> When I first reviewed this, before Jarkko merged it to blead, I
> thought
> it fixed the problem, and declared it good. Actually it's not
> correct:
> it's changed the problem into a different form. Sorry for not
> spotting
> it first time.
>

I've been following the discussion in this RT, as well as noting the filing of 5 other bug reports related to 'printf'. Those other reports are:

https://rt.perl.org/Ticket/Display.html?id=128888
https://rt.perl.org/Ticket/Display.html?id=128893
https://rt.perl.org/Ticket/Display.html?id=128889
https://rt.perl.org/Ticket/Display.html?id=128890
https://rt.perl.org/Ticket/Display.html?id=128909

(Each of the above is still in New status.)

I have to admit I'm confused as to where the discussion in going. In particular, I'm concerned about the fact that no one has cited any standards for the behavior of 'printf' that we presumably have failed to meet up until now and ought to be meeting going forward. It's not clear on what basis, other than personal knowledge, participants in the discussion are making their claims or making code changes. For those following along at home, this is unsatisfying.

Could we step back from the discussion for a moment and provide some links to standards we ought to be meeting?

Thank you very much.

--
James E Keenan (jke...@cpan.org)

[Zefram]

James E Keenan via RT wrote:
>In particular, I'm concerned about the fact that no one has cited any
>standards for the behavior of 'printf' that we presumably have failed
>to meet up until now and ought to be meeting going forward.

I've based my expectations on the C standard, to the extent that it's
not overridden by deliberate differences in Perl. I've mentioned it in
a couple of the bug reports. The actual standard isn't freely available
(in either sense of "free"), but a late draft of C99 is available at
<http://std.dkuug.dk/JTC1/SC22/WG14/www/docs/n843.pdf>. In that document,
the description of printf-style formatting starts on page 271, section
7.19.6.1 "The fprintf function".

-zefram

[Jarkko Hietaniemi via RT]

I've been following the discussion in this RT, as well as noting the
> filing of 5 other bug reports related to 'printf'. Those other
> reports are:
>
> https://rt.perl.org/Ticket/Display.html?id=128888
> https://rt.perl.org/Ticket/Display.html?id=128893
> https://rt.perl.org/Ticket/Display.html?id=128889
> https://rt.perl.org/Ticket/Display.html?id=128890
> https://rt.perl.org/Ticket/Display.html?id=128909
>
> (Each of the above is still in New status.)
>

> I have to admit I'm confused as to where the discussion in going. In

> particular, I'm concerned about the fact that no one has cited any
> standards for the behavior of 'printf' that we presumably have failed

> to meet up until now and ought to be meeting going forward. It's not
> clear on what basis, other than personal knowledge, participants in
> the discussion are making their claims or making code changes. For
> those following along at home, this is unsatisfying.
>
> Could we step back from the discussion for a moment and provide some
> links to standards we ought to be meeting?

Yeah, sorry. I see Zefram already pointed out some of the relevant docs.

In addition to the C99 (the first one in which the hexfp literals or the %a
came along), I'm following what the majority of platforms seems to have implemented.

There often seems to be some differences in the interpretation of the standards,
but if two our of three independent implementations seem to agree on a behavior,
I'm happy to follow it, unless of course the behavior clearly disagrees with some standard.

[Jarkko Hietaniemi via RT]

> When I first reviewed this, before Jarkko merged it to blead, I
> thought
> it fixed the problem, and declared it good. Actually it's not
> correct:
> it's changed the problem into a different form. Sorry for not
> spotting
> it first time.

Hrm. I'll try to take a look again tonight.

[Jarkko Hietaniemi via RT]

Zefram, before I go off and try to fix the new findings, could you please check t/op/sprintf2.t
and t/op/hexfp.t as of blead 73013786, and see that there are no accidentally wrong behaviors
in the newly added tests, especially the subnormals ones?

[Zefram]

Jarkko Hietaniemi via RT wrote:

>Zefram, before I go off and try to fix the new findings, could you
>please check t/op/sprintf2.t and t/op/hexfp.t as of blead 73013786,

sprintf2.t for double-double:

[ '%a', '-1', '-0x0p+0' ],

Expected result should be '-0x1p+0'.

sprintf2.t has a comment about tests for this ticket (#128843) for the
80-bit format, but it doesn't actually have any tests for values that
are subnormal in that format. It only tests 3e-320, which is normal
in 80-bit.

Other than that the %a tests look fine.

-zefram

[Jarkko Hietaniemi via RT]

Marking resolved until proven otherwise.