Pragma to disable scientific notation?
Ed Avis
output (e.g. 5e-6 instead of 0.000005) a source of bugs. Is there a way to
turn it off globally so that perl will never use it when stringifying numbers?
At the moment I am using this in my code
# Convert a number to a string, avoiding scientific notation.
sub num2str { new Math::BigFloat($_[0])->bstr }
but it would be handy to say instead
no scientific_notation;
and have perl's builtin stringification do the right thing. (printf() and
sprintf() would not be affected, and parsing of input that has scientific
notation would still work.)
Does such a pragma exist (I couldn't find anything when searching) and if not,
would it be considered as a feature request?
--
Ed Avis <e...@waniasset.com>
Zefram
>It's a matter of taste but personally I find 'scientific notation' for number
>output (e.g. 5e-6 instead of 0.000005) a source of bugs. Is there a way to
>turn it off globally so that perl will never use it when stringifying numbers?
There used to be, but it was deprecated in 5.8 and disappeared in 5.10:
$ perl5.8 -lwe '$#="%.6f"; print 0.1; print 5e-6'
Use of $# is deprecated at -e line 1.
0.100000
0.000005
$ perl5.10 -lwe '$#="%.6f"; print 0.1; print 5e-6'
$# is no longer supported at -e line 1.
0.1
5e-06
>but it would be handy to say instead
>
> no scientific_notation;
>
>and have perl's builtin stringification do the right thing.
There was a bit of talk a few weeks ago about a hypothetical "no
stringification;", which would turn default ref stringification into a
runtime error. Your desired pragma would run into the same problem as
that, in that what you're trying to override is some code embedded in
the middle of sv_2pv_flags(). It's not like overriding an op type.
You also run into the same action-at-a-distance problem. A numeric scalar
currently stringifies in a predictable way, and code that wants to supply
a string can generate a numeric scalar and rely on stringification to
do the expected thing. If the stringification result varies according
to when the stringification happens to be triggered, the string value of
that scalar gets broken for everyone else. Reintroducing $# or anything
equivalent to it would be a bad idea for this reason.
-zefram
Ed Avis
>>It's a matter of taste but personally I find 'scientific notation' for number
>>output (e.g. 5e-6 instead of 0.000005) a source of bugs. Is there a way to
>>turn it off
>There used to be, but it was deprecated in 5.8 and disappeared in 5.10:
>
>$ perl5.8 -lwe '$#="%.6f"; print 0.1; print 5e-6'
>Use of $# is deprecated at -e line 1.
>0.100000
>0.000005
Ah, that is a bit different to what I had in mind. It's more flexible in one
way, in that you can specify any format string, but is there a format string
which would guarantee the same behaviour as perl has now for non-scientific-
notation cases? In other words to print 0.1 as 0.1 without trailing zeroes.
I didn't think there was such a format string, which is why I am using
Math::BigFloat->bstr() instead of sprintf().
>what you're trying to override is some code embedded in
>the middle of sv_2pv_flags().
Yes, that's what I had in mind. Something quite primitive and so without any
significant runtime cost. I see that depending on platform, one of gconvert(),
gcvt() or sprintf() is called.
>You also run into the same action-at-a-distance problem. A numeric scalar
>currently stringifies in a predictable way,
Ironically, the reason I want to change the behaviour is to make it more
predictable. Scientific notation tends to pop up when you least expect it.
But I appreciate that there will be code that depends on the existing behaviour,
and while back in olden days perl was full of interesting global switches like
$[ and $#, we don't want to make any new ones.
If any 'no scientific_notation' pragma were introduced, it would have to be
lexically scoped.
--
Ed Avis <e...@waniasset.com>
Eric Brine
> Your desired pragma would run into the same problem as
> that, in that what you're trying to override is some code embedded in
> the middle of sv_2pv_flags(). It's not like overriding an op type.
>
Which goes to show it must be done in the core, not on cpan.
> You also run into the same action-at-a-distance problem. [...]
Reintroducing $# or anything
> equivalent to it would be a bad idea for this reason.
>
$# was global. pragmas are lexical. So
no scientific_notation;
func();
is the same as just
func();
...Or is it? What if func() is an XS function? Does it see the lexical scope
of the caller or some other lexical scope?
- Eric
Abigail
my $var1 = 2 / 100_000;
my $var2 = "2e-05";
say $var1;
say $var2;
{
no scientific_notation;
say $var1; # What should this print?
say $var2; # And this?
}
Isn't (s)printf already suited to do this job?
Abigail
Paul LeoNerd Evans
> There was a bit of talk a few weeks ago about a hypothetical "no
> stringification;", which would turn default ref stringification into a
> runtime error.
Hypothetical??
http://search.cpan.org/~pevans/stringification-0.01_002/lib/stringification.pm
It's a rather early work currently and doesn't catch anywhere near as
many things as it should, but those are just a matter of brute-force
copypasta on the ops it currently wraps.
But it's there, really.. :)
--
Paul "LeoNerd" Evans
leo...@leonerd.org.uk
ICQ# 4135350 | Registered Linux# 179460
http://www.leonerd.org.uk/
Paul LeoNerd Evans
> > Your desired pragma would run into the same problem as
> > that, in that what you're trying to override is some code embedded in
> > the middle of sv_2pv_flags(). It's not like overriding an op type.
>
> Which goes to show it must be done in the core, not on cpan.
This was roughly the approach I was going to take with
stringification.pm
The demonstration would go something like:
1. Write it in a hacky way
2. Find a number of Real World Bugs it has nevertheless caught
3. Bounce up and down saying "Here, useful ability"
4. Come up with a patch to sv_2pv_flags() that somehow manages to wrap
the same functionallity in a much better way
5. ???
6. Profit.
I'm still however at stage 1. ;)
Ed Avis
> my $var1 = 2 / 100_000;
> my $var2 = "2e-05";
> say $var1;
> say $var2;
> {
> no scientific_notation;
> say $var1; # What should this print?
> say $var2; # And this?
> }
What I had in mind is
no scientific_notation;
say $var1; # prints 0.00002
say $var2; # prints 2e-05
$var2 += 0;
say $var2; # prints 0.00002
In other words, only conversion of numbers to strings would be affected; if the
scalar already holds a string value then leave it alone.
>Isn't (s)printf already suited to do this job?
There are a couple of reasons why sprintf isn't quite right.
Firstly because what I am most concerned about is writing a program which works
fine and looks correct, until one day when it happens to get an input value
smaller than those previously used, which triggers scientific notation. It
would be an extraordinarily disciplined programmer who wrote an explicit
sprintf() in every single place where a floating point number is stringified in
order to get consistent formatting all the time; indeed if you have to sprintf()
everywhere it negates the value of using a loosely typed language like Perl.
Because easy things should be easy, it would be better to have some kind of
directive you can set once to indicate that scientific notation should not be
used in the following code. There is also a small performance penalty with
calling explicit sprintf('format string', x) rather than just stringifying.
Also, I don't believe any of the sprintf formats do the right thing by
themselves. I would like to emulate what Perl's built-in stringification does,
which seems like the right thing as long as it doesn't decide to go scientific.
So,
foreach (0 .. 4) {
my $x = 0.5 / (10 ** $_);
say $x, "\t", sprintf('%f', $x), "\t", sprintf('%g', $x);
}
outputs
0.5 0.500000 0.5
0.05 0.050000 0.05
0.005 0.005000 0.005
0.0005 0.000500 0.0005
5e-05 0.000050 5e-05
Neither the middle nor the right-hand column is what I'm aiming for, which is
0.5
0.05
0.005
0.0005
0.00005
I have in the past resorted to code like
$v = sprintf '%f', $x;
$v =~ s/0+\z// if $v =~ /[.]/;
It seems a bit cumbersome. I wish I could just grab whatever bit of code that's
deciding to use scientific notation when there are too many leading zeroes, and
tell it to stop.
--
Ed Avis <e...@waniasset.com>
Abigail
You'd have to pick one of two evils anyway:
- Do this globally
- Always force numerification of values first, to make sure you
aren't dealing with a value that already contains a string value.
Which is going to be cumbersome as well (because a simple
'say 0 + $var' isn't going to do it if $var eq "foo").
Abigail
Zefram
>In other words, only conversion of numbers to strings would be affected; if the
>scalar already holds a string value then leave it alone.
What about:
my $v = 2e-5;
{
no scientific_notation;
my $s = "$v";
}
say $v;
That is, does the custom stringification stick? If it does, you're
effectively creating dualvars all the time. I think the sane version
of this is that it does not stick, so only code within the scope of the
pragma sees non-standard behaviour.
I'd forgotten about the approach we came up with for LeoNerd's pragma (for
which this caching issue doesn't arise). Perhaps because CPAN.pm doesn't
admit to the existence of the stringification module. We identified
the main op types that invoke stringification, and the module modifies
ops of those types to perform differently. Stringification invoked
implicitly in any other op won't be caught, but this might be sufficient
for your purposes.
>It seems a bit cumbersome. I wish I could just grab whatever bit of code that's
>deciding to use scientific notation when there are too many leading zeroes, and
>tell it to stop.
I think your problem with having no appropriate printf format is the
much easier part of this to handle. You're having some difficulty
getting the effect you want at the moment because you're using %f
formats: this requires you to pick a specific number of digits after
the decimal point, and if you used enough then rounding is a pain.
(Consider sprintf("%.100f", 2e-5).) I think the better way to get what
you want is to use a %e format and convert from exponential notation:
this gives you a specific number of significant figures, which seems to
be what you want. Try this:
sub ed($) {
my($s, $a, $b, $e) =
sprintf("%.15e", $_[0]) =~
/\A([-+]?)([0-9]+)\.([0-9]+)e([-+]?[0-9]+)\z/;
$a = ("0" x (1-length($a)-$e)).$a if $e <= -length($a);
$b .= "0" x ($e - length($b)) if $e > length($b);
my $d = $e < 0 ?
substr($a, 0, length($a)+$e).".".substr($a, $e).$b :
$a.substr($b, 0, $e).".".substr($b, $e);
$d =~ s/\A0+(?!\.)//;
$d =~ s/\.?0*\z//;
return $s.$d;
}
Cumbersome? Well, don't write it out each time. Modularise it.
You could of course speed this up, if it matters, by writing it in XS.
With this subproblem solved, I'd prefer to describe the rest of your
request as seeking to use arbitrary Perl code for stringification.
In fact, LeoNerd's "no stringification" can be viewed as just another
case of this general problem, where his arbitrary Perl code dies in
certain cases.
-zefram
Ed Avis
>>In other words, only conversion of numbers to strings would be affected; if
>>the scalar already holds a string value then leave it alone.
>
>What about:
>
> my $v = 2e-5;
> {
> no scientific_notation;
> my $s = "$v";
> }
> say $v;
>
>That is, does the custom stringification stick?
I think that just *using* $v in a string context should not change its content
at all. I know that perl does have some cases where seemingly read-only
operations have these odd side effects, and they cause regular puzzlement and
discussions on this mailing list, but it would be best not to add any more.
So I would expect the above code to be equivalent to
my $v = 2e-5;
say $v;
>With this subproblem solved, I'd prefer to describe the rest of your
>request as seeking to use arbitrary Perl code for stringification.
I don't really see it that way. Arbitrary Perl code to override
stringification seems like a particularly propeller-head sort of feature,
useful for Perl golf and Acme modules, but not something you'd use in
production. Apart from anything else there would be a large performance
penalty. I was hoping to use the existing native code for formatting numbers,
but just flick a switch to turn off the heuristic it currently uses to decide
when to resort to scientific notation.
I realize that I am biased, and that it's natural for me to see the thing I
proposed as eminently reasonable and straightforward, while you as a developer
are more likely to find the general solution that encompasses both this idea
and all the other weird things that various users have thought of. But my hope
is that 'no scientific_notation' could be something one regularly puts at the
top of every program and module, alongside 'use strict' and so on, and has the
effect of removing one particular source of bugs without adding a runtime
penalty.
--
Ed Avis <e...@waniasset.com>
Zefram
> Arbitrary Perl code to override
>stringification seems like a particularly propeller-head sort of feature,
What you're asking to override with is fairly arbitrary. You're looking
for something akin to a printf format, but it happens to not actually
be available in printf at the moment. I think it would be a sane thing
to have available in printf, so by all means propose it to ISO/IEC
JTC1/SC22/WG14 for C1X. Supposing it were in printf, you're asking to
select which printf format is used for numeric stringification, which
is what $# did in a scope-horrific way. With all the parameterisation
available, there's not much to pick out either Perl's default or your
target form as special.
> Apart from anything else there would be a large performance
>penalty.
You're worrying about this prematurely, and as I said you can write it
in XS when this genuinely becomes an issue.
>but just flick a switch to turn off the heuristic it currently uses to decide
>when to resort to scientific notation.
If you get it into printf it would be a bit like that, though not as
simple as just skipping part of the logic.
>effect of removing one particular source of bugs without adding a runtime
>penalty.
Under your pragma you'll still have a major class of bugs in this area
due to the stringification performing rounding. If you're going to
pick out any single alternative stringification as the one to avoid
surprising numeric behaviour, it would have to be one that keeps
all the non-zero digits. Would you be happy for 2e-5 to stringify as
"0.00002000000000000000163606107828062619091724627651274204254150390625"?
(That's its exact value in IEEE double.)
-zefram
Abigail
> Zefram <zefram <at> fysh.org> writes:
>
> >>In other words, only conversion of numbers to strings would be affected; if
> >>the scalar already holds a string value then leave it alone.
> >
> >What about:
> >
> > my $v = 2e-5;
> > {
> > no scientific_notation;
> > my $s = "$v";
> > }
> > say $v;
> >
> >That is, does the custom stringification stick?
>
> I think that just *using* $v in a string context should not change its content
> at all.
The drawback of that would mean a slowdown of Perl, where it would
stringify numbers each and every time it's used in string context.
Basically, what you are asking for is disabling caching.
> at all. I know that perl does have some cases where seemingly read-only
> operations have these odd side effects, and they cause regular puzzlement and
> discussions on this mailing list, but it would be best not to add any more.
>
> So I would expect the above code to be equivalent to
>
> my $v = 2e-5;
> say $v;
>
> >With this subproblem solved, I'd prefer to describe the rest of your
> >request as seeking to use arbitrary Perl code for stringification.
>
> I don't really see it that way. Arbitrary Perl code to override
> stringification seems like a particularly propeller-head sort of feature,
> useful for Perl golf and Acme modules, but not something you'd use in
> production. Apart from anything else there would be a large performance
> penalty. I was hoping to use the existing native code for formatting numbers,
> but just flick a switch to turn off the heuristic it currently uses to decide
> when to resort to scientific notation.
Actually, when I first read your request, my immediate reaction was "well,
if anyone is going to implement that, it'd be a pity if they don't do the
next step and use a callback". Kind of what we have for stringification
overload of objects.
Abigail
Zefram
> - modify perl to not save cached string representation for read-only operations,
> likely causing a performance hit and possibly changing semantics in some
> corner cases.
The obvious and sane choice is to have this modification apply only
within the scope of your pragma. No semantics changed for code not
affected by the pragma. This is what I meant by the non-sticky option
in my discussion of this issue.
-zefram
Ed Avis
stringifying a scalar has a side effect: it sets the string slot of the scalar.
Therefore, if there is a pragma which affects stringification, code running with
the pragma might set the string slot, but then code running without the pragma
would nonetheless pick up what was cached in the slot from before.
So it's not possible to have a lexically scoped 'no scientific_notation' pragma
unless you also change the perl core so that code like
$v = 123;
$s = "$v";
no longer alters the internal representation $v in any way. (Code that
explicitly assigns a string value to $v would be fine.)
So there are three choices:
- make the new behaviour global, possibly breaking code that depends on getting
scientific notation;
- make it lexically scoped but with side effects, so that merely using a
scalar in string context affects what later code sees;
- modify perl to not save cached string representation for read-only operations,
likely causing a performance hit and possibly changing semantics in some
corner cases.
--
Ed Avis <e...@waniasset.com>
Ed Avis
Yes, sorry I was a bit slow on the uptake.
A further refinement would be that the string representation can still be cached
as normal if it isn't one of the cases changed by the pragma.
--
Ed Avis <e...@waniasset.com>
Zefram
>I suppose that for the particular case I had in mind, it would be sufficient to
>stop caching string values when they would ordinarily use scientific notation.
Globally? Any global change to this would break stuff. Of course,
if you *don't* change it then you run into the obvious problem when you
want to stringify something that's already been stringified normally,
which was alluded to up at the top of the thread.
>I get the feeling that scientific notation isn't the bugbear for others on this
>list that it is for me.
Yeah. If you really care about how your numbers are expressed in strings,
then you need to control that expression explicitly. You have run into
the limit of DWIM.
-zefram
Ed Avis
>>> my $v = 2e-5;
>>> {
>>> no scientific_notation;
>>> my $s = "$v";
>>> }
>>> say $v;
>>>
>>>That is, does the custom stringification stick?
>>
>>I think that just *using* $v in a string context should not change its
>>content at all.
>
>The drawback of that would mean a slowdown of Perl, where it would
>stringify numbers each and every time it's used in string context.
>Basically, what you are asking for is disabling caching.
Yes, I now realize that.
I suppose that for the particular case I had in mind, it would be sufficient to
stop caching string values when they would ordinarily use scientific notation.
That might be an acceptably small performance penalty. Whether it's worth doing
depends on how useful the 'no scientific_notation' pragma is held to be.
I get the feeling that scientific notation isn't the bugbear for others on this
list that it is for me.
--
Ed Avis <e...@waniasset.com>
Ed Avis
>Under your pragma you'll still have a major class of bugs in this area
>due to the stringification performing rounding. If you're going to
>pick out any single alternative stringification as the one to avoid
>surprising numeric behaviour, it would have to be one that keeps
>all the non-zero digits. Would you be happy for 2e-5 to stringify as
>"0.00002000000000000000163606107828062619091724627651274204254150390625"?
You make a good point, but what I am most concerned with is not any particular
algorithm for stringification but the fact that it flips from one to the other
when you're not expecting it.
So if perl prints 2e-3 as 0.002 and 2e-4 as 0.0002, 2e-5 and 2e-6 should follow
the same pattern.
In my personal view Perl's stringification of numbers is good and sensible most
of the time. It just has a slightly Jekyll and Hyde character where it switches
over to a different (and usually, IMHO, unwanted) representation based on a
heuristic which is not documented.
--
Ed Avis <e...@waniasset.com>
Zefram
>over to a different (and usually, IMHO, unwanted) representation based on a
>heuristic which is not documented.
It's a printf %g format, that's all. It selects between %e and %f
based on the value, and performs a little extra processing. All of
these formats are copied straight from FORTRAN, and have a distinguished
history. The rule for %g to select between %e and %f is documented in
printf(3), for example:
# g, G The double argument is converted in style f or e (or F or E for
# G conversions). The precision specifies the number of signifi-
# cant digits. If the precision is missing, 6 digits are given;
# if the precision is zero, it is treated as 1. Style e is used
# if the exponent from its conversion is less than -4 or greater
# than or equal to the precision. Trailing zeros are removed from
# the fractional part of the result; a decimal point appears only
# if it is followed by at least one digit.
%g's format switching is usually considered to be a desirable feature,
in that it saves you the notional bother of decoding scientific notation
for numbers small enough to fit your fixed field width, in situations
where you'd otherwise be using %e.
-zefram
Ed Avis
>The rule for %g to select between %e and %f is documented in printf(3)
Thanks, it is documented there for C code that calls printf("%g", x), but I
didn't think that perlfunc or other perl documentation documented the exact
behaviour of %g in Perl (in fact it is described as system-dependent). Is it
documented in perlnumber(1) or elsewhere that %g is what is used?
>%g's format switching is usually considered to be a desirable feature,
>in that it saves you the notional bother of decoding scientific notation
>for numbers small enough to fit your fixed field width, in situations
>where you'd otherwise be using %e.
Yes, %g is great if you think of it as giving scientific notation, but with the
occasional abbreviation to leave out the exponent part where it's not necessary.
If you mistakenly started using it as a handy alternative to %f which trims the
leading zeroes, you would be surprised when your program first produced an output
with the exponent. No C programmer would make such a mistake because in C you
explicitly choose which format to use. In Perl, you don't normally think about
it and so it can catch you out.
--
Ed Avis <e...@waniasset.com>
Abigail
Nothing stops you from using printf instead of print in Perl.
But I think you already dismissed that as not doing what you want.
Abigail
Ed Avis
>Nothing stops you from using printf instead of print in Perl.
>But I think you already dismissed that as not doing what you want.
It's not about print or printf but about everyday conversion of
numbers to strings. Yes, I could take some code such as
$s = "database code involving $x, $y";
and rewrite it instead as
$s = 'database code involving ' . safe_stringify($x) . ', '
. safe_stringify($y);
But that isn't a terribly practical way to write code.
My concern is with making the default safe and without surprises. I
have already used code like the above, so it's not that perl cannot be
coaxed into doing what is required. Rather, I feel there could be an
easier way to achieve the same effect - to add one line turning on the
pragma, and then relax and concentrate on solving the problem your
code addresses, rather than scrutinizing each time a number is used in
string form to make sure that scientific notation is either avoided or
handled correctly. Perhaps this wish is not practical to implement,
but I think it is legitimate. If it's laziness on my part to not wish
to rewrite code using printf and sprintf, I hope this can be viewed as
the constructive kind of laziness we associate with perl.
--
Ed Avis <e...@waniasset.com>
Mark Jason Dominus
> $s = "database code involving $x, $y";
>
> and rewrite it instead as
>
> $s = 'database code involving ' . safe_stringify($x) . ', '
> . safe_stringify($y);
use Interpolation (num => \&safe_stringify);
$s = "database code involving $num{$x}, $num{$y}";
Ed Avis
> use Interpolation (num => \&safe_stringify);
> $s = "database code involving $num{$x}, $num{$y}";
Thanks for the suggestion.
--
Ed Avis <e...@waniasset.com>