[AOLSERVER] Tcl rounding question

[William Scott Jordan]

Hey all!

This is really more of a tcl question, but I'm hoping that someone on
the list might have an explanation. Why does [format %.2f 18.005] round
down to "18.00" and [format %.2f 1.415] round up to "1.42"? Any
guesses? Am I missing something obvious here?

Tcl version 8.4, if it matters.

Thanks!

-William

--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <list...@listserv.aol.com> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

[Juan José del Río]

On Sat, 2008-05-03 at 15:31 -0700, William Scott Jordan wrote:
> Hey all!
>
> This is really more of a tcl question, but I'm hoping that someone on
> the list might have an explanation. Why does [format %.2f 18.005] round
> down to "18.00" and [format %.2f 1.415] round up to "1.42"? Any
> guesses? Am I missing something obvious here?
>

Hello William,

As far as I can see, it's just doing the correct math.

By convention, when you round:

1.00 = 1.0
1.01 = 1.0
1.02 = 1.0
1.03 = 1.0
1.04 = 1.0
1.05 = 1.1
1.06 = 1.1
1.07 = 1.1
1.08 = 1.1
1.09 = 1.1

and so on. This way you get .0 for 5 numbers, .1 for the other numbers,
and then the distribution is uniform. And that's not tcl-only , but they
way they teached me to round numbers in school ;-P

If you simply want to discard the decimal numbers... you'd try a method
different than format's :-)

Regards,

Juan José


-
Juan José del Río | Comercio online / e-commerce
(+34) 616 512 340 | juan...@simpleoption.com

Simple Option S.L.
Tel: (+34) 951 930 122
Fax: (+34) 951 930 122
http://www.simpleoption.com

[Dossy Shiobara]

On 2008.05.03, William Scott Jordan <sk...@SPEAKEASY.ORG> wrote:
> This is really more of a tcl question, but I'm hoping that someone on
> the list might have an explanation. Why does [format %.2f 18.005] round
> down to "18.00" and [format %.2f 1.415] round up to "1.42"? Any
> guesses? Am I missing something obvious here?

Classic floating-point precision and rounding issue. See:

http://en.wikipedia.org/wiki/Floating_point#Representable_numbers.2C_conversion_and_rounding

Read the third paragraph in that section.

--
Dossy Shiobara | do...@panoptic.com | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)

[Jeff Rogers]

William Scott Jordan wrote:
> Hey all!
>
> This is really more of a tcl question, but I'm hoping that someone on
> the list might have an explanation. Why does [format %.2f 18.005] round
> down to "18.00" and [format %.2f 1.415] round up to "1.42"? Any
> guesses? Am I missing something obvious here?
>
> Tcl version 8.4, if it matters.
>
> Thanks!
>
> -William

Set tcl_precision to 17 to see the fullest expansion of the value that
tcl will work with.

% set tcl_precision 17
17
% expr 18.005
18.004999999999999
% expr 1.415
1.415
%

The rounding I think is obvious at that point.

-J

[Bas Scheffers]

The plot thickens:

% format %.2f 18.0051
18.01

No ideas, though.

Bas.

[Alex]

It does explain it, but still results are not obvious :)

For example, this code:

=======
set bn [set rn 0]

for {set i 0} {$i < 1000} {incr i} {
set f "${i}.005"
set r [format %.2f $f]
set d [expr $r - $i]
if {$d > 0.0} {
incr bn
} else {
incr rn
}
}

puts "Rounded to 0.01 in $bn cases, to 0.0 in $rn cases"
============

produces results:

Rounded to 0.01 in 41 cases, to 0.00 in 959 cases

Thanks,
~ Alex.

[William Scott Jordan]

Yuck. Okay, so is there any practical work-around for getting X.XX5 to
consistently round up? I suppose I could do something like add
0.0000001 to any number that I'm rounding, but that seems pretty sloppy.
Is there a best practice for dealing with this?

-William

Dossy Shiobara wrote:
> On 2008.05.03, William Scott Jordan <sk...@SPEAKEASY.ORG> wrote:
>> This is really more of a tcl question, but I'm hoping that someone on
>> the list might have an explanation. Why does [format %.2f 18.005] round
>> down to "18.00" and [format %.2f 1.415] round up to "1.42"? Any
>> guesses? Am I missing something obvious here?
>
> Classic floating-point precision and rounding issue. See:
>
> http://en.wikipedia.org/wiki/Floating_point#Representable_numbers.2C_conversion_and_rounding
>
> Read the third paragraph in that section.
>

--

[Dossy Shiobara]

On 2008.05.03, William Scott Jordan <sk...@SPEAKEASY.ORG> wrote:
> Yuck. Okay, so is there any practical work-around for getting X.XX5 to
> consistently round up? I suppose I could do something like add
> 0.0000001 to any number that I'm rounding, but that seems pretty sloppy.
> Is there a best practice for dealing with this?

Best practice? Use integers, format as decimal value as needed. This
is often why you'll see financial applications store money values in
cents or hundredths of a cent, i.e., a dollar is stored as either "100"
(100 pennies) or "10000" ... if they deal in quantities that include
fractional cents. Only at the point where the data leaves the system,
either to a UI or to another system, do they format it with the decimal
point in place.

This approach generally eliminates all floating-point precision and
rounding issues and can even result in a performance increase in
compute-heavy applications where the machine's CPU handles integer math
faster than floating-point math.

--
Dossy Shiobara | do...@panoptic.com | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)

[Tom Jackson]

You have to remember that floating point math is done in base 2, but your are
inputing your numbers in base 10.

Another weired thing to keep in mind is that the default precision in Tcl has
changed in Tcl 8.5, so some answers are now different than they were in 8.4.

Another thing to keep in mind is that there is a difference between rounding
and actual significant digits. You should carry out your calculations with as
much precision as possible, then round to something less than that.

But, back to the question you have below. Tcl [format] uses a rounding
function, it isn't a truncation of the numbers.

If you use [expr {round(18.01)}] you get 18 (an integer).

The question is why doesn't Tcl offer a math function to round to some other
decimal precision?

The answer is simple. You should never 'round' an intermediate result. That
means you don't need this in Tcl. Final results can be formatted, and
[format] offers many more options than just the number of decimal points.

(Example of a problem with intermediate rounding:

Two step round:
round(18.449, 2) = 18.45
round(18.45, 1) = 18.5

One step round:
round(18.449, 1) = 18.4

)

tom jackson

[Bernhard van Woerden]

The only way to get what you expect is to do the rounding using integer
arithmetic.
The number 18.005 must be multiplied by 1000 to keep the implied precision.
Then do the rounding and reformat as a decimal.

TCL 8.5 has libtomath which promises arbitary precision integer arithetic
but 8.4 has wide which gives a fair bit of precision before it blows up.

Watch out
>expr 46341*46341
-2147479015

I've attached some procs that I use with 8.4 to round as I would expect
without having to keep everything as an integer.

[Tom Jackson]

Here is a helpful link, that explains the theory:

http://www2.hursley.ibm.com/decimal/

tom jackson