VB Calculation Problem

[youn...@gmail.com]

Hi All,

I used Excel VB to test this program, but found out this problem:

------
Private Sub CommandButton1_Click()
i = 0

x = 0

For i = 1 To 226
x = x + 0.1
Next i

MsgBox x

End Sub
-----

Why is the result of "x" not 22.6, but "22.600000000001" instead?

I also tried to use Javascript to perform the same loop and the result
is the same.

Will it be an OS's problem?

Thanks.

[Jan Hyde (VB MVP)]

youn...@gmail.com's wild thoughts were released on Mon, 25
Feb 2008 20:08:55 -0800 (PST) bearing the following fruit:

>Hi All,
>
>I used Excel VB to test this program, but found out this problem:
>
>------
>Private Sub CommandButton1_Click()
> i = 0
>
> x = 0
>
> For i = 1 To 226
> x = x + 0.1
> Next i
>
> MsgBox x
>
>End Sub
>-----
>
>Why is the result of "x" not 22.6, but "22.600000000001" instead?

You need to read up on floating point numbers and all will
become clear.

J

>I also tried to use Javascript to perform the same loop and the result
>is the same.
>
>Will it be an OS's problem?
>
>Thanks.

--
Jan Hyde

https://mvp.support.microsoft.com/profile/Jan.Hyde

[youn...@gmail.com]

On 2月26日, 下午5時09分, "Jan Hyde (VB MVP)"
<StellaDrin...@REMOVE.ME.uboot.com> wrote:
> young...@gmail.com's wild thoughts were released on Mon, 25

> https://mvp.support.microsoft.com/profile/Jan.Hyde- 隱藏被引用文字 -
>
> - 顯示被引用文字 -

But I have been working on the 1st decimal place only... why does it
return ~10 d.p. to me?
what do I have to do?

Thanks.

[Alok]

The problem alluded to by the earlier post is that .1 cannot be
represented exactly in binary, similar to how 1/3 cannot be
represented exactly in a finite decimal number. see

http://support.microsoft.com/kb/214118

from details, but it becomes a repeating fraction of the form:

.0001100011000111000111 (and so on)

This is truncated in the computer and leads to roundoff error, which
eventually adds up. For a more simple example paste the following in
the immediate window:

.1*100 - 10

As I understand it, the only solutions are to either stick with
integer arithmetic (in your case, store 10 times the values that you
are dealing with) or to round the values to some tolerance, say .
000001. The first is only really practical if you know how many digits
you will need, for instance if you are dealing with money with a known
number of fractional cents.

> >https://mvp.support.microsoft.com/profile/Jan.Hyde-隱藏被引用文字 -

[youn...@gmail.com]

> > Thanks.- 隱藏被引用文字 -
>
> - 顯示被引用文字 -

Thank you very much!