Server producing incorrect math?
Sugapablo
I have a simple loop that adds values.
It seems that after thousands of instances using this loop with no
problems whatsoever, when the loop tries the following, it messes up.
strTotal = strTotal + strNewLabor
When strTotal = 3.2 and strNewLabor = 1.7, it caluculates:
strTotal = 4.900001
Why in the world would somthing like this happen?
--
Sugapablo
------------------------------------
http://www.sugapablo.com <--music
http://www.sugapablo.net <--personal
Dave Anderson
>
> I have a simple loop that adds values.
>
> It seems that after thousands of instances using this loop with no
> problems whatsoever, when the loop tries the following, it messes up.
>
> strTotal = strTotal + strNewLabor
>
> When strTotal = 3.2 and strNewLabor = 1.7, it caluculates:
> strTotal = 4.900001
>
> Why in the world would somthing like this happen?
What is the binary representation of 3.2? How about 1.7? Each is an
approximation, so there is no expectation that adding them will improve
their precision. See the following:
http://support.microsoft.com/default.aspx?scid=kb;EN-US;q244699
This applies to every language that uses floating point numbers.
--
Dave Anderson
Unsolicited commercial email will be read at a cost of $500 per message. Use
of this email address implies consent to these terms. Please do not contact
me directly or ask me to contact you directly for assistance. If your
question is worth asking, it's worth posting.
Bob Barrows
> Here's a wierd one...
>
> I have a simple loop that adds values.
>
> It seems that after thousands of instances using this loop with no
> problems whatsoever, when the loop tries the following, it messes up.
>
> strTotal = strTotal + strNewLabor
>
> When strTotal = 3.2 and strNewLabor = 1.7, it caluculates:
> strTotal = 4.900001
>
> Why in the world would somthing like this happen?
http://support.microsoft.com/default.aspx?scid=kb;EN-US;42980
HTH,
Bob Barrows
Sugapablo
> http://support.microsoft.com/default.aspx?scid=kb;EN-US;q244699
> This applies to every language that uses floating point numbers.
Ok, so what would the fix be?
Bob Barrows
http://support.microsoft.com/default.aspx?scid=kb;en-us;165373
HTH,
Bob Barrows
Tom B
Single decimal
strTotal=int(strTotal*10)
strNewLabor=int(strNewLabor*10)
strTotal=(strTotal+strNewLabor)/10
"Sugapablo" <russR...@sugapablo.com> wrote in message
news:vmmdg0h...@corp.supernews.com...
WIlliam Morris
2 + 2 = 5 for extremely large values of 2
- Wm
--
William Morris
Product Development, Seritas LLC
"Sugapablo" <russR...@sugapablo.com> wrote in message
news:vmmci53...@corp.supernews.com...
Ray at <%=sLocation%>
Ray at work
"WIlliam Morris" <seamlyne...@hotmail.com> wrote in message
news:bkfgsp$1a2ga$1...@ID-205671.news.uni-berlin.de...
Dave Anderson
>
> How much accuracy do you want/need?
>
> Single decimal
>
> strTotal=int(strTotal*10)
> strNewLabor=int(strNewLabor*10)
> strTotal=(strTotal+strNewLabor)/10
Division by 10 is just another way to introduce computational error. Unless
the divisor is a power of two (or a whole divisor of the numerator), the
result will necessarily represent an *approximation*.
This is not a solution. It "works" most of the time, but not all of the
time.
Dave Anderson
>
> Ok, so what would the fix be?
There is no fix to computational error if you want to continue to represent
numbers in binary any more than there is a fix to the general problem that
some numbers cannot be represented with a finite number of decimals.
If you are asking what you can do for your particular problem, then the
answer depends on what you want. If, for example, you only care about what
is being displayed, then use one of the appropriate methods/functions. Note
that these both return STRINGS:
JScript:
http://msdn.microsoft.com/library/en-us/script56/html/js56jsmthtofixed.asp
VBScript:
http://msdn.microsoft.com/library/en-us/script56/html/vsfctformatnumber.asp
On the other hand, if you are trying to test for equality (or some other
relation), you will have to redefine those terms. Pick an arbitrary error
margin, such as 0.0001, and define equality to be "within the margin of
error", less than to be "NOT greater than by more than the margin of error",
etc. Your new system should work something like this:
JScript:
OLD NEW
A == B Math.abs(A-B) < e
A != B Math.abs(A-B) > e
A < B A < B+e
A > B A > B-e
VBScript:
OLD NEW
A = B Abs(A-B) < e
A <> B Abs(A-B) > e
A < B A < B+e
A > B A > B-e
Because of computational error, there is little difference in meaning
between > and >= or < and <=. The endpoints are simply too unreliable to
make that distinction.
Dave Anderson
>
> Intel Math:
>
> 2 + 2 = 5 for extremely large values of 2
In VBScript, Round(5/2) = 2, while in JScript, Math.round(5/2) == 3.
So it would appear 3 + 3 = 5 for extremely small values of 3, as well.
WIlliam Morris
of my mind.
- Wm
--
William Morris
Product Development, Seritas LLC
"Dave Anderson" <GTSPXO...@spammotel.com> wrote in message
news:#ZiSoFuf...@TK2MSFTNGP10.phx.gbl...
Aaron Bertrand [MVP]
One work around would be to use formatnumber to limit the number of
significant digits after the decimal place. I added this as a FAQ today:
Evertjan.
microsoft.public.inetserver.asp.general:
Since BCD [binary coded decimal] seems to be out of use in this time of
hardcoded floatingpointnumbercrunching pentiums, te best way to code money
is to use integers trough-out and define those values as cents.
Only when writing, you should use:
Function WriteCents(CentsIntegerValue)
CentsIntegerValue = cInt(CentsIntegerValue) ' to be sure
Response.Write FormatNumber( CentsIntegerValue/100, 2 )
End Function
--
Evertjan.
The Netherlands.
(Please change the x'es to dots in my emailaddress)