Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Improper rounding of float value by using round function in sybase

202 views
Skip to first unread message

Subind

unread,
Jun 4, 2009, 8:32:29 AM6/4/09
to
Hi,
I'm facing issue with rounding of any float value. These are the two
ways I tried, both gave different results.
The method 1 doesn't do the rounding. why is that so?
Can anybody please throw some light on this issue?

Method 1
------------------------
declare @b float
select @b=2.135
select round(@b,2)
Result:2.13

Method 2
------------------------
select round(2.135,2)
Result:2.14
------------------------

David Kerber

unread,
Jun 4, 2009, 9:16:10 AM6/4/09
to
In article <e3f37b75-fa66-422e-b582-b847ba9f13c8@
21g2000vbk.googlegroups.com>, subi...@gmail.com says...

Most likely, when 2.135 is stored into variable b, it can't be exactly
represented in binary floating point, so its internal representation is
equal to something like 2.134999999999999, which of course rounds to
2.13. If you declare b as a DECIMAL or equivalent type, it will
probably work.

In the second case, 2.135 is probably representated as a Decimal (or
other exact) data type, and therefore rounds correctly.

--
/~\ The ASCII
\ / Ribbon Campaign
X Against HTML
/ \ Email!

Remove the ns_ from if replying by e-mail (but keep posts in the
newsgroups if possible).

0 new messages