Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

simple numeric calculation doent work

0 views
Skip to first unread message

NZDeveloper

unread,
Dec 3, 2009, 7:55:38 PM12/3/09
to
why in gods name doesnt this simple sql calculation not work in a view

98/912500 = 0.0001073972603

if I do a simple view with the following calcualted field:

SELECT 98 / 912500 AS aaa
FROM dbo.tbl_CA_Cases

I get 0??????

If I explicitly convert to decimal

SELECT CONVERT(decimal(38, 16), 98) AS v, CONVERT(decimal(38, 16),
912500) AS vt, CONVERT(decimal(38, 16), 98) / CONVERT(decimal(38, 16),
912500)
AS aaa, CONVERT(decimal(38, 16), CONVERT(decimal
(38, 16), 98) / CONVERT(decimal(38, 16), 912500)) AS bbb
FROM dbo.tbl_CA_Cases

I get 0.000107

Why is it only going to 6 decimal places????

If I do
SELECT 118993 * 98 / 912500 * 0.8468 AS aaa
FROM dbo.tbl_CA_Cases

I get 10.1616 where did SQL get this value from??? the correct
calculation is 10.82169939200

If I do
SELECT CONVERT(decimal(38, 16), 118993) * 98 / 912500 * 0.8468 AS
aaa
FROM dbo.tbl_CA_Cases

I get 10.821699392000 whcih is correct

What exactly is SQL calculating?????????????????

I'm assuming there is some logical explanation.....

Cheers

Grant

Plamen Ratchev

unread,
Dec 3, 2009, 9:52:10 PM12/3/09
to
In the first case you get 0 because you have integer division. In the other examples you get different results because
of the different precision used in the dividend and divisor. See in the following article details on how the resulting
precision and scale are calculated:
http://msdn.microsoft.com/en-us/library/ms190476.aspx

--
Plamen Ratchev
http://www.SQLStudio.com

0 new messages