Strage difference between precision in literals

1 view
Skip to first unread message

Anton Shepelev

unread,
Apr 23, 2022, 12:04:16 PMApr 23
to
Hello, all

Can you please tell me why this query:

SELECT POWER(0.5, 2), POWER(0.50, 2)
Returns 0.3 0.25
Instead of 0.25 0.25

--
() ascii ribbon campaign -- against html e-mail
/\ http://preview.tinyurl.com/qcy6mjc [archived]

Erland Sommarskog

unread,
Apr 25, 2022, 3:58:35 PMApr 25
to
Anton Shepelev (anto...@gmail.com) writes:
> Hello, all
>
> Can you please tell me why this query:
>
> SELECT POWER(0.5, 2), POWER(0.50, 2)
> Returns 0.3 0.25
> Instead of 0.25 0.25
>

All say after me: "SQL Server is designed for maximum confusion!".

I know how it feels, I've run into exactly this thing myself. And
I thought, what!? Then I read the manual and learnt this:

The retun type of power() is the type of the first argument. The 0.3
has the type decimal(1, 1), and so the return type is numeric(1,1)
and you get 0.3. Intuitive? Maybe not. But this is what the manual
says.

By the way, this is how you can tell the type of a numeric literal:
DECLARE @s sql_variant = 0.3
SELECT sql_variant_property(@s, 'Basetype'),
sql_variant_property(@s, 'Precision'),
sql_variant_property(@s, 'Scale')

To avoid such surpises, say something like:

SELECT cast(power(cast @val as float) as decimal(10,3))




Anton Shepelev

unread,
Apr 26, 2022, 6:21:50 AMApr 26
to
Erland Sommarskog:

> Anton Sheplev:
>
> > Can you please tell me why this query:
> >
> > SELECT POWER(0.5, 2), POWER(0.50, 2)
> > Returns 0.3 0.25
> > Instead of 0.25 0.25
>
> All say after me: "SQL Server is designed for maximum
> confusion!".

Indeed, and after your epic article about error han-
dling.

> I know how it feels, I've run into exactly this thing
> myself. And I thought, what!? Then I read the manu-
> al ->

I was misled by the remark phrase in the documentation
that the first artument "is an expression of type float
or of a type that can be implicitly converted to float."
So I thought that since

CAST( 0.5 AS FLOAT) = CAST( 0.50 AS FLOAT )

the should be no difference between 0.5 and 0.50.

> -> and learnt this:
>
> The retun type of power() is the type of the first ar-
> gument. The 0.3 has the type decimal(1, 1), and so the
> return type is numeric(1,1) and you get 0.3. Intu-
> itive? Maybe not. But this is what the manual says.

Thank you, Erland. That explains it.

> By the way, this is how you can tell the type of a nu-
> meric literal:
> DECLARE @s sql_variant = 0.3
> SELECT sql_variant_property(@s, 'Basetype'),
> sql_variant_property(@s, 'Precision'),
> sql_variant_property(@s, 'Scale')

Yes, quite useful in debugging implicit and/or unintu-
itive type conversions.

> To avoid such surpises, say something like:
> SELECT cast(power(cast @val as float) as decimal(10,3))

That is what I did, albeit a bit plainer:

DECLARE @C_TWO FLOAT = 2 -- float constatnt two!
--- ...
POWER( @C_TWO, @lambda )

I think your outer `CAST' is superfluous because it can-
not increase actual precision.

--
() ascii ribbon campaign - against html e-mail
/\ http://preview.tinyurl.com/qcy6mjc [archived]

Erland Sommarskog

unread,
Apr 26, 2022, 2:26:53 PMApr 26
to
Anton Shepelev (anton.txt@g{oogle}mail.com) writes:
> I think your outer `CAST' is superfluous because it can-
> not increase actual precision.
>

Yeah, but from a distance I couldn't tell whether you want a decimal
or a float result, so I added the cast.
Reply all
Reply to author
Forward
0 new messages