Weird Float behaviour

65 views
Skip to first unread message

Stephan Henke

unread,
Nov 4, 2019, 6:15:38 AM11/4/19
to VA Smalltalk
Hello Instantiations team,

I noticed a strange behaviour while working with floats. The usecase I am working on is rounding 169.38125 to four digits after the comma. So I would expect 169.3813 as result. But I get 169.3812.
The actual rounding is done by a dll of our own but the problem appears to be about the float that is passed to the dll as a parameter.
Then I discovered that this is because 169.38125 does not have an exact float-64 representation but is casted to the nearest one (1.69381249999999994315658113919e2) which happens to be a lower one and that value is in principle then rounded down correctly.
But if I inspect 1.6938125 I see in the inspector the hex representation 40652C3333333333 which is to be inspected, but for 1.69381249999999994315658113919e2 I see 40652C3333333334, which I do not expect. and if I compare both numbers with < then

1.6938125 < 1.69381249999999994315658113919e2 evaluates to true which it should not do. I would expect here that either both numbers would be equal or that the former would be greater. I checked the method < and there a virtual machine primitive is called. Is this a bug?
We currently use VA 8.0.2 but will soon migrate to 9.2 64-bit. But I was able to reproduce the issue on both versions.

Mariano Martinez Peck

unread,
Nov 5, 2019, 8:56:26 AM11/5/19
to VA Smalltalk
Hi Stephan,

Maybe someone else from this forum may be able to help you. Off the top of my head I don't see anything obviously wrong. Maybe you prefer to use ScaledDecimals for what you are trying to do?

In any case, if you are a supported customer, feel free to contact customer support at vast-s...@instantiations.com and we will do our best to help you.

Best regards,



--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to va-smalltalk...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/2f9e935d-f6ef-4af5-9758-72cd74a76725%40googlegroups.com.


--
Mariano Martinez Peck
Software Engineer, Instantiations Inc.

Seth Berman

unread,
Nov 5, 2019, 9:03:13 AM11/5/19
to VA Smalltalk
Greetings Stephan,

Thanks for your question.  For a timely response from Instantiations, I would definitely propose vast-s...@instantiations.com.
This is a community forum, and while some of us from the company monitor it closely, our active support queue (with tracking system) is where all of us at the company are highly engaged and you will receive a quick response and dedication of resource.

I have created a case on your behalf and you will get a response from us shortly.

- Seth

On Tuesday, November 5, 2019 at 8:56:26 AM UTC-5, Mariano Martinez Peck wrote:
Hi Stephan,

Maybe someone else from this forum may be able to help you. Off the top of my head I don't see anything obviously wrong. Maybe you prefer to use ScaledDecimals for what you are trying to do?

In any case, if you are a supported customer, feel free to contact customer support at vast-support@instantiations.com and we will do our best to help you.

Best regards,



On Mon, Nov 4, 2019 at 8:15 AM 'Stephan Henke' via VA Smalltalk <va-sma...@googlegroups.com> wrote:
Hello Instantiations team,

I noticed a strange behaviour while working with floats. The usecase I am working on is rounding 169.38125 to four digits after the comma. So I would expect 169.3813 as result. But I get 169.3812.
The actual rounding is done by a dll of our own but the problem appears to be about the float that is passed to the dll as a parameter.
Then I discovered that this is because 169.38125 does not have an exact float-64 representation but is casted to the nearest one (1.69381249999999994315658113919e2) which happens to be a lower one and that value is in principle then rounded down correctly.
But if I inspect 1.6938125 I see in the inspector the hex representation 40652C3333333333 which is to be inspected, but for 1.69381249999999994315658113919e2 I see 40652C3333333334, which I do not expect. and if I compare both numbers with < then

1.6938125 < 1.69381249999999994315658113919e2 evaluates to true which it should not do. I would expect here that either both numbers would be equal or that the former would be greater. I checked the method < and there a virtual machine primitive is called. Is this a bug?
We currently use VA 8.0.2 but will soon migrate to 9.2 64-bit. But I was able to reproduce the issue on both versions.

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to va-smalltalk+unsubscribe@googlegroups.com.


--
Mariano Martinez Peck
Software Engineer, Instantiations Inc.

Hans-Martin Mosner

unread,
Nov 19, 2019, 8:27:10 AM11/19/19
to VA Smalltalk
Please note that this is not a bug with VA Smalltalk but with your understanding of how floating point numbers work. All binary floating point numbers will work the same, independently of the programming language in which they are used.
Float numbers have binary fractional parts, not decimal fractional parts, so as you already found out they cannot represent decimal fractions exactly (in general; all decimal fractions that are also finite binary fractions such as 0.5, 0.25, 0.125 and their integer multiples such as 0.75, 0.625 can be represented exactly). The 4/5 decimal rounding rule will therefore fail for numbers whose binary fractional representation is infinite.

If you want to round fractional numbers according to the standard decimal rounding rules, do not use Floating Point!

Use ScaledDecimal numbers or work with Fractions (which are always exact) and convert to a decimal representation when the numbers need to be displayed.

Cheers,
Hans-Martin

Stephan Henke

unread,
Nov 19, 2019, 10:17:12 AM11/19/19
to VA Smalltalk
Hallo Hans-Martin,

thank you for your comment. I already was on that point. The behaviour that I describe being weird ist that:

1.6938125 < 1.69381249999999994315658113919e2

That should not be true because the former value is larger.
Reply all
Reply to author
Forward
0 new messages