--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to va-smalltalk...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/2f9e935d-f6ef-4af5-9758-72cd74a76725%40googlegroups.com.
Hi Stephan,Maybe someone else from this forum may be able to help you. Off the top of my head I don't see anything obviously wrong. Maybe you prefer to use ScaledDecimals for what you are trying to do?
In any case, if you are a supported customer, feel free to contact customer support at vast-support@instantiations.com and we will do our best to help you.Best regards,
On Mon, Nov 4, 2019 at 8:15 AM 'Stephan Henke' via VA Smalltalk <va-sma...@googlegroups.com> wrote:
--Hello Instantiations team,I noticed a strange behaviour while working with floats. The usecase I am working on is rounding 169.38125 to four digits after the comma. So I would expect 169.3813 as result. But I get 169.3812.The actual rounding is done by a dll of our own but the problem appears to be about the float that is passed to the dll as a parameter.Then I discovered that this is because 169.38125 does not have an exact float-64 representation but is casted to the nearest one (1.69381249999999994315658113919e2) which happens to be a lower one and that value is in principle then rounded down correctly.But if I inspect 1.6938125 I see in the inspector the hex representation 40652C3333333333 which is to be inspected, but for 1.69381249999999994315658113919e2 I see 40652C3333333334, which I do not expect. and if I compare both numbers with < then1.6938125 < 1.69381249999999994315658113919e2 evaluates to true which it should not do. I would expect here that either both numbers would be equal or that the former would be greater. I checked the method < and there a virtual machine primitive is called. Is this a bug?We currently use VA 8.0.2 but will soon migrate to 9.2 64-bit. But I was able to reproduce the issue on both versions.
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to va-smalltalk+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/2f9e935d-f6ef-4af5-9758-72cd74a76725%40googlegroups.com.
--
Mariano Martinez Peck
Software Engineer, Instantiations Inc.
Email: mpeck@instantiations.com