Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

floating point precision: is this a bug?

19 views
Skip to first unread message

Irena Swanson

unread,
Sep 24, 2024, 11:11:45 PM9/24/24
to Macaulay2
I am running Macaulay2, version 1.24.05 on MacOSX.  After the following three lines:

d = 10594.48
e = 10514.44
d-e

M2 returns 80.0399999999936.  Interesting digits at the end.

If I type in

d = 10594.48, e = 10514.44, d-e

then the output is (10594.5, 10514.4, 80.04)

Irena

Doug Torrance

unread,
Sep 25, 2024, 6:45:18 AM9/25/24
to Macaulay2
That's the expected behavior.  We frequently see strange float point errors like this.  We can't represent d and e exactly using 64 binary bits, so they get rounded a tiny bit to nearby rational numbers with denominators that are powers of 2.

When floating point numbers are displayed as elements of some larger object like a sequence, then by default only the 6 most significant decimal digits are displayed.  You can change this behavior by setting the "printingPrecision" variable -- see https://macaulay2.com/doc/Macaulay2/share/doc/Macaulay2/Macaulay2Doc/html/_printing__Precision.html

Irena Swanson

unread,
Sep 25, 2024, 11:46:38 AM9/25/24
to maca...@googlegroups.com
Thank you, Doug.

Irena

--
You received this message because you are subscribed to the Google Groups "Macaulay2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to macaulay2+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/macaulay2/c8fbbb5a-1679-4f4a-b0ca-9dd7ae9d453fn%40googlegroups.com.

Thomas Kahle

unread,
Sep 27, 2024, 11:55:20 AM9/27/24
to Macaulay2

Hi Irena,

Often these spurious missing digits are correct according to the IEEE standard for floating point numbers: https://en.wikipedia.org/wiki/IEEE_754

If I do this in python (which is 100% IEEE correct) I get:

10594.48 - 10514.44

80.03999999999905

(which is different from your first M2 result).

Now, the classical example that everyone tries is this one here:
Python:

0.1+0.2

0.30000000000000004

M2:
i1 : 0.1+0.2
o1 = .3
o1 : RR (of precision 53)

And now for the fun part, your first example gives yet another result on my machine:
(Macaulay2, version 1.24.05 on MacOS)

i2 : d = 10594.48

o2 = 10594.48

o2 : RR (of precision 53)

i3 : e = 10514.44

o3 = 10514.44

o3 : RR (of precision 53)

i4 : d-e

o4 = 80.0399999999991

While the second is the same:

i7 : d = 10594.48, e = 10514.44, d-e

o7 = (10594.5, 10514.4, 80.04)

Of course I have not answered your question why these two ways to call the C++ library give different results. That I don’t know. I just wanted to say that sometimes these strange results can be correct according to a standard even if not mathematically correct. ¯_(ツ)_/¯

Best,
Thomas

--
You received this message because you are subscribed to the Google Groups "Macaulay2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to macaulay2+...@googlegroups.com.

Irena Swanson

unread,
Sep 27, 2024, 3:21:31 PM9/27/24
to maca...@googlegroups.com
Very interesting!  I did not know this.

Irena

Reply all
Reply to author
Forward
0 new messages