is LUA_NUMBER_FMT_N caused accuracy errors?

[Chenchen Zhuo (lunoctis)]

Lua 5.5.0-rc4 build with default configs (LUA_FLOAT_TYPE == LUA_FLOAT_DOUBLE)

in the lua(.exe), input

74.4 / 100

result: 0.74400000000000011

compare to Lua 5.4.6, the value of "LUA_NUMBER_FMT" is "%.14g"

but "%.15g" in Lua 5.5.0, and “LUA_NUMBER_FMT_N" is "%.17g"

after set "LUA_NUMBER_FMT" and "LUA_NUMBER_FMT_N" to "%.14g" and recompile

74.4 / 100

result: 0.744

[Roberto Ierusalimschy]

> after set "LUA_NUMBER_FMT" and "LUA_NUMBER_FMT_N" to "%.14g" and recompile
>
> 74.4 / 100
>
> result: 0.744

The problem is that both expressions are not equal :-)

> 74.4 / 100 == 0.744
false
> 74.4 / 100 == 0.74400000000000011
true

Python(3) and JavaScript (js) give similar results. (0.7440000000000001,
instead of 0.74400000000000011.)

-- Roberto

[Roberto Ierusalimschy]

For an explanation, see

https://www.phys.uconn.edu/~rozman/Courses/P2200_15F/downloads/floating-point-guide-2015-10-15.pdf

-- Roberto

[Chenchen Zhuo (lunoctis)]

thanks for reply, I have no further questions.