Floating point computation issue?

15 views
Skip to first unread message

ejlis...@yahoo.fr

unread,
Jun 7, 2012, 12:58:42 PM6/7/12
to js...@googlegroups.com, sha...@shantirao.com
Hi all,

Does anyone know why JSDB (win version, did not check others) and SQLite
do not get the same result in the following situation?

---

R:\>jsdb.exe
js>var db = new SQLite();
js>db.exec("select 999901/10.0", function(r){println(r.toString())});
999901/10.0=99990.1000000001
true
js>exit


R:\>SQLite.exe NUL "select 999901/10.0;"
99990.1

---

Regards,
Etienne


P.S. Keep up the good work Shanti!

Jason S

unread,
Jun 8, 2012, 9:50:47 PM6/8/12
to js...@googlegroups.com, sha...@shantirao.com, ejlis...@yahoo.fr
R:\>jsdb.exe

js>var db = new SQLite();
js>db.exec("select 999901/10.0", function(r){println(r.toString())});
999901/10.0=99990.1000000001
true
js>exit


R:\>SQLite.exe NUL "select 999901/10.0;"
99990.1


The former goes through Javascript (the call of r.toString()); the latter does not.

Shanti Rao

unread,
Jun 8, 2012, 11:42:54 PM6/8/12
to js...@googlegroups.com
That's what I thought at first, but it actually turned out to be quite subtle. A 32-bit float with a rounding error is promoted to double by the compiler in a call to sqlite3_snprintf(). The fix wouls be to change how JSDB converts SQLite values to text in sqlite_exec().

Dies anyone else need thus fixed?

Shanti

Reply all
Reply to author
Forward
0 new messages