Lack of precision in KDBStudio vs console

41 views
Skip to first unread message

trias10

unread,
May 15, 2012, 5:30:47 AM5/15/12
to Kdb+ Personal Developers
Pretty simple example, try executing the following line of code in
Studio:

1050421051106700f%2 xexp 100

It will return 0f


Now try the q console itself, it should return the correct answer of:
8.2863610123923659e-016


Is there any way to get Studio to display "8.2863610123923659e-016"
instead of truncating it to "0f"?

Charles Skelton

unread,
May 15, 2012, 5:42:47 AM5/15/12
to personal...@googlegroups.com
in $HOME/.studioforkdb/studio.properties

the property

DecimalFormat=...

can be set to whatever you like according to

but note that this affects every print of a float, inside tables too.
Reply all
Reply to author
Forward
0 new messages