Groups
Groups
Sign in
Groups
Groups
Kdb+ Personal Developers
Conversations
About
Send feedback
Help
Lack of precision in KDBStudio vs console
41 views
Skip to first unread message
trias10
unread,
May 15, 2012, 5:30:47 AM
5/15/12
Reply to author
Sign in to reply to author
Forward
Sign in to forward
Delete
You do not have permission to delete messages in this group
Copy link
Report message
Show original message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to Kdb+ Personal Developers
Pretty simple example, try executing the following line of code in
Studio:
1050421051106700f%2 xexp 100
It will return 0f
Now try the q console itself, it should return the correct answer of:
8.2863610123923659e-016
Is there any way to get Studio to display "8.2863610123923659e-016"
instead of truncating it to "0f"?
Charles Skelton
unread,
May 15, 2012, 5:42:47 AM
5/15/12
Reply to author
Sign in to reply to author
Forward
Sign in to forward
Delete
You do not have permission to delete messages in this group
Copy link
Report message
Show original message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to personal...@googlegroups.com
in $HOME/.studioforkdb/studio.properties
the property
DecimalFormat=...
can be set to whatever you like according to
http://docs.oracle.com/javase/1.4.2/docs/api/java/text/DecimalFormat.html
but note that this affects every print of a float, inside tables too.
Reply all
Reply to author
Forward
0 new messages