The value to be converted is 0.1 decimal.
Precision Hex value Decimal value Comments
single 3DCCCCCD 0.10000 value from SPTieee
double 3FB9999999999999 0.0999999999 value from AmigaBasic
double 3FB9DDDDA0000000 0.1010416522 value from IEEEDPFieee
As you can see AmigaBasic double precision (also IEEE) gives much more
accurate values than the output of IEEEDPFieee. Also, if 0.1 is calculated
by dividing 1 by 10 in double precision the answer is correct.
Has anybody noticed this before? Is there a fix for this?
________________________
Bob Fillmore, Systems Software & Communications BITNET: FILLMORE@EMRCAN
Computer Services Centre, BIX: bfillmore
Energy, Mines, & Resources Canada Voice: (613) 992-2832
588 Booth St., Ottawa, Ontario, Canada K1A 0E4 FAX: (613) 996-2953