Problem storing floats/decimals from Python using pymongo
Phillip B Oldham
floats are always slightly off in the DB.
Michael Dirolf
Python float type. The only way PyMongo could store Decimal instances
would be to convert them to this standard, so you'd really only be
storing floats anyway - I'd rather force users to do this conversion
explicitly so that they are aware that it is happening...
Phillip B Oldham
> MongoDB only supports IEEE 754 floating points - the same as the
> Python float type. The only way PyMongo could store Decimal instances
> would be to convert them to this standard, so you'd really only be
> storing floats anyway - I'd rather force users to do this conversion
> explicitly so that they are aware that it is happening...
So how would one ensure then that in the DB a value is stored as, say,
9.99?
> db.floats.save({ "value": 7.51 })
> db.floats.find()
{"_id" : ObjectId( "4a76a17d671c3ab3ef7d3bed") , "value" : 9.99}
Storing this using pymongo I'll get something like:
> db.floats.find()
{"_id" : ObjectId( "4a76a17d671c3ab3ef7d3bed") , "value" : 9.99}
{"_id" : "StoredByPymongo" , "value" : 9.9900000000000002}
How do I resolve that?
Michael Dirolf
This is a problem with either the IEEE float representation itself, or
with the way Python handles packing and unpacking that representation:
>>> import struct
>>> struct.unpack("d", struct.pack("d", 9.99))
(9.9900000000000002,)
>>> struct.unpack("d", struct.pack("d", 9.99)) == (9.99,)
True
The only way you can guarantee that what you're saving matches exactly
9.99 is to convert your Decimal instance to a string or to a Binary
representation that you know will accurately represent it. The problem
with this is that the database will no longer know that it is dealing
with a number, so other clients won't be able to properly handle it
and things like sorting might not work as expected.
Phillip B Oldham
> is dealing with a number, so other clients won't be able to properly
> handle it and things like sorting might not work as expected.
the DB representation of the data is 9.99 (or whatever value), but
will be working in both Javascript and Python. I lead to believe it's
common to use the Decimal class in python <3.0 in such situations,
hence my original question. Will there be any changes to the pymongo
library to work with such values? Or is this something I'll have to
handle with type conversions in the languages I use with our mongo
implementation?
Mike
through.
Well the db representation is 9.99 as an IEEE floating point (which is
common to MongoDB and Python as well as most other modern languages).
The problem is that 9.99 cannot be represented exactly with a double
precision floating point - to see this type ">>> 9.99" into a Python
interpreter. My point being that the result you get when you save 9.99
with Python is exactly the same as the result you'd get saving it with
JavaScript or any of the other languages (and as the data you're
working with when you type 9.99 into a Python program).
As to your question about Decimal - based on a quick look it is still
subject to a lot of the same issues as normal floats, just with a
different base (10 instead of 2) which allows it to represent decimal
numbers exactly. So support for Decimal would require supporting a new
type in the database specifically for Python (since most languages
don't have this concept) - this is probably not going to happen. Any
changes would have to occur at the database level, there is no way
that PyMongo can take Decimal("9.99") and save it as anything but the
inexact float version (or a string).
Hope this helps to clear things up a bit,
Mike
Phillip B Oldham
> Sorry for the delay - somehow I missed this message when it first came
> through.
> My point being that the result you get when you save 9.99
> with Python is exactly the same as the result you'd get saving it with
> JavaScript or any of the other languages (and as the data you're
> working with when you type 9.99 into a Python program).
save 9.99 it gets saved and returned as 9.99. There may be some stuff
happening in the background of which I'm not aware, or mongo might be
trimming the representation somehow, but it's "looks" like 9.99 and
"acts" like 9.99:
> db.test.save({'_id': 9.99})
> db.test.find()
{"_id" : 9.99}
There doesn't seem to be any string conversion there either. I'm not
sure what I'm missing from your explanation there.
I understand that the issue lies with Python. It's just annoying that
I'll have to work around this with the use of strings and casts or
(ints and division) in the various languages I'm working with, whereas
when working with MySQL we don't have to worry since the Python ORM we
use (Storm) converts to/from Decimal() automatically.
Michael Dirolf
>> Sorry for the delay - somehow I missed this message when it first
>> came
>> through.
>
> No worries - thanks for taking the time to answer!
>
>> My point being that the result you get when you save 9.99
>> with Python is exactly the same as the result you'd get saving it
>> with
>> JavaScript or any of the other languages (and as the data you're
>> working with when you type 9.99 into a Python program).
>
> This is a little confusing for me. From the mongo command-line if I
> save 9.99 it gets saved and returned as 9.99. There may be some stuff
> happening in the background of which I'm not aware, or mongo might be
> trimming the representation somehow, but it's "looks" like 9.99 and
> "acts" like 9.99:
>
>> db.test.save({'_id': 9.99})
>> db.test.find()
> {"_id" : 9.99}
>
> There doesn't seem to be any string conversion there either. I'm not
> sure what I'm missing from your explanation there.
The representation that JavaScript is using is exactly the same. You
can prove this by saving 9.99 in the Python driver and querying in
JavaScript, and vice-versa. The reason it looks like JavaScript is
handling this better is because it is doing some rounding when
displaying as a string. Python does the same thing when you explicitly
convert to a string, actually:
>>> 9.99
9.9900000000000002
>>> str(9.99)
'9.99'
So the difference is really only in how things are being displayed -
the underlying representations are the same in both languages.
> I understand that the issue lies with Python. It's just annoying that
> I'll have to work around this with the use of strings and casts or
> (ints and division) in the various languages I'm working with, whereas
> when working with MySQL we don't have to worry since the Python ORM we
> use (Storm) converts to/from Decimal() automatically.
Just took a quick look at the Storm code - they convert Decimal
instances to strings, as suggested previously. This is something I
don't want the driver to do automatically because there would be no
way of knowing whether a string in a result document should be treated
as a regular string or as a Decimal instance. Should be easy to add as
part of a higher layer like MongoKit though - something where you know
which fields are which types.