Decision on Decimals vs Floats?

[Andrew Durdin]

There are two open tickets regarding decimals vs floats: #200 and
#2365.
The current situation is untenable for anyone with a need for genuine
fixed-point decimal values, as there are too many cases (i.e. at least
one :-) where they might be converted to floats, with a loss of
accuracy. To summarise the problems:

Floats are a poor choice for dealing with currency values, but the
Decimal class is only standard in python 2.4. The models.FloatField
class actually refers to a DECIMAL column, and will return a Decimal
instance under 2.4, but a float under 2.3. Also, the
[old]forms.FloatField is the default form field for models.FloatField,
and will always return a float, never a Decimal instance.

Condensing the various comments to both tickets into a single proposal:

Bundle decimal.py with django as utils._decimal_local (as has been done
with the threading module).
Modify the database wrappers to ensure that decimal columns will return
Decimal instances, and float columns (if supported) will return floats.
Create separate models.FloatField and models.DecimalField.
Create separated forms.FloatField and forms.DecimalField (in newforms
and presumably in oldforms), and appropriate widgets if needed.

I would like to see this included in Django -- at present I am
maintaining my own fork of Django to support Decimals appropriately,
and that's not a pleasant long-term situation. As the plan is to have
some API-breaking changes between 0.96 and 1.0, I'd like to propose
this change for inclusion in 1.0. I will undertake the work required
to produce the patch + docs + tests, but I'd like a decision from the
core devs as to whether they would like to see this in 1.0 or not.

Cheers,

Andrew.

[Michael Radziej]

Hi Andrew,

Andrew Durdin wrote:
> There are two open tickets regarding decimals vs floats: #200 and
> #2365.
> The current situation is untenable for anyone with a need for genuine
> fixed-point decimal values, as there are too many cases (i.e. at least
> one :-) where they might be converted to floats, with a loss of
> accuracy. To summarise the problems:

> [...]

if I remember correctly, the goal of these changes is to introduce a new
field type, DecimalField. For an inclusion into 1.0, two aspects are
important:

a) Would this introduce an incompatible change of an API?

b) Would it hold up 1.0? In other words, is there an agreement about the
general idea and design, and is there a good patch available or to be
expected soon?

c) How can the decimal module be integrated? Has Adrian agreed?

Can you please report on this? Then I'd like to include it in the
feature list.

Michael

[Andrew Durdin]

Michael Radziej wrote:
>
> a) Would this introduce an incompatible change of an API?

Yes: models.FloatField would always return floats (not Decimals), and
introspection of DECIMAL columns would yield DecimalFields (not
FloatField). The parameters to FloatField would also change (no
precision, etc.).

I'm raising the issue again at this time because Jacob's comments in
http://groups.google.com/group/django-developers/msg/0c19a63a1e4648e5
about API changes between 0.96 and 1.0 would indicate that this is the
best timeframe for this change to be implemented.

> b) Would it hold up 1.0? In other words, is there an agreement about the
> general idea and design, and is there a good patch available or to be expected soon?

I have the beginnings of a beginnings of a patch in my fork; I will
produce a full patch if the proposal in my previous message is
approved. It would not take long to do, so should not delay 1.0.

> c) How can the decimal module be integrated? Has Adrian agreed?

The decimal module can be integrated by including decimal.py in
django/utils, as has been done with the python implementation of
threading.local (see django/utils/_threading_local.py). The decimal
module is already made available separately for use with Python 2.3
(see http://www.taniquetil.com.ar/facundo/bdvfiles/get_decimal.html).
To the best of my knowledge, Adrian has not yet commented on this
possibility -- which is actually why I posted this thread.

Andrew

[chris....@gmail.com]

For what it's worth, I think getting this minor wart fixed would be
very helpful. I know I spent a lot of time playing around with
FloatField and decimals. I think anyt kind of app that does anything
more than basic integer math is going to stumble with these confusing
field definitions. Since this might cause breakage, it makes a lot of
sense to be in 1.0.

Just my $.019999999999

[Michael Radziej]

Hi Andrew,

thanks for the information, I have added it to
http://code.djangoproject.com/wiki/VersionOneFeatures. And I'm convinced
it should be considered!

Michael

[Andrew Durdin]

I've added an updated patch for #2365 "models.FloatField should be
renamed"[1], against revision 4439 of the trunk.

Summary of this patch:

* The 'decimal' module from 2.4 is included under django.utils, to
ensure Python 2.3 compatibility
* db.FloatField handle only floats, and db.DecimalField handle only
decimals.
* oldforms.FloatField and oldforms.DecimalField are the defaults for
the db fields and do the Right Thing validation-wise
* newforms.FloatField and newforms.DecimalField ditto; but they also
support max_value and min_value arguments à la newforms.IntegerField

This patch includes tests for both the database fields and the form
fields.

Important notes:

This patch BREAKS the old behaviour of db.FloatFields with various
database backends -- this is a good thing, because they were
inconsistent with different Python versions and also wouldn't handle
full round trips properly.

I've tested this patch with MySQL 4.1 and Sqlite 3.1.3 under Python
2.3.5 and 2.4.1 on OS X 10.4.8. The changes to the MSSQL, Oracle, and
Postgresql backends have NOT been tested; I don't have an install of
MSSQL or Oracle available, and was unable to get Postgresql running on
my machines.

So if anyone has MSSQL, Oracle, or Postgresql running and can try out
this patch and run the tests[2], please do! and post the results
here. There might be some additional typecasting needed, particularly
for Python 2.3 compatibility.

Cheers,

Andrew

[1] Come to think of it, the ticket title should be renamed.

[2] The widget tests in forms seem to be broken at the moment with a
UnicodeDecodeError; I commented out the widgets half of the forms
tests so the rest would run.

[Jacob Kaplan-Moss]

On 1/28/07 8:18 AM, Andrew Durdin wrote:
> * The 'decimal' module from 2.4 is included under django.utils, to
> ensure Python 2.3 compatibility

In order for this to be accepted, we need to know:

* What is the license for the decimal module, and is said license compatible
with the BSD license that Django uses?

* Are the authors of the decimal module OK with us distributing it with Django?

Our policy for including third party libraries requires an answer to both
those questions. We won't include anything if it's not legal to do so, and we
won't include anything that the original author doesn't want included.

Can somebody volunteer to answer those questions?

> This patch BREAKS the old behaviour of db.FloatFields with various
> database backends -- this is a good thing, because they were
> inconsistent with different Python versions and also wouldn't handle
> full round trips properly.

Can you explain more clearly what the breakage is? We'll need to write up a
transition path for any backwards breakage.

Other than that this looks good; thanks!

Jacob

[Todd O'Bryan]

The decimal module is part of Python from 2.4 onward, so it uses the
Python license.