DecimalField and 'decimal' module.
Graham Dumpleton
about SVN version of Django supporting a DecimalField database type.
What is the relationship between this database field type and any
particular database adapters ability to store or return values in some
native database decimal type and convert it to the decimal.Decimal
type?
The reason I am asking is that there were recently problems found with
pyscopg2 module and its support for use of decimal.Decimal type. In
practice this only affects people who use mod_python or mod_wsgi and
run two different Django application instances in two different Python
sub interpreter instances in the one process. The actual details of
the pyscopg2 problem can be found at:
http://www.initd.org/tracker/psycopg/ticket/192
If Django does not use such type conversions done by a database
adapter module then all would be okay, but if it does some how rely on
them, I can see in the future some really obscure problems coming up
if people use psycopg2.
Just want to be forewarned since I usually end up dealing with the
more obscure mod_python questions. :-(
Graham
Russell Keith-Magee
>
> Rather obscure question here. I noted a comment over on the user list
> about SVN version of Django supporting a DecimalField database type.
> What is the relationship between this database field type and any
> particular database adapters ability to store or return values in some
> native database decimal type and convert it to the decimal.Decimal
> type?
The DecimalField/FloatField change was made on May 20, in [5302]
The change was made to clarify the FloatField type - the old
implementation was called float field, but used a Decimal as the db
representation.
In the new code, the DecimalField uses a Python Decimal to store
values internal to Django; a numeric(max_digits, decimal_places) on
the PostgreSQL backend, and converts the Decimal into a string (using
%*.f) for use in any INSERT/UPDATE/SELECT statement.
When retrieving data for a DecimalField in a SELECT, Django expects
that the response coming from the DB backend will be a Python Decimal.
More details here:
http://code.djangoproject.com/wiki/BackwardsIncompatibleChanges#RenamedFloatFieldtoDecimalField
> The reason I am asking is that there were recently problems found with
> pyscopg2 module and its support for use of decimal.Decimal type. In
> practice this only affects people who use mod_python or mod_wsgi and
> run two different Django application instances in two different Python
> sub interpreter instances in the one process. The actual details of
> the pyscopg2 problem can be found at:
>
> http://www.initd.org/tracker/psycopg/ticket/192
Ok - this is the 'mod_python caching' problem I mentioned on a
different thread the other day. I got a few details wrong at the time,
but please allow me to now gesticulate wildly at the aforementioned
URL, yelling "that one! that one!" :-)
At the time, I said that we fixed the problem by updating mod_python -
it turns out that I was mistaken. We still have the problem, but we
worked around it by updating some of our Django applications so that a
common Django install was being used.
> If Django does not use such type conversions done by a database
> adapter module then all would be okay, but if it does some how rely on
> them, I can see in the future some really obscure problems coming up
> if people use psycopg2.
If I'm understanding the ticket correctly, Django _is_ using the
problematic type conversions.
Are you able to provide advice/instructions for a workaround for those
using mod_python/mod_wsgi (if such a workaround exists)?
Yours,
Russ Magee %-)
Graham Dumpleton
wrote:
> On 7/23/07, Graham Dumpleton <Graham.Dumple...@gmail.com> wrote:
>
>
>
> > Rather obscure question here. I noted a comment over on the user list
> > about SVN version of Django supporting a DecimalField database type.
> > What is the relationship between this database field type and any
> > particular database adapters ability to store or return values in some
> > native database decimal type and convert it to the decimal.Decimal
> > type?
>
> The DecimalField/FloatField change was made on May 20, in [5302]
>
> The change was made to clarify the FloatField type - the old
> implementation was called float field, but used a Decimal as the db
> representation.
>
> In the new code, the DecimalField uses a Python Decimal to store
> values internal to Django; a numeric(max_digits, decimal_places) on
> the PostgreSQL backend, and converts the Decimal into a string (using
> %*.f) for use in any INSERT/UPDATE/SELECT statement.
>
> When retrieving data for a DecimalField in a SELECT, Django expects
> that the response coming from the DB backend will be a Python Decimal.
>
>
> > The reason I am asking is that there were recently problems found with
> > pyscopg2 module and its support for use of decimal.Decimal type. In
> > practice this only affects people who use mod_python or mod_wsgi and
> > run two different Django application instances in two different Python
> > sub interpreter instances in the one process. The actual details of
> > the pyscopg2 problem can be found at:
>
> >http://www.initd.org/tracker/psycopg/ticket/192
>
> Ok - this is the 'mod_python caching' problem I mentioned on a
> different thread the other day. I got a few details wrong at the time,
> but please allow me to now gesticulate wildly at the aforementioned
> URL, yelling "that one! that one!" :-)
>
> At the time, I said that we fixed the problem by updating mod_python -
> it turns out that I was mistaken. We still have the problem, but we
> worked around it by updating some of our Django applications so that a
> common Django install was being used.
>
> > If Django does not use such type conversions done by a database
> > adapter module then all would be okay, but if it does some how rely on
> > them, I can see in the future some really obscure problems coming up
> > if people use psycopg2.
>
> If I'm understanding the ticket correctly, Django _is_ using the
> problematic type conversions.
>
> Are you able to provide advice/instructions for a workaround for those
> using mod_python/mod_wsgi (if such a workaround exists)?
No solution if using mod_python alone.
If using mod_wsgi you could leave your main performance oriented
Django instance to run in the main Apache child processes (using
embedded mode of mod_wsgi) and push secondary instances of Django into
distinct daemon processes (using daemon mode of mod_wsgi).
In mod_wsgi there is a solution as it allows use of Apache child
processes like mod_python as well as daemon processes like with
mod_fastcgi. I'll update the Django documentation for mod_wsgi
explaining the potential for this problem and include an example
configuration.
If Django is going to be affected by this, also looks like I better
suggest a patch to the pyscopg2 people so it might get fixed sooner
rather than later.
Thanks for the feedback on Django, I quickly looked at the source code
and couldn't see how things hung together myself to know whether it
might be affected or not.
Graham
Gábor Farkas
>
> The reason I am asking is that there were recently problems found with
> pyscopg2 module and its support for use of decimal.Decimal type. In
> practice this only affects people who use mod_python or mod_wsgi and
> run two different Django application instances in two different Python
> sub interpreter instances in the one process. The actual details of
> the pyscopg2 problem can be found at:
>
> http://www.initd.org/tracker/psycopg/ticket/192
does this also affect psycopg1?
gabor
Graham Dumpleton
Looking at psycopg1 code, the answer would appear to be that it is not
affected. Changes to support Decimal appear to be psycopg2 specific.
Graham
oggie rob
>
> Looking at psycopg1 code, the answer would appear to be that it is not
> affected. Changes to support Decimal appear to be psycopg2 specific.
>
> Graham
The problems I was seeing the other day were repeatable both using
strictly psycopg1 (in fact, that was where I first saw the server
error). The issue wrt Decimal is that the Django svn version uses
Decimal and older versions use float, but because psycopg is "cached"
it will convert it into the type that is initialized. In other words,
its not psycopg itself - its the fact that it isn't re-initialized
between requests.
The recommendation I would make for now is that applications using new
Django builds should use psycopg2 and applications using older builds
should use psycopg1. This way there are two executables that don't
clash in any way. I have done this myself and it eliminated all the
problems I had with float/Decimal compatibility & Unicode.
-rob
Gábor Farkas
>>> does this also affect psycopg1?
>> Looking at psycopg1 code, the answer would appear to be that it is not
>> affected. Changes to support Decimal appear to be psycopg2 specific.
>>
>> Graham
>
> The problems I was seeing the other day were repeatable both using
> strictly psycopg1 (in fact, that was where I first saw the server
> error). The issue wrt Decimal is that the Django svn version uses
> Decimal and older versions use float, but because psycopg is "cached"
> it will convert it into the type that is initialized. In other words,
> its not psycopg itself - its the fact that it isn't re-initialized
> between requests.
are you sure this is the same issue?
from what i saw in the ticket
http://www.initd.org/tracker/psycopg/ticket/192,
it seems that it can fail even when both sub-interpreters run the same
version of python/django...
gabor
oggie rob
> Django builds should use psycopg2 and applications using older builds
> should use psycopg1. This way there are two executables that don't
> clash in any way. I have done this myself and it eliminated all the
> problems I had with float/Decimal compatibility & Unicode.
>
> -rob
Forgot to add - this is strictly a temporary solution. If there are
any other changes to the backend, this may not work with older
versions anymore.
-rob
oggie rob
>
>
> it seems that it can fail even when both sub-interpreters run the same
> version of python/django...
>
> gabor
Sorry, gabor I didn't see this earlier.
Yes, I'm pretty certain it is the same issue.
-rob
Graham Dumpleton
Hmmm, I'm not sure how psycopg1 in isolation could be a cause of
problems as it doesn't anywhere in its source code support the new
'decimal.Decimal' type. Thus, the exact same problem as identified so
far with pyscopg2 couldn't be what was being encountered with
psycopg1.
Looking at the code for psycopg2 again though, it is possible that
psycopg2 makes similar mistakes in how it deals with the mxDateTime
module as well, as it imports that from module initialiser and caches
stuff from it as well. In that case though, the type objects are
implemented in C code, unlike 'decimal.Decimal', so that they are
shared between sub interpreters I didn't think would cause a problem.
How psycopg1 handles mxDateTime is a bit different, but presuming it
wouldn't be different.
One problem that could arise is if, even in different sub
interpreters, two Django instances separately tried to use version 1
and 2 of pyscopg. This is because the Python wrappers for one wouldn't
match the loaded C extension module since it is named the same for
both versions and would only be loaded once for the whole process. The
errors in this case would have been quite dramatic though given the
probably significant changes between the API of the C extension module
between the two versions.
Do you remember the specific Python exceptions you were getting? The
Python exceptions which were being encountered when this was
originally found were things like isinstance() checks failing because
the class type for the instance of a decimal.Decimal didn't match the
identity of the decimal.Decimal type from that interpreter, because
the instance was creating using the type object from another
interpreter. Apart from failure of checks of this sort, the
decimal.Decimal instances otherwise worked fine.
Looking at the psycopg code, I know there are other problems with the
code, but they would only arise where one was doing strange stuff with
destroying and recreating Python sub interpreters, a feature that is
only possible in mod_wsgi and is unlikely being used by anyone at this
point.
Graham
oggie rob
> problems as it doesn't anywhere in its source code support the new
> 'decimal.Decimal' type. Thus, the exact same problem as identified so
> far with pyscopg2 couldn't be what was being encountered with
> psycopg1.
I'm sorry, I should have been clearer. To be honest I would think it
were possible to avoid some of these problems by modifying Django
somehow, but I haven't looked very closely at how, yet. However I
think I do understand how this is happening at present.
Let me describe the situation I was in previously and how I saw
problems:
- Application "Santa Cruz" built with 0.91 using psycopg1. Running
for just under a couple of years and accessed regularly.
- Application "Curriculum" built with trunk using psycopg1. Added
about 2 weeks ago & accessed infrequently.
The first problem I saw was when I was doing a (python) sum on a
FloatField in the Santa Cruz application. It was saying that it
couldn't add Float and Decimal properly. From my very cursory look
just now at trunk/django/db/backends/postgresql/creation.py,
FloatField is registered as a "double precision" type in the database.
However, in 0.91, it is listed as a "numeric(%(max_digits)s, %
(decimal_places)s)" - which translates into a DecimalField in the
trunk version. So if I visit my Curriculum application, then go to
Santa Cruz, I get the error about adding Float and Decimal because the
introspection function is "initialized" by trunk.
The second problem I saw was more prevalent because it happened much
more frequently! After visiting the Curriculum application, a trip to
the Santa Cruz application would win me improperly quoted SQL. The
reason in that case was because my old 0.91 code wasn't set to
translate unicode, and after visiting Curriculum it was returning
u'xyz' data instead of 'xyz'. I couldn't even log in!
I figure the root cause of both of these problems is data that is
accessed in two interpreters, and is propagated by having two versions
that expect different results at that point (i.e. 0.91 FloatField
instrospection expects a float, not a Decimal). Sometimes it takes a
while to be seen, as in my sum call (which I do about once or twice a
month, normally). The thing is most of the role of psycopg seems to be
kind of like a Singleton or static method object, so it is not always
seen, but is obvious when you are using the combinations I was, or the
one you reported on the initd site.
Conversely, I can see how this hasn't been raised as a major problem
before now, since the 'static'-like nature of psycopg doesn't normally
show any issues. But I don't like to be the guinea pig and find out
how extensively this affects what you can do!
> One problem that could arise is if, even in different sub interpreters,
> two Django instances separately tried to use version 1 and 2 of pyscopg.
> This is because the Python wrappers for one wouldn't match the loaded
> C extension module since it is named the same for both versions and
> would only be loaded once for the whole process.
Are you sure this would happen? If that were the case I would have
expected my box to blow up by now. I wonder if the extensions are
named differently, or if the namespace avoids the problem (i.e.
"import psycopg" vs "import psycopg2")? I haven't seen any problems so
far (cross fingers).
-rob
Graham Dumpleton
> > One problem that could arise is if, even in different sub interpreters,
> > two Django instances separately tried to use version 1 and 2 of pyscopg.
> > This is because the Python wrappers for one wouldn't match the loaded
> > C extension module since it is named the same for both versions and
> > would only be loaded once for the whole process.
>
> Are you sure this would happen? If that were the case I would have
> expected my box to blow up by now. I wonder if the extensions are
> named differently, or if the namespace avoids the problem (i.e.
> "import psycopg" vs "import psycopg2")? I haven't seen any problems so
> far (cross fingers).
If the C extension module resides within the package, ie., thus is
within parent package namespace, then probably wouldn't occur, ie.,
both would be loaded. I would though still be concerned if the two
different versions of the C extension use the same C symbol names for
certain global non static data. If this occurs the dynamic linker may
in the second loaded module, actually use global data variables from
the first to be loaded module rather than its own. If the way such
global data was initialised was different to how expected, it may
feasibly cause subtle strange problems with one more of the other.
Graham
Carl Karsten
> - Application "Santa Cruz" built with 0.91 using psycopg1. Running
> for just under a couple of years and accessed regularly.
> - Application "Curriculum" built with trunk using psycopg1. Added
> about 2 weeks ago & accessed infrequently.
> Are you sure this would happen? If that were the case I would have
> expected my box to blow up by now. I wonder if the extensions are
> named differently, or if the namespace avoids the problem (i.e.
> "import psycopg" vs "import psycopg2")? I haven't seen any problems so
> far (cross fingers).
"Curriculum" use psycopg1 or 2?
seems like that was part of your point.
Carl K
oggie rob
>
> seems like that was part of your point.
>
> Carl K
Yeah I didn't mention this post that *now* Curriculum is using
psycopg2 and Santa Cruz is using psycopg1, with no apparent problems
(sorry, I did mention it earlier but forgot to clarify). But as Graham
says, there may still be some subtle risks that I'm unaware. And of
course, it won't work beyond two projects!
-rob