Rather obscure question here. I noted a comment over on the user list about SVN version of Django supporting a DecimalField database type. What is the relationship between this database field type and any particular database adapters ability to store or return values in some native database decimal type and convert it to the decimal.Decimal type?
The reason I am asking is that there were recently problems found with pyscopg2 module and its support for use of decimal.Decimal type. In practice this only affects people who use mod_python or mod_wsgi and run two different Django application instances in two different Python sub interpreter instances in the one process. The actual details of the pyscopg2 problem can be found at:
If Django does not use such type conversions done by a database adapter module then all would be okay, but if it does some how rely on them, I can see in the future some really obscure problems coming up if people use psycopg2.
Just want to be forewarned since I usually end up dealing with the more obscure mod_python questions. :-(
On 7/23/07, Graham Dumpleton <Graham.Dumple...@gmail.com> wrote:
> Rather obscure question here. I noted a comment over on the user list > about SVN version of Django supporting a DecimalField database type. > What is the relationship between this database field type and any > particular database adapters ability to store or return values in some > native database decimal type and convert it to the decimal.Decimal > type?
The DecimalField/FloatField change was made on May 20, in [5302]
The change was made to clarify the FloatField type - the old implementation was called float field, but used a Decimal as the db representation.
In the new code, the DecimalField uses a Python Decimal to store values internal to Django; a numeric(max_digits, decimal_places) on the PostgreSQL backend, and converts the Decimal into a string (using %*.f) for use in any INSERT/UPDATE/SELECT statement.
When retrieving data for a DecimalField in a SELECT, Django expects that the response coming from the DB backend will be a Python Decimal.
> The reason I am asking is that there were recently problems found with > pyscopg2 module and its support for use of decimal.Decimal type. In > practice this only affects people who use mod_python or mod_wsgi and > run two different Django application instances in two different Python > sub interpreter instances in the one process. The actual details of > the pyscopg2 problem can be found at:
Ok - this is the 'mod_python caching' problem I mentioned on a different thread the other day. I got a few details wrong at the time, but please allow me to now gesticulate wildly at the aforementioned URL, yelling "that one! that one!" :-)
At the time, I said that we fixed the problem by updating mod_python - it turns out that I was mistaken. We still have the problem, but we worked around it by updating some of our Django applications so that a common Django install was being used.
> If Django does not use such type conversions done by a database > adapter module then all would be okay, but if it does some how rely on > them, I can see in the future some really obscure problems coming up > if people use psycopg2.
If I'm understanding the ticket correctly, Django _is_ using the problematic type conversions.
Are you able to provide advice/instructions for a workaround for those using mod_python/mod_wsgi (if such a workaround exists)?
> On 7/23/07, Graham Dumpleton <Graham.Dumple...@gmail.com> wrote:
> > Rather obscure question here. I noted a comment over on the user list > > about SVN version of Django supporting a DecimalField database type. > > What is the relationship between this database field type and any > > particular database adapters ability to store or return values in some > > native database decimal type and convert it to the decimal.Decimal > > type?
> The DecimalField/FloatField change was made on May 20, in [5302]
> The change was made to clarify the FloatField type - the old > implementation was called float field, but used a Decimal as the db > representation.
> In the new code, the DecimalField uses a Python Decimal to store > values internal to Django; a numeric(max_digits, decimal_places) on > the PostgreSQL backend, and converts the Decimal into a string (using > %*.f) for use in any INSERT/UPDATE/SELECT statement.
> When retrieving data for a DecimalField in a SELECT, Django expects > that the response coming from the DB backend will be a Python Decimal.
> > The reason I am asking is that there were recently problems found with > > pyscopg2 module and its support for use of decimal.Decimal type. In > > practice this only affects people who use mod_python or mod_wsgi and > > run two different Django application instances in two different Python > > sub interpreter instances in the one process. The actual details of > > the pyscopg2 problem can be found at:
> Ok - this is the 'mod_python caching' problem I mentioned on a > different thread the other day. I got a few details wrong at the time, > but please allow me to now gesticulate wildly at the aforementioned > URL, yelling "that one! that one!" :-)
> At the time, I said that we fixed the problem by updating mod_python - > it turns out that I was mistaken. We still have the problem, but we > worked around it by updating some of our Django applications so that a > common Django install was being used.
> > If Django does not use such type conversions done by a database > > adapter module then all would be okay, but if it does some how rely on > > them, I can see in the future some really obscure problems coming up > > if people use psycopg2.
> If I'm understanding the ticket correctly, Django _is_ using the > problematic type conversions.
> Are you able to provide advice/instructions for a workaround for those > using mod_python/mod_wsgi (if such a workaround exists)?
No solution if using mod_python alone.
If using mod_wsgi you could leave your main performance oriented Django instance to run in the main Apache child processes (using embedded mode of mod_wsgi) and push secondary instances of Django into distinct daemon processes (using daemon mode of mod_wsgi).
In mod_wsgi there is a solution as it allows use of Apache child processes like mod_python as well as daemon processes like with mod_fastcgi. I'll update the Django documentation for mod_wsgi explaining the potential for this problem and include an example configuration.
If Django is going to be affected by this, also looks like I better suggest a patch to the pyscopg2 people so it might get fixed sooner rather than later.
Thanks for the feedback on Django, I quickly looked at the source code and couldn't see how things hung together myself to know whether it might be affected or not.
> The reason I am asking is that there were recently problems found with > pyscopg2 module and its support for use of decimal.Decimal type. In > practice this only affects people who use mod_python or mod_wsgi and > run two different Django application instances in two different Python > sub interpreter instances in the one process. The actual details of > the pyscopg2 problem can be found at:
On Jul 23, 7:33 pm, Gábor Farkas <ga...@nekomancer.net> wrote:
> Graham Dumpleton wrote:
> > The reason I am asking is that there were recently problems found with > > pyscopg2 module and its support for use of decimal.Decimal type. In > > practice this only affects people who use mod_python or mod_wsgi and > > run two different Django application instances in two different Python > > sub interpreter instances in the one process. The actual details of > > the pyscopg2 problem can be found at:
> Looking at psycopg1 code, the answer would appear to be that it is not > affected. Changes to support Decimal appear to be psycopg2 specific.
> Graham
The problems I was seeing the other day were repeatable both using strictly psycopg1 (in fact, that was where I first saw the server error). The issue wrt Decimal is that the Django svn version uses Decimal and older versions use float, but because psycopg is "cached" it will convert it into the type that is initialized. In other words, its not psycopg itself - its the fact that it isn't re-initialized between requests.
The recommendation I would make for now is that applications using new Django builds should use psycopg2 and applications using older builds should use psycopg1. This way there are two executables that don't clash in any way. I have done this myself and it eliminated all the problems I had with float/Decimal compatibility & Unicode.
oggie rob wrote: >>> does this also affect psycopg1? >> Looking at psycopg1 code, the answer would appear to be that it is not >> affected. Changes to support Decimal appear to be psycopg2 specific.
>> Graham
> The problems I was seeing the other day were repeatable both using > strictly psycopg1 (in fact, that was where I first saw the server > error). The issue wrt Decimal is that the Django svn version uses > Decimal and older versions use float, but because psycopg is "cached" > it will convert it into the type that is initialized. In other words, > its not psycopg itself - its the fact that it isn't re-initialized > between requests.
> The recommendation I would make for now is that applications using new > Django builds should use psycopg2 and applications using older builds > should use psycopg1. This way there are two executables that don't > clash in any way. I have done this myself and it eliminated all the > problems I had with float/Decimal compatibility & Unicode.
> -rob
Forgot to add - this is strictly a temporary solution. If there are any other changes to the backend, this may not work with older versions anymore.
> > Looking at psycopg1 code, the answer would appear to be that it is not > > affected. Changes to support Decimal appear to be psycopg2 specific.
> > Graham
> The problems I was seeing the other day were repeatable both using > strictly psycopg1 (in fact, that was where I first saw the server > error). The issue wrt Decimal is that the Django svn version uses > Decimal and older versions use float, but because psycopg is "cached" > it will convert it into the type that is initialized. In other words, > its not psycopg itself - its the fact that it isn't re-initialized > between requests.
> The recommendation I would make for now is that applications using new > Django builds should use psycopg2 and applications using older builds > should use psycopg1. This way there are two executables that don't > clash in any way. I have done this myself and it eliminated all the > problems I had with float/Decimal compatibility & Unicode.
Hmmm, I'm not sure how psycopg1 in isolation could be a cause of problems as it doesn't anywhere in its source code support the new 'decimal.Decimal' type. Thus, the exact same problem as identified so far with pyscopg2 couldn't be what was being encountered with psycopg1.
Looking at the code for psycopg2 again though, it is possible that psycopg2 makes similar mistakes in how it deals with the mxDateTime module as well, as it imports that from module initialiser and caches stuff from it as well. In that case though, the type objects are implemented in C code, unlike 'decimal.Decimal', so that they are shared between sub interpreters I didn't think would cause a problem. How psycopg1 handles mxDateTime is a bit different, but presuming it wouldn't be different.
One problem that could arise is if, even in different sub interpreters, two Django instances separately tried to use version 1 and 2 of pyscopg. This is because the Python wrappers for one wouldn't match the loaded C extension module since it is named the same for both versions and would only be loaded once for the whole process. The errors in this case would have been quite dramatic though given the probably significant changes between the API of the C extension module between the two versions.
Do you remember the specific Python exceptions you were getting? The Python exceptions which were being encountered when this was originally found were things like isinstance() checks failing because the class type for the instance of a decimal.Decimal didn't match the identity of the decimal.Decimal type from that interpreter, because the instance was creating using the type object from another interpreter. Apart from failure of checks of this sort, the decimal.Decimal instances otherwise worked fine.
Looking at the psycopg code, I know there are other problems with the code, but they would only arise where one was doing strange stuff with destroying and recreating Python sub interpreters, a feature that is only possible in mod_wsgi and is unlikely being used by anyone at this point.
> Hmmm, I'm not sure how psycopg1 in isolation could be a cause of > problems as it doesn't anywhere in its source code support the new > 'decimal.Decimal' type. Thus, the exact same problem as identified so > far with pyscopg2 couldn't be what was being encountered with > psycopg1.
I'm sorry, I should have been clearer. To be honest I would think it were possible to avoid some of these problems by modifying Django somehow, but I haven't looked very closely at how, yet. However I think I do understand how this is happening at present.
Let me describe the situation I was in previously and how I saw problems: - Application "Santa Cruz" built with 0.91 using psycopg1. Running for just under a couple of years and accessed regularly. - Application "Curriculum" built with trunk using psycopg1. Added about 2 weeks ago & accessed infrequently.
The first problem I saw was when I was doing a (python) sum on a FloatField in the Santa Cruz application. It was saying that it couldn't add Float and Decimal properly. From my very cursory look just now at trunk/django/db/backends/postgresql/creation.py, FloatField is registered as a "double precision" type in the database. However, in 0.91, it is listed as a "numeric(%(max_digits)s, % (decimal_places)s)" - which translates into a DecimalField in the trunk version. So if I visit my Curriculum application, then go to Santa Cruz, I get the error about adding Float and Decimal because the introspection function is "initialized" by trunk.
The second problem I saw was more prevalent because it happened much more frequently! After visiting the Curriculum application, a trip to the Santa Cruz application would win me improperly quoted SQL. The reason in that case was because my old 0.91 code wasn't set to translate unicode, and after visiting Curriculum it was returning u'xyz' data instead of 'xyz'. I couldn't even log in!
I figure the root cause of both of these problems is data that is accessed in two interpreters, and is propagated by having two versions that expect different results at that point (i.e. 0.91 FloatField instrospection expects a float, not a Decimal). Sometimes it takes a while to be seen, as in my sum call (which I do about once or twice a month, normally). The thing is most of the role of psycopg seems to be kind of like a Singleton or static method object, so it is not always seen, but is obvious when you are using the combinations I was, or the one you reported on the initd site.
Conversely, I can see how this hasn't been raised as a major problem before now, since the 'static'-like nature of psycopg doesn't normally show any issues. But I don't like to be the guinea pig and find out how extensively this affects what you can do!
> One problem that could arise is if, even in different sub interpreters, > two Django instances separately tried to use version 1 and 2 of pyscopg. > This is because the Python wrappers for one wouldn't match the loaded > C extension module since it is named the same for both versions and > would only be loaded once for the whole process.
Are you sure this would happen? If that were the case I would have expected my box to blow up by now. I wonder if the extensions are named differently, or if the namespace avoids the problem (i.e. "import psycopg" vs "import psycopg2")? I haven't seen any problems so far (cross fingers).
On Jul 24, 12:15 pm, oggie rob <oz.robhar...@gmail.com> wrote:
> > One problem that could arise is if, even in different sub interpreters, > > two Django instances separately tried to use version 1 and 2 of pyscopg. > > This is because the Python wrappers for one wouldn't match the loaded > > C extension module since it is named the same for both versions and > > would only be loaded once for the whole process.
> Are you sure this would happen? If that were the case I would have > expected my box to blow up by now. I wonder if the extensions are > named differently, or if the namespace avoids the problem (i.e. > "import psycopg" vs "import psycopg2")? I haven't seen any problems so > far (cross fingers).
If the C extension module resides within the package, ie., thus is within parent package namespace, then probably wouldn't occur, ie., both would be loaded. I would though still be concerned if the two different versions of the C extension use the same C symbol names for certain global non static data. If this occurs the dynamic linker may in the second loaded module, actually use global data variables from the first to be loaded module rather than its own. If the way such global data was initialised was different to how expected, it may feasibly cause subtle strange problems with one more of the other.
oggie rob wrote: > - Application "Santa Cruz" built with 0.91 using psycopg1. Running > for just under a couple of years and accessed regularly. > - Application "Curriculum" built with trunk using psycopg1. Added > about 2 weeks ago & accessed infrequently. > Are you sure this would happen? If that were the case I would have > expected my box to blow up by now. I wonder if the extensions are > named differently, or if the namespace avoids the problem (i.e. > "import psycopg" vs "import psycopg2")? I haven't seen any problems so > far (cross fingers).
Yeah I didn't mention this post that *now* Curriculum is using psycopg2 and Santa Cruz is using psycopg1, with no apparent problems (sorry, I did mention it earlier but forgot to clarify). But as Graham says, there may still be some subtle risks that I'm unaware. And of course, it won't work beyond two projects!
|
