[Django] #20414: Handling of numbers under oracle is slow
Django
----------------------------------------------+----------------------------
Reporter: shai | Owner: shai
Type: Bug | Status: new
Component: Database layer (models, ORM) | Version: master
Severity: Normal | Keywords: oracle
Triage Stage: Unreviewed | performance
Easy pickings: 0 | Has patch: 1
| UI/UX: 0
----------------------------------------------+----------------------------
With current code, for all number columns, the Oracle backend asks
cx_Oracle to return the numbers as strings, and then transforms them back
to numbers on the way out. All this is done because Oracle supports higher
precision (and a wider range of integers) than the Python `float` type;
some values, if they are to be reported accurately, must be returned as
`decimal.Decimal`s, and cx_Oracle doesn't apparently do that on its own.
With cx_Oracle 5, though, it is possible to use a much better approach, on
a per-column (rather than per-row) basis, selecting the right output type
when there is enough information to do so, and making most of the
decisions once-per-query instead of once-per-row.
An original patch for this was written by Ian Kelly; I improved it and
made a pull-request in the stretch towards 1.5, but there was no ticket
for it and so it was never merged. I hope we can finally merge it into
1.6.
There was a little saga of discussions around this (mostly Anssi, Ian and
myself, https://groups.google.com/d/topic/django-
developers/4BNkJyGez9A/discussion). Some other issues are also mentioned
there. But this is the main one.
So now there is a ticket. Things changed, and the pull-request needed
updating. The new one is at https://github.com/django/django/pull/1071.
--
Ticket URL: <https://code.djangoproject.com/ticket/20414>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: shai | Owner: shai
Type: Bug | Status: new
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: oracle performance | Needs documentation: 0
Has patch: 1 | Patch needs improvement: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Changes (by russellm):
* needs_better_patch: => 0
* needs_docs: => 0
* needs_tests: => 0
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/20414#comment:1>
Django
Reporter: shai | Owner: shai
Type: Bug | Status: new
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: oracle performance | Needs documentation: 0
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
Comment:
Newest pull request appears to be
[https://github.com/django/django/pull/1279 here]. It at least needs to be
updated to reflect the fact that it won't be added in 1.6 as currently
documented.
--
Ticket URL: <https://code.djangoproject.com/ticket/20414#comment:2>
Django
Reporter: shai | Owner: shai
Type: Bug | Status: new
(models, ORM) | Resolution:
Severity: Normal | Triage Stage: Accepted
Keywords: oracle performance | Needs documentation: 0
Has patch: 1 | Patch needs improvement: 1
Needs tests: 0 | UI/UX: 0
Easy pickings: 0 |
-------------------------------------+-------------------------------------
Comment (by shai):
Well, I seemed to have botched that branch -- I need to fix it. Until
then, I'll close the PR.
Anyway, later tests I ran showed this to be a lot less promising than I
had thought. The reason is that cx_Oracle, internally, does this return-
number-as-string-and-transform-to-Python-int dance whenever the database
column can hold more than 9 digits; Django's Oracle backend uses
`number(11)` for ints. I'll revisit this when I finally get to meddling
with cx_Oracle internals (this is needed mostly for Python 3 support,
which is currently quite completely broken).
--
Ticket URL: <https://code.djangoproject.com/ticket/20414#comment:3>
Django
Reporter: Shai Berger | Owner: Shai
| Berger
Type: Bug | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: oracle performance | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by felixxm):
I think it's already fixed in
https://github.com/django/django/commit/e06cab260049bb58eafdc4f60ac50a5f3759c38c.
--
Ticket URL: <https://code.djangoproject.com/ticket/20414#comment:4>
Django
Reporter: Shai Berger | Owner: Shai
| Berger
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: oracle performance | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: felixxm (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/20414#comment:5>
Django
Reporter: Shai Berger | Owner: Shai
| Berger
Component: Database layer | Version: 2.0
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: oracle performance | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* version: master => 2.0
* resolution: => fixed
Comment:
Replying to [comment:4 felixxm]:
> I think it's already fixed in
https://github.com/django/django/commit/e06cab260049bb58eafdc4f60ac50a5f3759c38c.
Well, it needed https://github.com/oracle/python-
cx_Oracle/commit/b1d1a93a8749eb40e9a8f76d94b6b2434c84399b first, but yes,
since both are in, I believe the issue should be closed. If there's any
more issues with integers performance on Oracle, they're unrelated to
this.
--
Ticket URL: <https://code.djangoproject.com/ticket/20414#comment:6>