[Django] #16063: Problem with searching in m2m fields in inherited model
Django
----------------------+-------------------------------
Reporter: esizikov | Owner: nobody
Type: Bug | Status: new
Milestone: | Component: contrib.admin
Version: 1.2 | Severity: Normal
Keywords: | Triage Stage: Unreviewed
Has patch: 1 | Easy pickings: 0
----------------------+-------------------------------
Django 1.2.5
Models:
{{{
class Client(models.Model):
name = models.CharField(_('name'), max_length=256)
name2 = models.CharField(_('unofficial or obsolete name'),
max_length=256, blank=True, null=True)
contact_person = models.CharField(_('contact person'), max_length=256,
blank=True, null=True)
...
class ClientOffice(models.Model):
name = models.CharField(_('name'), max_length=256)
name2 = models.CharField(_('unofficial or obsolete name'),
max_length=256, blank=True, null=True)
...
client = models.ForeignKey(Client, verbose_name=_('client'))
...
}}}
and admin options like these:
{{{
class ClientAdmin(admin.ModelAdmin):
search_fields = ('name', 'name2', 'contact_person',
'clientoffice__name', 'clientoffice__name2')
...
}}}
Numbers:
{{{
>>> Client.objects.count()
10907
>>> ClientOffice.objects.count()
16952
}}}
Now, if we try searching for clients in admin by a search query containig
several words (>3), got django/admin stalled.
The problem is going to be that each word in the search query leads to
additional JOIN in final SQL query beacause of {{{ qs = qs.filter(...) }}}
pattern. The attached patch is for Django 1.2.5, but adopting for the
current SVN trunk is trivial.
--
Ticket URL: <https://code.djangoproject.com/ticket/16063>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Type: Bug | Status: new
Milestone: | Component: contrib.admin
Version: 1.2 | Severity: Normal
Patch needs improvement: 0 | Easy pickings: 0
--------------------------------------+-------------------------------
Changes (by julien):
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
This seems related to #13902 and #15819. Are you able to test if this gets
fixed by using Django 1.3 or trunk?
--
Ticket URL: <https://code.djangoproject.com/ticket/16063#comment:1>
Django
Type: Bug | Status: new
Milestone: | Component: contrib.admin
Version: 1.2 | Severity: Normal
Triage Stage: Accepted | Has patch: 1
Needs documentation: 0 | Needs tests: 1
Changes (by aaugustin):
* needs_tests: 0 => 1
* stage: Unreviewed => Accepted
Comment:
Looking at the code, the part that is modified by the patch is not
modified in current trunk. The problematic pattern is still there:
{{{
for bit in self.query.split():
...
qs = qs.filter(...)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/16063#comment:2>
Django
Reporter: esizikov | Owner: nobody
Type: | Status: new
Cleanup/optimization | Component: contrib.admin
Milestone: | Severity: Normal
Version: 1.2 | Keywords:
Resolution: | Has patch: 1
Triage Stage: Accepted | Needs tests: 1
Needs documentation: 0 | Easy pickings: 0
Patch needs improvement: 0 |
-------------------------------------+-------------------------------------
Changes (by aaugustin):
* type: Bug => Cleanup/optimization
--
Ticket URL: <https://code.djangoproject.com/ticket/16063#comment:3>
Django
Reporter: esizikov | Owner: nobody
Cleanup/optimization | Component: contrib.admin
Milestone: | Severity: Normal
Version: 1.2 | Keywords:
Resolution: | Has patch: 1
Triage Stage: Accepted | Needs tests: 1
Needs documentation: 0 | Easy pickings: 0
Patch needs improvement: 0 |
-------------------------------------+-------------------------------------
Comment (by esizikov):
I have checked with Django 1.3 and can now confirm that the problem is
still there. Django's ORM in 1.3 still generates SELECT with JOIN's for
each {{{ qs = qs.filter(...) }}} call, which is going to be made per term
in a search query.
--
Ticket URL: <https://code.djangoproject.com/ticket/16063#comment:4>
Django
Reporter: Evgeny Sizikov | Owner: nobody
Type: Cleanup/optimization | Status: new
Component: contrib.admin | Version: 1.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
--------------------------------------+------------------------------------
Changes (by goblinJoel):
* cc: goblinJoel (added)
Comment:
This is old, but I think it's a good idea. It's still a problem in Django
1.11, and I didn't see any changes in the 2.x code that looked relevant.
It can end up devouring storage space larger than the db itself and suck
up CPU to the point where the database system becomes unresponsive or runs
out of storage.
I'm not sure whether I should be making unilateral edits, so I suggest
renaming to something like:
**Multi-word admin searches hang with foreign keys in search_fields**
Related:
* https://github.com/django/django/pull/7277 with a similar solution
* https://groups.google.com/forum/#!topic/django-developers/V8M49P326dI
thread related to above PR
* https://code.djangoproject.com/ticket/27864 more recent attempt to
mitigate by limiting how many search terms you can use
--
Ticket URL: <https://code.djangoproject.com/ticket/16063#comment:6>
Django
Reporter: Evgeny Sizikov | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Component: contrib.admin | Version: 1.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Jacob Walls):
* owner: nobody => Jacob Walls
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/16063#comment:7>
Django
-------------------------------------+-------------------------------------
Reporter: Evgeny Sizikov | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
--
Ticket URL: <https://code.djangoproject.com/ticket/16063#comment:8>
Django
-------------------------------------+-------------------------------------
Reporter: Evgeny Sizikov | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 1 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak):
#21241 was a duplicate, see for a related discussion.
--
Ticket URL: <https://code.djangoproject.com/ticket/16063#comment:9>
Django
-------------------------------------+-------------------------------------
Reporter: Evgeny Sizikov | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_tests: 1 => 0
Comment:
[https://github.com/django/django/pull/15022 new PR]
[https://groups.google.com/g/django-developers/c/V8M49P326dI mailing list]
--
Ticket URL: <https://code.djangoproject.com/ticket/16063#comment:10>
Django
-------------------------------------+-------------------------------------
Reporter: Evgeny Sizikov | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"948d6d880500bc925e23fde29e0f7ff7cc52cac5" 948d6d88]:
{{{
#!CommitTicketReference repository=""
revision="948d6d880500bc925e23fde29e0f7ff7cc52cac5"
Refs #16063 -- Added tests for searching against multiple related fields
in admin changelist.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/16063#comment:11>
Django
-------------------------------------+-------------------------------------
Reporter: Evgeny Sizikov | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
* needs_docs: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/16063#comment:12>
Django
-------------------------------------+-------------------------------------
Reporter: Evgeny Sizikov | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
* needs_docs: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/16063#comment:13>
Django
-------------------------------------+-------------------------------------
Reporter: Evgeny Sizikov | Owner: Jacob
Type: | Walls
Cleanup/optimization | Status: assigned
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/16063#comment:14>
Django
-------------------------------------+-------------------------------------
Reporter: Evgeny Sizikov | Owner: Jacob
Type: | Walls
Component: contrib.admin | Version: 1.2
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"76ccce64cc3d66cfec075651c3d2239fda747dc2" 76ccce6]:
{{{
#!CommitTicketReference repository=""
revision="76ccce64cc3d66cfec075651c3d2239fda747dc2"
Fixed #16063 -- Adjusted admin changelist searches spanning multi-valued
relationships.
This reduces the likelihood of admin searches issuing queries with
excessive joins.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/16063#comment:15>