[Django] #27066: Possible Memory leak while evaluating a QuerySet
Django
-------------------------------------+-------------------------------------
Reporter: RamezIssac | Owner: nobody
Type: Uncategorized | Status: new
Component: Database layer | Version: 1.8
(models, ORM) | Keywords: queryset evaluation
Severity: Normal | memory leak
Triage Stage: Unreviewed | Has patch: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
I discovered this as i have a 150K rows QuerySet,
I checked its existence by the not efficient
{{{
if queryset:
pass
}}}
However, as soon as execution hits this line, memory consumption jumps big
time and it never goes back to normal unless uwsgi process gets restarted.
This information has been noticed using htop showing the relevant uwsgi
process.
Using query.exists() is working perfectly good.
--
Ticket URL: <https://code.djangoproject.com/ticket/27066>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: RamezIssac | Owner: nobody
Type: Uncategorized | Status: new
Component: Database layer | Version: 1.8
(models, ORM) |
Keywords: queryset evaluation | Triage Stage:
memory leak | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: => 0
* needs_tests: => 0
* needs_docs: => 0
Comment:
As [https://docs.djangoproject.com/en/stable/ref/models/querysets/#when-
querysets-are-evaluated the documentation says]:
Testing a `QuerySet` in a boolean context, such as using `bool()`, `or`,
`and` or an `if` statement, will cause the query to be executed. If there
is at least one result, the `QuerySet` is `True`, otherwise `False`. If
you only want to determine if at least one result exists (and don’t need
the actual objects), it’s more efficient to use `exists()`.
Can you provide more details if that doesn't address your concerns?
--
Ticket URL: <https://code.djangoproject.com/ticket/27066#comment:1>
Django
-------------------------------------+-------------------------------------
Reporter: RamezIssac | Owner: nobody
Type: Uncategorized | Status: new
Component: Database layer | Version: 1.8
(models, ORM) |
Keywords: queryset evaluation | Triage Stage:
memory leak | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by RamezIssac):
Thank you dear time for your quick reply,
What i'm referring to here is when i made the mistake of using
`bool(queryset)` , the server memory jumped and '''never''' returned back
to normal even after the function exit.
Something is not right here.
The only way to bring the memory back to normal was by restarting the
uwsgi process running the django instance.
It's not about that the line `bool(queryset)` takes longer time to
execute, or is heavy on memory;
But this line have a ''long lasting side effect'' on server memory
consumption.
--
Ticket URL: <https://code.djangoproject.com/ticket/27066#comment:2>
Django
-------------------------------------+-------------------------------------
Reporter: RamezIssac | Owner: nobody
Type: Uncategorized | Status: new
Component: Database layer | Version: 1.8
(models, ORM) |
Keywords: queryset evaluation | Triage Stage:
memory leak | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by timgraham):
I'm not intimately familiar with memory management in Python but unless
you can provide details as to why, I'm doubtful that Django is at fault
for this behavior. Maybe this post about [http://chase-
seibert.github.io/blog/2013/08/03/diagnosing-memory-leaks-python.html
Diagnosing Memory "Leaks" in Python] helps.
--
Ticket URL: <https://code.djangoproject.com/ticket/27066#comment:3>
Django
-------------------------------------+-------------------------------------
Reporter: RamezIssac | Owner: nobody
Type: Uncategorized | Status: new
Component: Database layer | Version: 1.8
(models, ORM) |
Keywords: queryset evaluation | Triage Stage:
memory leak | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by claudep):
You don't run with DEBUG=True, right?
--
Ticket URL: <https://code.djangoproject.com/ticket/27066#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: RamezIssac | Owner: nobody
Type: Uncategorized | Status: new
Component: Database layer | Version: 1.8
(models, ORM) |
Keywords: queryset evaluation | Triage Stage:
memory leak | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by RamezIssac):
@claudep Lol Nope ! :D
@Tim I'll review the document you attached and get back to you with more
detailed/insightful information.
--
Ticket URL: <https://code.djangoproject.com/ticket/27066#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: RamezIssac | Owner: nobody
Component: Database layer | Version: 1.8
(models, ORM) |
Keywords: queryset evaluation | Triage Stage:
memory leak | Unreviewed
Has patch: 0 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => needsinfo
--
Ticket URL: <https://code.djangoproject.com/ticket/27066#comment:6>