Use of FileField fields causes memory consumption to grow without releasing it
34 views
Skip to first unread message
Gustavo Narea
May 13, 2011, 5:57:58 AM5/13/11
to Django users
Hello,
I've noticed that when you iterate over a query set and use a
FileField instance from each model instance, the memory used by the
process increases every time the loop finishes. This happens on the
Web and command-line interfaces.
Say you have the following model:
"""
class Bug(Model):
patch = FileField(upload_to="whatever")
"""
The following loop will make the process use a lot more memory every
time it's run:
"""
for bug in Bug.objects.all()[:1000]:
print "- %s" % bug.patch.name
"""
But the following loop won't cause memory to grow at all:
"""
for bug in Bug.objects.all()[:1000]:
print "- %s" % bug.__dict__['patch']
"""
The mere act of using "bug.patch" causes the problem, regardless of
what member of "patch" you use (e.g., "name", "url"). The same happens
with ImageField and presumably any other subclass of FileField.
According to the "strace" utility, the process doesn't even try to
access the file in any way. In fact, if I remove the files from the
file system, the problem is still present. I've also checked the
Postgres statement logs and the only query issued looks absolutely
fine.
I'm using Python 2.5, Django 1.1.4 and Ubuntu (it also happens on
Debian boxes). I got DEBUG set to False.
I'll continue to look at this to see what's exactly going on here, but
I'm posting here to see if someone could shed some lights. I'll post
my findings.
Cheers.
- Gustavo Narea.
I've noticed that when you iterate over a query set and use a
FileField instance from each model instance, the memory used by the
process increases every time the loop finishes. This happens on the
Web and command-line interfaces.
Say you have the following model:
"""
class Bug(Model):
patch = FileField(upload_to="whatever")
"""
The following loop will make the process use a lot more memory every
time it's run:
"""
for bug in Bug.objects.all()[:1000]:
print "- %s" % bug.patch.name
"""
But the following loop won't cause memory to grow at all:
"""
for bug in Bug.objects.all()[:1000]:
print "- %s" % bug.__dict__['patch']
"""
The mere act of using "bug.patch" causes the problem, regardless of
what member of "patch" you use (e.g., "name", "url"). The same happens
with ImageField and presumably any other subclass of FileField.
According to the "strace" utility, the process doesn't even try to
access the file in any way. In fact, if I remove the files from the
file system, the problem is still present. I've also checked the
Postgres statement logs and the only query issued looks absolutely
fine.
I'm using Python 2.5, Django 1.1.4 and Ubuntu (it also happens on
Debian boxes). I got DEBUG set to False.
I'll continue to look at this to see what's exactly going on here, but
I'm posting here to see if someone could shed some lights. I'll post
my findings.
Cheers.
- Gustavo Narea.
Gustavo Narea
May 13, 2011, 8:15:20 AM5/13/11
to Django users
OK, I've found that if I use .iterator() on the query set, the amount of
memory used doesn't grow. So I suppose FileField is somehow (indirectly)
referencing something in the retrieved data.
memory used doesn't grow. So I suppose FileField is somehow (indirectly)
referencing something in the retrieved data.
BTW, we've created a fresh Django project with "django-admin
startproject" and the problem is present. It looks like a bug in Django.
--
Gustavo Narea.
Software Developer.
2degrees, Ltd. <http://dev.2degreesnetwork.com/>.
Gustavo Narea
May 13, 2011, 11:16:54 AM5/13/11
to Django users
I've filled a bug report:
http://code.djangoproject.com/ticket/16022
http://code.djangoproject.com/ticket/16022
On 13/05/11 10:57, Gustavo Narea wrote:
Reply all
Reply to author
Forward
0 new messages