Importer memory leak

3 views
Skip to first unread message

Greg Taylor

unread,
Mar 10, 2010, 11:04:50 AM3/10/10
to Django and EVE Online
Some of you may have noticed that the django-eve-db importers sucked
all of your memory while importing. This is due to the fact that
Django caches all SQL queries while DEBUG = True, but the view system
usually purges this cache automatically. Since our importers don't use
views, this purging never happened, so memory usage increased
continuously.

The solution was to manually clear the cache like so:
from django import db
db.reset_queries()

You will notice that db.reset_queries() has made appearances in many
of the importers that have a ton of rows. Right now I've just slapped
them at the end of each iteration in for loops. If this ends up being
a performance killer, we'll look into only resetting queries after X
number of rows.

It's also important to note that a few smaller importers don't have
the query reset statements. In the case of 100 or less rows, it's
generally not really worth even touching.

So, import away without fear of bringing your computer down to a halt.
My memory usage stayed constant while running the importers, and I
hope for the same result from everyone else.

Reply all
Reply to author
Forward
0 new messages