MemoryError when adding a lot of data

[Doug Blank]

I'm filling my Django tables with data through a regular Python
program (not through the browser). After it runs for a few hours, I
get:

10557896: ERROR: gramps.py: line 121: Unhandled exception
Traceback (most recent call last):
...
event = models.Event.objects.get(handle=ref)
File "/usr/lib/python2.6/site-packages/django/db/models/manager.py",
line 120, in get
return self.get_query_set().get(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/django/db/models/query.py",
line 300, in get
num = len(clone)
File "/usr/lib/python2.6/site-packages/django/db/models/query.py",
line 81, in __len__
self._result_cache = list(self.iterator())
File "/usr/lib/python2.6/site-packages/django/db/models/query.py",
line 238, in iterator
for row in self.query.results_iter():
File "/usr/lib/python2.6/site-packages/django/db/models/sql/query.py",
line 287, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/lib/python2.6/site-packages/django/db/models/sql/query.py",
line 2369, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python2.6/site-packages/django/db/backends/util.py",
line 22, in execute
sql = self.db.ops.last_executed_query(self.cursor, sql, params)
File "/usr/lib/python2.6/site-packages/django/db/backends/__init__.py",
line 217, in last_executed_query
return smart_unicode(sql) % u_params
MemoryError

The line that looks suspicious to me is the "self._result_cache =
list(self.iterator())". Does it look like that might be the problem?
I'm not manually doing anything other than adding data, changing data,
and calling .save() (eg, I'm not doing anything with transactions in
my code).

I looked at the chapter on cache from the book:

http://www.djangobook.com/en/beta/chapter14/

but I'm not sure how this is related. Although, if there was a Python
command to turn off caching while importing, that might be something
to try.

Any ideas appreciated,

-Doug

[Doug Blank]

Some additional data:

I'm using Django 1.1 on Fedora11 with sqlite backend. I get the same
kind of spiking of memory usage if I just enter:

>>> Person.objects.all().delete()

Memory usage continues to grow, and it doesn't seem to be able to be
garbage collected. What could cause this? Anything I can set or issue
to make Django clean up/use less memory? I am running out of memory,
just deleting the data! Something seems to be very wrong...

-Doug

[Karen Tracey]

On Fri, Nov 20, 2009 at 6:19 PM, Doug Blank <doug....@gmail.com> wrote:

Some additional data:

I'm using Django 1.1 on Fedora11 with sqlite backend. I get the same
kind of spiking of memory usage if I just enter:

>>> Person.objects.all().delete()

Memory usage continues to grow, and it doesn't seem to be able to be
garbage collected. What could cause this? Anything I can set or issue
to make Django clean up/use less  memory? I am running out of memory,
just deleting the data! Something seems to be very wrong...

Are you running this script with settings that have DEBUG set to True?

See: http://docs.djangoproject.com/en/dev/faq/models/#why-is-django-leaking-memory

Karen

[Doug Blank]

On Fri, Nov 20, 2009 at 10:22 PM, Karen Tracey <kmtr...@gmail.com> wrote:
> On Fri, Nov 20, 2009 at 6:19 PM, Doug Blank <doug....@gmail.com> wrote:
>>
>> Some additional data:
>>
>> I'm using Django 1.1 on Fedora11 with sqlite backend. I get the same
>> kind of spiking of memory usage if I just enter:
>>
>> >>> Person.objects.all().delete()
>>
>> Memory usage continues to grow, and it doesn't seem to be able to be
>> garbage collected. What could cause this? Anything I can set or issue
>> to make Django clean up/use less  memory? I am running out of memory,
>> just deleting the data! Something seems to be very wrong...
>>
>
> Are you running this script with settings that have DEBUG set to True?

I did have DEBUG = False in settings.py, but I also confirmed that it
was the connections that were growing. On further inspection, I found
I had this tucked away in my non-browser, reloadable Python code:

from django.conf import settings
import web.settings as default_settings
try:
settings.configure(default_settings, DEBUG=True)
except RuntimeError:
# already configured; ignore
pass

which was causing part of the problem. It does seem that:

Table.objects.all().delete()

is "leaking" memory (eg, continues to use memory) and is very slow.
I'm trying to find a better (faster, less memory) method similar to
the manage.py command sql_flush...

Thank you very much for the pointer! Part of the problem solved...

-Doug

> See:
> http://docs.djangoproject.com/en/dev/faq/models/#why-is-django-leaking-memory
>
> Karen
ooglegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=.
>

[Doug Blank]

On Sat, Nov 21, 2009 at 7:44 AM, Doug Blank <doug....@gmail.com> wrote:

[snip]

> It does seem that:
>
> Table.objects.all().delete()
>
> is "leaking" memory (eg, continues to use memory) and is very slow.
> I'm trying to find a better (faster, less memory) method similar to
> the manage.py command sql_flush...

Is this a "better" way to flush a table, programmatically?

from django.db import connection, transaction
models = [Person, User, ...]
cursor = connection.cursor()
flush_tables = []
for model in models:
flush_tables.append(model._meta.db_table)
statements = connection.ops.sql_flush(no_style(),
flush_tables,

connection.introspection.sequence_list())
for statement in statements:
cursor.execute(statement)
transaction.commit_unless_managed()

It would be nice if flush() were a method on Model, yes?

-Doug