SQL Alchemy Out of memory (re-posted)

[Harish Tejwani]

Michael suggested to repost this.
----

We are running on a fairly beefy Linux machine.... and SQLAlchemny
0.6.5

We have started getting getting Out of Memory errors like below (MySQL
runs on same machine).

Any ideas or suggestion why this could be happening and how to go
about identifying root cause of this


Out of memory (Needed 18653760 bytes)
Traceback (most recent call last):
File "/home/topsight/code/topsight-multicore.py", line 36, in
<module>
rules[rule].execute()
File "/home/topsight/code/objects.py", line 1259, in execute
runresult.flag(self.id, update_flagtime=self.update_flagtime)
File "/home/topsight/code/objects.py", line 669, in flag
for e in self.all():
File "/opt/python26/lib/python2.6/site-packages/SQLAlchemy-0.6.5-
py2.6.egg/sqlalchemy/orm/query.py", line 1576, in all
return list(self)
File "/opt/python26/lib/python2.6/site-packages/SQLAlchemy-0.6.5-
py2.6.egg/sqlalchemy/orm/query.py", line 1790, in instances
fetch = cursor.fetchall()
File "/opt/python26/lib/python2.6/site-packages/SQLAlchemy-0.6.5-
py2.6.egg/sqlalchemy/engine/base.py", line 2490, in fetchall
l = self.process_rows(self._fetchall_impl())
File "/opt/python26/lib/python2.6/site-packages/SQLAlchemy-0.6.5-
py2.6.egg/sqlalchemy/engine/base.py", line 2459, in _fetchall_impl
self._non_result()
File "/opt/python26/lib/python2.6/site-packages/SQLAlchemy-0.6.5-
py2.6.egg/sqlalchemy/engine/base.py", line 2464, in _non_result
"This result object does not return rows. "
sqlalchemy.exc.ResourceClosedError: This result object does not return
rows. It has been closed automatically.

[Michael Bayer]

so because its a query that is a SELECT and then MySQLdb is not returning a description, I'm suspecting you might be loading a very large BLOB or a very large result set on a particular query. I'd look into logging what you are sending to the database, and particularly using the debugger or similar at that point if the error is reproducible in a development environment, to see what was passed.

> --
> You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
> To post to this group, send email to sqlal...@googlegroups.com.
> To unsubscribe from this group, send email to sqlalchemy+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
>

[Harish Tejwani]

Turning on logging gives lots of logs, anyway we also thought that
self.all() is returning a very large data-set (we don't have BLOBs)

Are there any limits and is there anyway we can increase the memory
here, maybe of python process itself

Is there any other solution you suggest if select itself very returns
large rows (some times) but is limited size most of time? How to
handle that in application = we would like to keep logic generic

[Michael Bayer]

usually applying limit() to the select is the solution for that