Something in Pyramid is preventing gc on SQLAlchemy objects
Vlad K.
Hi all.
I have a Pyramid bootstrapped script that has to process thousands of
rows (on relatively complex model relationships) per call from command
line. Pyramid is bootstrapped to reuse models and helper functions from
the main web application this script is part of.
The problem I'm having is that in each iteration and/or individual
transaction, the memory keeps growing, seems like SQLAlchemy objects are
not released. A self-sufficient example script is here:
https://gist.github.com/d669e958c54869c69831
Without bootstrapping Pyramid (comment out line 266 and set env=None or
something), the gcdelta is 0 or negative. With Pyramid bootstrapped
(note that nothing else touches the application or request, only
bootstrap is called) the gcdelta is in thousands per 200 iterations.
Fedora 16 x86_64
PostgreSQL 9.1
Pyramid 1.2.1 (except sqlalchemy and psycopg2, all dependencies are
default given with 1.2.1 installation)
SQLAlchemy 0.7.5
Psycopg2 2.4.4
Transaction 1.1.1
Zope.Sqlalchemy 0.6.1
--
.oO V Oo.
Vlad K.
/facepalm
Okay, debugtoolbar does that. If I disable it from config, everything
works fine. My fault. Been hunting for this memory leak for two days
straight and it never occurred to me that even the command line
bootstrapped Pyramid, when using development.ini, will have the debug
toolbar on.
Sorry for wasting your time, folks.
I'm sending this email to the SQLAlchemy list too, where I asked about
this problem, for further reference in case anyone else has the same
problem....
.oO V Oo.
Vlad K.
I've replied to this but sent both to SQLAlchemy list and here, seems it
never got here.
The problem was in the debug toolbar. It never occurred to me that using
a bootstrapped Pyramid from a command line script would use debug
toolbar (because I used development.ini which had it on), and debug
toolbar caused all the SQL objects to remain in memory and never get
gc-collected for the life of the script ("request", more precisely).
So, if anyone finds himself processing large amounts of data from a
Pyramid bootstrapped command line script, be aware of teh debug toolbar.
Thanks,
.oO V Oo.
On 02/23/2012 02:24 PM, Vlad K. wrote:
>