sqlalchemy gc and memory leak
drakkan
inspecting the gc I see a sqlalchemy memory leak in my application,
here is the output from the gc:
<class 'sqlalchemy.engine.base.Connection'>: 2 -> 3 (+1)
<class 'sqlalchemy.engine.base.RootTransaction'>: 2 -> 3 (+1)
<class 'sqlalchemy.util.LRUCache'>: 1 -> 2 (+1)
<class 'sqlalchemy.util.PopulateDict'>: 2 -> 3 (+1)
<type 'tuple'>: 5987 -> 5990 (+3)
<class 'sqlalchemy.sql.expression._BindParamClause'>: 73 -> 80 (+7)
<type 'dict'>: 8943 -> 8954 (+11)
<class 'sqlalchemy.sql.expression.Insert'>: 1 -> 2 (+1)
<class 'sqlalchemy.util.OrderedDict'>: 180 -> 181 (+1)
<type 'instancemethod'>: 490 -> 487 (-3)
<class 'sqlalchemy.dialects.sqlite.base.SQLiteCompiler'>: 2 -> 3 (+1)
<type 'collections.defaultdict'>: 34 -> 35 (+1)
<type 'weakref'>: 3830 -> 3831 (+1)
<type 'list'>: 2953 -> 2963 (+10)
every time I make some database object a reference is added to
sqlalchemy objects and never released. I'm sure the problem is my
application and not sa, however I would like to know how to force
sqlalchemy to delete objects references.
I'm using these function to query gc:
def gcHistogram(self):
import gc
result = {}
for o in gc.get_objects():
t = type(o)
count = result.get(t, 0)
result[t] = count + 1
print len(result)
return result
def diffHists(self,h1, h2):
for k in h1:
if k in h2:
if h1[k] != h2[k]:
print "%s: %d -> %d (%s%d)" % (
k, h1[k], h2[k], h2[k] > h1[k] and "+" or "", h2[k] - h1[k])
thanks
Nicola
Michael Bayer
SQLA doesn't maintain strong references to anything at the module level. The objects you have above appear to be related to the "compiled cache" used by an individual mapper. This is an LRU-enabled dictionary that holds onto Insert constructs and their compiled form up to approximately 100 elements. The dictionary is associated with a mapper, which in turn is associated with your mapped class. If you dereference your mapped class (and all instances of that class), the mapper and the LRU cache will be gone. You can also say mapper._compiled_cache.clear() which will probably remove the Insert/SQLiteCompiler objects above.
As for Sessions, if you close(), commit(), rollback(), or expire_all() the session object, all user objects inside are weakly referenced by the session (though related objects will have strong references to each other).
>
> thanks
> Nicola
>
> --
> You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
> To post to this group, send email to sqlal...@googlegroups.com.
> To unsubscribe from this group, send email to sqlalchemy+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
>
drakkan
after some other tests I can see some objects released:
<class 'sqlalchemy.util.OrderedDict'>: 239 -> 199 (-40)
<type 'instancemethod'>: 630 -> 531 (-99)
<class 'sqlalchemy.sql.expression.ColumnCollection'>: 147 -> 107 (-40)
<class 'sqlalchemy.sql.util.ClauseAdapter'>: 98 -> 28 (-70)
<class 'sqlalchemy.orm.util._ORMJoin'>: 84 -> 24 (-60)
<type 'weakref'>: 3851 -> 3825 (-26)
<class 'sqlalchemy.util.IdentitySet'>: 14 -> 4 (-10)
<class 'sqlalchemy.sql.util.ColumnAdapter'>: 7 -> 2 (-5)
<class 'sqlalchemy.orm.strategies.EagerLazyOption'>: 42 -> 12 (-30)
<class 'sqlalchemy.sql.util.AnnotatedAlias'>: 14 -> 4 (-10)
<class 'sqlalchemy.sql.compiler._CompileLabel'>: 234 -> 96 (-138)
<class 'sqlalchemy.util.OrderedSet'>: 799 -> 464 (-335)
<class 'sqlalchemy.sql.expression._BinaryExpression'>: 175 -> 115
(-60)
<type 'set'>: 2390 -> 1370 (-1020)
<class 'sqlalchemy.sql.expression._generated_label'>: 542 -> 277
(-265)
<class 'sqlalchemy.dialects.sqlite.base.SQLiteCompiler'>: 11 -> 8 (-3)
<class 'sqlalchemy.engine.base.Connection'>: 6 -> 3 (-3)
<class 'sqlalchemy.engine.base.RootTransaction'>: 6 -> 3 (-3)
<class 'sqlalchemy.util.UniqueAppender'>: 14 -> 4 (-10)
<class 'sqlalchemy.orm.collections.InstrumentedList'>: 14 -> 4 (-10)
<type 'function'>: 11147 -> 11142 (-5)
<class 'sqlalchemy.orm.query.QueryContext'>: 7 -> 2 (-5)
<class 'sqlalchemy.orm.collections.CollectionAdapter'>: 14 -> 4 (-10)
<class 'sqlalchemy.sql.expression.Alias'>: 49 -> 14 (-35)
<class 'sqlalchemy.sql.expression.ClauseList'>: 45 -> 25 (-20)
<class 'sqlalchemy.orm.query.Query'>: 7 -> 2 (-5)
<class 'sqlalchemy.orm.attributes.QueryableAttribute'>: 14 -> 4 (-10)
<class 'sqlalchemy.schema.Column'>: 619 -> 299 (-320)
<type 'list'>: 4208 -> 3304 (-904)
<class 'sqlalchemy.orm.state.InstanceState'>: 24 -> 7 (-17)
<class 'sqlalchemy.orm.query._MapperEntity'>: 7 -> 2 (-5)
<type 'tuple'>: 7058 -> 6291 (-767)
<class 'sqlalchemy.sql.expression.ColumnSet'>: 86 -> 46 (-40)
<class 'sqlalchemy.sql.expression.Select'>: 15 -> 5 (-10)
<type 'dict'>: 10985 -> 9474 (-1511)
<class 'sqlalchemy.orm.properties.Comparator'>: 78 -> 68 (-10)
<class 'sqlalchemy.orm.util.ORMAdapter'>: 42 -> 12 (-30)
<class 'explorsrv.models.samodels.ConfigurazioneQuickCall'>: 10 -> 3
(-7)
<type 'cell'>: 336 -> 331 (-5)
<class 'explorsrv.models.samodels.ConfigurazioneDocking'>: 7 -> 2 (-5)
<class 'sqlalchemy.orm.util.AliasedClass'>: 42 -> 12 (-30)
<class 'sqlalchemy.schema.ForeignKey'>: 87 -> 37 (-50)
<class 'sqlalchemy.util.PopulateDict'>: 60 -> 22 (-38)
for example explorsrv.models.samodels.ConfigurazioneQuickCall is a
related object and after some database activity 7 references were
collected, as you can see from the gc graph above.
after each database task I call a method such this:
@staticmethod
def tryCommitAndClose(session,ritorno=None):
try:
session.commit()
except:
session.rollback()
session.close()
return ritorno
else:
session.close()
return True
and this should release the session resources, so in summary my
sqlalchemy usage should be safe and even if the memory isn't
immediatly released it should not leak, I understand you correctly?
Michael Bayer
When we do memory tests we typically take an application loop we'd like to test and then run it about 50 times, counting the total size of gc, and we then look to make sure that gc isn't growing over time. A "sawtooth" pattern in gc growth is common - meaning gc grows for 6-10 iterations, then it goes down again. This is actually in some cases indpendent of calls to gc.collect(). Newer versions of Pysqlite for example have some internal references that are cleared out periodically which cause this symptom to appear. If you want to try our test fixture the profile function is at http://www.sqlalchemy.org/trac/browser/test/aaa_profiling/test_memusage.py#L32 .