Debugging memory leaks

[Anupama Goparaju]

Hi,

We are trying to debug memory leaks in our flask app that is using SQLAlchemy version 1.3.16. When i was using tracemalloc python library and comparing the memory snapshots,

I found the below lines repetitively appearing in the snapshot differences: Could you please help us in providing us some pointers if we aren't using the lib properly or is there some caching going on that is causing this similar to what is mentioned here - https://giters.com/sqlalchemy/sqlalchemy/issues/7242.

sqlalchemy/util/langhelpers.py:1429: size=3687 KiB (+18.5 KiB), count=3395 (+17), average=1112 B
sqlalchemy/sql/visitors.py:95: size=22.8 KiB (+5376 B), count=147 (+56), average=159 B
sqlalchemy/sql/schema.py:1346: size=2301 KiB (+3944 B), count=10157 (+17), average=232 B

sqlalchemy/sql/schema.py:1345: size=2301 KiB (+3944 B), count=10157 (+17), average=232 B

Thanks,

Anupama

 

[Mike Bayer]

1.3.16 doesn't have a lot of caching going on compared to the 1.4 series.  it's expected that memory in an ORM application will grow for a few iterations as there is some caching of things, but a "leak" is defined as growth that never stops.   you'd want to verify that the growth you see does in fact continue without stopping until it gets into the tens or hundreds of gigs without ever going down, I usually use OS-level memory tools like top to verify this.

the line numbers shown seem to indicate schema objects like Table / Column / ForeignKey instances are being created.  These are usually created for a MetaData collection at the module level and would not be constructed on a continuous basis so you might want to look for patterns where Table objects are being created per-request, something like that.

--

SQLAlchemy -

The Python SQL Toolkit and Object Relational Mapper

 

http://www.sqlalchemy.org/

 

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description.

---

You received this message because you are subscribed to the Google Groups "sqlalchemy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/dc943ea2-e124-4710-a6dc-a0bc1348b85bn%40googlegroups.com.

[Anupama Goparaju]

Thanks.

Do you have a minimalistic example on the lifecycle of table objects? Or best practices of create, use, and destroy them?

How do we identify if they are continuously created and not destroyed?

[Mike Bayer]

Table objects are normally created at the module level and are persistent for the full lifetime of the application.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/8ca5b723-de09-4ae5-b1c0-4bd96ffcd9f9n%40googlegroups.com.