Understanding sqlalchemy memory usage and releasing unused memory

[Manav Goel]

I wanted to get idea about how much memory my sqlalchemy objects are taking.
I used pympler library and used its class tracking function.
I am using declarative base for defining the model and hence tables. I am pretty new to web development and python in general. So some questions may not be entirely due to sqlalchemy but due to my less knowledge about python.

My User model which contain 13 column attributes, 1 association proxy for many to many connections and 6 one to many attributes set using lazy='dynamic'. When I tracked its memory usage only 6 attributes were loaded as others were set deffered and memory used was 1.6 mb.

I tried to do rough calculations and thought if one object was taking this much then loading 100 will take 160mb.
Then I created 50 user objects and then tracked memory and to my surprise total memory consumed was only 2 mb!
What is the reason of this?? Are this objects sharing some base which is is around 1.5 mb and actual object is just in kbs?

I tracked another type of object defined using 5 fields took only about 30kb and another object containing only 2 fields was 90 kb. Why is this different behavior in sizes?

Reading about memory usage and sqlalchemy in this group and other places that once python process accumulates memory then they release only once closed I am little confused.
Does it mean when I have loaded 100 objects and their usage is over, after that its memory will not be released back to system??

Final question expiring an object creates a weak reference and can be collected by gc if not used. Does expunging an object does the same?
mainly how can i make memory taken by object free after usage is over.

Thas quite a lot of questions. Thank you for reading.

Regards,
Manav Goel

[Gunnlaugur Briem]

Hi Manav,

the final question did touch SQLAlchemy: the one about expiring and expunging.

1. "expiring an object creates a weak reference" - this is inaccurate. All object references held by the the session (except for objects whose addition, deletion or attribute change is not yet flushed) are weak references in the first place, see http://docs.sqlalchemy.org/en/latest/orm/session.html#session-attributes and expiring an object just marks its attributes out of date, it does not change the session's reference to the object itself

2. "Does expunging an object do the same" - expunging an object means the session no longer holds a reference (weak or otherwise) to that object. But the reference was weak in the first place (unless the object was in new, dirty or deleted), so expunging neither helps nor hurts in getting the object collected.

Your other questions do not involve SQLAlchemy, and that's why nobody here is answering them. You might find some other group where questions about python basics and process memory management are in scope ... but the One True Way to learn these things is to grit your teeth and google and read. That may be more work than you were hoping, but such is life. :) Also, this talk is good: http://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2011-dude-where-s-my-ram-a-deep-dive-into-how-python-uses-memory-4896725

- Gulli

[Manav Goel]

Hey
Thanks for the answer even after my half researched questions
which I should not have done. :)
Yeah I realised that after posting the question and I dig
deep into python memory issues and have improved knowledge a lot.
In fact objgraph module is a great one for understanding
these issues. It helped in understanding sqlalchemy a lot by drawing
great detailed graphs.

On Feb 7, 3:18 pm, Gunnlaugur Briem <gunnlau...@gmail.com> wrote:
> Hi Manav,
>
> the final question did touch SQLAlchemy: the one about expiring and
> expunging.
>
> 1. "expiring an object creates a weak reference" - this is inaccurate. All
> object references held by the the session (except for objects whose
> addition, deletion or attribute change is not yet flushed) are weak
> references in the first place,

> seehttp://docs.sqlalchemy.org/en/latest/orm/session.html#session-attributes

> and expiring an object just marks its attributes out of date, it does not
> change the session's reference to the object itself
>
> 2. "Does expunging an object do the same" - expunging an object means the
> session no longer holds a reference (weak or otherwise) to that object. But
> the reference was weak in the first place (unless the object was in new,
> dirty or deleted), so expunging neither helps nor hurts in getting the
> object collected.
>
> Your other questions do not involve SQLAlchemy, and that's why nobody here
> is answering them. You might find some other group where questions about
> python basics and process memory management are in scope ... but the One
> True Way to learn these things is to grit your teeth and google and read.
> That may be more work than you were hoping, but such is life. :) Also, this
> talk is

> good:http://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2011-dude-where-s...
>
> - Gulli