Just *when* are objects deallocated ?

[Nathann Cohen]

Hello everybody !!!

There has been a bug report [1, 2] on "ask Sage" about a memory leak with CPLEX. I checked, and it is True. But I do not think I know how to patch that.

Here is the thing : the dominating_set function (among many others) uses Linear Programming, and so creates a LP object when it is called. The LP object *should* disappear by itself when the method returns its result, but I observed a STRANGE behaviour...

sage: graphs.PetersenGraph().dominating_set()
IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
[1, 4, 5]
sage: graphs.PetersenGraph().dominating_set()
[1, 4, 5]
sage: graphs.PetersenGraph().dominating_set()
[1, 4, 5]
sage: graphs.PetersenGraph().dominating_set()
[1, 4, 5]
sage: graphs.PetersenGraph().dominating_set()
[1, 4, 5]
sage: graphs.PetersenGraph().dominating_set()
[1, 4, 5]
sage: graphs.PetersenGraph().dominating_set()
[1, 4, 5]
sage: # At this moment I just *do nothing* for a few seconds.....
sage: graphs.PetersenGraph().dominating_set()
__dealloc__ called !
__dealloc__ called !
__dealloc__ called !
__dealloc__ called !
__dealloc__ called !
__dealloc__ called !
__dealloc__ called !
[1, 4, 5]
sage: graphs.PetersenGraph().dominating_set()
[1, 4, 5]
sage: graphs.PetersenGraph().dominating_set()
[1, 4, 5]
sage: graphs.PetersenGraph().dominating_set()
[1, 4, 5]
sage: graphs.PetersenGraph().dominating_set()
[1, 4, 5]
sage: graphs.PetersenGraph().dominating_set()
[1, 4, 5]
sage: graphs.PetersenGraph().dominating_set()
[1, 4, 5]
sage: graphs.PetersenGraph().dominating_set()
[1, 4, 5]
sage: exit
Exiting Sage (CPU time 0m0.27s, Wall time 0m13.47s).
__dealloc__ called !
__dealloc__ called !
__dealloc__ called !
__dealloc__ called !
__dealloc__ called !
__dealloc__ called !
__dealloc__ called !
__dealloc__ called !

Well. With this kind of behaviour, no wonder a loops ends up eating a lot of memory, but then I do not know what to do anymore... Should I call "del" explicitely at the end of each function ? Thank you for your help O_o;;;

Nathann

[1] http://ask.sagemath.org/question/1170/memory-blowup-with-milp
[2] http://ask.sagemath.org/question/1191/memory-blowup-2

[Volker Braun]

Python uses a garbage collector to reclaim memory. Objects don't die just because they fall out of scope, they will be around until the next GC cycle.

You can force the garbage collector with

import gc

gc.collect()

but that is a rather expensive operation so you don't want to do call it all the time.

[William Stein]

On Wed, Feb 29, 2012 at 9:55 AM, Volker Braun <vbrau...@gmail.com> wrote:
> Python uses a garbage collector to reclaim memory. Objects don't die just
> because they fall out of scope, they will be around until the next GC cycle.

Minor note: At least when using Cython, sometimes objects will be
freed when they fall out of scope if they are not part of a circular
reference cycle. It is useful to look at

python-2.7.2.p1/src/Include/object.h

to see what Py_DECREF actually does.

> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to
> sage-devel+...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org

--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

[Dima Pasechnik]

In gmane.comp.mathematics.sage.devel, you wrote:
> ------=_Part_115_6111097.1330538107790
> Content-Type: text/plain; charset=ISO-8859-1

>
> Python uses a garbage collector to reclaim memory. Objects don't die just
> because they fall out of scope, they will be around until the next GC cycle.
>
> You can force the garbage collector with
>
> import gc
> gc.collect()
>
> but that is a rather expensive operation so you don't want to do call it
> all the time.

to distinguish a real memory leak (which is quite possible with Cython/C/C++) from a stuff that
is "good" recyclible garbage, you might try using
get_memory_usage()

I'd run graphs.PetersenGraph().dominating_set() in a loop, where I'd
call gc.collect() followed up by print get_memory_usage(),
and see if the latter grows (sometimes one need like 10 or 100 iterations to
see difference).

(That's how we dug up leaks in mpmath Cython interface)

HTH,
Dima

> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
>

> ------=_Part_115_6111097.1330538107790
> Content-Type: text/html; charset=ISO-8859-1
> Content-Transfer-Encoding: quoted-printable
>
> Python uses a garbage collector to reclaim memory. Objects don't die just b=
> ecause they fall out of scope, they will be around until the next GC cycle.=
> <div><br></div><div>You can force the garbage collector with</div><div><br>=
> </div><div>import gc</div><div>gc.collect()<br><div><br>but that is a rathe=
> r expensive operation so you don't want to do call it all the time.</div><d=
> iv><br></div><div><br></div><div><br><br>On Wednesday, February 29, 2012 6:=
> 30:47 AM UTC-8, Nathann Cohen wrote:<blockquote class=3D"gmail_quote" style=
> =3D"margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: =
> 1ex;">Hello everybody !!!<br><br>There has been a bug report [1, 2] on "ask=
> Sage" about a memory leak with CPLEX. I checked, and it is True. But I do =
> not think I know how to patch that.<br><br>Here is the thing : the dominati=
> ng_set function (among many others) uses Linear Programming, and so creates=
> a LP object when it is called. The LP object *should* disappear by itself =
> when the method returns its result, but I observed a STRANGE behaviour...<b=
> r>
> <br>sage: graphs.PetersenGraph().<wbr>dominating_set()<br>IBM ILOG License =
> Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing=
> CPLEX 12 with option(s): "e m b q ".<br>[1, 4, 5]<br>sage: graphs.Petersen=
> Graph().<wbr>dominating_set()<br>
> [1, 4, 5]<br>sage: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5=
> ]<br>sage: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5]<br>sag=
> e: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5]<br>sage: graph=
> s.PetersenGraph().<wbr>dominating_set()<br>
> [1, 4, 5]<br>sage: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5=
> ]<br>sage: # At this moment I just *do nothing* for a few seconds.....<br>s=
> age: graphs.PetersenGraph().<wbr>dominating_set()<br>__dealloc__ called !<b=
> r>__dealloc__ called !<br>
> __dealloc__ called !<br>__dealloc__ called !<br>__dealloc__ called !<br>__d=
> ealloc__ called !<br>__dealloc__ called !<br>[1, 4, 5]<br>sage: graphs.Pete=
> rsenGraph().<wbr>dominating_set()<br>[1, 4, 5]<br>sage: graphs.PetersenGrap=
> h().<wbr>dominating_set()<br>
> [1, 4, 5]<br>sage: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5=
> ]<br>sage: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5]<br>sag=
> e: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5]<br>sage: graph=
> s.PetersenGraph().<wbr>dominating_set()<br>
> [1, 4, 5]<br>sage: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5=
> ]<br>sage: exit<br>Exiting Sage (CPU time 0m0.27s, Wall time 0m13.47s).<br>=
> __dealloc__ called !<br>__dealloc__ called !<br>__dealloc__ called !<br>__d=
> ealloc__ called !<br>
> __dealloc__ called !<br>__dealloc__ called !<br>__dealloc__ called !<br>__d=
> ealloc__ called !<br><br>Well. With this kind of behaviour, no wonder a loo=
> ps ends up eating a lot of memory, but then I do not know what to do anymor=
> e... Should I call "del" explicitely at the end of each function ? Thank yo=
> u for your help O_o;;;<br>
> <br>Nathann<br><br>[1] <a href=3D"http://ask.sagemath.org/question/1170/mem=
> ory-blowup-with-milp" target=3D"_blank">http://ask.sagemath.org/<wbr>questi=
> on/1170/memory-blowup-<wbr>with-milp</a><br>[2] <a href=3D"http://ask.sagem=
> ath.org/question/1191/memory-blowup-2" target=3D"_blank">http://ask.sagemat=
> h.org/<wbr>question/1191/memory-blowup-2</a>
> </blockquote></div></div>
>
> <p></p>
>
> -- <br />
> To post to this group, send an email to sage-...@googlegroups.com<br />
> To unsubscribe from this group, send an email to sage-devel+unsubscribe@goo=
> glegroups.com<br />
> For more options, visit this group at <a href=3D"http://groups.google.com/g=
> roup/sage-devel">http://groups.google.com/group/sage-devel</a><br />
> URL: <a href=3D"http://www.sagemath.org">http://www.sagemath.org</a><br />
>
> ------=_Part_115_6111097.1330538107790--
>

[Nathann Cohen]

Thank you for your answers ! I forward the result to ask.sage.

But this sure is something that I will meet again sooner or later :-)

Nathann

[Robert Bradshaw]

On Wed, Feb 29, 2012 at 10:02 AM, William Stein <wst...@gmail.com> wrote:
> On Wed, Feb 29, 2012 at 9:55 AM, Volker Braun <vbrau...@gmail.com> wrote:
>> Python uses a garbage collector to reclaim memory. Objects don't die just
>> because they fall out of scope, they will be around until the next GC cycle.
>
> Minor note:  At least when using Cython, sometimes objects will be
> freed when they fall out of scope if they are not part of a circular
> reference cycle.   It is useful to look at
>
>   python-2.7.2.p1/src/Include/object.h
>
> to see what Py_DECREF actually does.

In general (not just Cython) a Python object is deallocated as soon as
it's no longer referenced (via the Py_DECREF macro), which is often as
soon as they fall out of scope. The garbage collector is used to
reclaim cyclically-referenced objects. Note also that ipython (used at
the Sage prompt) also keeps a lot of references to old results around,
making it difficult to do such analysis at the command line.

- Robert

[Nathann Cohen]

> In general (not just Cython) a Python object is deallocated as soon as
> it's no longer referenced (via the Py_DECREF macro), which is often as
> soon as they fall out of scope. The garbage collector is used to
> reclaim cyclically-referenced objects. Note also that ipython (used at
> the Sage prompt) also keeps a lot of references to old results around,
> making it difficult to do such analysis at the command line.

Hmmm.... So you mean that the object that is only used inside of the
function and does not get deallocated may be cyclically referenced ?
Is there any way to know who has a reference toward him ?

Hey !!!! Actually, you are right !! When you are using an Linear
Program in Sage you also define variables ! Variables point toward the
LP object, and the LP object points toward the variables... Is that
the cycle I should get rid of so that the LP object are deallocated as
soon as the function ends ???

But that would be messy.... :-/

Nathann

[Jason Grout]

I think weak references are the typical answer to a situation like this.

http://docs.python.org/library/weakref.html

http://docs.cython.org/src/reference/extension_types.html#weak-referencing

And thanks for working on this. I have a friend that was computing
things like the chromatic number on all graphs of 10 vertices, and it
was crashing because memory usage kept climbing.

Jason

[Jean-Pierre Flori]

Dear Nathann,

You can have a look at trac tickets #715, #11521, #12313 and #12357 where Simon and I had a look at this kind of problems.
Maybe you can draw some inspiration from them.

Best,

[Nathann Cohen]

My God !! How could I have found that without you ?? Thank you
sooooooooo much !!! :-)

There is a patch available at this address [1] to fix the problem. One
of the two references was nto even used nor useful, so the patch is 10
lines long and avoids this cyclic reference --> the objects are now
deallocated at the right time. Great ! :-)

Thaaaaaank you everybody !! :-)

Nathann

[1] http://trac.sagemath.org/sage_trac/ticket/12616#comment:3