157 views

Skip to first unread message

Feb 29, 2012, 9:30:47 AM2/29/12

to Sage devel

Hello everybody !!!

There has been a bug report [1, 2] on "ask Sage" about a memory leak with CPLEX. I checked, and it is True. But I do not think I know how to patch that.

Here is the thing : the dominating_set function (among many others) uses Linear Programming, and so creates a LP object when it is called. The LP object *should* disappear by itself when the method returns its result, but I observed a STRANGE behaviour...

sage: graphs.PetersenGraph().dominating_set()

IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: # At this moment I just *do nothing* for a few seconds.....

sage: graphs.PetersenGraph().dominating_set()

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: exit

Exiting Sage (CPU time 0m0.27s, Wall time 0m13.47s).

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

Well. With this kind of behaviour, no wonder a loops ends up eating a lot of memory, but then I do not know what to do anymore... Should I call "del" explicitely at the end of each function ? Thank you for your help O_o;;;

Nathann

[1] http://ask.sagemath.org/question/1170/memory-blowup-with-milp

[2] http://ask.sagemath.org/question/1191/memory-blowup-2

There has been a bug report [1, 2] on "ask Sage" about a memory leak with CPLEX. I checked, and it is True. But I do not think I know how to patch that.

Here is the thing : the dominating_set function (among many others) uses Linear Programming, and so creates a LP object when it is called. The LP object *should* disappear by itself when the method returns its result, but I observed a STRANGE behaviour...

sage: graphs.PetersenGraph().dominating_set()

IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: # At this moment I just *do nothing* for a few seconds.....

sage: graphs.PetersenGraph().dominating_set()

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: graphs.PetersenGraph().dominating_set()

[1, 4, 5]

sage: exit

Exiting Sage (CPU time 0m0.27s, Wall time 0m13.47s).

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

__dealloc__ called !

Well. With this kind of behaviour, no wonder a loops ends up eating a lot of memory, but then I do not know what to do anymore... Should I call "del" explicitely at the end of each function ? Thank you for your help O_o;;;

Nathann

[1] http://ask.sagemath.org/question/1170/memory-blowup-with-milp

[2] http://ask.sagemath.org/question/1191/memory-blowup-2

Feb 29, 2012, 12:55:07 PM2/29/12

to sage-...@googlegroups.com

Python uses a garbage collector to reclaim memory. Objects don't die just because they fall out of scope, they will be around until the next GC cycle.

You can force the garbage collector with

import gc

gc.collect()

but that is a rather expensive operation so you don't want to do call it all the time.

but that is a rather expensive operation so you don't want to do call it all the time.

Feb 29, 2012, 1:02:14 PM2/29/12

to sage-...@googlegroups.com

On Wed, Feb 29, 2012 at 9:55 AM, Volker Braun <vbrau...@gmail.com> wrote:

> Python uses a garbage collector to reclaim memory. Objects don't die just

> because they fall out of scope, they will be around until the next GC cycle.

> Python uses a garbage collector to reclaim memory. Objects don't die just

> because they fall out of scope, they will be around until the next GC cycle.

Minor note: At least when using Cython, sometimes objects will be

freed when they fall out of scope if they are not part of a circular

reference cycle. It is useful to look at

python-2.7.2.p1/src/Include/object.h

to see what Py_DECREF actually does.

> --

> To post to this group, send an email to sage-...@googlegroups.com

> To unsubscribe from this group, send an email to

> sage-devel+...@googlegroups.com

> For more options, visit this group at

> http://groups.google.com/group/sage-devel

> URL: http://www.sagemath.org

--

William Stein

Professor of Mathematics

University of Washington

http://wstein.org

Feb 29, 2012, 3:25:45 PM2/29/12

to sage-...@googlegroups.com

In gmane.comp.mathematics.sage.devel, you wrote:

> ------=_Part_115_6111097.1330538107790

> Content-Type: text/plain; charset=ISO-8859-1

>

> Python uses a garbage collector to reclaim memory. Objects don't die just

> because they fall out of scope, they will be around until the next GC cycle.

>

> You can force the garbage collector with

>

> import gc

> gc.collect()

>

> but that is a rather expensive operation so you don't want to do call it

> all the time.

> ------=_Part_115_6111097.1330538107790

> Content-Type: text/plain; charset=ISO-8859-1

>

> Python uses a garbage collector to reclaim memory. Objects don't die just

> because they fall out of scope, they will be around until the next GC cycle.

>

> You can force the garbage collector with

>

> import gc

> gc.collect()

>

> but that is a rather expensive operation so you don't want to do call it

> all the time.

to distinguish a real memory leak (which is quite possible with Cython/C/C++) from a stuff that

is "good" recyclible garbage, you might try using

get_memory_usage()

I'd run graphs.PetersenGraph().dominating_set() in a loop, where I'd

call gc.collect() followed up by print get_memory_usage(),

and see if the latter grows (sometimes one need like 10 or 100 iterations to

see difference).

(That's how we dug up leaks in mpmath Cython interface)

HTH,

Dima

> --

> To post to this group, send an email to sage-...@googlegroups.com

> To unsubscribe from this group, send an email to sage-devel+...@googlegroups.com

> For more options, visit this group at http://groups.google.com/group/sage-devel

> URL: http://www.sagemath.org

>

> ------=_Part_115_6111097.1330538107790

> Content-Type: text/html; charset=ISO-8859-1

> Content-Transfer-Encoding: quoted-printable

>

> Python uses a garbage collector to reclaim memory. Objects don't die just b=

> ecause they fall out of scope, they will be around until the next GC cycle.=

> <div><br></div><div>You can force the garbage collector with</div><div><br>=

> </div><div>import gc</div><div>gc.collect()<br><div><br>but that is a rathe=

> r expensive operation so you don't want to do call it all the time.</div><d=

> iv><br></div><div><br></div><div><br><br>On Wednesday, February 29, 2012 6:=

> 30:47 AM UTC-8, Nathann Cohen wrote:<blockquote class=3D"gmail_quote" style=

> =3D"margin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: =

> 1ex;">Hello everybody !!!<br><br>There has been a bug report [1, 2] on "ask=

> Sage" about a memory leak with CPLEX. I checked, and it is True. But I do =

> not think I know how to patch that.<br><br>Here is the thing : the dominati=

> ng_set function (among many others) uses Linear Programming, and so creates=

> a LP object when it is called. The LP object *should* disappear by itself =

> when the method returns its result, but I observed a STRANGE behaviour...<b=

> r>

> <br>sage: graphs.PetersenGraph().<wbr>dominating_set()<br>IBM ILOG License =

> Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing=

> CPLEX 12 with option(s): "e m b q ".<br>[1, 4, 5]<br>sage: graphs.Petersen=

> Graph().<wbr>dominating_set()<br>

> [1, 4, 5]<br>sage: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5=

> ]<br>sage: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5]<br>sag=

> e: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5]<br>sage: graph=

> s.PetersenGraph().<wbr>dominating_set()<br>

> [1, 4, 5]<br>sage: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5=

> ]<br>sage: # At this moment I just *do nothing* for a few seconds.....<br>s=

> age: graphs.PetersenGraph().<wbr>dominating_set()<br>__dealloc__ called !<b=

> r>__dealloc__ called !<br>

> __dealloc__ called !<br>__dealloc__ called !<br>__dealloc__ called !<br>__d=

> ealloc__ called !<br>__dealloc__ called !<br>[1, 4, 5]<br>sage: graphs.Pete=

> rsenGraph().<wbr>dominating_set()<br>[1, 4, 5]<br>sage: graphs.PetersenGrap=

> h().<wbr>dominating_set()<br>

> [1, 4, 5]<br>sage: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5=

> ]<br>sage: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5]<br>sag=

> e: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5]<br>sage: graph=

> s.PetersenGraph().<wbr>dominating_set()<br>

> [1, 4, 5]<br>sage: graphs.PetersenGraph().<wbr>dominating_set()<br>[1, 4, 5=

> ]<br>sage: exit<br>Exiting Sage (CPU time 0m0.27s, Wall time 0m13.47s).<br>=

> __dealloc__ called !<br>__dealloc__ called !<br>__dealloc__ called !<br>__d=

> ealloc__ called !<br>

> __dealloc__ called !<br>__dealloc__ called !<br>__dealloc__ called !<br>__d=

> ealloc__ called !<br><br>Well. With this kind of behaviour, no wonder a loo=

> ps ends up eating a lot of memory, but then I do not know what to do anymor=

> e... Should I call "del" explicitely at the end of each function ? Thank yo=

> u for your help O_o;;;<br>

> <br>Nathann<br><br>[1] <a href=3D"http://ask.sagemath.org/question/1170/mem=

> ory-blowup-with-milp" target=3D"_blank">http://ask.sagemath.org/<wbr>questi=

> on/1170/memory-blowup-<wbr>with-milp</a><br>[2] <a href=3D"http://ask.sagem=

> ath.org/question/1191/memory-blowup-2" target=3D"_blank">http://ask.sagemat=

> h.org/<wbr>question/1191/memory-blowup-2</a>

> </blockquote></div></div>

>

> <p></p>

>

> -- <br />

> To post to this group, send an email to sage-...@googlegroups.com<br />

> To unsubscribe from this group, send an email to sage-devel+unsubscribe@goo=

> glegroups.com<br />

> For more options, visit this group at <a href=3D"http://groups.google.com/g=

> roup/sage-devel">http://groups.google.com/group/sage-devel</a><br />

> URL: <a href=3D"http://www.sagemath.org">http://www.sagemath.org</a><br />

>

> ------=_Part_115_6111097.1330538107790--

>

Mar 1, 2012, 2:57:05 PM3/1/12

to sage-...@googlegroups.com

Thank you for your answers ! I forward the result to ask.sage.

But this sure is something that I will meet again sooner or later :-)

Nathann

Mar 2, 2012, 2:45:48 AM3/2/12

to sage-...@googlegroups.com

On Wed, Feb 29, 2012 at 10:02 AM, William Stein <wst...@gmail.com> wrote:

> On Wed, Feb 29, 2012 at 9:55 AM, Volker Braun <vbrau...@gmail.com> wrote:

>> Python uses a garbage collector to reclaim memory. Objects don't die just

>> because they fall out of scope, they will be around until the next GC cycle.

>

> Minor note: At least when using Cython, sometimes objects will be

> freed when they fall out of scope if they are not part of a circular

> reference cycle. It is useful to look at

>

> python-2.7.2.p1/src/Include/object.h

>

> to see what Py_DECREF actually does.

> On Wed, Feb 29, 2012 at 9:55 AM, Volker Braun <vbrau...@gmail.com> wrote:

>> Python uses a garbage collector to reclaim memory. Objects don't die just

>> because they fall out of scope, they will be around until the next GC cycle.

>

> Minor note: At least when using Cython, sometimes objects will be

> freed when they fall out of scope if they are not part of a circular

> reference cycle. It is useful to look at

>

> python-2.7.2.p1/src/Include/object.h

>

> to see what Py_DECREF actually does.

In general (not just Cython) a Python object is deallocated as soon as

it's no longer referenced (via the Py_DECREF macro), which is often as

soon as they fall out of scope. The garbage collector is used to

reclaim cyclically-referenced objects. Note also that ipython (used at

the Sage prompt) also keeps a lot of references to old results around,

making it difficult to do such analysis at the command line.

- Robert

Mar 2, 2012, 4:24:24 AM3/2/12

to sage-...@googlegroups.com

> In general (not just Cython) a Python object is deallocated as soon as

> it's no longer referenced (via the Py_DECREF macro), which is often as

> soon as they fall out of scope. The garbage collector is used to

> reclaim cyclically-referenced objects. Note also that ipython (used at

> the Sage prompt) also keeps a lot of references to old results around,

> making it difficult to do such analysis at the command line.

> it's no longer referenced (via the Py_DECREF macro), which is often as

> soon as they fall out of scope. The garbage collector is used to

> reclaim cyclically-referenced objects. Note also that ipython (used at

> the Sage prompt) also keeps a lot of references to old results around,

> making it difficult to do such analysis at the command line.

Hmmm.... So you mean that the object that is only used inside of the

function and does not get deallocated may be cyclically referenced ?

Is there any way to know who has a reference toward him ?

Hey !!!! Actually, you are right !! When you are using an Linear

Program in Sage you also define variables ! Variables point toward the

LP object, and the LP object points toward the variables... Is that

the cycle I should get rid of so that the LP object are deallocated as

soon as the function ends ???

But that would be messy.... :-/

Nathann

Mar 2, 2012, 4:34:22 AM3/2/12

to sage-...@googlegroups.com

I think weak references are the typical answer to a situation like this.

http://docs.python.org/library/weakref.html

http://docs.cython.org/src/reference/extension_types.html#weak-referencing

And thanks for working on this. I have a friend that was computing

things like the chromatic number on all graphs of 10 vertices, and it

was crashing because memory usage kept climbing.

Jason

Mar 2, 2012, 4:44:19 AM3/2/12

to sage-...@googlegroups.com

Dear Nathann,

You can have a look at trac tickets #715, #11521, #12313 and #12357 where Simon and I had a look at this kind of problems.

Maybe you can draw some inspiration from them.

Best,

You can have a look at trac tickets #715, #11521, #12313 and #12357 where Simon and I had a look at this kind of problems.

Maybe you can draw some inspiration from them.

Best,

Mar 2, 2012, 5:08:15 AM3/2/12

to sage-...@googlegroups.com

My God !! How could I have found that without you ?? Thank you

sooooooooo much !!! :-)

sooooooooo much !!! :-)

There is a patch available at this address [1] to fix the problem. One

of the two references was nto even used nor useful, so the patch is 10

lines long and avoids this cyclic reference --> the objects are now

deallocated at the right time. Great ! :-)

Thaaaaaank you everybody !! :-)

Nathann

[1] http://trac.sagemath.org/sage_trac/ticket/12616#comment:3

Reply all

Reply to author

Forward

0 new messages

Search

Clear search

Close search

Google apps

Main menu