how to enlarge memory limits

[Daniel Krenn]

I'm tying to do a length computation, which unfortunately terminates by

TypeError: ECL says: Memory limit reached. Please jump to an outer
pointer, quit program and enlarge the
memory limits before executing the program again.

How can do this enlarging?

FWI: What I have are a lot of (not too small) symbolic expressions in
which I substitute some variables by RIF-elements. I don't see a problem
with physical memory: I work on a 24 GB machine and the python process
uses about 10% (and at least 50% are free). I'm using Sage 6.2.

Daniel

[Dima Pasechnik]

have a look at
http://ecls.sourceforge.net/new-manual/re86.html#table.memory.limits
and
http://trac.sagemath.org/ticket/6772
where one of these limits was removed.
Perhaps removing other limits will help.

[Nils Bruin]

On Friday, August 8, 2014 3:02:03 AM UTC-7, Dima Pasechnik wrote:

have a look at
http://ecls.sourceforge.net/new-manual/re86.html#table.memory.limits
and
http://trac.sagemath.org/ticket/6772
where one of these limits was removed.
Perhaps removing other limits will help.

Actually, the change there did not make it into the maxima_lib interface that calculus uses:

sage: from sage.libs.ecl import *
sage: ecl_eval("(ext:get-limit 'ext:heap-size)")
<ECL: 1073741824>

so doing

sage: import sage.libs.ecl
sage: sage.libs.ecl.ecl_eval("(ext:set-limit 'ext:heap-size 0)")

might solve your problem. I'm not sure if we should do the same. Since maxima_lib runs in the same process as the rest of sage, it might be problematic to let ecl munch away all memory and starve python (for instance when a ulimit has been set). I don't think the boehm garbage collector will ever "give back" once allocated memory to the operating system.

[Daniel Krenn]

Am 2014-08-08 um 17:18 schrieb Nils Bruin:
> On Friday, August 8, 2014 3:02:03 AM UTC-7, Dima Pasechnik wrote:
> so doing
>
> sage: import sage.libs.ecl
> sage: sage.libs.ecl.ecl_eval("(ext:set-limit 'ext:heap-size 0)")
>
> might solve your problem.

Many thanks, this solved my original problem; I can run my computation
for the given parameter.

Now I have a new problem: I want to run it for different parameters, but
the memory is eaten up. It is not freed after each run.
I checked my code and nothing except the result (one RIF element) is
stored; I have removed all caches and even ran the garbage collector,
but no effect. There is still a huge amount of memory blocked (about 16 GB).
How can I free this memory or at least find out which object / software
package takes this memory?

Daniel

[Dima Pasechnik]

valgrind?
there could well be a memory leak somewhere...

[Dima Pasechnik]

On 2014-08-08, Nils Bruin <nbr...@sfu.ca> wrote:
> ------=_Part_203_2052776100.1407511132909
> Content-Type: text/plain; charset=UTF-8

if this is the case then how one would to to reclaim the memory?
And why this is not a memory leak?
Would at least restarting ecl release the memory?
(it's not clear how the latter
can be done; would maxima_calculus.reset() do the job?)

[Nils Bruin]

On Sunday, August 10, 2014 6:31:03 AM UTC-7, Dima Pasechnik wrote:

if this is the case then how one would to to reclaim the memory?
And why this is not a memory leak?

 

It's not necessarily  a leak from the perspective of ecl: the memory obtained from the operating system may well sit there, ready to be filled with new lisp objects. From the python perspective you could call this a memory leak, though. But it's a virtually unavoidable one if you place multiple memory management systems in the same process that are not designed to cooperate: each thinks they have a complete view of the world but they don't. It may well be that ecl/boehm does unmap memory pages every now and again. But really, in our case the same scenarios can arise with pari/gp, libsingular, libgap

Would at least restarting ecl release the memory?

Possibly.
 

(it's not clear how the latter can be done; would maxima_calculus.reset() do the job?)

restarting ecl is currently not a supported operation, because it would be a terribly difficult operation to do correctly: all EclObjects in python memory would somehow have to be "invalidated", a state that currently doesn't even exist for them. That also means *finding* those objects (or knowing that there are none-which will never happen because we have plenty of global EclObjects), which would require an extra expensive facility.

Regardless, of this, symbolics leak anyway:

sage: for i in range(10000000): _=SR.var("t%d"%i)

will steadily eat memory. Probably the pynac symbol table doesn't get garbage collected. Even if that wouldn't leak then

sage: for i in range(10000000): _=maxima_calculus(SR.var("t%d"%i))

and

sage: from sage.interfaces.maxima_lib import sr_to_max
sage: for i in range(10000000): _=sr_to_max(SR.var("t%d"%i))

would leak for sure via slightly different mechanisms, but for the same reason: we need to keep symbols alive in python and in maxima, depending on the lifetime of objects in the other. To garbage collect these, you'd need cycle detection *across the python/ecl boundary*. So currently, symbols that have been transported across to maxima have eternal life. It seems that symbols in SR proper already have eternal life, so it doesn't really matter. Solving this leak would be really quite hard, so we should probably call it a "feature".

The solution is to restart sage early and often. It has good facilities for it, see @parallel.

[Dima Pasechnik]

On 2014-08-10, Nils Bruin <nbr...@sfu.ca> wrote:
> ------=_Part_313_345193229.1407685130291
> Content-Type: text/plain; charset=UTF-8

>
> On Sunday, August 10, 2014 6:31:03 AM UTC-7, Dima Pasechnik wrote:
>>
>> if this is the case then how one would to to reclaim the memory?
>> And why this is not a memory leak?
>
>
> It's not necessarily a leak from the perspective of ecl: the memory
> obtained from the operating system may well sit there, ready to be filled
> with new lisp objects. From the python perspective you could call this a
> memory leak, though. But it's a virtually unavoidable one if you place
> multiple memory management systems in the same process that are not
> designed to cooperate: each thinks they have a complete view of the world
> but they don't. It may well be that ecl/boehm does unmap memory pages every
> now and again. But really, in our case the same scenarios can arise with
> pari/gp, libsingular, libgap

hmm, AFAIK GAP will call its GC now and then, thus libGAP better be
able to cope with this...

>
> Would at least restarting ecl release the memory?
>>
>
> Possibly.

my understanding is that it is possible to register with ECL external pointers to
Lisp objects, so if such a pointer goes away (i.e. the corresponding Python object is gone)
then it will become possible for ECL to collect its garbage related to this Lisp
object (i.e. calling in ECL (system:gc <level>) will do the right thing).

>
>> (it's not clear how the latter can be done; would maxima_calculus.reset()
>> do the job?)
>>
>
> restarting ecl is currently not a supported operation, because it would be
> a terribly difficult operation to do correctly: all EclObjects in python
> memory would somehow have to be "invalidated", a state that currently
> doesn't even exist for them. That also means *finding* those objects (or
> knowing that there are none-which will never happen because we have plenty
> of global EclObjects), which would require an extra expensive facility.

if the registration I talk about above is implemented, I imagine that
one can also create a database of EclObjects to maintain.
Although this is some extra cost, indeed.

[Volker Braun]

Afaik all library interfaces (in particular libGAP) have only Sage->Library references. So Sage objects can keep library objects alive, but not the other way round. Once the Sage objects get collected, the library GC will do its part. Of course you can still have memory leaks, either in Sage or in the library, but there is no insurmountable obstacle.

[Nils Bruin]

On Sunday, August 10, 2014 1:06:44 PM UTC-7, Dima Pasechnik wrote:

hmm, AFAIK GAP will call its GC now and then, thus libGAP better be
able to cope with this...

It will garbarge collect just fine. It's a question on the side of the memory manager whether unmapping memory pages or keeping them around for meeting the next allocation requests is better. Getting pages from and releasing pages to the operating system is generally quite expensive, so memory managers specialized for handling many (small) allocs/deallocs tend to keep an ample amount of memory around (never mind that fragmentation might lead to many barely filled pages, none of which *could* be returned. There's plenty of memory available to the memory manager, but memory allocations via other managers might fail.

This problem is not a memory leak in any one of the memory managers individually, but it's a "memory page mapping" leak, which is only a leak when considered on the level of multiple long-lived processes or when you put multiple memory managers in the same process.

my understanding is that it is possible to register with ECL external pointers to
Lisp objects, so if such a pointer goes away (i.e. the corresponding Python object is gone)
then it will become possible for ECL to collect its garbage related to this Lisp
object (i.e. calling in ECL (system:gc <level>) will do the right thing).

Yes, we do that via a very simple mechanism: All EclObjects register their pointer to a lisp pointer to a global double-linked list within ecl/boehm's scope, and deregister that link upon their __dealloc__. This is an extremely cheap and effective mechanism, which was recommended by the main ecl developer at the time. EclObject by itself doesn't leak. The corresponding lisp objects have their lifetime bounded below by the referring EclObject, but not more than that (and with a mark/sweep collector like boehm, that's all you need).
 

if the registration I talk about above is implemented, I imagine that
one can also create a database of EclObjects to maintain.
Although this is some extra cost, indeed.

Do you mean: also have a way to link "back" from ecl to python objects, i.e., let lisp objects impose lower bounds on lifetimes of certain python objects? Then cycle detection indeed becomes a problem, and doing that across the python/ecl boundary would require very careful coordination of *both* garbage collectors. This would entail a significant rewrite of *both*.
 
The leak with symbols, however, would not even be solved by that. Certainly, currently, symbols in Pynac are immortal anyway, so there are no lifetimes to coordinate. Probably, given the age of Maxima, symbols are effectively immortal there too (maxima symbols are instantiated as lisp symbols, which are normally interned. If you don't manually unintern them, they would likely be immortal), so even solving the Pynac side would probably not solve the pynac/maxima leak, because maxima would still leak. It's rather fundamental: The "state" of computer algebra systems like maxima normally consists of all computations done until this moment (maxima has an unlimited history by default--we disable that). That includes all instantiated symbols.

Finally, the string-based interface means that we fundamentally cannot tie lifetimes of python objects unambiguously to ecl objects: every (interned) symbol is alive, because it can be retrieved via its name. You'd have to be very careful in deciding when a symbol (and all its assumptions and other properties, including its translations to other computer algebra systems) can be discarded. Currently there is nothing to manage, because symbols are immortal anyway. If they weren't, you'd either need to artificially put lower bounds on their existence or be prepared to reinstall things like assumptions on them any time you are translating a symbol from one system to another.

[Nils Bruin]

On Sunday, August 10, 2014 2:13:53 PM UTC-7, Nils Bruin wrote:

maxima symbols are instantiated as lisp symbols, which are normally interned. If you don't manually unintern them, they would likely be immortal)

Indeed, they are immortal. Changing this would mean changing common lisp, or at least hacking it rather badly. Both of the following loops see ecl run out of memory:

for i in [1..10000000]:
   _=maxima_calculus("t%d"%i)
  
for i in [1..10000000]:
   _=EclObject("t%d"%i)

(you'll have to trust me or check yourself that in neither case we're forcing ecl to keep the symbols alive; CL semantics demand ecl to do it)

On the plus side, that means we don't have to bother with solving this issue in the maxima interfaces. It's just intrinsic to maxima.

[Daniel Krenn]

Am 2014-08-10 um 15:30 schrieb Dima Pasechnik:
> On 2014-08-08, Nils Bruin <nbr...@sfu.ca> wrote:
>> ------=_Part_203_2052776100.1407511132909
>> Content-Type: text/plain; charset=UTF-8

> Would at least restarting ecl release the memory?
> (it's not clear how the latter
> can be done; would maxima_calculus.reset() do the job?)

In my case, this (maxima_calculus.reset()) gives the following:

;;;
;;; Detected access to protected memory, also kwown as 'bus or
segmentation fault'.
;;; Jumping to the outermost toplevel prompt
;;;

Traceback (most recent call last):
...
TypeError: ECL says: Detected access to an invalid or protected memory
address.


Daniel

[Nils Bruin]

On Sunday, August 10, 2014 11:16:19 PM UTC-7, Daniel Krenn wrote:

In my case, this (maxima_calculus.reset()) gives the following:

See

http://maxima.sourceforge.net/docs/manual/en/maxima_4.html

for what the "reset()" command does: not that much. The description doesn't make it sound it should help you much with memory leaks. It should also not lead to a segfault (although there may be variables that get reset to values that sage doesn't expect, so it may well be that sage will act funnily when you execute reset). I wasn't able to reproduce the segfault. Would you have a concise code fragment that can reliably reproduce the segfault? It may be worth investigating, because it might point to a bug elsewhere.

[Daniel Krenn]

Am 2014-08-11 um 09:24 schrieb Nils Bruin:
> On Sunday, August 10, 2014 11:16:19 PM UTC-7, Daniel Krenn wrote:
>
> In my case, this (maxima_calculus.reset()) gives the following:
>

> [...] I wasn't able to

> reproduce the segfault. Would you have a concise code fragment that can
> reliably reproduce the segfault? It may be worth investigating, because
> it might point to a bug elsewhere.

Ok, tried again (at least on a partial calculation) but didn't get the
error again. Now I get

[errormsg,?rhs,?lhs,piece,nolabels,display2d,lispdisp,?tr\\-unique,domai\
n,multiplicities,keepfloat,?genvar,features,?odds]

(Since I played around a bit before, I cannot exclude that I did
something weird before the first .reset())

Daniel

[Daniel Krenn]

Am 2014-08-10 um 17:38 schrieb Nils Bruin:
> On Sunday, August 10, 2014 6:31:03 AM UTC-7, Dima Pasechnik wrote:
> (it's not clear how the latter can be done; would
> maxima_calculus.reset() do the job?)

Tried again (since I had troubles before; see other posting), but memory
is not released.

Daniel

[Nils Bruin]

As expected.  If you want to get some idea of what is taking memory at the sage side you could do something like

import gc
from collections import Counter
gc.collect()
pre={id(c) for c in gc.get_objects()}

<your computation>

gc.collect()
post=Counter(type(o) for o in gc.get_objects() if id(o) not in pre)
sorted(post.iteritems(),key=lambda t: t[1])

It'll give you a list of python object types that occur in memory, sorted by count. This might give you an indication of what is leaking on the python side. It may well be nothing, in which case it's probably maxima leaking or ecl keeping freed memory mapped for future use. It's likely that the memory behaviour you're observing isn't easily changed.

[Daniel Krenn]

Am 2014-08-11 um 16:19 schrieb Nils Bruin:
> On Monday, August 11, 2014 4:46:45 AM UTC-7, Daniel Krenn wrote:
> As expected. If you want to get some idea of what is taking memory at
> the sage side you could do something like
>
> import gc
> from collections import Counter
> gc.collect()
> pre={id(c) for c in gc.get_objects()}
>
> <your computation>
>
> gc.collect()
> post=Counter(type(o) for o in gc.get_objects() if id(o) not in pre)
> sorted(post.iteritems(),key=lambda t: t[1])

(<class 'sage.symbolic.function_factory.NewSymbolicFunction'>, 1)
(<type 'builtin_function_or_method'>, 1)
(<type 'set'>, 1)
(<type 'module'>, 1)
(<type 'sage.structure.coerce_actions.LeftModuleAction'>, 1)
(<type 'sage.misc.function_mangling.ArgumentFixer'>, 1)
(<type 'sage.rings.real_double.ToRDF'>, 1)
(<class 'sage.symbolic.function_factory.NewSymbolicFunction'>, 1)
(<type 'sage.rings.real_mpfr.double_toRR'>, 2)
(<type 'sage.structure.coerce_maps.NamedConvertMap'>, 2)
(<type 'sage.categories.morphism.CallMorphism'>, 3)
(<type 'type'>, 3)
(<type 'frame'>, 4)
(<type 'sage.structure.coerce_maps.DefaultConvertMap_unique'>, 4)
(<type 'getset_descriptor'>, 4)
(<type 'staticmethod'>, 7)
(<class 'sage.structure.dynamic_class.DynamicMetaclass'>, 7)
(<type 'sage.misc.constant_function.ConstantFunction'>, 12)
(<type 'list'>, 12)
(<type 'function'>, 16)
(<type 'dict'>, 16)
(<type 'cell'>, 24)
(<class 'weakref.KeyedRef'>, 147)
(<type 'weakref'>, 151)
(<type 'sage.rings.complex_interval.ComplexIntervalFieldElement'>, 2175)
(<type 'sage.rings.real_mpfi.RealIntervalFieldElement'>, 8563)
(<type 'sage.symbolic.expression.Expression'>, 28257)
(<type 'tuple'>, 38713)
(<type 'sage.rings.rational.Rational'>, 3555901)

> It'll give you a list of python object types that occur in memory,
> sorted by count. This might give you an indication of what is leaking on
> the python side. It may well be nothing, in which case it's probably
> maxima leaking or ecl keeping freed memory mapped for future use. It's
> likely that the memory behaviour you're observing isn't easily changed.

It doesn't look like the results above help (but maybe I just interpret
them wrongly)

Daniel

[Nils Bruin]

On Tuesday, August 12, 2014 7:29:01 AM UTC-7, Daniel Krenn wrote:

(<type 'sage.rings.complex_interval.ComplexIntervalFieldElement'>, 2175)
(<type 'sage.rings.real_mpfi.RealIntervalFieldElement'>, 8563)

Numbers of course depend on the context, but this doesn't look alarming. You can probably check yourself if you expect the number of RIF and CIF elements to be of this order. They certainly wouldn't keep anything in ECL alive.

(<type 'sage.symbolic.expression.Expression'>, 28257)

This could be a large number (although expressions are recursive data structures, so one complicated expression can cause a lot of them to exist). Expressions could keep things in ecl alive, but there would have to be EclObjects in memory for that and you're finding none. This suggests that anything you're sending over to maxima gets sent there via the strings-based interface.
 

(<type 'tuple'>, 38713)
(<type 'sage.rings.rational.Rational'>, 3555901)

that is a rather huge number. Are you keeping a large number of tuples with rational numbers in memory somewhere (roughly 100 numbers per tuple on average)? If not, then you may want to look where these are coming from.

It doesn't look like the results above help (but maybe I just interpret
them wrongly)

Well, it excludes EclObject, so that is telling us that any "leaking" to maxima would have to occur strings-based. The strings-based maxima interface uses temporary symbols to tie translated results to in maxima. As we know, generating symbols in common lisp almost always causes a memory leak in that the symbol names get interned and never forgotten, this would be the most likely reason to have leaking behaviour. We can find this out by defining:

maxima_calculus(1) # make sure maxima_calculus is defined
from sage.libs.ecl import ecl_eval
 def ecl_symbols():
    return set(ecl_eval("""(let ((lst ())) (do-all-symbols (s lst) (push s lst)) lst)"""))

Now let's do some tests to see if we're collecting any symbols:

A=ecl_symbols()
for i in range(10):
   _= integrate(sin(x),1,10)
   B = ecl_symbols()
   print B.difference(A)
   A= B

The above works fine: we get some symbols that do get defined initially, but once they exist, no new symbols arise.

for i in range(10):
   _= sin(x).simplify()
   B = ecl_symbols()
   print B.difference(A)
   A= B

Here we see the culprit! "simplify" still uses the strings-based interface and it is creating a new "temporary" symbol every time: $sage6, $sage7 etc. This leaks.

To plug this, we should either try and reuse these symbols or dive into CL to see if we can really unintern them (which is going to be the same problem, because we'd have to know when it's safe to unintern, i.e., safe to recycle.

Either that or someone sits down and finally removes the strings-based uses from our maxima_lib interface.

So, my guess is that you're just making a lot of calls to maxima that end up using the strings-based interface, rather than the binary interface, and that the memory blow-up you're experiencing is the accumulation of (unbound) `$sage...` symbols. This is certainly something that would always happen, albeit slowly. However, you could of course be running into a different leak as well, so it may be worth playing around with the above yourself a bit.

[Nils Bruin]

On Tuesday, August 12, 2014 7:29:01 AM UTC-7, Daniel Krenn wrote:

It doesn't look like the results above help (but maybe I just interpret
them wrongly)

Possible fix now on:  http://trac.sagemath.org/ticket/16809

[Dima Pasechnik]

On 2014-08-12, Nils Bruin <nbr...@sfu.ca> wrote:
> ------=_Part_5256_754630738.1407883007654
> Content-Type: text/plain; charset=UTF-8

perhaps if this still doesn't help one can call ECL's garbade collector:
sage.libs.ecl.ecl_eval("(system:gc 3)")

(3 is the most expensive options, 2 less expensive, 1 is the cheapest, according to
http://ecls.sourceforge.net/ecl/The-garbage-collector.html)

although this might collect too much, as the registration of CL objects we talked about
in this thread is not implemented, right?



>

[Daniel Krenn]

Am 2014-08-13 um 08:18 schrieb Dima Pasechnik:
> On 2014-08-12, Nils Bruin <nbr...@sfu.ca> wrote:
>> ------=_Part_5256_754630738.1407883007654
>> Content-Type: text/plain; charset=UTF-8
>>
>> On Tuesday, August 12, 2014 7:29:01 AM UTC-7, Daniel Krenn wrote:
>>>
>>>
>>> It doesn't look like the results above help (but maybe I just interpret
>>> them wrongly)
>>>
>>> Possible fix now on: http://trac.sagemath.org/ticket/16809
>
> perhaps if this still doesn't help one can call ECL's garbade collector:
> sage.libs.ecl.ecl_eval("(system:gc 3)")
>

> although this might collect too much, as the registration of CL objects we talked about
> in this thread is not implemented, right?

Tried, but it didn't release the memory (still 3 GB or so blocked)

Daniel

[Daniel Krenn]

Am 2014-08-12 um 17:42 schrieb Nils Bruin:
> On Tuesday, August 12, 2014 7:29:01 AM UTC-7, Daniel Krenn wrote:
> (<type 'sage.symbolic.expression.Expression'>, 28257)
>
>
> This could be a large number (although expressions are recursive data
> structures, so one complicated expression can cause a lot of them to
> exist).

I'm using a lot of symbolic expressions, so this does not worry me.

> Expressions could keep things in ecl alive, but there would have
> to be EclObjects in memory for that and you're finding none. This
> suggests that anything you're sending over to maxima gets sent there via
> the strings-based interface.

Ok.

> (<type 'tuple'>, 38713)
> (<type 'sage.rings.rational.Rational'>, 3555901)
>
> that is a rather huge number. Are you keeping a large number of tuples
> with rational numbers in memory somewhere (roughly 100 numbers per tuple
> on average)? If not, then you may want to look where these are coming from.

Actually, no. I don't use rationals much. I have to check the code very
carefully to see, where they might be hidden.
On the other hand, my calculation uses about 3.5 Gigabytes of Memory, so
this would mean about 1 Kilobyte per rational, which are quite large
numbers (and I'm not sure, I'm using that large numbers). But as I said,
I will check.

> [...] So, my guess is that you're just making a lot of calls to maxima that

> end up using the strings-based interface, rather than the binary
> interface, and that the memory blow-up you're experiencing is the
> accumulation of (unbound) `$sage...` symbols. This is certainly
> something that would always happen, albeit slowly. However, you could of
> course be running into a different leak as well, so it may be worth
> playing around with the above yourself a bit.

I will (answers follow, but maybe only in two weeks or so...)

Thanks for your input.

Daniel