memory leak

[Vincent Delecroix]

Hello,

Some friend just sent an e-mail to me mentioning a memory leak. Here is
a minimal example

sage: x = polygen(ZZ)
sage: K = NumberField(x**3 - 2, 'cbrt2', embedding=RR(1.2599))
sage: w = K.gen()
sage: import resource
sage: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
180720
sage: for _ in range(10000): test = w > 1
sage: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
183452
sage: for _ in range(10000): test = w > 1
sage: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
184552
sage: for _ in range(10000): test = w > 1
sage: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
185832

The memory usage should *not* grow at all but it does very fast. I am
new to valgrind and was not able to get anything from it. Any help
appreciated.

Vincent

[Christian Nassau]

I also found valgrind not very helpful here, but good old
code-dissection leads me to believe that the problem might originate in
the polynomial evaluation in the _richcmp_ routine in

src/sage/rings/number_field/number_field_element.pyx

That's because the following code shows the same leak:

x = polygen(ZZ)

K = NumberField(x**3 - 2, 'cbrt2', embedding=RR(1.2599))

w = K.gen()
wp = w.polynomial()
app = K._get_embedding_approx(3)
import resource
print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
for _ in range(20000): test = wp(app)
print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
for _ in range(20000): test = wp(app)
print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
for _ in range(20000): test = wp(app)
print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

HTH,
Christian

[Vincent Delecroix]

Thanks Christian!

More specifically, it only concerns the NTL implementation backend. Here
is a more direct example

sage: R = PolynomialRing(ZZ, 'x', implementation='NTL')
sage: x = R.gen()
sage: p = x**2 - 3
sage: for _ in range(10000): a = p(2)
sage: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
171648
sage: for _ in range(10000): a = p(2)
sage: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
172968
sage: for _ in range(10000): a = p(2)
sage: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
174552

Vincent

[Nils Bruin]

On Wednesday, March 23, 2016 at 9:17:58 AM UTC-7, vdelecroix wrote:

Hello,

Some friend just sent an e-mail to me mentioning a memory leak. Here is
a minimal example

sage: x = polygen(ZZ)
sage: K = NumberField(x**3 - 2, 'cbrt2', embedding=RR(1.2599))
sage: w = K.gen()
sage: import resource
sage: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
180720
sage: for _ in range(10000): test = w > 1
sage: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

I tried python-level analysis:

import gc
from collections import Counter
gc.collect()
pre={id(c) for c in gc.get_objects()}

x = polygen(ZZ)

K = NumberField(x**3 - 2, 'cbrt2', embedding=RR(1.2599))

w = K.gen()
gc.collect()
pre={id(c) for c in gc.get_objects()}
for _ in range(20000): test = w > 1
gc.collect()
gc.collect()
post=Counter(type(o) for o in gc.get_objects() if id(o) not in pre)
sorted(post.iteritems(),key=lambda t: t[1])

C=[c for c in gc.get_objects() if id(c) not in pre and type(c) is tuple and type(c[0]) is sage.rings.real_mpfi.RealIntervalFieldElement]

 This finds a whole bunch of one-element tuples with a realintervalfield element. If I use objgraph_showbackrefs(C[100]) or something similar, I only  get the references via C! So it seems somehow these objects survived "gc.collect" without actually having objects referring to them. That would point towards an incref/decref error somewhere.

[Vincent Delecroix]

I am not sure about the Python analysis (though it might be some other
problem).

With the code (which does not involve number fields anymore)
{{{
import gc
import resource

R = PolynomialRing(ZZ, 'x', implementation='NTL')

x = R.gen()

p = x**2 - 3

gc.collect()
n0 = len(gc.get_objects())
mem0 = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

def stats():
global n0, mem0
gc.collect()
print len(gc.get_objects()) - n0,
print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss - mem0

for _ in range(10):
stats()
for _ in range(5000): a = p(2)
}}}

The result I got is

57 0
68 508
68 1300
63 2620
63 2620
63 3356
63 4148
58 4676
58 5468
58 6236

Hence the memory usage (right column) is not reflecting any problem from
the garbage collection (left column).

An other important remark is that if we use the FLINT implementation
everything is fine. In other words using

R = PolynomialRing(ZZ, 'x', implementation='FLINT')

I got

60 0
54 0
54 0
54 0
54 0
54 0
49 0
49 0
49 0
49 0

Vincent

[Nils Bruin]

On Wednesday, March 23, 2016 at 12:08:29 PM UTC-7, vdelecroix wrote:

I am not sure about the Python analysis (though it might be some other
problem).

I observed the same thing. It looks like we're looking at two different memory leaks in the same example:

1) A plain old malloc-type memory leak in NTL (valgrind could help for this)
2) An incref/decref problem with tuples of real interval elements somewhere.

[Vincent Delecroix]

Strangely enough, it does not seem to be NTL related... I was able to
reproduce a somehow minimal Cython example without any use of Sage.
. See the report on Cython mailing list

https://groups.google.com/forum/#!topic/cython-users/g10b0911qq0

The reason why everything is fine with FLINT version is that FLINT
polynomials actually implement their own __call__. So it is a bug with
the __call__ of Polynomial.

I opened #20268 to track the issue.

Best,
Vincent

[Nils Bruin]

For the python-level leak, I think it's down to this:

x = polygen(ZZ)
v = x # or 1/3 or RR(3) but not, say 3

pre={id(c) for c in gc.get_objects()}

for _ in range(20000): test =  sage.rings.polynomial.polynomial_element.Polynomial.__call__(x,v)

It ends up letting a lot of 1-element tuples of the form (v,) lie around. Note that the call in question is an unbound method call,
(in the original example, where there was a number field element comparison, things do boil down to polynomial evaluation in sage.rings.number_field.number_field_element.NumberFieldElement._richcmp_ ), which is a relatively unusual thing to do, so there might be room there for a cython error, but then it's strange that the problem does not occur when v is an integer.

[Vincent Delecroix]

Fixed in Cython. I provided a backport of the patch in Sage at #20268
(needs review).

Volker: Once it is reviewed, could we have a new stable Sage release
with the fix?

Vincent