Memory leak in integer multiplication in sage?

[Maarten Derickx]

Hi All,

tldr: the bottom of this post contains example code of which I would like the results on some other systems.

I recently encountered a memory leak in the relatively innocently looking code:

d = 27

M = 109

for i in range(10000):

    for a in srange(M):

        for r in srange(d):

            tmp = r*a

However it doesn't seem to be on all platforms that sage supports. For example on cocalc (ubuntu 20.04) one gets:

~$ sage

┌────────────────────────────────────────────────────────────────────┐

│ SageMath version 9.2, Release Date: 2020-10-24                     │

│ Create a "Sage Worksheet" file for the notebook interface.         │

│ Enhanced for CoCalc.                                               │

│ Using Python 3.8.5. Type "help()" for help.                        │

└────────────────────────────────────────────────────────────────────┘

sage: import gc 

....: gc.collect() 

....: mem = get_memory_usage() 

....: d = 27 

....: M = 109 

....: for i in range(10000): 

....:     for a in srange(M): 

....:         for r in srange(1,d): 

....:             tmp = r*a 

....: gc.collect() 

....: print("memory usage 10k:", get_memory_usage(mem)) 

....: mem = get_memory_usage() 

....: for i in range(20000): 

....:     for a in srange(M): 

....:         for r in srange(1,d): 

....:             tmp = r*a 

....: gc.collect() 

....: print("memory usage 20k:", get_memory_usage(mem)) 

....:                                                                                                                     

434

0

memory usage 10k: 9.15234375

0

memory usage 20k: 21.3984375

                    

showing clear indication of a memory leak. While on my own laptop (OS X 10.13.6) I get:

~ mderickx$ sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.2, Release Date: 2020-10-24 │
│ Using Python 3.8.5. Type "help()" for help. │
└────────────────────────────────────────────────────────────────────┘
sage: import gc

....: gc.collect() 

....: mem = get_memory_usage() 

....: d = 27 

....: M = 109 

....: for i in range(10000): 

....:     for a in srange(M): 

....:         for r in srange(1,d): 

....:             tmp = r*a 

....: gc.collect() 

....: print("memory usage 10k:", get_memory_usage(mem)) 

....: mem = get_memory_usage() 

....: for i in range(20000): 

....:     for a in srange(M): 

....:         for r in srange(1,d): 

....:             tmp = r*a 

....: gc.collect() 

....: print("memory usage 20k:", get_memory_usage(mem)) 

....:  

278
0
memory usage 10k: 0.0
0
memory usage 20k: 0.0

so here the memory leak does not occur. However I don't have any other systems to which I have access to. So I was wondering if people could execute the code below on some other systems and report back here to see where the leaking is actually occurring. Since I do not have access to a plain vanilla sage install on ubuntu, I am especially interested if on ubuntu the non cocalc enhanced version of sage also produces a memory leak for the following code:

import gc

gc.collect()

mem = get_memory_usage()

d = 27

M = 109

for i in range(10000):

    for a in srange(M):

        for r in srange(1,d):

            tmp = r*a

gc.collect()

print("memory usage 10k:", get_memory_usage(mem))

mem = get_memory_usage()

for i in range(20000):

    for a in srange(M):

        for r in srange(1,d):

            tmp = r*a

gc.collect()

print("memory usage 20k:", get_memory_usage(mem))

[Antonio Rojas]

On Arch, using system libraries:

0

0

memory usage 10k: 10.69921875

0

memory usage 20k: 21.40234375

El jueves, 18 de marzo de 2021 a las 11:56:33 UTC+1, m.derick...@gmail.com escribió:

[Jean-Florent Raymond]

With 9.3.beta8 on arch I get:

588
0
memory usage 10k: 10.1875
0
memory usage 20k: 21.26953125

[David Joyner]

This is in sage 9.1.rc5 running on ubuntu 20.4 on a dell inspiron laptop.

It's an old version of sage but I hope it helps:

 sage: gc.collect()

....: mem = get_memory_usage()
....: d = 27
....: M = 109
....: for i in range(10000):
....:     for a in srange(M):
....:         for r in srange(1,d):
....:             tmp = r*a
....: gc.collect()
....: print("memory usage 10k:", get_memory_usage(mem))
....: mem = get_memory_usage()
....: for i in range(20000):
....:     for a in srange(M):
....:         for r in srange(1,d):
....:             tmp = r*a
....: gc.collect()
....: print("memory usage 20k:", get_memory_usage(mem))
....:  

176
0
memory usage 10k: 10.0546875

0
memory usage 20k: 21.3984375

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/5882460b-842c-49c4-b33b-fae1c7986db9n%40googlegroups.com.

[Byeongsu Yu]

On MacBook Air Mid 2012, with Sage 9.2,

39

0

memory usage 10k: 5.0

0

memory usage 20k: 11.0

[Maarten Derickx]

Hi Guys,

Thanks for the replies. I think this is enough info to know that it happens on multiple systems and that it's not just the cocalc enhanced version of sage. I have created:
https://trac.sagemath.org/ticket/31511#ticket for this.

In the meantime I found the problem is already with the srange command. So the integer multiplication seems to be a red herring. 

~$ sage

┌────────────────────────────────────────────────────────────────────┐

│ SageMath version 9.2, Release Date: 2020-10-24                     │

│ Create a "Sage Worksheet" file for the notebook interface.         │

│ Enhanced for CoCalc.                                               │

│ Using Python 3.8.5. Type "help()" for help.                        │

└────────────────────────────────────────────────────────────────────┘

sage: M = 3000 

....: import gc 

....: gc.collect() 

....: mem = get_memory_usage() 

....: for i in range(10000): 

....:     R = srange(M) 

....: gc.collect() 

....: print("memory usage 10k:", get_memory_usage(mem)) 

....: mem = get_memory_usage() 

....: for i in range(20000): 

....:     R = srange(M) 

....: gc.collect() 

....: print("memory usage 20k:", get_memory_usage(mem))                                                                   

434

0

memory usage 10k: 884.4296875

0

memory usage 20k: 1770.71875

[Vincent Delecroix]

Maarten, in your example your can not fairly compare the first
and second run. For example R is not allocated at your first
call of "mem = get_memory_usage()". Also Integers have a pool
which is likely not being filled at startup. Ignoring the first
run, I do not notice any difference.

(both on system sage 9.2 and compiled sage 9.3.beta8 on archlinux)

sage: M = 3000
....: import gc

....: _ = gc.collect()

....: mem = get_memory_usage()
....: for i in range(10000):
....: R = srange(M)

....: _ = gc.collect()

....: print("memory usage 10k:", get_memory_usage(mem))
....: mem = get_memory_usage()
....: for i in range(20000):
....: R = srange(M)

....: _ = gc.collect()

....: print("memory usage 20k:", get_memory_usage(mem))

....: mem = get_memory_usage()
....: for i in range(20000):
....: R = srange(M)

....: _ = gc.collect()
....: print("memory usage 30k:", get_memory_usage(mem))

....: mem = get_memory_usage()
....: for i in range(20000):
....: R = srange(M)

....: _ = gc.collect()
....: print("memory usage 40k:", get_memory_usage(mem))


memory usage 10k: 885.4453125
memory usage 20k: 1771.9375
memory usage 30k: 1771.8046875
memory usage 40k: 1771.80859375

>>> plain vanilla sage install on ubuntu, *I am especially interested if on

>>> ubuntu the non cocalc enhanced version of sage also produces a memory leak

>>> for the following code:*

>>> <https://groups.google.com/d/msgid/sage-devel/5882460b-842c-49c4-b33b-fae1c7986db9n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
>

[Vincent Delecroix]

Sorry, your example indeed looks fishy.

[Maarten Derickx]

Hi Vincent,

Thanks for testing and good that you realized that indeed looked fishy indeed.

I think your extension of my code with multiple loops actually confirms that there is a problem even after warming up:

the prints:

memory usage 30k: 1771.8046875
memory usage 40k: 1771.80859375

mean that you are using 1.7 GB memory extra after the 3th and the 4th loop (notice you chance the baseline of memory usage every loop so the 1.7 GB is not the total memory usage of sage, it is the new memory usage relative to the previous loop). So that in the end sage was using around 885+3*1777 MB = 6.1 GB of memory in your session.

Also,  the largest object ever created are a range object of size 10000 (which in python 3 is actually consuming almost no memory since under the hood it behaves very much like xrange from python 2) and an srange object of size 3000. Neither of these should occupy an amount of memory in the GB range.

Also one thing that is maybe easy to miss, I doubled the size of the loop between the first loop and the second loop, which explains the 885 vs 1771.

[Volker Braun]

This is presumably the same memory leak as in https://trac.sagemath.org/ticket/31340

[Dima Pasechnik]

On Thu, Mar 18, 2021 at 8:29 PM Volker Braun <vbrau...@gmail.com> wrote:
>
> This is presumably the same memory leak as in https://trac.sagemath.org/ticket/31340

yes, and it can be fixed by just removing custom memory allocators in ZZ.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/e7f09639-ec31-4e78-afae-f83c36d10063n%40googlegroups.com.