Memory usage by doctests

[Jeroen Demeyer]

Hello sage-devel,

While working on #9343, I discovered that some of the doctests (in
particular, related to elliptic curves) use lots of memory (more than 1GB).

I know that Sage has a "# long time" option for tests which take a long
time. Maybe there should be an option "# huge memory" or something to
indicate that.

Is there any policy on how much memory a reasonable doctest should use?

Jeroen.

[David Kirkby]

What particular test is using a lot of RAM? On what operating system?
32-bit or 64-bit?

Although not recently, I've built Sage and it passed all doctests on a
machine with 1.5 GB RAM. Only yesterday I built is on a machine with
only 2 GB RAM. These were all 32-bit Solaris builds on SPARC.

There is some memory leaks on Sage. When I linked Sage on Solaris
using a debugging memory tool, I discovered there are memory leaks
even before we get the

sage:

prompt. Some of the functions called had 'pari in the name, so
probably related to Pari, though I'm not saying Pari is at fault.

Dave

[Jeroen Demeyer]

On 2010-07-21 18:41, David Kirkby wrote:
> What particular test is using a lot of RAM?

In particular sage/schemes/elliptic_curves/heegner.py (but also others
from elliptic_curves) from vanilla sage 4.5.1:

$ ulimit -v 1900000 # Use at most 1,945,600,000 bytes of virtual memory
$ sage -t sage/schemes/elliptic_curves/heegner.py
File "[...]/sage/schemes/elliptic_curves/heegner.py", line 4646:
sage: H.optimal_embeddings(-7, 1, R)
Exception raised:
Traceback (most recent call last):
[...]
MemoryError: Unable to allocate 1024000000 bytes memory for PARI.

> On what operating system? 32-bit or 64-bit?

$ uname -a
Linux arcanis 2.6.32-gentoo-r7 #5 SMP Thu Jun 10 23:07:26 CEST 2010
x86_64 Intel(R) Core(TM)2 Duo CPU T5870 @ 2.00GHz GenuineIntel GNU/Linux

[Dr. David Kirkby]

I set that ulimit and it passed the test ok on Solaris 10 (SPARC). The memory
kept climbing, but *only* reached 695 MB. Still a lot I would agree.

I did a test the other day on a 64-bit build of Sage on Solaris, by using a
different malloc - one designed for testing for memory leaks. There is certainly
something that looks to be Pari related there - see below. Note how one of the
calls which is generating a memory leak has 'pari' in the name?

kirkby@t2:[~/sage-4.5-hacked-for-64-bit-solaris] $ mdb core
Loading modules: [ libumem.so.1 libc.so.1 libuutil.so.1 ld.so.1 ]
> > ::findleaks -dv
findleaks: maximum buffers => 38054
findleaks: actual buffers => 37054
findleaks:
findleaks: potential pointers => 10532388
findleaks: dismissals => 3532123 (33.5%)
findleaks: misses => 4985446 (47.3%)
findleaks: dups => 1977768 (18.7%)
findleaks: follows => 37051 ( 0.3%)
findleaks:
findleaks: elapsed wall time => 8 seconds
findleaks:
BYTES LEAKED VMEM_SEG CALLER
65536 1 ffffffff73c60000 MMAP
------------------------------------------------------------------------
Total 1 oversized leak, 65536 bytes

CACHE LEAKED BUFCTL CALLER
000000010015e028 1 00000001026e2020
gen.so`__pyx_f_4sage_4libs_4pari_3gen__new_gen+0x68
0000000100168028 1 00000001039e6e10
integer.so`__pyx_f_4sage_5rings_7integer_fast_tp_new+0xb0
----------------------------------------------------------------------
Total 2 buffers, 24 bytes

mmap(2) leak: [ffffffff73c60000, ffffffff73c70000), 65536 bytes
umem_alloc_8 leak: 1 buffer, 8 bytes
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
1026e2020 1014f7fe0 6880239fd06b2d 1
10015e028 0 0
libumem.so.1`umem_cache_alloc+0x148
libumem.so.1`umem_alloc+0x5c
libumem.so.1`malloc+0x40
gen.so`__pyx_f_4sage_4libs_4pari_3gen__new_gen+0x68

gen.so`__pyx_f_4sage_4libs_4pari_3gen_12PariInstance_new_gen+0x28

gen.so`__pyx_pf_4sage_4libs_4pari_3gen_12PariInstance___init__+0x8d0
libpython2.6.so.1.0`type_call+0x88
libpython2.6.so.1.0`PyObject_Call+0x60
gen.so`initgen+0x194c
libpython2.6.so.1.0`_PyImport_LoadDynamicModule+0x94
libpython2.6.so.1.0`import_submodule+0xc8
libpython2.6.so.1.0`load_next+0x98
libpython2.6.so.1.0`import_module_level+0x1f8
libpython2.6.so.1.0`PyImport_ImportModuleLevel+0x28
libpython2.6.so.1.0`builtin___import__+0x8c

umem_alloc_16 leak: 1 buffer, 16 bytes
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
1039e6e10 1039df140 6880256324a377 1
100168028 0 0
libumem.so.1`umem_cache_alloc+0x21c
libumem.so.1`umem_alloc+0x5c
libumem.so.1`malloc+0x40
integer.so`__pyx_f_4sage_5rings_7integer_fast_tp_new+0xb0
integer.so`__pyx_f_4sage_5rings_7integer_8int_to_Z__call_+0x68

coerce.so`__pyx_f_4sage_9structure_6coerce_24CoercionModel_cache_maps_canonical_coercion+0x2fc

element.so`__pyx_f_4sage_9structure_7element_7Element__richcmp+0x248

integer.so`__pyx_pf_4sage_5rings_7integer_7Integer___richcmp__+0x20
libpython2.6.so.1.0`try_rich_compare+0x5c
libpython2.6.so.1.0`PyObject_RichCompare+0x64
libpython2.6.so.1.0`PyEval_EvalFrameEx+0x2838
libpython2.6.so.1.0`PyEval_EvalCodeEx+0x988
libpython2.6.so.1.0`function_call+0x158
libpython2.6.so.1.0`PyObject_Call+0x60
libpython2.6.so.1.0`instancemethod_call+0x150

> >

[John Cremona]

Perhaps Jeroen is using pari 2.4.3 and that behaves differently?

The pari library does its own memory management. Not for the faint-hearted!

John

> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to
> sage-devel+...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org
>

[Dr. David Kirkby]

On Jul 22, 9:41 am, John Cremona <john.crem...@gmail.com> wrote:
> Perhaps Jeroen is using pari 2.4.3 and that behaves differently?
>
> The pari library does its own memory management.  Not for the faint-hearted!
>
> John

He would have to say. I used the Pari 2.3.5 in Sage 4.5.

My result was obtained by starting sage, then waiting for the

sage:

prompt. At that point, I run 'gcore' to cause a core dump. Then I
inspected the core dump I'd created. So memory had already leaked by
the point that one got the sage prompt.

Dave

[Jeroen Demeyer]

On 2010-07-22 10:41, John Cremona wrote:
> Perhaps Jeroen is using pari 2.4.3 and that behaves differently?

No, this was the old PARI from a non-patched Sage 4.5.1. As I said in
#9343, the new PARI seems to use even more memory than the old PARI, but
both of them use a lot.

> The pari library does its own memory management. Not for the faint-hearted!

Actually, it is a very simple-minded memory management. But a
malloc-debugger is pointless to debug PARI memory leaks.

Jeroen.