[Python-Dev] tracemalloc: add an optional memory limit

[Victor Stinner]

Hi,

The PEP 454 (tracemalloc module) has been implemented in Python 3.4
beta 1. Previously, I also wrote a pyfailmalloc project to test how
Python behaves on memory allocation failures. I found various bugs
using this tool.

I propose to add an optional memory limit feature to the
tracemallocmodule. It would allow to set an arbitrary limit (ex: 100
MB) to check how your application behaves on systems with a few free
memory (ex: embedded devices).

What do you think of this enhancement? Would it fit Python 3.4
timeline? It's a new feature, but for a module which was introduced in
Python 3.4. I wanted to propose directly the feature in the PEP, but
Python was not ready for that (Python failed to handle correctly
memory allocation failures).

On Linux, it's possible to limit globally the "address space" of a
process, but this value is not convinient. For example, the address
space includes shared memory and read-only memory mappings of files,
whereas this memory should not reduce the memory available for other
applications. tracemalloc only counts memory directly allocated by
Python and so private memory which reduces directly the memory
available for other applications.

Technically, it's interesting to implement this feature in
tracemalloc. No new code should be added to count how many bytes were
allocated by Python, tracemalloc must already do that (indirectly).
It's simple to modify tracemalloc to make malloc() fails if an
allocation would make the counter greater than the limit.

I wrote a patch in the following issue to add an optional memory_limit
parameter to tracemalloc.start():
http://bugs.python.org/issue19817

The patch is short:

Doc/library/tracemalloc.rst | 6 +++-
Lib/test/test_tracemalloc.py | 11 +++++++-
Modules/_tracemalloc.c | 56 ++++++++++++++++++++++++++++++++++++-------
3 files changed, 63 insertions(+), 10 deletions(-)

--

I tried to fix all bugs in Python to handle correctly memory
allocation failures. When the memory limit is very low (less than 1024
bytes), there are still some corner cases where Python may crash.
These corner cases can be fixed. For example, I proposed a patch to
make PyErr_NoMemory() more reliable (which is probably the most
critical bug of remaining bugs):
http://bugs.python.org/issue19835

I fixed a reference leak in the unittest module (#19880) to reduce the
risk of PyErr_NoMemory() crash when I use the Python test suite to
test the memory limit feature (see issue #19817). I'm not sure that
PyErr_NoMemory() bug affects real applications.

Victor
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Serhiy Storchaka]

09.12.13 03:56, Victor Stinner написав(ла):

> On Linux, it's possible to limit globally the "address space" of a
> process, but this value is not convinient. For example, the address
> space includes shared memory and read-only memory mappings of files,
> whereas this memory should not reduce the memory available for other
> applications. tracemalloc only counts memory directly allocated by
> Python and so private memory which reduces directly the memory
> available for other applications.

But tracemalloc doesn't count memory allocated besides Python allocators
(e.g. memory for executable, static variables and stack, memory
allocated by extensions and C lib, unallocated but not returned to OS
dynamical memory). When you want investigate how your program works on
low memory, e.g. 500 MB, ulimit -v 500000 is much closer to real life
than tracemalloc.

Limiting memory in tracemalloc can be used only for testing obscure
corner cases in Python interpreter itself. It will be very rarely used
since all bugs will be fixed first time (and thank you for your current
work). So there is more sense implemented it as auxiliary utility than
as feature of the stdlib module.

[Victor Stinner]

Hi,

2013/12/9 Serhiy Storchaka <stor...@gmail.com>:

> But tracemalloc doesn't count memory allocated besides Python allocators
> (e.g. memory for executable, static variables and stack, memory allocated by
> extensions and C lib, unallocated but not returned to OS dynamical memory).
> When you want investigate how your program works on low memory, e.g. 500 MB,
> ulimit -v 500000 is much closer to real life than tracemalloc.

Well, both limits are useful, but also (completly?) different.

"memory for executable, static variables and stack," : these values
are fixed, so they don't matter at all. Substract them from the RSS
when you choose the memory limit for tracemalloc.

"memory allocated by extensions" : I started to modify some Python
extensions (ex: zlib, bzip2, lzma) to reuse Python memory allocation,
so the memory is also traced by tracemalloc. Memory allocated by other
extensions is not traced.

"and C lib," : correct, this memory is not traced by tracemalloc and
so not counted in the memory limit.

"unallocated but not returned to OS dynamical memory" : it's really
hard to estimate and understand the fragmentation of the heap memory
:-( But this may only concern short peak, and the free memory can be
reused later. If the peak is still lower than the limit, the
fragmentation does not matter

> Limiting memory in tracemalloc can be used only for testing obscure corner
> cases in Python interpreter itself.

In my previous job in embedded device, we hard an *hard* limit for
memory. Python had its own dedicated memory: an hardcoded 30 MB chunk
of memory, reserved for Python. Only memory directly allocated by
Python was stored in this 30 MB chunk. Other allocations ("memory
allocated by extensions, C lib" in your list) was allocated in the
system memory. And I was only concerned of memory directly allocated
by Python.

So being able to run the application on a PC and set a "Python" memory
limit of 30 MB makes sense in such use case.

> It will be very rarely used since all
> bugs will be fixed first time (and thank you for your current work). So
> there is more sense implemented it as auxiliary utility than as feature of
> the stdlib module.

The memory limit is used to test if the application doesn't leak
memory nor allocate more than the limit. Said differently, it's a
convinient tool to simulate an embeded device :-)

Victor

[Antoine Pitrou]

On Mon, 9 Dec 2013 10:28:17 +0100
Victor Stinner <victor....@gmail.com> wrote:
> Hi,
>
> 2013/12/9 Serhiy Storchaka <stor...@gmail.com>:
> > But tracemalloc doesn't count memory allocated besides Python allocators
> > (e.g. memory for executable, static variables and stack, memory allocated by
> > extensions and C lib, unallocated but not returned to OS dynamical memory).
> > When you want investigate how your program works on low memory, e.g. 500 MB,
> > ulimit -v 500000 is much closer to real life than tracemalloc.
>
> Well, both limits are useful, but also (completly?) different.
>
> "memory for executable, static variables and stack," : these values
> are fixed, so they don't matter at all. Substract them from the RSS
> when you choose the memory limit for tracemalloc.

Stack consumption is O(number of threads).

> "unallocated but not returned to OS dynamical memory" : it's really
> hard to estimate and understand the fragmentation of the heap memory
> :-( But this may only concern short peak, and the free memory can be
> reused later. If the peak is still lower than the limit, the
> fragmentation does not matter

I don't understand the statement: "if the peak is still lower than the
limit, the fragmentation does not matter". Serhiy's point is precisely
that the peak's computation doesn't account for OS-level fragmentation.

That said, I agree the feature can be useful. Unlike ulimit, it's a
best-effort thing, but it also works at a much more useful level than
"size of the virtual address space".

Regards

Antoine.