I'm trying to understand how Python handles memory usage and dynamic
object loading and unloading. Problem to solve? Build a very low
memory footprint (non-GUI) Python application for Unix/Windows. Why
use Python? End user flexibility.
So far it seems that (on Windows):
(1) memory increases when you import <module>. Expected behaviour.
(2) memory does not release when you del <module>. Why not? Is Python
not releasing the memory or is it the OS not releasing memory?
So then I tried embedded Python in C:
(1) c application base 1.6MBytes (a good number)
(2) Call Python function from c application - memory up to 2.6MByte
(OK number)
(3) Python function exits but memory usage stays at 2.6MByte
Why is memory not releasing? How can I tell if this Python behaviour
or OS behaviour?
C code snippet:
#include <Python.h>
int main(int argc, char *argv[])
{
int y = 0;
printf("Started...\n");
for (int count=0;count < 999999999;count++) {
y += 1;
}
printf("Initing Py\n");
Py_Initialize();
PyRun_SimpleString("from time import sleep\n"
"print 'I will sleep for a while'\n"
"sleep(10)\n"
"print 'Exiting Py'\n");
Py_Finalize();
for (int count=0;count < 999999999;count++) {
y += 1;
}
return 0;
}
Thanks, Leonard
> I'm trying to understand how Python handles memory usage and dynamic
> object loading and unloading. Problem to solve? Build a very low
> memory footprint (non-GUI) Python application for Unix/Windows. Why
> use Python? End user flexibility.
>
> So far it seems that (on Windows):
> (1) memory increases when you import <module>. Expected behaviour.
> (2) memory does not release when you del <module>. Why not? Is Python
> not releasing the memory or is it the OS not releasing memory?
>
> So then I tried embedded Python in C:
> (1) c application base 1.6MBytes (a good number)
> (2) Call Python function from c application - memory up to 2.6MByte
> (OK number)
> (3) Python function exits but memory usage stays at 2.6MByte
>
> Why is memory not releasing? How can I tell if this Python behaviour
> or OS behaviour?
this is how most runtime systems handle memory. they have a certain amount
of space allocated, and if they run out, they ask the OS for more, but
they never give it back. i think this is what most implementations of
malloc/free do; it's certainly what java does.
what you want is some way to tell python to give back some or all of its
spare memory. i'm not aware of such a thing myself.
tom
--
unconstrained by any considerations of humanity or decency
> I'm trying to understand how Python handles memory usage and dynamic
> object loading and unloading. Problem to solve? Build a very low
> memory footprint (non-GUI) Python application for Unix/Windows. Why
> use Python? End user flexibility.
>
> So far it seems that (on Windows):
> (1) memory increases when you import <module>. Expected behaviour.
> (2) memory does not release when you del <module>. Why not? Is Python
> not releasing the memory or is it the OS not releasing memory?
[...]
On almost all OSes, processes hold on to all memory that has been
allocated by the OS. Memory released by a process can only be reused
*by that same process*. The OS only gets it back when the process
exits.
John
This is not true any more these days!!! Consider the following program:
#include <stdio.h>
#include <stdlib.h>
int main()
{
char* blobbone;
printf("No blob yet, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");
blobbone = malloc(1024*1024);
printf("I have the blob, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");
free(blobbone);
printf("No more blob, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");
printf("Bye bye.\n");
return 0;
}
Using Ctrl-Z to suspend and Enter to continue, on Linux, we see:
[alex@lancelot sae]$ gcc mem.c
[alex@lancelot sae]$ ./a.out
No blob yet, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 73 1 1330 276 0.0 ./a.out
[alex@lancelot sae]$ fg
./a.out
I have the blob, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 82 1 2362 320 0.0 ./a.out
[alex@lancelot sae]$ fg
./a.out
No more blob, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 82 1 1334 316 0.0 ./a.out
[alex@lancelot sae]$ fg
./a.out
Bye bye.
[alex@lancelot sae]$
See? the DRS grows from 1330 to 2362 K when the blob is allocated,
then drops back to 1334 when the blob is freed again. I'm pretty
sure on Windows, with a good modern VC++ version, you'll see the
same, apart from the slightly higher difficulty of the "measure
me now" task;-).
And in Python...? *SAME THING*!!! Watch:
print "No blob yet, measure me now, then press enter...",
x = raw_input()
print
blobbone = 1024*256*[None]
print "I have the blob, measure me now, then press enter...",
x = raw_input()
print
del blobbone
print "No more blob, measure me now, then press enter...",
x = raw_input()
print
print "Bye bye."
[alex@lancelot sae]$ python mem.py
No blob yet, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 3864 2504 0.4 python mem.py
[alex@lancelot sae]$ fg
python mem.py
I have the blob, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 4892 3536 0.6 python mem.py
[alex@lancelot sae]$ fg
python mem.py
No more blob, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 3864 2508 0.4 python mem.py
[alex@lancelot sae]$ fg
python mem.py
Bye bye.
[alex@lancelot sae]$
See? Just the same behavior -- DRS grows 3864->4892, then shrinks
back to 3864 when the blob is dropped.
The explanation of the behavior observed by the original poster is
quite different from the one you posit: rather, it is that
"del module" does NOT drop the module -- the module remains in
memory, fully available and loaded, and it is pointed to by
the appropriate entry sys.modules['module'] in case you want it
back. That's all there is to it -- modules, as opposed to, say,
lists, are NOT so easily disposed of:-).
Alex
Well, you live and learn. :-)
John