How can I free memory?
janko hauser
Hello, I'm working with the numerical extension of python. If I create
a huge array, say with:
a=array(arange(1000000)).reshape(1000,1000)
and try to free the memory with
a=[]
the process gives only a small amount of memory free. If I do this
again, python allocates even more memory. Is this the right way?
I'm working on a Linux-box
TIA
Janko
Skip Montanaro
Hello, I'm working with the numerical extension of python. If I create
a huge array, say with:
a=array(arange(1000000)).reshape(1000,1000)
and try to free the memory with
a=[]
the process gives only a small amount of memory free. If I do this
again, python allocates even more memory. Is this the right way?
There are a couple possibilities. First, if there are any other references
to the object, then rebinding a to another value won't cause that item to be
released. That doesn't seem to be the case here, but just to be sure, just
before you reassign a, execute
import sys
print sys.getrefcnt(a)
Note that there will be one reference due to its use as an argument to
sys.getrefcnt. If the count is greater than 2, there are other references
outstanding to the storage, and reassigning a (or executing "del a") won't
free the object.
More likely, however, you are getting bitten by the way your system's malloc
manages the heap. Like most interpreted systems Python is a pretty heavy
user of malloc. You may have released whatever large block of storage was
associated with the reshaped array, only to have it get carved up by malloc
calls in the interpreter before you got a chance to try and reuse it.
You might try building Python with a different version of malloc to see if
it exhibits better behavior. Doug Lea's malloc
(ftp://g.oswego.edu/pub/misc/malloc.c) treats large and small allocations
differently (by default allocating blocks larger than 128Kbytes using mmap),
so when looking for space on the heap it won't break up a 1M block on the
free list to satisfy a request for, say, 256 bytes.
I'd be a little surprised if your Linux system didn't already use Doug's
malloc, however. It has a CPP macro that is set when it is used as part of
Linux libc so that some names can get properly mangled.
Hey, Guido, how about making all names private that start and end with lower
case letters and have all upper case letters in between (like mALLOc)?
That's the way they do it in Linux. :-)
Skip Montanaro | Musi-Cal: http://concerts.calendar.com/
sk...@calendar.com | Conference Calendar: http://conferences.calendar.com/
(518)372-5583 | Python: http://www.python.org/