munmap releases memory from process but does not free system memory
Darryl Brown
I mmap a file of a given (large) size, then when it is full I munmap
it. I do this repeatedly, iterating over a new file each time, so I
end up with a set of files that I have written into.
The problem I am running into is that when I munmap the file it does
not appear to free up system memory. I am watching in "top" while I
step over my munmap call in a debugger and there are two things I take
note of. First, my process size is reduces by the size of the file I
just munmap'ped. Second, the (global) free system memory does not
change (nor does swap space get freed up). Not until the process
exits does the system memory get freed up - then it frees up the sum
of all the files that I have mmap'ped during my execution. FYI - I
also notice that the system free space does not change when I mmap a
file. It apparently only changes when I actually write into the file
(I haven't proven this, but that is what it looks like is happening).
This is a problem because I am running out of memory and swap (I go
through about 10-12 gigs of these files per day). Is there a way to
force the system to free this memory while the process is still alive?
Are there likely flaws in the way I am freeing this space? Is it not
a reasonable goal to expect the system to free up system memory that
has been munmap'ped?
Thanks for any ideas! Cheers,
darryl
Stephen L.
When you `open()' and _after_ you `mmap()' the file,
do you close the file descriptor from the `open()'?
Another possiblity is that if you're using `malloc()'
for intermediate results before you write to your
`mmap()'d file, `free()' _generally_ does not return
the memory back to the system (which usually shows
up as used swap space).
Depending on your Unix, there might be other tools
different from "top" that might provide more
insight into your problem.
HTH,
-
Stephen
elia Mazzawi
loop {
if (there is work) fork
child:
do the work
exit
parent:
waitpid
}
similar to apache's strategy of overcomming memory leaks in mod's like mod_perl :p
James Antill
> I am running into a problem while "munmap"ping files.
>
> I mmap a file of a given (large) size, then when it is full I munmap it.
> I do this repeatedly, iterating over a new file each time, so I end up
> with a set of files that I have written into.
It's not obvious what you mean by "writing to", is the file empty before
... or are you altering it?
> The problem I am running into is that when I munmap the file it does not
> appear to free up system memory. I am watching in "top" while I step
> over my munmap call in a debugger and there are two things I take note
> of. First, my process size is reduces by the size of the file I just
> munmap'ped.
Hmm, this would seem to indicate that you are passing the right values to
munmap() ... or at least values that are close.
> Second, the (global) free system memory does not
> change (nor does swap space get freed up). Not until the process exits
> does the system memory get freed up - then it frees up the sum of all
> the files that I have mmap'ped during my execution.
This seems weird, as Steven said this would mean you aren't closing the
file descriptor ... but it seems unlikely that this would be in swap
(unless ... are you mapping with MAP_PRIVATE, and altering the data?)
> also notice that the system free space does not change when I mmap a
> file. It apparently only changes when I actually write into the file (I
> haven't proven this, but that is what it looks like is happening).
This would indicate to me that you are using MAP_PRIVATE ... free space
shouldn't change when you do this otherwise.
> This is a problem because I am running out of memory and swap (I go
> through about 10-12 gigs of these files per day). Is there a way to
> force the system to free this memory while the process is still alive?
What Unix are you using?
> Are there likely flaws in the way I am freeing this space? Is it not
> a reasonable goal to expect the system to free up system memory that has
> been munmap'ped?
It certainly doesn't happen on my machine (Linux-2.4.x).
You could try http://www.and.org/vstr/examples/ex_hexdump.c.html with the
--mmap option, which could tell you if it's just your code or not.
--
James Antill -- ja...@and.org
Need an efficient and powerful string library for C?
http://www.and.org/vstr/