Memory consumption of multiprocessing.Pool

[Wolodja Wentland]

Hi all,

I have a problem with the memory consumption of multiprocessing.Pool()'s
worker processes. I have a parent process that has to handle big data
structures and would like to use a pool of processes for computations.

The problem is, that all worker processes have the same memory
requirement as the parent one, although they do *not* use any the parent
processes data structures. The following snippet illustrates this
behaviour:

--- snip ---
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import multiprocessing
import time

def worker(el):
time.sleep(10)

def main():
print 'Init Pool 1'
p_pool = multiprocessing.Pool()

print 'Call pool.map()'
p_pool.map(worker, range(multiprocessing.cpu_count()))

print 'Allocate memory'
eat_memory = range(1000000)

print 'Call pool.map()'
p_pool.map(worker, range(multiprocessing.cpu_count()))

print 'Delete pool 1'
del p_pool

print 'Init Pool 2'
p_pool = multiprocessing.Pool()

print 'Call pool.map()'
p_pool.map(worker, range(multiprocessing.cpu_count()))

print 'Delete pool 2'
del p_pool

if __name__ == '__main__':
main()
--- snip ---

You will see that the memory consumption of the worker processes will be
roughly the same for the first two calls to p_pool.map(), but rise for
the third.

How can I make sure that 'eat_memory' does not use any memory in the
pool processes? This is important as I don't always know when a pool is
instanciated and the pool processes should *not* have the same memory
requirements as the parent process.

Am I missing something here?

--
.''`. Wolodja Wentland <went...@cl.uni-heidelberg.de>
: :' :
`. `'` 4096R/CAF14EFC
`- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC

[Aahz]

In article <mailman.1926.1260872...@python.org>,

Wolodja Wentland <went...@cl.uni-heidelberg.de> wrote:
>
>I have a problem with the memory consumption of multiprocessing.Pool()'s
>worker processes. I have a parent process that has to handle big data
>structures and would like to use a pool of processes for computations.
>
>The problem is, that all worker processes have the same memory
>requirement as the parent one, although they do *not* use any the parent
>processes data structures.

What I would do is put your current parent process into its own
subprocess and have the main process handle communication between the
memory-heavy subprocess and the worker processes.
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/

Weinberg's Second Law: If builders built buildings the way programmers wrote
programs, then the first woodpecker that came along would destroy civilization.