Memory error during multiprocessing

[tom]

Hi all,

I'am currently running a simulation with many different parameters
using the multiprocessing package. In general, the simulation runs
fine. However, despite clearing the Brian objects [using clear(True)]
there is a continuous memory build up which ultimately results is a
memory error.

A kind of 'work around' is to explicitly terminate the pool (see
below); let's say computing the first 100 parameters, and generating a
new pool thereafter with the next 100 parameters. This is, however,
not very efficient.
As I just recently moved to Python I'am sure I missed something. Are
there any suggestions to prevent the memory build up?

I'am using Python(x,y) 2.6.5.1 on 64Bit Win7.

The basic structure of the simulation is like this:
...
import multiprocessing
...
def myBrianSim( current_parameters ):
clear(True)
# build the simulation and set up the monitors
# run the simulation
# plot and save graphs
# save relevant monitor and custom data

if __name__==’__main__’:
# generate 'param_list'
pool = multiprocessing.Pool()
pool.map(myBrianSim, param_list)



Thanks in advance
tom

[Dan Goodman]

Hi Tom,

I've had some similar issues where even though the Python objects have
been deleted, Python's garbage collection doesn't get called often
enough. After calling clear(True), you could try doing:

import gc
gc.collect()

This forces Python to do a garbage collection.

This solved a similar problem I had, but it may be that your issue is
something else that is specific to multiprocessing. If it doesn't work -
perhaps you could post a minimal example that reproduces the problem?

Also, you may already know this, but to use Brian efficiently, you
should have as many neurons as possible in the network. If your
simulation runs with different parameter sets are independent and each
one is a smallish network, you could probably gain in efficiency by
vectorising your simulation (e.g. if you have N neurons and M runs you
can make a single run with N*M neurons). Our initial experiments with
this show that you need to have around 10000+ neurons in the network
before using multiprocessing to subdivide the computations becomes
worthwhile, and you don't get significant gains until you have quite a
few more than that.

Dan

On 08/06/2010 22:00, tom wrote:
> Hi all,
>
> I'am currently running a simulation with many different parameters
> using the multiprocessing package. In general, the simulation runs
> fine. However, despite clearing the Brian objects [using clear(True)]
> there is a continuous memory build up which ultimately results is a
> memory error.
>
> A kind of 'work around' is to explicitly terminate the pool (see
> below); let's say computing the first 100 parameters, and generating a
> new pool thereafter with the next 100 parameters. This is, however,
> not very efficient.
> As I just recently moved to Python I'am sure I missed something. Are
> there any suggestions to prevent the memory build up?
>
> I'am using Python(x,y) 2.6.5.1 on 64Bit Win7.
>
> The basic structure of the simulation is like this:
> ...
> import multiprocessing
> ...
> def myBrianSim( current_parameters ):
> clear(True)
> # build the simulation and set up the monitors
> # run the simulation
> # plot and save graphs
> # save relevant monitor and custom data
>

> if __name__==�__main__�:

[tom]

Hi Dan,

thank you for the fast response. As the speed in general was
acceptable when running the simulation on all 8 CPUs in parallel and
I'am still in testing phase, I have to admit that I did not vectorize
the simulation so far to keep the code more intuitive.

I've already tried garbage collection and it had no significant effect
on memory build up. However, since my last post I rewrote some of the
code (for other reasons). When I then tried to reproduce the memory
build up, the memory seems to be freed (at least partially) every now
and then. Unfortunately, I could not reconstruct what was responsible
for the build up. If I find the reason I'll post it here, maybe it is
useful for others as well.

Thanks again
tom

> > if __name__== __main__ :

[Dan Goodman]

Hi Tom,

Of course - no need to spend time optimising your code to use
vectorisation if it's already fast enough! :)

Another possible issue might be that clear(True) only clears object that
would be included in the network if you called run(). So, for example,
if you did something like:

def f(p):
... run simulation ...
return results

for p in params:
clear(True)
x = f(p)
...

This might still have a memory build up, because clear(True) only clears
objects that are created in the for p in params loop, i.e. no objects at
all (although garbage collection in the for loop should still fix that
problem).

Dan

[tom]

Hi Dan,

yes, indeed under this conditions a build up could occur. However,
this isn't an issue in my case. I'am still trying find the reason why
the re-coded version seems not to have this problem.

tom


Dan Goodman schrieb:

[tom]

Just a final comment:

It seemed that some of the processes that started during
multiprocessing ended with an exception and did not free the memory
thereafter.

Best
Tom