Potentially unreliable simulation times using pyNN.neuron and MPI parallelism
23 views
Skip to first unread message
Carlos Fernandez Musoles
Apr 6, 2017, 4:04:42 AM4/6/17
to Neural Ensemble
Hello all,
I am trying to run benchmark simulations on a cluster computer system using PyNN and NEURON as backend simulator. My model is a relatively simple one based on Vogels and Abbot's but with a higher excitatory to inhibitory connections ratio (7 to 1). My benchmark consists of recording simulation time of 1 second running of the neuronal model (with random initialisation) across increasing number of MPI processes (starting with 1 process, all the way up to 64).
What I would expect to see is a drastic decrease in sim time due to parallelism, followed by a flat section and potentially increase in sim time due to inter-processor communication overhead (hence, some sort of limits to parallelism speedup).
However, what I am seeing is substantially different. There is an initial expected speed up up to certain amount of cores, but then the simulation times as reported by PyNN are surprising, fluctuating to 100x the times of the previous iteration, but often going back. To make my point a little clearer, below there is an example of the data I am seeing (for statistical significance I run each iteration several times; times have been rounded here for clarity).
I believe the jumps in sim time are too steep to attribute to communication overhead alone. The network does not have a very high activity level (approx 8-10 spikes per second average) to justify that. We are investigating if there is an issue with the hardware infrastructure, but I wanted to ask here if anyone knows of anything from the simulator perspective that could be causing this discrepancies.
I'd appreciate your help.
Thank you very much,
Carlos
EXAMPLE OF THE DATA REPORTED
Simulation with 1 MPI process
48
47
49
Simulation with 2 MPI processes
23
22
22
Simulation with 36 processes
1
2
144
Simulation with 37 processes
78
85
111
Simulation with 38 processes
137
79
2
I am trying to run benchmark simulations on a cluster computer system using PyNN and NEURON as backend simulator. My model is a relatively simple one based on Vogels and Abbot's but with a higher excitatory to inhibitory connections ratio (7 to 1). My benchmark consists of recording simulation time of 1 second running of the neuronal model (with random initialisation) across increasing number of MPI processes (starting with 1 process, all the way up to 64).
What I would expect to see is a drastic decrease in sim time due to parallelism, followed by a flat section and potentially increase in sim time due to inter-processor communication overhead (hence, some sort of limits to parallelism speedup).
However, what I am seeing is substantially different. There is an initial expected speed up up to certain amount of cores, but then the simulation times as reported by PyNN are surprising, fluctuating to 100x the times of the previous iteration, but often going back. To make my point a little clearer, below there is an example of the data I am seeing (for statistical significance I run each iteration several times; times have been rounded here for clarity).
I believe the jumps in sim time are too steep to attribute to communication overhead alone. The network does not have a very high activity level (approx 8-10 spikes per second average) to justify that. We are investigating if there is an issue with the hardware infrastructure, but I wanted to ask here if anyone knows of anything from the simulator perspective that could be causing this discrepancies.
I'd appreciate your help.
Thank you very much,
Carlos
EXAMPLE OF THE DATA REPORTED
Simulation with 1 MPI process
48
47
49
Simulation with 2 MPI processes
23
22
22
Simulation with 36 processes
1
2
144
Simulation with 37 processes
78
85
111
Simulation with 38 processes
137
79
2
Andrew Davison
Sep 9, 2017, 9:17:30 AM9/9/17
to Neural Ensemble
Hi Carlos,
Sorry to take so long to reply to this. I don't really know what is causing the fluctuating simulation times. We are working on improving the performance of pyNN.nest and pyNN.neuron with MPI at the moment; I'll let you know once we make some progress on this (will be several weeks, at least).
Best regards,
Andrew
Reply all
Reply to author
Forward
0 new messages