Multiprocessing limits

[Martin Karas]

Hello everybody,

how does DEAP divide workload to multiple processes? What would happen if i had 128 cores and a population consisting of only 40 individuals? At how many cores adding more cores wont be beneficial?

Thanks in advance.

Martin

[Derek Tishler]

Multiprocess recreates your global scope (so the code in global, the stuff outside 'main', is run on each process in the pool hence why we load the data for eval or perform setup in global).

Instead of evaluating individuals 1-by-1 in a loop, multiprocess uses map to batch out the work to the pool.

In your example only 40 of the 128 processes in pool would be utilized via the map to pool since there are only 40 individuals to evaluate in each eval step per generation. You can define the pool to only use 40 processes with Pool(processes=40).

You may also notice, if you use 128 size pop and 128(or any systems max n proc) processes, that you dont always get a full speedup as there are other things going on in the system as well as overhead for each eval process. The speedup may converge at a lower number of processes(such as 80% of max n proc), something worth testing as to avoid loosing time or energy.

Also, most people accidently tests MP(multiprocessing) with very very fast evaluations. This is going to make it appear slower than if not using MP due to the overhead. Make sure to create a compute heavy(or just not super trivial) eval when testing to time the speedup. And note only eval is being 'multiprocess'ed' not things like Selection so not everything is expected to speed up by switching to MP.