Deap with PyPy and multiprocessing

[kramer65]

Hello,


I just wanted to use deap with pypy, but unfortunately this resulted in an import error for deap.

After a while I realised I maybe needed to install deap again within pypy, which I did by simply doing "pypy setup.py install". Maybe it is a good idea to mention this in the docs.

It is now running, but deap doesn't seem to run well with both pypy and multiprocessing enabled. I guess this is not deap's fault, but a problem with pypy and multiprocessing (although on this blog one Alex Gaynor says something that with Python version 2.7 multiprocessing will be part of pypy).

Anyway, I thought I could mention it so you guys are aware of the issue.


Regards,
Hielke

[Félix-Antoine Fortin]

Hello,

Could you be more specific on what is the problem?
Is it running slower? Is it not running at all?

The blog post you mentionned refers to PyPy 1.4. Since then, they have
released PyPy 1.8 which implements most of Python 2.7, including
multiprocessing. You can look at the running time of every example of
DEAP 0.8 with PyPy 1.8 at :
http://deap.gel.ulaval.ca/speed/results.html?branch=default&version=pypy

mpga_onemax is using multiprocessing, and it is actually running a
little bit faster with PyPy, but this is too trivial, and the results
might not be representative of the expected time with a real problem.

We already are aware of some issues with PyPy. Some GP examples runs
far slower on PyPy, mostly because of the multiples call to the eval
function.

Regards,
Félix-Antoine

[kramer65]

Hi Félix-Antoine,


Excuse me for not getting back to you on this. After my last post I got into some hefty error-horror, which (after a lot of searches throughout my code) turned out to be a hardware issue. A memtest finally revealed the source of the errors. After fixing that I just went back into my program again and got back to the pypy issue.

Maybe I should have been a bit more hesitant before posting this issue though. I checked the oneMax example, and that indeed works with pypy. My evolution strategy also works with pypy, but only one cpu-core is being used. I made two screenshots for you. First I ran my program through python (screenshot 1) and then I ran the same program without any changes with pypy (screenshot 2). As you can see the exact same program only uses one core with pypy.

I don't know if this issue is solely restricted to my specific case. I am running Python version 2.7.3 on Ubuntu 12.04 with deap version 8.1. If you are not able to recreate this, it must be an issue specific to my program. In that case I excuse myself for all the fuzz.. :-)


Kind regards,
Hielke


Op maandag 30 april 2012 03:00:03 UTC+2 schreef Félix-Antoine Fortin het volgende:

Op maandag 30 april 2012 03:00:03 UTC+2 schreef Félix-Antoine Fortin het volgende:

[Francois-Michel De Rainville]

Hello Hielke,

After looking at processor usage for the mpga_onemax example, I realize that what you observe is probably normal. Using Pypy the evaluation function might be too quick thus all processing time is consumed when cloning the individual. To confirm this you can profile the execution (python -m without multiprocesing) and look at the time used for each function. Note that cloning the individuals is not parallelized (this would be stange since we need to serialize, clone, serialize them).

I hope this helps,

François-Michel

[Francois-Michel De Rainville]

The command for profiling is

python -m profile your_executable.py

or

pypy -m profile your_executable.py

under python cProfile instead of profile should be faster.

Regards,

FM

[kramer65]

Hi François-Michel,


Unfortunately the evaluation function is the majority of the processing power. The screenshot I showed you with all four cores at 100% (this one) is of an evolution strategy with mu=10 and lambda=100 and only 5 generations. The cloning is only a very small part of this evolution.

However, it turned out that even though Pypy only uses one core, it is still faster than using Python with the multiprocessing module. So for now I'll stick with pypy.. :-)


Kind regards,
Hielke



Op zondag 6 mei 2012 23:03:35 UTC+2 schreef François-Michel De Rainville het volgende:

[Marc-André Gardner]

Hello,

As I was intrigued by your problem, I ran some test to figure it out.

Unfortunately, none of them have been conclusive. I tried with three different problems : a GA problem (sorting network), a GP problem (artificial ant) and an ES problem (the fctmin example provided with DEAP).
Each of them was tuned so they spent most of their computing time in the evaluation function, thus allowing us to see if the distribution works well.

I ran those tests on two separate platforms (both on Linux), one high-end workstation (8 cores, 12 GB RAM, 64 bits) and one "low-end" computer (2 cores, 2 GB RAM, 32 bits), with Python 2.7 and Pypy 1.8.

I was not able to reproduce your bug : each time, pypy scales adequatly, as cPython.

The included printscreen shows that correct behavior (it was taken from an ES run) : the small concavities are the non-parallelisable work (selection, cloning, etc.), but we can see that all of the 8 CPU cores are 100% loaded when it comes to evaluate individuals, which is correct.

More specifically on ES, I should state an important point (which can be generalized to every usage of multiprocessing.Pool.map() with a very small number of items) : as multiprocessing does not implement a load balancing algorithm (it just separates the work into approximately equal chunks), having a small number of individuals like in ES combined with an high standard deviation in evaluation times, may conduct to a performance decrease.

If you have any remaining information which may help us to find out why your algorithm is not distributed well with multiprocessing, please let us know (although I do not believe that it would be a DEAP bug).

Regards,

Marc-André Gardner

[kramer65]

Hi Marc-André,


Thanks for your further research into the issue. I had another look over my code and for now I wouldn't know why this issue happens here. In my code I first load about 1 GB of data from an sqlite database into memory. This however, is done before the evolution begins and should not be a bottleneck (also because it does work well in cPython). The evaluation function then takes the individuals and loops over the data, trying to find patterns in the data and returning a general measure of fitness in the end. Although evaluation times can vary slightly, the standard deviation should not be high (but I haven't specifically tested this).

What I'm going to do is clone my repository and I try to minimise the evaluation function (take out functions) in order to see where possible sources of this issue may be. If I am unable to find the source of the issue and I end up with a really small version of my program I could send it to you so that you can maybe test it on your system? If it still doesn't show any issues on your system it should be clear that the source lies within my setup. Would that be an idea?


Kind regards,
Hielke




Op dinsdag 8 mei 2012 21:01:32 UTC+2 schreef Marc-André Gardner het volgende:

[Marc-André Gardner]

Hi Hielke,

Code minimization is surely one way to look at the issue. The less imbricated and complicated functions a program has, the better we can debug it...

I have no problem to test a specific version of your program, however, in the interest of other DEAP users, I would ask you to allow us to publish the results of those tests and the conclusions we can take from them on this mailing list -- assuming, of course, that the results are interesting for other developpers.
We just want that every information about DEAP remains accessible to everyone.

Also, of course, although I'm sure that your work is scientifically very interesting, we would like you to help us to focus on the DEAP/multprocessing/pypy issue, if there is one, not your code itself -- in other words, just send us a minimum test case from which we can reproduce your problem.

Have a good day (debugging DEAP?) ;-)

Marc-André Gardner