How to use multiprocessing with feasible(constraints) function?
156 views
Skip to first unread message
Nikhil Soni
Jul 23, 2020, 8:01:21 AM7/23/20
to deap-users
Hi,
The multiprocessing module throws a pickling error when I try to decorate the evaluate function with a feasible function that has constraints in it.
How to solve it?
Derek Tishler
Jul 23, 2020, 12:22:35 PM7/23/20
to deap-users
Ok...my example(I applied it arbitrarily to onemax_mp.py example) here gets a bit messy so please bear with me. I could us a hand cleaning refining as I think I got mixed up trying to combine Ray's ActorPool with decorators.
A big note her, like with Scoop, is parallel can be really slow for inexpensive evals due to network overhead so expect the below example to run slower in Ray mode(change at top via use_old_method bool). If you actually wanted to use it for a case of quick evals and large pops,you would need to batch out groups individuals to remote workers to iterate though.
A big note her, like with Scoop, is parallel can be really slow for inexpensive evals due to network overhead so expect the below example to run slower in Ray mode(change at top via use_old_method bool). If you actually wanted to use it for a case of quick evals and large pops,you would need to batch out groups individuals to remote workers to iterate though.
I posted here for more discussion regarding ray vs scoop: https://github.com/DEAP/deap/issues/404
You can get around this pickle issue for things like ephemeral constants or delta penalties by using Ray vs multiprocessing or SCOOP. This will required a newer version( > 3.5 i think) of python. I switched to ray as scoop out of development and unsupported and I required cluster support. I highly suggest Ray!
I was able to test...but it was a bit of a hack job(very messy to apply the decorator...I must have overcomplicated it). To recreate map via ray I use the ActorPool, but it is pop eval specific vs useable for islands and other map cases in Deap examples. Ray, like scoop, has overheard so this is expectedly slower for inexpensive evaluations. but I was able to show that it has same behavior with the old eval+decorator vs using ray+decorator:
Example of using Ray to scale population evaluations and get around pickle issues for deap, such as DeltaPenalty:
You can get around this pickle issue for things like ephemeral constants or delta penalties by using Ray vs multiprocessing or SCOOP. This will required a newer version( > 3.5 i think) of python. I switched to ray as scoop out of development and unsupported and I required cluster support. I highly suggest Ray!
I was able to test...but it was a bit of a hack job(very messy to apply the decorator...I must have overcomplicated it). To recreate map via ray I use the ActorPool, but it is pop eval specific vs useable for islands and other map cases in Deap examples. Ray, like scoop, has overheard so this is expectedly slower for inexpensive evaluations. but I was able to show that it has same behavior with the old eval+decorator vs using ray+decorator:
Example of using Ray to scale population evaluations and get around pickle issues for deap, such as DeltaPenalty:
Derek Tishler
Jul 23, 2020, 3:28:18 PM7/23/20
to deap-users
I updated the gist i posted before in order to have a simple, hopefully general way, to replace pool and scoop's map in deap such that if working with multiple cpus, the work will be automatically batched out to workers.
https://gist.github.com/DMTSource/72adc855e5e972d472338485da1841a7
https://gist.github.com/DMTSource/72adc855e5e972d472338485da1841a7
Serguey Maximov
Apr 13, 2021, 2:14:32 PM4/13/21
to deap-...@googlegroups.com
Hi Derek,
Have you run your "onemax_ray" example in Windows? I have problems running my code and your example, and I get this error: "INFO services.py:1173 -- View the Ray dashboard at http://127.0.0.1:8265".
It seems that Ray was developed for Linux and only relatively recently is available for Windows so I guess the error could be related to that.
I am trying to use your Ray based alternative as I need to constrain my function.
Best,
Serguey
--
You received this message because you are subscribed to the Google Groups "deap-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to deap-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/deap-users/a5a9f362-80b6-40bf-a4f8-b6c3f26ed1bfo%40googlegroups.com.
Derek Tishler
Apr 13, 2021, 9:40:57 PM4/13/21
to deap-users
Serguey,
That may take some searching of that error if its a window specific issue(github issues may have a fix but could be tricky), as you say win support for ray is new and I am unfamiliar with its usage.
This sort of thing used to come up alot for me before switching to linux many years ago(X develops great on linux but windows is broken, used to be Tensorflow was like this originally), but for windows users you can always use an approach of using a program like Docker to run such things; an Anaconda ready linux image or a Deap ready image should be available on DockerHub to nearly instantly boot up a emulator on your current OS.
-Derek
That may take some searching of that error if its a window specific issue(github issues may have a fix but could be tricky), as you say win support for ray is new and I am unfamiliar with its usage.
This sort of thing used to come up alot for me before switching to linux many years ago(X develops great on linux but windows is broken, used to be Tensorflow was like this originally), but for windows users you can always use an approach of using a program like Docker to run such things; an Anaconda ready linux image or a Deap ready image should be available on DockerHub to nearly instantly boot up a emulator on your current OS.
-Derek
Reply all
Reply to author
Forward
0 new messages