customized evaluation function with multiprocessing
t.k
Hello,
Would like to start off with a big thankyou to the developers of deap. Although I don’t specialize in programming, I have successfully prepared the backbone for my code within a few days. Very helpful!
Would like a bit of help
regarding; how to obtain the ID number for each individual. The ID number I’m referring to is the global
ID, meaning that the 1st individual from
the initial population will be #0, and if each generation has a total of 100 individuals, then the 1st
individual from the following generation would be #100. The reason why I
want this information is so I can save each and every individual
(including
dominated individuals) along with the corresponding fitness as it gets
evaluated (even while the optimization is still running). Flicking through the posts in this group, I suppose a
lot of people are not interested in non-pareto-front individuals,
however when each individual takes a long time to
compute, each and every result is precious.
-----------------------------------------------------------------------------------------------------------
Using the knapsack.py example code as a starting point; I altered the evaluate function to print the following information into a csv:
global ID for each individual / generation number of each individual / individual / fitness
※ originally for the ID and generation number I simply added a counter inside the evaluate function and incremented the values as required
>> however after adding multiprocessing.Pool(2) in main() function, the counters couldn’t increment properly ..
so was wondering how to pass ID and generation for each individual to the evaluate function (properly).
※will be using more workers later
-----------------------------------------------------------------------------------------------------------
also ... I found that occasionally ... some individuals couldn't get added to the csv properly and instead just added a blank row ..
>> possibly caused by two workers trying to write into the csv at the same time? From my guess, this problem might not happen if evaluations don't get executed at the same time even with a large number of workers. Any other ideas regarding what I should check ?
-----------------------------------------------------------------------------------------------------------
On a side note, from my understanding, by using multiprocessing, this divides the “current” generation in progress among the workers (CPUs or threads), is this correct?
I would like to be able to run the “evaluate” function on multiple individuals from the current generation in parallel, as each calculation I plan on running from this function takes quite a while.
Thank you in advance.
Best Regards