Custom Generator and Evaluator with NSGA2 and multiprocessing crashes

[jensk...@gmail.com]

I am new to evolutionary computation and inspyred, and I am currently trying to modify and understand the supplied examples (which are great). Ultimately, my goal is to utilize the NSGA2 algorithm to optimize an external model. As far as I understand, for that I need to write a custom generator and evaluator. Additionally, since the model is computationally expensive, I'd like to use multiprocessing (similar to this thread).

I started with modifying the NSGA2 example to have the generator and evaluator in separate functions and pulled the evaluation out of the 'main' function. This all works well and I started looking into multiprocessing.

My first question is related to the multiprocessing example. Executing this example works. However, I noticed that whatever number I entered for the argument 'mp_num_cpus=8' (e.g. 1,2,4), it was always using 4 processes due to my 4 CPUs (I was printing the 'nprocs' variable from evaluators.parallel_evaluation_mp). In none of inspyred's files I found the argument 'mp_num_cpus'. Isn't it thus necessary to replace 'mp_num_cpus=' with 'mp_nprocs=' to achieve the expected behaviour?

Then, I was trying to merge the multiprocessing example with the modified NSGA2. My code now looks like this:

from random import Random
from time import time
import inspyred
import math
   
def CustomGenerator(random, args): #from the class Kursawe (benchmark.py)
    nr_inputs = args.get('nr_inputs', 10)
    return_generator = [random.uniform(-5.0, 5.0) for _ in range(nr_inputs)]
    return return_generator
   
def CustomEvaluator(candidates, args): #from the class Kursawe (benchmark.py)
    fitness = []
    for c in candidates:
        f1 = sum([-10 * math.exp(-0.2 * math.sqrt(c[i]**2 + c[i+1]**2)) for i in range(len(c) - 1)])
        f2 = sum([math.pow(abs(x), 0.8) + 5 * math.sin(x)**3 for x in c])
        fitness.append(inspyred.ec.emo.Pareto([f1, f2]))
    return fitness  
     
prng = Random()
prng.seed(time())
ea = inspyred.ec.emo.NSGA2(prng)
ea.variator = [inspyred.ec.variators.blend_crossover,
               inspyred.ec.variators.gaussian_mutation]
ea.terminator = inspyred.ec.terminators.generation_termination
dimensions=3
final_pop = ea.evolve(generator=CustomGenerator,
                      evaluator=inspyred.ec.evaluators.parallel_evaluation_mp,
                      mp_evaluator=CustomEvaluator,
                      mp_nprocs=4,
                      #evaluator=CustomEvaluator,
                      pop_size=100,
                      maximize=False,
                      nr_inputs = 3,
                      bounder=inspyred.ec.Bounder([-5.0]*dimensions, [5.0]*dimensions),
                      max_generations=80)

final_arc = ea.archive
print('Best Solutions: \n')
for f in final_arc:
    print(f)

As I mentioned before, when I use the example without multiprocessing it works well. However, when executing the code as posted, python hangs (without error) in 'inspyred.ec.evaluators.parallel_evaluation_mp' when attempting to execute 'pool.join()'. 'pool.close()' is executed. In addition, the number of python processes increase linearly until the computer becomes unresponsive, even after stopping the execution or closing my python IDE. This makes a restart necessary. Also this and this should not be an issue since it should be fixed for Python 2.7.3 (which I am using in Windows 8 and Spyder, in case that makes a difference)? Has anybody encountered this or knows what I am doing wrong?

I really appreciate your help! Thanks a lot for looking into this.
Jens

[Aaron Garrett]

First of all, that was a really easy-to-follow description of your issues. Thanks for taking the time to be thorough.

1. The mp_num_cpus is not correct in the example. The examples have been fixed, but I haven't pushed them out to the documentation site yet. Your solution is the right one (change to mp_nprocs).

2. You bigger problem is very easily fixed, and it is caused from a limitation in Windows that is documented in the Python library (http://docs.python.org/2/library/multiprocessing.html#windows). Essentially, you have to have a "main" portion of your script. Both of the examples had it, but you might have thought it was just extraneous when you made your simpler working example. In this case, though, the multiprocessing won't work without it. I think that should fix your issues. I've run your script successfully with that change.

Let me know if you have other questions.

Aaron Garrett, Ph.D.
Assistant Professor
Computer Science

131 Ayers Hall
Jacksonville State University
Jacksonville, AL  36265
agar...@jsu.edu
256-782-5364

--
You received this message because you are subscribed to the Google Groups "Inspyred" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inspyred+u...@googlegroups.com.
To post to this group, send email to insp...@googlegroups.com.
Visit this group at http://groups.google.com/group/inspyred.
For more options, visit https://groups.google.com/d/optout.

[jensk...@gmail.com]

Thanks a lot for the quick and helpful response! Works like a charm now! Seeing the link you provided, I really should have done some more reading on multiprocessing... Thanks also for the offer to help in case I run into trouble again.