Skip to first unread message
uqzcpo
Jul 7, 2012, 8:34:34 PM7/7/12
to insp...@googlegroups.com
I wonder if the inspyred support the pause operation,
sometimes i don't know how many generations do i need
to achieve the task, so i want to stop the evolve process
and check the solutions then decide whether to go on
or stop there.
I also have another question, in the "Evaluating Individuals
Concurrently" part of the documentation,I found that
paralle evaluation is supported, but in the example, the
evaluation process is to solve a simple equation, but in
my case, the evaluation process is to call a CFD solver, which
is a commercial CFD software, usually a single evaluation process
needs 1 hour, so i really hope the parallel evaluation support
this kind of evaluator. Do you have some experence with this?
sometimes i don't know how many generations do i need
to achieve the task, so i want to stop the evolve process
and check the solutions then decide whether to go on
or stop there.
I also have another question, in the "Evaluating Individuals
Concurrently" part of the documentation,I found that
paralle evaluation is supported, but in the example, the
evaluation process is to solve a simple equation, but in
my case, the evaluation process is to call a CFD solver, which
is a commercial CFD software, usually a single evaluation process
needs 1 hour, so i really hope the parallel evaluation support
this kind of evaluator. Do you have some experence with this?
Aaron Garrett
Jul 7, 2012, 8:51:44 PM7/7/12
to insp...@googlegroups.com
For pausing, there's no built-in mechanism like you find in PyEvolve for that. It's possible to create one, and it can be relatively easy to do so, depending on what information you want in order to make your decision. The simplest thing to do would be to create a custom terminator. Take a look at the user_termination function as an example to build from. You'd just want to output some info and prompt the user as to whether to continue or not. If you had more quantified criteria, you could just make a custom terminator that automatically checked those criteria directly without involving the user.
For parallel evaluation, I have quite a bit of experience with problems like you describe. Are you running the evaluations on a network of computers? If so, then you can use the Parallel Python evaluator to do that for you, but it would require that your CFD software be installed on every node and be executable from the command line. And you'd need to be able to get the output to send back to the controller node. For instance, suppose that you were able to say
run_cfd my_parameters
in a terminal and your solver output a file called result.out that had all of the information you want in it. Then you could just have your evaluator function do something like this:
subprocess.call(['run_cfd', param1, param2, param3, ..., paramN])
with open('result.out') as resultfile:
# read in the answer from the resultfile
return answer
If you can give me more detail about what you're trying to do, I'll help you get it working or give you some guidance as to what will work better.
-- Aaron Garrett
--
You received this message because you are subscribed to the Google Groups "Inspyred" group.
To view this discussion on the web visit https://groups.google.com/d/msg/inspyred/-/Kvz-YBdnBCUJ.
To post to this group, send email to insp...@googlegroups.com.
To unsubscribe from this group, send email to inspyred+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/inspyred?hl=en.
uqzcpo
Jul 7, 2012, 10:21:59 PM7/7/12
to insp...@googlegroups.com
Thanks for your reply Aaron Garrett!
Below is what i am trying to do:
In the turbomachinary, blade is a key component, the blade geometry are defined by a set of discrete points which are represented by the XYZ coordinates.
The CFD slover can evaluate the performance of the balde, it output a result file that had all of the information i want. What i want to do with inspyred is to
search a blade whose geometry are brilliant that will lead to great performance, like high efficiency.
In the generator function, a set of (X,Y,Z) coordinates is returned as candidate like this:
def generate_frofile(random, args):
size = args.get('num_input', 9)
return [(random.uniform(a1,a1), random.uniform(b1,c1), random.uniform(d1,d1)),
(random.uniform(a2,a2), random.uniform(b2,c2), random.uniform(d2,d2)),
(random.uniform(a3,a3), random.uniform(b3,c3), random.uniform(d3,d3))]
You can see that I fixed the X and Z coordinate, I only want the Y coordinate to be changable.
In the evaluator function, the candidates which contain the blade geometry is sent to the mesher to generate mesh, after that, the mesh is sent to the CFD solver, and the CFD solver is executed with a script, and the fitness will be generated from the output file.
I already have a piece of code which can evaluate the candidates in sequence, but it is too slow, I have a computer with two Quad-Core CPU, so most of the computation resource is wasted. I didn't run the evaluations on a network of computers, just a single computer.
The CFD slover have the parallel coomputation commands itself, if my population size is 50, i can evaluate 10 individuals a time, but i have to write some extra code to handle this situation. I wonder if the Inspyred built in parallel mechanism can allocate the tasks to different process. For example, a candiate contain 50 individuals, if i use the
Below is what i am trying to do:
In the turbomachinary, blade is a key component, the blade geometry are defined by a set of discrete points which are represented by the XYZ coordinates.
The CFD slover can evaluate the performance of the balde, it output a result file that had all of the information i want. What i want to do with inspyred is to
search a blade whose geometry are brilliant that will lead to great performance, like high efficiency.
In the generator function, a set of (X,Y,Z) coordinates is returned as candidate like this:
def generate_frofile(random, args):
size = args.get('num_input', 9)
return [(random.uniform(a1,a1), random.uniform(b1,c1), random.uniform(d1,d1)),
(random.uniform(a2,a2), random.uniform(b2,c2), random.uniform(d2,d2)),
(random.uniform(a3,a3), random.uniform(b3,c3), random.uniform(d3,d3))]
You can see that I fixed the X and Z coordinate, I only want the Y coordinate to be changable.
In the evaluator function, the candidates which contain the blade geometry is sent to the mesher to generate mesh, after that, the mesh is sent to the CFD solver, and the CFD solver is executed with a script, and the fitness will be generated from the output file.
I already have a piece of code which can evaluate the candidates in sequence, but it is too slow, I have a computer with two Quad-Core CPU, so most of the computation resource is wasted. I didn't run the evaluations on a network of computers, just a single computer.
The CFD slover have the parallel coomputation commands itself, if my population size is 50, i can evaluate 10 individuals a time, but i have to write some extra code to handle this situation. I wonder if the Inspyred built in parallel mechanism can allocate the tasks to different process. For example, a candiate contain 50 individuals, if i use the
"parallel_evaluation_mp", can it call the CFD slover in different process at one time?I used the NSGA-II algorithm, i have two objective functions. And i also want t know, in my case what variator, mutator is recommanded.
Aaron Garrett
Jul 8, 2012, 2:05:19 PM7/8/12
to insp...@googlegroups.com
OK. If I understand you right, it seems like your representation needs to change a little. If the x and z components don't ever change, then they shouldn't be in the representation. So you just need a triple for the three y values as the representation.
You can certainly have the parallel_evaluator_mp send jobs to multiple cores. In your case, you just need to make sure that you can call and process the output from your solver in Python, and then you can just put that code in the actual evaluator function that is given to the parallel evaluator. That will be very similar to the example in the documentation, I think, and very similar to what I included in the previous email.
I don't know the particular mutation for your needs, but since it's basically just real numbers, I'd start with Gaussian mutation to see how that works.
Aaron Garrett, Ph.D.
Assistant Professor
Computer Science
Jacksonville State University
Jacksonville, AL 36265
agar...@jsu.edu
256-782-5364
http://mcis.jsu.edu/faculty/agarrett
Aaron Garrett, Ph.D.
Assistant Professor
Computer Science
Jacksonville State University
Jacksonville, AL 36265
agar...@jsu.edu
256-782-5364
http://mcis.jsu.edu/faculty/agarrett
--
You received this message because you are subscribed to the Google Groups "Inspyred" group.
To view this discussion on the web visit https://groups.google.com/d/msg/inspyred/-/NQ63W5dyzc8J.
uqzcpo
Jul 9, 2012, 9:59:12 AM7/9/12
to insp...@googlegroups.com
I'am now trying to modify my code as you advised, and after several days I will feed you back, thanks a lot.
uqzcpo
Jul 10, 2012, 12:17:24 AM7/10/12
to insp...@googlegroups.com
Hi, Aaron Garrett
I think I have a big trouble with parallel_evaluation_mp, because it needs all arguments to the evaluation function must be pickleable. But in my evaluation function, I have to creat folders to save the data of every individual in each generation. I modified the evaluator in the example "parallel_evaluation_mp_example.py" as follows:
def evaluate_rastrigin(candidates, args):
global generation_id
os.mkdir("D:\test\generation%d" % generation_id)
individual_id = 0
fitness = []
for cs in candidates:
fit = 10 * len(cs) + sum([((x - 1)**2 - 10 *
math.cos(2 * math.pi * (x - 1)))
for x in cs])
fitness.append(fit)
os.mkdir("D:\test\generation%d\individual%d" % (generation_id,individual_id))
individual_id += 1
generation_id += 1
return fitness
but failed.
So, I don't know how to handle this situation, cause i have to read and write data in the evaluator function.
I think I have a big trouble with parallel_evaluation_mp, because it needs all arguments to the evaluation function must be pickleable. But in my evaluation function, I have to creat folders to save the data of every individual in each generation. I modified the evaluator in the example "parallel_evaluation_mp_example.py" as follows:
def evaluate_rastrigin(candidates, args):
global generation_id
os.mkdir("D:\test\generation%d" % generation_id)
individual_id = 0
fitness = []
for cs in candidates:
fit = 10 * len(cs) + sum([((x - 1)**2 - 10 *
math.cos(2 * math.pi * (x - 1)))
for x in cs])
fitness.append(fit)
os.mkdir("D:\test\generation%d\individual%d" % (generation_id,individual_id))
individual_id += 1
generation_id += 1
return fitness
but failed.
So, I don't know how to handle this situation, cause i have to read and write data in the evaluator function.
Aaron Garrett
Jul 10, 2012, 12:32:23 AM7/10/12
to insp...@googlegroups.com
I think it's the global that's causing the problem. You should just pass that in as an integer in the args variable
args['generation_id'] = whatever
and modify it the same way. Just make sure you initially pass it in as a 0 in the call to the evolve method.
If all you need are temporary directories with unique names, though, you can use the tempfile.mkdtemp function in the standard Python library. That may solve your problem without needing any extra parameters.
-- Aaron Garrett
--
You received this message because you are subscribed to the Google Groups "Inspyred" group.
To view this discussion on the web visit https://groups.google.com/d/msg/inspyred/-/3iXZZECYyBgJ.
uqzcpo
Jul 11, 2012, 2:42:57 AM7/11/12
to insp...@googlegroups.com
Hi Aaron Garrett.
I wrote another piece of code to generate mesh using pp。
The 'stator1.geomTurbo' file represent geometry, the 'stator1_mesh.py' file is the script to generate mesh, the function 'mesh_pre()' generate the .p file from the .geomTurbo file.
The function 'mesh_gen()' call the commercial software from command line to generate the mash. I found that the mesh_pre() works well, but the mesh_gen() can't output the mesh.
I google the pp with os.popen, but still can not work out.
Here is the code:
import sys, time
import pp
import os
def mesh_pre(filename):
"""gnerate .py file from .geomTurbo file"""
w = open("%s_mesh.py" % filename[:7], 'w')
w.writelines("igg_script_version(1)\n"
"a5_open_template('D:/863/stator_numeca/stator01/mesh_thiner/mesh_thiner.trb')\n"\
"a5_get_row_number()\n"\
"row(1).load_geometry('D:/my_python_program/inspyred/%s')\n"\
"row(1).select()\n"\
"a5_start_3d_generation()\n"\
"row(1).unselect()\n"\
"a5_save_project('D:/my_python_program/inspyred/%s_mesh/%s.trb')" % (filename,filename[:7],filename[:7]))
w.close()
return True
def mesh_gen(filename):
"""generate mesh from .py file"""
os.popen(r'C:\NUMECA_SOFTWARE\Fine86_1\bin\igg.exe -autogrid5 -batch -script D:\my_python_program\inspyred\%s' % filename)
return True
# tuple of all parallel python servers to connect with
ppservers = ()
#ppservers = ("10.0.0.1",)
if len(sys.argv) > 1:
ncpus = int(sys.argv[1])
# Creates jobserver with ncpus workers
job_server = pp.Server(ncpus, ppservers=ppservers)
else:
# Creates jobserver with automatically detected number of workers
job_server = pp.Server(ppservers=ppservers)
print "Starting pp with", job_server.get_ncpus(), "workers"
start_time = time.time()
# The following submits 8 jobs and then retrieves the results
inputs = ('stator1.geomTurbo', 'stator2.geomTurbo', 'stator3.geomTurbo', 'stator4.geomTurbo')
jobs = [(input, job_server.submit(mesh_pre,(input,),(),())) for input in inputs]
for input, job in jobs:
print "mesh_pre done: ", input, "is", job()
inputs = ('stator1_mesh.py', 'stator2_mesh.py', 'stator3_mesh.py', 'stator4_mesh.py')
jobs = [(input, job_server.submit(mesh_gen,(input,),(),('os',))) for input in inputs]
for input, job in jobs:
print "mesh_gen done: ", input, "is", job()
print "Time elapsed: ", time.time() - start_time, "s"
job_server.print_stats()
This program can run without error, but no mesh obtained, it means the commercial software does not work at all, but when I run the cammand individually it works.
I wrote another piece of code to generate mesh using pp。
The 'stator1.geomTurbo' file represent geometry, the 'stator1_mesh.py' file is the script to generate mesh, the function 'mesh_pre()' generate the .p file from the .geomTurbo file.
The function 'mesh_gen()' call the commercial software from command line to generate the mash. I found that the mesh_pre() works well, but the mesh_gen() can't output the mesh.
I google the pp with os.popen, but still can not work out.
Here is the code:
import sys, time
import pp
import os
def mesh_pre(filename):
"""gnerate .py file from .geomTurbo file"""
w = open("%s_mesh.py" % filename[:7], 'w')
w.writelines("igg_script_version(1)\n"
"a5_open_template('D:/863/stator_numeca/stator01/mesh_thiner/mesh_thiner.trb')\n"\
"a5_get_row_number()\n"\
"row(1).load_geometry('D:/my_python_program/inspyred/%s')\n"\
"row(1).select()\n"\
"a5_start_3d_generation()\n"\
"row(1).unselect()\n"\
"a5_save_project('D:/my_python_program/inspyred/%s_mesh/%s.trb')" % (filename,filename[:7],filename[:7]))
w.close()
return True
def mesh_gen(filename):
"""generate mesh from .py file"""
os.popen(r'C:\NUMECA_SOFTWARE\Fine86_1\bin\igg.exe -autogrid5 -batch -script D:\my_python_program\inspyred\%s' % filename)
return True
# tuple of all parallel python servers to connect with
ppservers = ()
#ppservers = ("10.0.0.1",)
if len(sys.argv) > 1:
ncpus = int(sys.argv[1])
# Creates jobserver with ncpus workers
job_server = pp.Server(ncpus, ppservers=ppservers)
else:
# Creates jobserver with automatically detected number of workers
job_server = pp.Server(ppservers=ppservers)
print "Starting pp with", job_server.get_ncpus(), "workers"
start_time = time.time()
# The following submits 8 jobs and then retrieves the results
inputs = ('stator1.geomTurbo', 'stator2.geomTurbo', 'stator3.geomTurbo', 'stator4.geomTurbo')
jobs = [(input, job_server.submit(mesh_pre,(input,),(),())) for input in inputs]
for input, job in jobs:
print "mesh_pre done: ", input, "is", job()
inputs = ('stator1_mesh.py', 'stator2_mesh.py', 'stator3_mesh.py', 'stator4_mesh.py')
jobs = [(input, job_server.submit(mesh_gen,(input,),(),('os',))) for input in inputs]
for input, job in jobs:
print "mesh_gen done: ", input, "is", job()
print "Time elapsed: ", time.time() - start_time, "s"
job_server.print_stats()
This program can run without error, but no mesh obtained, it means the commercial software does not work at all, but when I run the cammand individually it works.
Aaron Garrett
Jul 11, 2012, 8:41:25 AM7/11/12
to insp...@googlegroups.com
OK. I can't test your specific issue because I don't have that software. But here is my mock-up of what you're trying to do. First, you should just use the multiprocessing module instead of pp, since it's all on the same machine. So you'd use the parallel_evaluation_mp function in inspyred, instead. I'm using multiprocessing in this example code.
Aaron Garrett, Ph.D.
Assistant Professor
Computer Science
Jacksonville State University
Jacksonville, AL 36265
agar...@jsu.edu
256-782-5364
http://mcis.jsu.edu/faculty/agarrett
Second, my fake.py represents your program that I don't have. The test.py is my version of what I think you're trying to do. See if this helps you figure it out.
Aaron Garrett, Ph.D.
Assistant Professor
Computer Science
Jacksonville State University
Jacksonville, AL 36265
agar...@jsu.edu
256-782-5364
http://mcis.jsu.edu/faculty/agarrett
--
You received this message because you are subscribed to the Google Groups "Inspyred" group.
To view this discussion on the web visit https://groups.google.com/d/msg/inspyred/-/2NMhDP_NwxkJ.
uqzcpo
Jul 12, 2012, 8:01:29 AM7/12/12
to insp...@googlegroups.com
Hi, Aaron Garrett.
Thank you so much! I used the the multiprocessing module, and now my computer is fully used.
But another prolem is when I use a big pop_size like 50, then I found 1 individual is missed by the computer, it was not be computed, the index of the individual is not fixed, it may individual5 or individual23. Did you come across this kind of error. How can I make sure that every individual is cpmputed, and a proper fitness is obtained.
Thank you so much! I used the the multiprocessing module, and now my computer is fully used.
But another prolem is when I use a big pop_size like 50, then I found 1 individual is missed by the computer, it was not be computed, the index of the individual is not fixed, it may individual5 or individual23. Did you come across this kind of error. How can I make sure that every individual is cpmputed, and a proper fitness is obtained.
Aaron Garrett
Jul 12, 2012, 10:19:41 AM7/12/12
to insp...@googlegroups.com
I have never seen that happen. I suspect it's an issue with your evaluation function. When you say that the individual is missed, what does that mean? What is its fitness? Are you certain that your evaluation function isn't returning the bad value itself?
Aaron Garrett, Ph.D.
Assistant Professor
Computer Science
Jacksonville State University
Jacksonville, AL 36265
agar...@jsu.edu
256-782-5364
http://mcis.jsu.edu/faculty/agarrett
Aaron Garrett, Ph.D.
Assistant Professor
Computer Science
Jacksonville State University
Jacksonville, AL 36265
agar...@jsu.edu
256-782-5364
http://mcis.jsu.edu/faculty/agarrett
--
You received this message because you are subscribed to the Google Groups "Inspyred" group.
To view this discussion on the web visit https://groups.google.com/d/msg/inspyred/-/BgUVThsIsdUJ.
uqzcpo
Jul 15, 2012, 9:10:55 AM7/15/12
to insp...@googlegroups.com
The problem is that the CFD soler is not stable itself，I wrote a piece code to check the results, if some individuals was ignored by the solver, it would recompute the individual.
uqzcpo
Jul 15, 2012, 9:19:17 AM7/15/12
to insp...@googlegroups.com
Until now I still have no idea how to write a terminator to pause the evaluation, if my evaluation process was stopped by accident like power off, how can i go on from where i stopped last time. Can you show a example?
Aaron Garrett
Jul 15, 2012, 1:31:39 PM7/15/12
to insp...@googlegroups.com
That's a harder problem. Did the whole machine crash, or just the module that is calculating your fitnesses? If it's just the fitness evaluator, you should just write your evaluation function to test and re-evaluate any candidates for which the thing crashed. If it's the whole machine, that's a bit trickier. My first thought would be to pickle what you can and try to rebuild as much as possible when you run it again. Here's a pretty thrown-together example of the idea. There may certainly be better solutions.
Aaron Garrett, Ph.D.
Assistant Professor
Computer Science
Jacksonville State University
Jacksonville, AL 36265
agar...@jsu.edu
256-782-5364
http://mcis.jsu.edu/faculty/agarrett
import inspyred
import os
import pickle
import random
import time
def save_termination(population, num_generations, num_evaluations, args):
with open('ec.pkl', 'wb') as pickle_file:
random_state = args['_ec']._random.getstate()
archive = args['_ec'].archive
pickle.dump(population, pickle_file, -1)
pickle.dump(archive, pickle_file, -1)
pickle.dump(random_state, pickle_file, -1)
pickle.dump(num_generations, pickle_file, -1)
pickle.dump(num_evaluations, pickle_file, -1)
return False
if __name__ == '__main__':
max_gens = 300
prng = random.Random()
problem = inspyred.benchmarks.Ackley(2)
if os.path.exists('ec.pkl'):
with open('ec.pkl', 'rb') as pickle_file:
population = pickle.load(pickle_file)
archive = pickle.load(pickle_file)
random_state = pickle.load(pickle_file)
num_generations = pickle.load(pickle_file)
num_evaluations = pickle.load(pickle_file)
prng.setstate(random_state)
ea = inspyred.ec.EvolutionaryComputation(prng)
ea.selector = inspyred.ec.selectors.tournament_selection
ea.variator = [inspyred.ec.variators.uniform_crossover,
inspyred.ec.variators.gaussian_mutation]
ea.replacer = inspyred.ec.replacers.steady_state_replacement
ea.terminator = [inspyred.ec.terminators.generation_termination, save_termination]
ea.observer = inspyred.ec.observers.stats_observer
ea.archive = archive
final_pop = ea.evolve(seeds=[p.candidate for p in population],
generator=problem.generator,
evaluator=problem.evaluator,
pop_size=100,
bounder=problem.bounder,
maximize=problem.maximize,
tournament_size=7,
num_selected=2,
max_generations=max_gens - num_generations,
mutation_rate=0.2)
else:
prng.seed(1234)
ea = inspyred.ec.EvolutionaryComputation(prng)
ea.selector = inspyred.ec.selectors.tournament_selection
ea.variator = [inspyred.ec.variators.uniform_crossover,
inspyred.ec.variators.gaussian_mutation]
ea.replacer = inspyred.ec.replacers.steady_state_replacement
ea.terminator = [inspyred.ec.terminators.generation_termination, save_termination]
ea.observer = inspyred.ec.observers.stats_observer
final_pop = ea.evolve(generator=problem.generator,
evaluator=problem.evaluator,
pop_size=100,
bounder=problem.bounder,
maximize=problem.maximize,
tournament_size=7,
num_selected=2,
max_generations=20,#max_gens, # Normally, this would be max_gens, but we want a "crash".
mutation_rate=0.2)
Aaron Garrett, Ph.D.
Assistant Professor
Computer Science
Jacksonville State University
Jacksonville, AL 36265
agar...@jsu.edu
256-782-5364
http://mcis.jsu.edu/faculty/agarrett
On Sun, Jul 15, 2012 at 8:19 AM, uqzcpo <shuai.s...@gmail.com> wrote:
Until now I still have no idea how to write a terminator to pause the evaluation, if my evaluation process was stopped by accident like power off, how can i go on from where i stopped last time. Can you show a example?
--
You received this message because you are subscribed to the Google Groups "Inspyred" group.
To view this discussion on the web visit https://groups.google.com/d/msg/inspyred/-/uWm1-OmqOGYJ.
Reply all
Reply to author
Forward
0 new messages