how to save all the individuals fitness values

[matthieu gaudet]

Hi everybody,

I'm using the cma-es module of DEAP. I would like to save to a file all the individuals evaluated during a optimisation.

My idea till now is to create a hof which length is the product of the number of individuals in a population by the number of generations. At the end of the algo, I would just have to save the hof to a file.

Would it work? How to save the hof to a file?

Thanks for your help.

Regards

Matthieu

[Rodolphe Coulon]

Hey Mathieu,

I *think* you can do this with a hall of fame set to record the pareto front: halloffame = tools.ParetoFront()

class deap.tools.ParetoFront([similar])

The Pareto front hall of fame contains all the non-dominated individuals that ever lived in the population. That means that the Pareto front hall of fame can contain an infinity of different individuals.

Other wise, you can always iterate your generations and record whatever you need to at each step of the evolution.

[matthieu gaudet]

Hi Rodolphe,

a big thanks to you for your fast answer. It seems that I may use this indeed... good good... but ! How can I save it to a file?

Thanks in advance

Regards

M

[Rodolphe Coulon]

Let me know if you want an example on how to iterate through generations, I can dig something out for you !

[Rodolphe Coulon]

No worries, I'm surprised I can help a little. Giving back is the least I can do as this discussion board thought me everything I know about DEAP.

At the end of your evolution, you can iterate through your Pareto hall of fame:

for instance, if you want to save each individual as a number to a row in a csv file:

# do all your generations, then:

for ind in halloffame:

    # append a list, let's say A. This depends on what you want to save to the file, really.

with open(nameCsv, "w") as output:
    writer = csv.writer(output, lineterminator='\n')
    writer.writerow(# list of strings with column names)
    writer.writerows(A)

[matthieu gaudet]

That would be nice... how can I iterate using the CMA-ES module?

Acutally, it would be nice if I show my main function. Where should I report it? I read that I should use a special platform for that?

Thanks a lot for your help

M

[François-Michel De Rainville]

The Pareto Hall of Fame will discard any dominated individual. This mean in one objective problem it will keep a single individual. What you can do is decorate the generate method with a loging decorator.

def logIndividuals(file):

    def decorator(func):

        def wrapper(*args, **kargs):

            individuals = func(*args, **kargs)

            for ind in individuals:

                pickle.dump(ind, file)

            return individuals

        return wrapper

    return decorator

and

toolbox.decorate("generate", logIndividuals(open("cma-ind.pkl", "w")))

To read it just do this

You might want to dump only the numpy.array(ind) or even list(ind) if you don't want to load all your class for reading.

--
You received this message because you are subscribed to the Google Groups "deap-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to deap-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rodolphe Coulon]

https://gist.github.com to share your code.

But check, if you haven't done so yet, my other message, I explain how to iterate through the pareto hall of fame after the evolution (and save to a csv), that might be enough for you?

[matthieu gaudet]

Thanks again. I have a look to that and let you know if I managed to do it.

Cheers

M

[matthieu gaudet]

Hi there,

just a little update on "how to save the entire eval values calculated?"

I'm creating a hof as big as the number of individuals that will be generated:

hof = tools.HallOfFame(popSize*NbGen)

and I'm saving it in a file as followed:

with open("test.csv", "a") as myfile:  
   
        for ind in range(0,len(hof)):
   
            print("%i %s, %s" % (ind,hof[ind], hof[ind].fitness.values), file=myfile)
   
    myfile.close()

Hope it helped

Cheers

M

[Yannick Hold-Geoffroy]

Hello,

Creating a Hall of Fame as big as your population won't achieve what you want: between two generations, if a new individual is worse than every entry in the HoF (which keeps every generation), it won't be recorded. Furthermore, your population will grow during your generations, individuals will be removed and new one created, which will lead to better individuals replacing worse one. This won't achieve the goal you were after, namely to save every evaluation.

After seeing your code, I see that you create an HoF as big as a population theoretically would grow during these populations. The problem with this is the speed: handling this HoF will be painfully slow because it will still keep individuals from the first generation and will try to order the new entries according to its whole content.

Your best bet will be to either put your evaluation saving feature in the evaluation function, either at the end of it or, even better, as a decorator (the solution proposed by François-Michel). Another solution would be to implement yourself the evolutionary loop (replace eaSimple by your code, for example) which saves the population at each iteration (named generation). For better speed, you could record the individuals in a simple FIFO (list or deque, for example) and use pickle, json or csvwriter to dump the data structure at the end of the program.

Hope it helped,

Yannick Hold

--

[matthieu gaudet]

Thanks yannick,

all that seems quiet nice but for a beginner in python and DEAP, it's not a piece of cake.

However, I will have a reason to fight a little bit with the code that is certainly the best way to learn things.

Thanks anyway for your advices.

Regards

Matthieu

[matthieu gaudet]

Hi to all of you,

as proposed in the previous message, I decided to integrate the process of saving individuals to a file in the evaluation loop using the following code:

with open("test.csv", "a") as myfile:  
   

      print("%f,%f,%f,%s" % (var[0],var[1],var[2], ind_result), file=myfile)
   
  myfile.close()

var[0] to var [2] being the gene of the individual to save and ind_result, the fitness value.

It works like a charm... only difference with the previous code... the individuals are not ordered but I guess open office calc will do it without problem.

Thanks again for your help

Regards

M

To unsubscribe from this group and stop receiving emails from it, send an email to deap-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "deap-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to deap-users+unsubscribe@googlegroups.com.

[François-Michel De Rainville]

You should have a look at the csv module which write csv files, it should be preferred to a print in a file.

To unsubscribe from this group and stop receiving emails from it, send an email to deap-users+...@googlegroups.com.

[matthieu gaudet]

Thanks a lot for this advice. I'll modify my script this way.

Regards