multiprocessing in DEAP

[Naoki]

Hi.

I am trying  Multiprocessing for python on a windows machine. However, the four process occurs, had not which only one operation (CPU = 0% in the Task Manager but the moving process 25%).If I want to multi-process, whether there are things to do to something else

Code

import random
import numpy
import matplotlib.pyplot
import time
import multiprocessing

from deap import algorithms
from deap import base
from deap import creator
from deap import tools
# from docutils.utils.punctuation_chars import delimiters
IND_INIT_SIZE = 3000 
# MIN_ENERGY = 237178.013392/3600 
MIN_ENERGY =7255 
MIN_POWER = 303.4465137486
NBR_ITEMS = 3000 

# Create the item dictionary: item name is an integer, and value is
# a (weight, value) 2-uple.
items = {}
# Create random items and store them in the items' dictionary.
for i in range(NBR_ITEMS):
    items[i] = random.choice([[10,5],[10,10]])

creator.create("Fitness", base.Fitness, weights=(-1.0, -1.0))
creator.create("Individual", set, fitness=creator.Fitness)

toolbox = base.Toolbox()

# Attribute generator
toolbox.register("attr_item", random.randrange, NBR_ITEMS)

# Structure initializers
toolbox.register("individual", tools.initRepeat, creator.Individual, toolbox.attr_item, IND_INIT_SIZE)
toolbox.register("population", tools.initRepeat, list, toolbox.individual)

def evalKnapsack(individual):
    energy = 0.0
    power = 0.0
    for item in individual:
        energy += items[item][1]
        power += items[item][0]
    if power < MIN_POWER or energy < MIN_ENERGY:
        return 100000000000,1000000000000
    return energy, power

def cxSet(ind1, ind2):
    """Apply a crossover operation on input sets. The first child is the
    intersection of the two sets, the second child is the difference of the
    two sets.
    """
    temp = set(ind1)                # Used in order to keep type
    ind1 &= ind2                    # Intersection (inplace)
    ind2 ^= temp                    # Symmetric Difference (inplace)
    return ind1, ind2

def mutSet(individual):
    """Mutation that pops or add an element."""
    for var in range(0,3000):
        if random.random() < 0.5:
            if len(individual) > 0:     # We cannot pop from an empty set
                individual.remove(random.choice(sorted(tuple(individual))))
            else:
                individual.add(random.randrange(NBR_ITEMS))
    return individual,

toolbox.register("evaluate", evalKnapsack)
toolbox.register("mate", cxSet)
toolbox.register("mutate", mutSet)
toolbox.register("select", tools.selSPEA2)
toolbox.register("map", pool.map)

def main():
#     random.seed(64)
    NGEN = 5
    MU = 75
    LAMBDA = 75
    CXPB = 0.6
    MUTPB = 0.3

    pop = toolbox.population(n=MU)
    hof = tools.ParetoFront()
    pool = multiprocessing.Pool(3)
    toolbox.register("map", pool.map)
    stats = tools.Statistics(lambda ind: ind.fitness.values)
    stats.register("avg", numpy.mean, axis=0)
    stats.register("std", numpy.std, axis=0)
    stats.register("min", numpy.min, axis=0)
    stats.register("max", numpy.max, axis=0)

    algorithms.eaMuPlusLambda(pop, toolbox, MU, LAMBDA, CXPB, MUTPB, NGEN, stats,
                              halloffame=hof)
    return pop, stats, hof
if __name__ == '__main__':  
    for var in range(0,5):
        start = time.time()
        pop, stats, hof= main()
        lischp=[]
        lisclp=[]
        libatthp=[]
        libattlp=[]
        ligoukei=[]
        for ind in hof:
            itemslist=[]
            print ind, ind.fitness
            for k in ind:
                itemslist.append(items[k])
            schpkazu=itemslist.count([10,5])
            lischp.append(schpkazu)
            battlpkazu=itemslist.count([10,10])
            libattlp.append(battlpkazu)
        print libatthp
        print lischp
        print libattlp
        print lisclp
        ligoukei.append(ind.fitness)
        print ligoukei
        with open('battlpcazu.csv',mode='a')as fb:
            numpy.savetxt(fb,libattlp,fmt="%.0f",delimiter=",")
        with open('schpcazu.csv',mode='a')as fc:
            numpy.savetxt(fc,lischp,fmt="%.0f",delimiter=",")

        elapsed_time = time.time() - start
        print ("elapsed_time:{0}".format(elapsed_time)) + "[sec]"

[Naoki]

I spite of specifying the four processes, python processes are five occurred.
Therefore, it is processing to move the processing and DEAP itself that produces the process seems to have become separate reason is I do not know.
Please help me.

2016年1月19日火曜日 18時36分02秒 UTC+9 Naoki:

[François-Michel De Rainville]

I think your evaluation function is too short to be able to see any improvement using multiprocessing.

--
You received this message because you are subscribed to the Google Groups "deap-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to deap-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Naoki]

@François
In other words, what that does not make sense even if the multiprocessing?
However, I want to and I think done in the multi-process, if possible.
Should I be in how the case?

2016年1月20日水曜日 22時23分19秒 UTC+9 François-Michel De Rainville:

[François-Michel De Rainville]

At first glance your script is ok. To test it, you can add a sleep of 0.1 second in the evaluation function, the running time should be significantly lower when using multiprocessing than the serial version.

It is normal to not seen any processing in slave processes when the evaluation is very fast. There is an overhead when using multiprocessing. The individuals need to be pickled, transferred, unpickled, then the result also need to be pickled transferred and unpickled. These operations are mostly done by the master process while the slave processes just compute the sum of 3000 integers (which is very fast).

Best regards,

François-Michel

[Naoki]

@François

The code in question is not, or is that not multi-process for the number of such NGEN is small?

But I was running in NGEN = 1000, MU = 5000, etc., but I have produced five process in spite of the specified to be before and unchanged in 4 process. The other one in the same way it did not go well with CPU = 0%.

2016年1月20日水曜日 23時10分11秒 UTC+9 François-Michel De Rainville:

[François-Michel De Rainville]

It is not the number of generation, but the time it takes to complete 1 evaluation.

[Naoki]

@François

I'm sorry.

But I really do not know where to do should I rewrite.
As shown in this question, it was rewritten the algorithm you can rewrite the following code.https://groups.google.com/forum/?hl=ja#!topic/deap-users/nFhq1dPUWkg

But I did not work well.
What is the problem?

2016年1月21日木曜日 0時51分41秒 UTC+9 François-Michel De Rainville:

[François-Michel De Rainville]

Your code appears to be correct.

You cannot have any performance improvement using multiprocessing and you current evaluation function because it is too short.

On my computer, Python adds 3000 integers in about 120 micro seconds which is very short compared to the rest of the algorithm. Look at this for explanations https://en.wikipedia.org/wiki/Amdahl%27s_law

[Naoki]

@François
Now the parameter's the test.
In order to become a case as follows actually to do, without parallelization can I not end my
Simulation (when you try before in my PC took two weeks per loop.)

CODE

IND_INIT_SIZE = 30000
MIN_ENERGY =72550 
MIN_POWER = 303000.4465137486
NBR_ITEMS = 30000
・・・
 NGEN = More than 1000
 MU = More　than 5000
 LAMBDA = More than 5000

2016年1月21日木曜日 3時53分05秒 UTC+9 François-Michel De Rainville:

[François-Michel De Rainville]

I think we miss understood somewhere. For a single individuals if the time taken to pickle, transmit, unpickle the individuals, pickle, transmit, unpickle the result is greater than the time it take to compute you will not see any gain to parallelize. With the provided evaluation function:

ind_size = 30000

pop_size = 10000

def evalKnapsack(individual):
    energy = 0.0
    power = 0.0
    for item in individual:
        energy += items[item][1]
        power += items[item][0]
    if power < MIN_POWER or energy < MIN_ENERGY:
        return 100000000000,1000000000000
    return energy, power

The evaluation time is approximately 1.2 milliseconds per individual (12 seconds for a 10 000 individuals population) which is well below the overhead of parallelizing it. Maybe you bottleneck is somewhere else.

Best regards,
François-Michel

[Naoki]

@François

First of all thank you for me various answer to my question.

In other words, what that my code is not a multi-process for it is too easy to to multi-process?

But if so, when I wrote the code, as follows: Why does the four process is produced as the accompanying image.

Also, as the bottleneck where it is or would you think?

It takes 18 hours close as to be terminated even if the above code in this environment (CPU=Xeon X5670 2.93 GHz(6core)&mem=45gb), to me it is helpless.

CODE：
def main():
・・・

　  pool = multiprocessing.Pool(3)
    toolbox.register("map", pool.map)

・・・

    NGEN =100
    MU = 250
    LAMBDA = 250
・・・

2016年1月21日木曜日 4時28分33秒 UTC+9 François-Michel De Rainville:

[François-Michel De Rainville]

Not using multiprocessing you can profile your code : https://docs.python.org/2/library/profile.html

python -m profile script.py

Try increasing the number of individual gradually.

But it is probably SPEA2 which as the greater complexity. I think it is close too O(mn^2), where m is the number of objectives and n the number of individuals. 

[Naoki]

@François

Thank you.　I tried profile.
It seems bottle-neck should be "DEAP", guessing from the result (show in pasted figure) .
Is there anyway to solve this and make program faster?

Also, I defined number of process to 3, but there are 4 process and only one of them worked.
I can't understand the reason of this.
Do you have any clue?




2016年1月21日木曜日 20時50分42秒 UTC+9 François-Michel De Rainville:

[Naoki]

@François

Postscript
The　parametar is

NGEN = 5
MU = 50
LAMBDA = 50

2016年1月21日木曜日 22時58分41秒 UTC+9 Naoki:

[François-Michel De Rainville]

To make your program faster: change the selection method.

[Naoki]

Thanks,
I try that.

2016年1月24日日曜日 22時11分14秒 UTC+9 François-Michel De Rainville: