DEAP with scoop performance drop

Agris Nikitenko

unread,

Jun 3, 2019, 1:18:04 PM6/3/19

to deap-users

Hi,

I have managed to run deap example on scoop using 2xRPi as a cluster simulator.

I have noticed that actaully the performace drops using more cores on a single machine or several machines.

The cores are loaded evenly and seems to be wokring properly on both machines.

Can anyone explain whay?

evalOneMax() has been changed a little to provide some "work" for fitness evaluation function. Everything else is Copy/Paste from: https://github.com/soravux/scoop/blob/master/examples/deap_ga_onemax.py

The code I am running :

import array

import random
import numpy
from deap import algorithms
from deap import base
from deap import creator
from deap import tools
from timeit import default_timer as timer
from scoop import futures


creator.create("FitnessMax", base.Fitness, weights=(1.0,))
creator.create("Individual", array.array, typecode='b', fitness=creator.FitnessMax)
toolbox = base.Toolbox()
# Attribute generator
toolbox.register("attr_bool", random.randint, 0, 1)
# Structure initializers
toolbox.register("individual", tools.initRepeat, creator.Individual, toolbox.attr_bool, 100)
toolbox.register("population", tools.initRepeat, list, toolbox.individual)


def evalOneMax(individual):
    #random.seed(123)
    k = sum(individual)
    maxr = 0
    for i in range(1,1000):
        x = random.randint(0,5)
        if(x > maxr):
            maxr = x
    return k + maxr,

toolbox.register("evaluate", evalOneMax)
toolbox.register("mate", tools.cxTwoPoint)
toolbox.register("mutate", tools.mutFlipBit, indpb=0.05)
toolbox.register("select", tools.selTournament, tournsize=3)
toolbox.register("map", futures.map)

def main():
    random.seed(64)
    pop = toolbox.population(n=1000)
    hof = tools.HallOfFame(1)
    stats = tools.Statistics(lambda ind: ind.fitness.values)
    stats.register("avg", numpy.mean)
    stats.register("std", numpy.std)
    stats.register("min", numpy.min)
    stats.register("max", numpy.max)
    pop, log = algorithms.eaSimple(pop, toolbox, cxpb=0.5, mutpb=0.2, ngen=40,
                                   stats=stats, halloffame=hof, verbose=True)

    return pop, log, hof


if __name__ == "__main__":
    tic = timer()
    main()
    print(timer() - tic)

François-Michel De Rainville

unread,

Jun 3, 2019, 2:08:04 PM6/3/19

to deap-users

I think the overhead of communication is still much more significant than the evaluation time. 1000 is still a very small number.

--
You received this message because you are subscribed to the Google Groups "deap-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to deap-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/deap-users/8f727cca-b66c-4015-967d-237c5d5dd30a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Agris Nikitenko

unread,

Jun 3, 2019, 2:28:22 PM6/3/19

to deap-users

I agree, will check it. Thanks

Agris Nikitenko

unread,

Jun 4, 2019, 6:32:15 AM6/4/19

to deap-users

I checked with more intensive work increasing the loop constant to 10000.

So, it works!!! In a single core variant it took 1449 secs, while in 8 core (2x RPi) it took 249 secs.