Question about using DEAP on a linux cluster

Devin S

unread,

Nov 30, 2016, 5:30:29 PM11/30/16

to deap-users

Hi everyone

Can DEAP work on a Linux cluster?

My program is very computationally expensive. I want to parallel evaluation part on a Linux cluster.

Are there any example or tutorial about how to use DEAP library on Linux cluster, or any help documents related?

Thank you!

EBo

unread,

Nov 30, 2016, 6:18:05 PM11/30/16

to deap-...@googlegroups.com

Short answer is yes. I have used DEAP and SCOOP on Linux CLUSTERS at
home and at work at NASA. Unfortunately with my schedule I will not be
able to go through the long answer...

François-Michel De Rainville

unread,

Nov 30, 2016, 6:42:14 PM11/30/16

to deap-users

I use DEAP on Linux machines every day. You can parallelize DEAP with SCOOP on many cluster management programs. Just look at SCOOP documentation.

--
You received this message because you are subscribed to the Google Groups "deap-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to deap-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Markus Q

unread,

Dec 1, 2016, 3:52:13 AM12/1/16

to deap-users

Since scoop uses pickle and not dill/cloudpickle you may also want to have a look at dask.distributed together with https://github.com/dask/dask-drmaa

Am Donnerstag, 1. Dezember 2016 00:42:14 UTC+1 schrieb François-Michel De Rainville:

I use DEAP on Linux machines every day. You can parallelize DEAP with SCOOP on many cluster management programs. Just look at SCOOP documentation.

Le 30 nov. 2016 5:30 PM, "Devin S" <lpc...@gmail.com> a écrit :

Hi everyone

Can DEAP work on a Linux cluster?
My program is very computationally expensive. I want to parallel evaluation part on a Linux cluster.
Are there any example or tutorial about how to use DEAP library on Linux cluster, or any help documents related?

Thank you!

--
You received this message because you are subscribed to the Google Groups "deap-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to deap-users+...@googlegroups.com.

Devin S

unread,

Dec 11, 2016, 5:13:46 PM12/11/16

to deap-users

Thank you very much!

I have figured out how to use it on Linux system. But I meet a new problem.

My problem:

I use DEAP NSGA2 for multiobjective optimization. It works by parallelism and fast on cluster! O(∩_∩)O

But when I use it on one of my programs, DEAP cannot get a clear and continuous Pareto front which I got before. What I got is some gathered point group distributed on several regions, instead of a clear Pareto front curve. Small part of these point group overlap with Pareto front I got before and other distribute on wrong regions.

I think the possible reason is variety loss owing to bad parameter settings, improper mutation, crossover methods, or compact algorithm I used. Expect developers can give me some suggestions on how to use deap more effectively and properly!

My Program:

def nsga_ii(toolbox, stats=None, verbose=False):

pop = toolbox.population(n=toolbox.pop_size)

pop = toolbox.select(pop, len(pop))

return algorithms.eaMuPlusLambda(pop, toolbox, mu=toolbox.pop_size,

lambda_=toolbox.pop_size,

cxpb=0.5,

mutpb=0.5,

stats=stats,

ngen=10000,

verbose=verbose)

def uniform(low, up, size=None):

return [np.random.uniform(a, b) for a, b in zip(low, up)]

creator.create("FitnessMin", base.Fitness, weights=(-1.0,-1.0, -1.0))

creator.create("Individual", list, fitness=creator.FitnessMin)

NDIM=5

BOUND_LOW=[0.001, 0.001, 0.001, 0.001, 0.001]

BOUND_UP=[10.0, 25.0 ,10.0, 5.0, 5.0]

toolbox = base.Toolbox()

toolbox.register("attr_float", uniform, BOUND_LOW, BOUND_UP, NDIM)

toolbox.register("individual", tools.initIterate, creator.Individual, toolbox.attr_float)

toolbox.register("evaluate",fPID)

toolbox.register("population", tools.initRepeat, list, toolbox.individual)

toolbox.register("mate", tools.cxSimulatedBinaryBounded, low=BOUND_LOW, up=BOUND_UP, eta=20.0)

toolbox.register("mutate", tools.mutPolynomialBounded, low=BOUND_LOW, up=BOUND_UP, eta=20.0, indpb=1.0/NDIM)

toolbox.register("select", tools.selNSGA2)

toolbox.pop_size = 1000

toolbox.register("map", futures.map)

def main():

ddddd=time.time()

res, logbook = nsga_ii(toolbox)

fronts = tools.emo.sortLogNondominated(res, len(res))

with open('population.txt','a') as pfile:

for row in res:

a=fPID(row)

pfile.write(str(row[0])+','+str(row[1])+','+str(row[2])+','+str(row[3])+','+str(row[4])+','+str(a[0])+','+str(a[1])+','+str(a[2])+'\n')

pfile.close()

if name == "main":

main()

Thank you very much !

在 2016年11月30日星期三 UTC-5下午5:30:29，Devin S写道：

Reply all

Reply to author

Forward