Using DEAP with scoop: do all toolbox and creator statements need to be global?

[Matt von Hippel]

Hi all,

I've been trying to write a script to do genetic programming with DEAP that can toggle between different PrimitiveSet()'s based on user input. The script works fine without parallelization, but when parallelized with scoop it runs into pickling errors, for example:

"

scoop._comm.scoopexceptions.ReferenceBroken: This element could not be pickled: (b'127.0.0.1:54540', 3):partial([<deap.gp.Primitive object at 0x7feefc3bf790>, <deap.gp.Primitive object at 0x7feefc3bf5b0>, <deap.gp.Primitive object at 0x7feefc3bf5b0>, <deap.gp.Primitive object at 0x7feefc3bf5b0>, <deap.gp.Terminal object at 0x7feefc3cdf80>, <deap.gp.Terminal object at 0x7feefc3cdf80>, <deap.gp.Terminal object at 0x7feefc3cdf00>, <deap.gp.Primitive object at 0x7feefc3bf5b0>, <deap.gp.Terminal object at 0x7feefc3cdd00>, <deap.gp.Terminal object at 0x7feefc3cde00>, <deap.gp.Primitive object at 0x7feefc3bf5b0>, <deap.gp.Primitive object at 0x7feefc3bf5b0>, <deap.gp.Primitive object at 0x7feefc3bf5b0>, <deap.gp.Terminal object at 0x7feefc3cde80>, <deap.gp.Terminal object at 0x7feefc3cdf80>, <deap.gp.Terminal object at 0x7feefc3cdc40>, <deap.gp.Primitive object at 0x7feefc3bf5b0>, <deap.gp.Primitive object at 0x7feefc3bf5b0>, <deap.gp.Terminal object at 0x7feefc3cdf80>, <deap.gp.Terminal object at 0x7feefc3cdd80>, <deap.gp.Terminal object at 0x7feefc3cdd80>],){}=None.

"

After reading a few related threads on here, it sounds like this issue could be caused by using toolbox.register in a function instead of within the global scope. The problem is, many of the functions I need to register in the toolbox depend on the PrimitiveSet, which cannot be determined before main() is run because it depends on user input. Do all toolbox statements need to be in the global scope in order to use scoop? (And does the same apply to creator statements?) And is there a way around this? I'm currently using algorithms.eaSimple() to run the genetic algorithm, so ideally I'd like to be able to pass that function a toolbox and population.

Cheers,

Matt