Pickling Error

[Grant Tregonning]

Hey,

I am trying to parallelise some of my GA framework using multi-processing:

fdist_values = Dist_Lookup.fdist_lookup

# Creating an aggregate of the shortest paths    
agg_dist=0
for dev_site in Proposed_Sites:
    
    processors = 4
    pool = multiprocessing.Pool(processors)
    # for site in fdist_values:
    def func(site):
        if dev_site==site[0]:
            # Add the shortest path to this site to an aggregate variable
            # Dividing by 1000 to convert to kilometres
            return site[1]/1000

    agg_dist = sum(pool.map(func, fdist_values))

fdist = agg_dist/len(Proposed_Sites)

return fdist

However, the above returns the following error message:

File "ga.py", line 344, in <module>
    start_run()
  File "ga.py", line 297, in start_run
    hof = Genetic_Algorithm()
  File "ga.py", line 163, in Genetic_Algorithm
    stats=stats, halloffame=hof)
  File "C:\Users\b6051089\AppData\Local\Continuum\anaconda2\envs\spatopt\lib\site-packages\deap\algorithms.py", line 303, in eaMuPlusLambda
    fitnesses = toolbox.map(toolbox.evaluate, invalid_ind)
  File "ga.py", line 113, in Evaluate
    Dist_Fit = Eval.Calc_fdist(Proposed_Sites, scenario.greenspace_development)
  File "evaluate.py", line 175, in Calc_fdist
    agg_dist = sum(pool.map(func, fdist_values))
  File "C:\Users\b6051089\AppData\Local\Continuum\anaconda2\envs\spatopt\lib\multiprocessing\pool.py", line 253, in map
    return self.map_async(func, iterable, chunksize).get()
  File "C:\Users\b6051089\AppData\Local\Continuum\anaconda2\envs\spatopt\lib\multiprocessing\pool.py", line 572, in get
    raise self._value
cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

Process finished with exit code 1

Any ideas? Thanks!

[Derek Tishler]

Seems like a scope issue, as you are working inside a loop. The workers wont be able to distribute this properly in a new process(s). Can you move the func into a global scope and pass it what it needs to do work. Also careful if on windows.