Can't run DEAP program with SCOOP (continuation from Github)

[Glen van den Bergen]

Hello,

Referring to the issue I raised on the Github page for DEAP.

https://github.com/DEAP/deap/issues/57

Felix mentions that the problem may be due to creator.create being called from somewhere other than the global scope of the main module. However, I call it after the import statements like so.

from deap import base, creator, tools, algorithms

...

from scoop import futures

...

creator.create("FitnessMax", base.Fitness, weights=(1.0,))

creator.create("Individual", list, fitness=creator.FitnessMax)

...

def main(argv):
    ...
    toolbox.register("map", futures.map)
    ...

if __name__ == '__main__':

    main(sys.argv[1:])

I followed the examples provided in the DEAP documentation, but still can't get my program to run in parallel on multiple workers. I get this traceback.

Traceback (most recent call last):

  File "/Applications/Canopy.app/appdata/canopy-1.1.0.1371.macosx-x86_64/Canopy.app/Contents/lib/python2.7/runpy.py", line 162, in _run_module_as_main

    "__main__", fname, loader, pkg_name)

  File "/Applications/Canopy.app/appdata/canopy-1.1.0.1371.macosx-x86_64/Canopy.app/Contents/lib/python2.7/runpy.py", line 72, in _run_code

    exec code in run_globals

  File "/Users/gvdb/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scoop/bootstrap/__main__.py", line 302, in <module>

    b.main()

  File "/Users/gvdb/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scoop/bootstrap/__main__.py", line 92, in main

    self.run()

  File "/Users/gvdb/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scoop/bootstrap/__main__.py", line 290, in run

    futures_startup()

  File "/Users/gvdb/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scoop/bootstrap/__main__.py", line 271, in futures_startup

    run_name="__main__"

  File "/Users/gvdb/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scoop/futures.py", line 64, in _startup

    result = _controller.switch(rootFuture, *args, **kargs)

  File "/Users/gvdb/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scoop/_control.py", line 253, in runController

    raise future.exceptionValue

IndexError: string index out of range

Any help and advice you can provide would be appreciated. And thanks for responding so promptly on Github, even though I posted it in the wrong location. :)

-GvdB

[Félix-Antoine Fortin]

Hi Glen,

Could you share a bit more of your code with us? To do so, please use https://gist.github.com. 

If not, have you been able to run one DEAP example with SCOOP? I doubt the string IndexError is coming from DEAP.

Regards,

Félix-Antoine

--
You received this message because you are subscribed to the Google Groups "deap-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to deap-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Glen van den Bergen]

Hi Felix,

Yes, I have been able to run a DEAP example with SCOOP; deap_ga_onemax.py. But for some reason mine won't work. I was reading through the documentation on using multiple processors and saw there's a note saying that pickling of the lambda function is not yet available in Python. I used lambda two times in my code. Could that be the reason?

Please find the Python application at the gist below. Apologies, it's a bit of a mess and might be hard to follow.

https://gist.github.com/gvdb/c35ded645cfafea345fb

Once again, thanks for your prompt replies and assistance.

[Félix-Antoine Fortin]

Ok, two things

1- Can you confirm it works without scoop ?

2- When you launch with scoop, every worker share the same filesystem?

3- With scoop, have you tried commenting the line 715 to 727? I would want to isolate DEAP algorithm from the rest to make sure that the problem happens during the algorithm execution and not the initialization. 

Félix-Antoine Fortin

[Félix-Antoine Fortin]

Ok, I am pretty sure I found the culprit.

You declare global variables on line 620:

https://gist.github.com/gvdb/c35ded645cfafea345fb#file-gistfile1-py-L620

If you reference any of these variable in functions that are called by workers, for example your evaluation function, your program will crash. The reason is that these variables are only initialized for the process that calls your main function. The workers only have access to the global scope state as it is before executing "if __name__ == '__main__'".

To share the value of these variables, you will need to refactor you code to use the SCOOP object sharing API. It is covered briefly by SCOOP documentation:

http://scoop.readthedocs.org/en/latest/usage.html#object-sharing-api

The other option is to pass these variables as parameters of your functions instead of using the evil global keyword.

Good luck!

Félix-Antoine

[Félix-Antoine Fortin]

To complete my answer, the problem is with the global variable G_aaSeq, which is used by buildNewPeptide, the latter being called by your evaluation function. Simply make G_assSeq a parameter of your evaluation function, and register its value in the toolbox and it should work.

Félix-Antoine Fortin

[Glen van den Bergen]

Do you mean like this?

toolbox.register("deap_aaSeq", G_aaSeq)

toolbox.register("evaluate", evaluateTotalMoment, deap_aaSeq=toolbox.deap_aaSeq)

def evaluateTotalMoment(individual, deap_aaSeq):
    structure = buildNewPeptide(individual, deap_aaSeq)  
    ...

When I try that, I get the following traceback.

Traceback (most recent call last):

  File "/Applications/Canopy.app/appdata/canopy-1.1.0.1371.macosx-x86_64/Canopy.app/Contents/lib/python2.7/runpy.py", line 162, in _run_module_as_main

    "__main__", fname, loader, pkg_name)

  File "/Applications/Canopy.app/appdata/canopy-1.1.0.1371.macosx-x86_64/Canopy.app/Contents/lib/python2.7/runpy.py", line 72, in _run_code

    exec code in run_globals

  File "/Users/gvdb/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scoop/bootstrap/__main__.py", line 302, in <module>

    b.main()

  File "/Users/gvdb/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scoop/bootstrap/__main__.py", line 92, in main

    self.run()

  File "/Users/gvdb/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scoop/bootstrap/__main__.py", line 290, in run

    futures_startup()

  File "/Users/gvdb/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scoop/bootstrap/__main__.py", line 271, in futures_startup

    run_name="__main__"

  File "/Users/gvdb/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scoop/futures.py", line 64, in _startup

    result = _controller.switch(rootFuture, *args, **kargs)

  File "/Users/gvdb/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/scoop/_control.py", line 253, in runController

    raise future.exceptionValue

TypeError: the first argument must be callable

I looked for an example of an evaluate function that has multiple parameters, but couldn't find any in the documentation.

[Glen van den Bergen]

Sorry, to be clearer, here's the result when I run it without SCOOP.

Traceback (most recent call last):

  File "/Users/gvdb/Dropbox/Public/BIOC7009/project/amp-hmv/evolvePeptide.py", line 740, in <module>

    main(sys.argv[1:])

  File "/Users/gvdb/Dropbox/Public/BIOC7009/project/amp-hmv/evolvePeptide.py", line 679, in main

    toolbox.register("deap_aaSeq", G_aaSeq)

  File "/Users/gvdb/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/deap/base.py", line 76, in register

    pfunc = partial(function, *args, **kargs)

TypeError: the first argument must be callable

[Glen van den Bergen]

Sorry again, I just realised my error.

You meant register it for evaluate, like this.

toolbox.register("evaluate", evaluateTotalMoment, deap_aaSeq=G_aaSeq)

And then have the custom evaluation function like this.

def evaluateTotalMoment(individual, deap_aaSeq):
    structure = buildNewPeptide(individual, deap_aaSeq)
    ...

It seems to be running now, and using multiple workers.

Thanks for all your help Felix, and sorry for asking you to debug my awful code. :)

DEAP is an excellent package and has saved a lot of time in my research.