Using DEAP with SCOOP

1,258 views
Skip to first unread message

Anjum Sayed

unread,
Nov 21, 2015, 5:02:45 PM11/21/15
to deap-users
Hi,

I'm trying to use DEAP with SCOOP. I'm starting my script like this:

if __name__ == "__main__":
    toolbox = base.Toolbox()
    toolbox.register("map", futures.map)

But i'm getting this error:

/usr/bin/python3.4 -m scoop /home/anjum/Python/Trading/Forex/backtest/genetic.py
[2015-11-21 21:30:23,990] launcher  INFO    SCOOP 0.7 1.1 on linux using Python 3.4.3 (default, Oct 14 2015, 20:28:29) [GCC 4.8.4], API: 1013
[2015-11-21 21:30:23,990] launcher  INFO    Deploying 2 worker(s) over 1 host(s).
[2015-11-21 21:30:23,990] launcher  INFO    Worker distribution: 
[2015-11-21 21:30:23,991] launcher  INFO       127.0.0.1:	1 + origin
Started at 2015-11-21 21:30:25.519308
[2015-11-21 21:30:25,521] scoopzmq  (b'127.0.0.1:56736') ERROR   An instance could not find its base reference on a worker. Ensure that your objects have their definition available in the root scope of your program.
Can't get attribute 'Individual' on <module 'deap.creator' from '/usr/local/lib/python3.4/dist-packages/deap/creator.py'>
[2015-11-21 21:30:25,522] scoopzmq  (b'127.0.0.1:56340') ERROR   A worker exited unexpectedly. Read the worker logs for more information. SCOOP pool will now shutdown.
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/scoop/_comm/scoopzmq.py", line 188, in _recv
    thisFuture = pickle.loads(msg[1])
AttributeError: Can't get attribute 'Individual' on <module 'deap.creator' from '/usr/local/lib/python3.4/dist-packages/deap/creator.py'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.4/dist-packages/scoop/bootstrap/__main__.py", line 302, in <module>
    b.main()
  File "/usr/local/lib/python3.4/dist-packages/scoop/bootstrap/__main__.py", line 92, in main
    self.run()
  File "/usr/local/lib/python3.4/dist-packages/scoop/bootstrap/__main__.py", line 290, in run
    futures_startup()
  File "/usr/local/lib/python3.4/dist-packages/scoop/bootstrap/__main__.py", line 271, in futures_startup
    run_name="__main__"
  File "/usr/local/lib/python3.4/dist-packages/scoop/futures.py", line 64, in _startup
    result = _controller.switch(rootFuture, *args, **kargs)
  File "/usr/local/lib/python3.4/dist-packages/scoop/_control.py", line 207, in runController
    future = execQueue.pop()
  File "/usr/local/lib/python3.4/dist-packages/scoop/_types.py", line 320, in pop
    self.updateQueue()
  File "/usr/local/lib/python3.4/dist-packages/scoop/_types.py", line 343, in updateQueue
    for future in self.socket.recvFuture():
  File "/usr/local/lib/python3.4/dist-packages/scoop/_comm/scoopzmq.py", line 279, in recvFuture
    received = self._recv()
  File "/usr/local/lib/python3.4/dist-packages/scoop/_comm/scoopzmq.py", line 197, in _recv
    raise ReferenceBroken(e)
scoop._comm.scoopexceptions.ReferenceBroken: Can't get attribute 'Individual' on <module 'deap.creator' from '/usr/local/lib/python3.4/dist-packages/deap/creator.py'>
[2015-11-21 21:30:26,185] launcher  (127.0.0.1:49737) INFO    Root process is done.
[2015-11-21 21:30:26,185] launcher  (127.0.0.1:49737) INFO    Finished cleaning spawned subprocesses.

Process finished with exit code 0


Does anyone know what I'm doing wrong?

Anjum Sayed

unread,
Nov 21, 2015, 6:06:03 PM11/21/15
to deap-users
I've also tried:

creator.create("FitnessMax", base.Fitness, weights=(1.0, 1.0))
creator.create("Individual", gp.PrimitiveTree, fitness=creator.FitnessMax)


toolbox = base.Toolbox()
toolbox.register("map", futures.map)

if __name__ == "__main__":
    pset = gp.PrimitiveSetTyped("MAIN", [x], bool)
    pset.addEphemeralConstant("rand_int", lambda: random.randint(1, 10), int)
    ......

Which gives this error:
[2015-11-21 23:03:00,525] launcher  INFO    SCOOP 0.7 1.1 on linux using Python 3.4.3 (default, Oct 14 2015, 20:28:29) [GCC 4.8.4], API: 1013
[2015-11-21 23:03:00,525] launcher  INFO    Deploying 2 worker(s) over 1 host(s).
[2015-11-21 23:03:00,525] launcher  INFO    Worker distribution: 
[2015-11-21 23:03:00,525] launcher  INFO       127.0.0.1:	1 + origin
Started at 2015-11-21 23:03:01.914771
/usr/local/lib/python3.4/dist-packages/deap/creator.py:141: RuntimeWarning: A class named 'FitnessMax' has already been created and it will be overwritten. Consider deleting previous creation of that class or rename it.
  RuntimeWarning)
/usr/local/lib/python3.4/dist-packages/deap/creator.py:141: RuntimeWarning: A class named 'Individual' has already been created and it will be overwritten. Consider deleting previous creation of that class or rename it.
  RuntimeWarning)
[2015-11-21 23:03:01,992] scoopzmq  (b'127.0.0.1:55560') ERROR   An instance could not find its base reference on a worker. Ensure that your objects have their definition available in the root scope of your program.
Can't get attribute 'rand_int' on <module 'deap.gp' from '/usr/local/lib/python3.4/dist-packages/deap/gp.py'>
[2015-11-21 23:03:01,993] scoopzmq  (b'127.0.0.1:53412') ERROR   A worker exited unexpectedly. Read the worker logs for more information. SCOOP pool will now shutdown.
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/scoop/_comm/scoopzmq.py", line 188, in _recv
    thisFuture = pickle.loads(msg[1])
AttributeError: Can't get attribute 'rand_int' on <module 'deap.gp' from '/usr/local/lib/python3.4/dist-packages/deap/gp.py'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.4/dist-packages/scoop/bootstrap/__main__.py", line 302, in <module>
    b.main()
  File "/usr/local/lib/python3.4/dist-packages/scoop/bootstrap/__main__.py", line 92, in main
    self.run()
  File "/usr/local/lib/python3.4/dist-packages/scoop/bootstrap/__main__.py", line 290, in run
    futures_startup()
  File "/usr/local/lib/python3.4/dist-packages/scoop/bootstrap/__main__.py", line 271, in futures_startup
    run_name="__main__"
  File "/usr/local/lib/python3.4/dist-packages/scoop/futures.py", line 64, in _startup
    result = _controller.switch(rootFuture, *args, **kargs)
  File "/usr/local/lib/python3.4/dist-packages/scoop/_control.py", line 207, in runController
    future = execQueue.pop()
  File "/usr/local/lib/python3.4/dist-packages/scoop/_types.py", line 320, in pop
    self.updateQueue()
  File "/usr/local/lib/python3.4/dist-packages/scoop/_types.py", line 343, in updateQueue
    for future in self.socket.recvFuture():
  File "/usr/local/lib/python3.4/dist-packages/scoop/_comm/scoopzmq.py", line 279, in recvFuture
    received = self._recv()
  File "/usr/local/lib/python3.4/dist-packages/scoop/_comm/scoopzmq.py", line 197, in _recv
    raise ReferenceBroken(e)
scoop._comm.scoopexceptions.ReferenceBroken: Can't get attribute 'rand_int' on <module 'deap.gp' from '/usr/local/lib/python3.4/dist-packages/deap/gp.py'>
[2015-11-21 23:03:02,624] launcher  (127.0.0.1:45690) INFO    Root process is done.
[2015-11-21 23:03:02,625] launcher  (127.0.0.1:45690) INFO    Finished cleaning spawned subprocesses.

Process finished with exit code 0

Thanks in advance!
Anjum

Anjum Sayed

unread,
Nov 25, 2015, 4:12:44 PM11/25/15
to deap-users
Ok, so I seem to have got things moving by moving a bunch of my code to the global namespace as discussed here: https://github.com/DEAP/deap/issues/92

But I'm still getting an issue which stems from the use of ephemeral constants:

[2015-11-25 21:04:04,417] launcher  INFO    SCOOP 0.7 1.1 on linux using Python 3.4.3 (default, Oct 14 2015, 20:28:29) [GCC 4.8.4], API: 1013
[2015-11-25 21:04:04,417] launcher  INFO    Deploying 2 worker(s) over 1 host(s).
[2015-11-25 21:04:04,417] launcher  INFO    Worker distribution: 
[2015-11-25 21:04:04,418] launcher  INFO       127.0.0.1: 1 + origin
Traceback (most recent call last):
  File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.4/dist-packages/scoop/bootstrap/__main__.py", line 302, in <module>
    b.main()
  File "/usr/local/lib/python3.4/dist-packages/scoop/bootstrap/__main__.py", line 92, in main
    self.run()
  File "/usr/local/lib/python3.4/dist-packages/scoop/bootstrap/__main__.py", line 290, in run
    futures_startup()
  File "/usr/local/lib/python3.4/dist-packages/scoop/bootstrap/__main__.py", line 271, in futures_startup
    run_name="__main__"
  File "/usr/local/lib/python3.4/dist-packages/scoop/futures.py", line 64, in _startup
    result = _controller.switch(rootFuture, *args, **kargs)
  File "/usr/local/lib/python3.4/dist-packages/scoop/_control.py", line 253, in runController
    raise future.exceptionValue
  File "/usr/local/lib/python3.4/dist-packages/scoop/_control.py", line 127, in runFuture
    future.resultValue = future.callable(*future.args, **future.kargs)
  File "/usr/lib/python3.4/runpy.py", line 240, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/usr/lib/python3.4/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/anjum/Python/backtest/genetic.py", line 89, in <module>
    pset.addEphemeralConstant("rand_int", lambda: random.randint(0, 10), int)
  File "/usr/local/lib/python3.4/dist-packages/deap/gp.py", line 398, in addEphemeralConstant
    raise Exception("Ephemerals with different functions should "
Exception: Ephemerals with different functions should be named differently, even between psets.
[2015-11-25 21:04:08,972] launcher  (127.0.0.1:48569) INFO    Root process is done.
[2015-11-25 21:04:08,973] launcher  (127.0.0.1:48569) INFO    Finished cleaning spawned subprocesses.

I'm defining the individuals and toolbox bits outside the main() part as per the onemax_island_scoop.py example. but it appears that there's some issue around how the ephemerals are named between workers. Does anyone know where the issue might lie?

Thanks!
Anjum

On Saturday, 21 November 2015 22:02:45 UTC, Anjum Sayed wrote:

Anjum Sayed

unread,
Nov 26, 2015, 10:01:34 AM11/26/15
to deap-users
Using the symbolic regression example gives the same error. I've posted this as an issue on GitHub: https://github.com/DEAP/deap/issues/108

Thanks,
Anjum

On Saturday, 21 November 2015 22:02:45 UTC, Anjum Sayed wrote:
Reply all
Reply to author
Forward
0 new messages