Re: Scoop issues?

[Jason Zutty]

I have traced the program error to the following lines:

<script src="http://pastebin.com/embed_js.php?i=ck8YPzxh"></script>

I think the problem may be that the fitnesses variable is a generator object, and can not be handled the way I am handling it in the for loop.  

Is this correct?

Thanks,

Jason

On Monday, January 7, 2013 11:05:03 AM UTC-5, Jason Zutty wrote:

Hi All,

I am just getting up and running with DEAP, and I am enjoying it very much.  I was referred to this group from twitter, so here is my current issue:

I have attempted to implement SCOOP.  I installed the module, and am calling my code through the scoop module (python -m scoop mycode.py).

I have the following code:

if __name__ == '__main__':

    toolbox.register("map",futures.map)

    main()

my issue is I get the following error:

Traceback (most recent call last):

  File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main

    "__main__", fname, loader, pkg_name)

  File "C:\Python27\lib\runpy.py", line 72, in _run_code

    exec code in run_globals

  File "C:\Python27\lib\site-packages\scoop\bootstrap\__main__.py", line 99, in <module>

    run_name="__main__"))

  File "C:\Python27\lib\site-packages\scoop\futures.py", line 51, in _startup

    result = _controller.switch(rootFuture, *args, **kargs)

  File "C:\Python27\lib\site-packages\scoop\_control.py", line 83, in runController

    future = execQueue.pop()

  File "C:\Python27\lib\site-packages\scoop\_types.py", line 261, in pop

    self.updateQueue()

  File "C:\Python27\lib\site-packages\scoop\_types.py", line 284, in updateQueue

    for future in self.socket.recvFuture():

  File "C:\Python27\lib\site-packages\scoop\_comm.py", line 62, in recvFuture

    yield self._recv()

  File "C:\Python27\lib\site-packages\scoop\_comm.py", line 58, in _recv

    return pickle.loads(msg[1])

AttributeError: 'module' object has no attribute 'Individual'

Exception TypeError: "'NoneType' object is not callable" in <zmq.core.socket.Socket object at 0x00000000061499A8> ignored

Exception TypeError: "'NoneType' object is not callable" in <zmq.core.socket.Socket object at 0x0000000006149A08> ignored

My code runs fine without the scoop module.  Any ideas what might cause this?

Thanks,

Jason 

[Félix-Antoine Fortin]

Hi Jason,

We are glad that your enjoying DEAP, and we will make sure you enjoy SCOOP as well ;).

The key to your problem lies in the following line of the error stack :

AttributeError: 'module' object has no attribute 'Individual'

What you need to know, is that SCOOP workers only have to elements that are initialized in the global scope of your main program. It is the only part of the program that is run on every worker before they start pulling jobs from the broker.

A quick example with deap :

from deap import base

from deap import creator

from scoop import futures

creator.create("Fitness", base.Fitness, weights=(1.0,))

…

def main():

creator.create("Individual", list, fitness=creator.Fitness)

…

if __name__ == "__main__":

toolbox.register("map", futures.map)

main()

The type "creator.Fitness" will be accessible to every worker, as it is created in the global scope. However, the "creator.Individual" type will not, as the main function is only executed on the root node.

Having not seen your code, I can only guess that you have done something similar. However, the error is unequivocal, the type "Individual" is not available on the worker node that just received the job.

You can take a look at the example of DEAP using scoop : https://code.google.com/p/scoop/source/browse/examples/deap_ga_onemax.py

Feel free to ask more questions if I left something unclear.

Félix-Antoine Fortin

[Jason Zutty]

Thanks so much for the quick reply.  You were right that I had all those declarations inside my main() function.  

However, when I moved them in to the global scope, my issue remained.  Here is my full code:

<script src="http://pastebin.com/embed_js.php?i=59HzKzCV"></script>

Thanks,

Jason

[Jason Zutty]

Actually, it's working now.  It may have been the debugger.

Thank you!

[Félix-Antoine Fortin]

Actually, it's working now.  It may have been the debugger.

No problem! Your report actually made us realized that the documentation of SCOOP does not clearly explained the subtleties regarding the initialization of types and functions in the global scope of the main program. I have pointed it to the main SCOOP developer and it should be corrected shortly.

If you encounter anymore problem with either SCOOP or DEAP, do not hesitate to contact us.

Best regards,

Félix-Antoine Fortin

[Jason Zutty]

Also, it may be worth noting, that while scoop buys much greater power when using multiple machines, I am only using one.  I found that using a multiprocessing pool instead of scoop completed my code in 68 seconds, while using scoop my code completed in 391 seconds. Is there anything I can do to speed up scoop further? 

[Félix-Antoine Fortin]

scoop completed my code in 68 seconds, while using scoop my code completed in 391 seconds. Is there anything I can do to speed up scoop further? 

Is it still with the code on pastebin ? I asked Yannick (SCOOP main developer) to have a look. Normally, the performances should be on par.

We should get back to you on this subject soon.

Félix-Antoine

[Jason Zutty]

I tweaked it a bit, but basically yes.

[Yannick.Hold-Geoffroy]

Hello M. Zutty,

Your provided code was executing instantly on the computers I've tried. I modified your line 111 to have a substantial number of generations and tried using the development branch of SCOOP as well as the stable release (0.5.3) versus multiprocessing. Both gave a ~70% working time (~30% speedup) on 2 cores versus a serial run in my case. Multiprocessing as well as SCOOP game similar performances in the runs I've made, on both versions of SCOOP tested.

Have you tried to reproduce this problem on another system? Can you provide us your test scripts that executes in 68 seconds on multiprocessing and 391 seconds on SCOOP? On what operating system are you performing your tests? How many cores are involved? What Python version was used?

Have a nice day,

Yannick Hold

[Jason Zutty]

Here is a more up to date version.  It is currently configured for multiprocessing.

<script src="http://pastebin.com/embed_js.php?i=X25nuw4b"></script>

To change it back over just uncomment the scoop import and the futures.map line in if __name__ == '__main__', and comment the multiprocessing lines.

I have not tried to reproduce the results on another system.  I have scoop 0.5.3 and deap 0.9-rc1

I am using an i7, with 8 threads.

I am using python 2.7

Thanks,
Jason

[Yannick Hold-Geoffroy]

Hello M. Zutty,

I've tried your last example on several machines and setups. Most of the time, performances are pretty similar (27m 19.7s for multiprocessing versus 27m 6.7s for SCOOP using 4 physical cores) using the latest development code for SCOOP on ZeroMQ 2.2.0 and Python 2.7.3. Most of my tests were using Linux.

Since your problem generates a plethora of small futures (or small tasks), you would benefit greatly of Future (task) buffering. This feature was only added recently in the 0.6 development version of SCOOP, which explains why you get poor performances compared to its newer version or the multiprocessing module. I would suggest you to try the development code of SCOOP (available at our googlecode) or wait for the 0.6 release that is due for this week. If you still obtain your described performances using this version, please contact us on our mailing list.

On an i7, running multiprocessing using 4 or 8 cores doesn't exhibit a tremendous speedup (25s on a ~27m 15s run). This is explainable because an i7 Sandy Bridge only have 4 physical cores, the other 4 are virtual cores emulated by HyperThreading. While running your program using SCOOP, I've found that running on the amount of physical cores (4) showed to be on par with the multiprocessing module (results shown in the first paragraph) while running on every virtual core available (8) showed a 10% slowdown compared to the multiprocessing run (28m 40s for SCOOP). Can you confirm or infirm that running the latest development code of SCOOP on the physical number of cores is faster than running it using the number of available virtual cores? From what I deduce empirically, running at 4 or 5 workers is the best for a 4 core i7 with HyperThreading activated for your problem.

While we strive to get the best performances possible in every possible case while staying as simple as possible, SCOOP is still a work in progress and aimed at scaling notably at multiple networked resources. As you saw, switching from multiprocessing to SCOOP is simple.

Have a nice day,

Yannick Hold