Figuring out the cause of SCOOP errors

[stefano franchi]

I am just beginning to use Scoop and on my first run of a DEAP-based GA algorithm I got several errors. The simulation works fine when I use the standard map function in my evaluation. When I switch to  scoop.futures.map I get a crash. The most relevant part of the message to console includes the following lines:

1. [2015-03-12 15:33:37,894] scoopzmq  (127.0.0.1:51756) WARNING Pickling Error: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
2. [2015-03-12 15:33:37,895] scoopzmq  (127.0.0.1:58058) ERROR   An instance could not find its base reference on a worker. Ensure that your objects have their definition available in the root scope of your program.
3, 'module' object has no attribute 'Individual'
4. [2015-03-12 15:33:37,895] scoopzmq  (127.0.0.1:51756) WARNING Pickling Error: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
5, [2015-03-12 15:33:37,896] scoopzmq  (127.0.0.1:51756) WARNING Pickling Error: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
6. [2015-03-12 15:33:37,896] scoopzmq  (127.0.0.1:51756) ERROR   A worker exited unexpectedly. Read the worker logs for more information. SCOOP pool will now shutdown.


I wonder:

1. Where are the workers' logs referenced to in the last error line? Do I have to turn on logging explicitly?

2. (Related) How can I figure out which unpicklable function I am apparently using that's causing the warnings in lines 1, 4, and 5? (I have three workers running on a 4 cores, CPU, so I guess it's the same line in a function called by my evaluation function, unless the identical port number after the local host implies that's the same worker?).

3. And finally, how to locate the source of the problem at line 3? Or is line 3 adding information to the previous error message in line (2)?

Thanks for the help,

Stefano

[Yannick Hold-Geoffroy]

Hello Stefano,

This particular error happens, as you guessed, when you try to pickle (on line 1 of your log) or unpickle (line 2) something unpicklable. Without any minimal code reproducing the error, it's hard to help you.

To answer your questions:

1. To get the workers' logs, you must pass the --debug flag before your executable name when you execute your program. In your case, I doubt the workers' logs would contain any more informations...

2. The warning is issued here. The problem is that at this point in the code, there's no easy way to determine the name of the problematic variable, so that would not help you much. You're right on the last part of your statement: if the port is the same, it's the same worker.

3. You could remove the exception at the line I've showed you (that would imply to clone the code source, modify the line and install this version using python setup.py develop or install). The problematic line would be then shown by the Python interpreter.

I reckon this is pretty tedious, I shall look to improve the reporting of these kind of problems in the future.

Hope it helps,

Yannick Hold

[stefano franchi]

Hi Yannick,

sorry for the lack of communication. I was finally able to get back to this project after classes ended, and I hope you'll still be able to help.

 

On Thu, Mar 26, 2015 at 5:45 PM, Yannick Hold-Geoffroy <yanni...@gmail.com> wrote:

Hello Stefano,

This particular error happens, as you guessed, when you try to pickle (on line 1 of your log) or unpickle (line 2) something unpicklable. Without any minimal code reproducing the error, it's hard to help you.

It's pretty hard to make a minimal example out of my project, but I wonder if I am not just stuck in a silly problem deriving from my ignorance of scoop.
Here is what I am doing:
I am using DEAP as recommended by the documentation, and my main script registers my fitness evaluation function in the toolbox with:

        self.toolbox.register("evaluate", self.evaluateGenomeFitness)

The evaluate function is then called with a standard:

fitnesses = list(self.toolbox.map(self.toolbox.evaluate, pop))

where pop is my population, which

When I look into the scoopzmq module, the error occurs in the sendFuture function (as you said), when trying to pickle the future. If i try to examine the future whose pickling fails I get the following:

('127.0.0.1:61807', 1):evaluate([0.45633846411220114, 0.31550441670394125, 0.46363608616776597, 0.3086209926362171, 0.013001842474000158, 0.479695830013139, 0.30240862309421357, 0.9165722604554994, 0.3919531148867911
, 0.3953272247666064, 0.028132711860598292, 0.19049827858512436, 1, 1, 1, 0.9373284201625616, 0.8533281554592619, 0.3064599033759874, 0.12908296866898905, 0.8413429259128087, 0.3096371515372175, 0.6618273189040453, 0.6294721895068692, 0.6311620990955914, 0.060221303510594
74, 0.6944992188457483, 0.5307969460415565, 0.6989416016954564, 0.9038258443890377, 0.4099931916830303, 0.14681802646464226, 0.25002287756824615, 0.4914022393561672, 0.48036429493834343, 0.45169332285883645, 0.842574097858232, 0.04598407925003645, 0.7535516976972665, 0.8468250631248619, 0.5286025069455741, 0, 0.8679229202047375, 0.4063290126830873, 0.8106516227766005, 0.10307882551748548, 0.7492834355935499, 0.22904989356350403, 0.44244994135760873, 0.3137342131591817, 0.9420434568819893, 0.8108710305376252, 0.8446610939161083, 0.14431289
49190028, 0.5522889707569374, 0.7600279012630509, 0.6822277039300773, 0.6833004887260508, 0.31444861136096725, 0.2723612722164602, 0.9558687908677073],){}=None

where all the floats are my 64 genes long genome.

I am not sure why I see 'evaluate' in the future, instead of the correct function that should be used (evaluateGenomeFitness). Perhaps that's the problem? Or is it a problem with pickle.dumps not reporting correctly where it fails?

Thanks for the help,

Stefano

--

__________________________________________________
Stefano Franchi

stefano...@gmail.com
http://stefano.cleinias.org