Potential issue with using scoop

[Alan Teesdale]

I don't know if this is a known issue, but I've run into an issue using scoop/multithreading library. The issue seems to be that I can't pass the toolbox into a function that is being used to multithread.

I think I've narrowed it down to the simplest working example:

```

t = Toolbox()
def test(toolbox: Toolbox, x) -> int:
    return x**2

t.register("map", futures.map)
t.register("test", test, toolbox)
if __name__ == "__main__":
    print(list(t.map(t.test, [1, 2, 3, 4])))

```

This code fails to run when the run by through `python -m scoop ....` but does work if you run it without using scoop `python ...`

When this fails it returns a rather large error message that ends with

```

raise ReferenceBroken("This element could not be pickled: "
scoop._comm.scoopexceptions.ReferenceBroken: This element could not be pickled: (b'127.0.0.1:64452', 1
):partial(1,){}=None.

```

I suspect this is because the toolbox can't be pickled.

```

t = Toolbox()
def test(toolbox: Toolbox, x) -> int:
    return x**2

t.register("map", futures.map)
t.register("test", test, toolbox)

if __name__ == "__main__":
    import pickle
    pickle.dumps(t)

```

The above code panics with the following error

```

Traceback (most recent call last):
  File "... .py", line 54, in <module>
    pickle.dumps(t)
_pickle.PicklingError: Can't pickle <function map at 0x733e05b294e0>: it's not the same object as scoop.futures.map```

Thanks,

Alan

[François-Michel De Rainville]

The best way to handle this is by making the toolbox global for every node and not relying to transferring it over network. There seem to be something that is not pickleable in your toolbox.

--
You received this message because you are subscribed to the Google Groups "deap-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to deap-users+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/deap-users/73a12818-556e-42de-8fdc-76236b799619n%40googlegroups.com.

[Alan Teesdale]

That is definitely correct, and that was the solution I ended up going with, but it felt worthwhile mentioning. Upon further investigation it appears to be actually an issue with adding futures.map to the tool box that makes it non pickleable.

```

from deap.base import Toolbox
import pickle
from scoop import futures

t = Toolbox()

t.register("map", futures.map)

if __name__ == "__main__":
    import pickle
    pickle.dumps(t)```

this code crashes with the following message

```

Traceback (most recent call last):

  File "/home/loaf/Sandbox/test.py", line 11, in <module>
    pickle.dumps(t)
_pickle.PicklingError: Can't pickle <function map at 0x7198f8614360>: it's not the same object as sco
op.futures.map

```

Versions if they're relevant

Python: 3.12.7

deap: 1.4.1

scoop: 0.7.2.0

Thanks,

Alan

[Alan Teesdale]

As a final note for what I decided to do is instead of passing in the whole toolbox, which couldn't be pickled due to the above error, I passed in just the function I needed (the compile function in my case). Interestingly I noticed some performance improvements when passing in the function rather using a global tool box when using scoop (very non-scientific).