Issues using scoop for gp

20 views
Skip to first unread message

Alan Teesdale

unread,
Nov 22, 2024, 5:17:10 AM11/22/24
to deap-users
I'm using scoop for multithreading for GP, and I'm getting this error semi-randomly `_pickle.UnpicklingError: pickle data was truncated` (full error below). I suspect this is because the trees are getting large that when pickled scoop will only send part of the byte string and thus it's failing to unpickle things.

I had 2 questions, is this actually the issue, and if so is there a fix for this?

If it matters I've the program seems to run fine using the multiprocessing library, but I would like to see if I can get scoop to work.

```
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/loaf/Documents/summer_research/first_attempt/.venv/lib/python3.12/site-packages/scoop/bootstrap/__main__.py", line 298, in <module>
    b.main()
  File "/home/loaf/Documents/summer_research/first_attempt/.venv/lib/python3.12/site-packages/scoop/bootstrap/__main__.py", line 92, in main
    self.run()
  File "/home/loaf/Documents/summer_research/first_attempt/.venv/lib/python3.12/site-packages/scoop/bootstrap/__main__.py", line 285, in run
    futures_startup()
  File "/home/loaf/Documents/summer_research/first_attempt/.venv/lib/python3.12/site-packages/scoop/bootstrap/__main__.py", line 261, in futures_startup
    return futures._startup(
           ^^^^^^^^^^^^^^^^^
  File "/home/loaf/Documents/summer_research/first_attempt/.venv/lib/python3.12/site-packages/scoop/futures.py", line 71, in _startup
    result = _controller.switch(rootFuture, *args, **kargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/loaf/Documents/summer_research/first_attempt/.venv/lib/python3.12/site-packages/scoop/_control.py", line 258, in runController
    future = execQueue.pop()
             ^^^^^^^^^^^^^^^
  File "/home/loaf/Documents/summer_research/first_attempt/.venv/lib/python3.12/site-packages/scoop/_types.py", line 359, in pop
    self.updateQueue()
  File "/home/loaf/Documents/summer_research/first_attempt/.venv/lib/python3.12/site-packages/scoop/_types.py", line 382, in updateQueue
    for future in self.socket.recvFuture():
                  ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/loaf/Documents/summer_research/first_attempt/.venv/lib/python3.12/site-packages/scoop/_comm/scoopzmq.py", line 352, in recvFuture
    received = self._recv()
               ^^^^^^^^^^^^
  File "/home/loaf/Documents/summer_research/first_attempt/.venv/lib/python3.12/site-packages/scoop/_comm/scoopzmq.py", line 239, in _recv
    thisFuture = pickle.loads(msg[1])
                 ^^^^^^^^^^^^^^^^^^^^
_pickle.UnpicklingError: pickle data was truncated```

Thanks,
Alan
Reply all
Reply to author
Forward
0 new messages