Failure in launching workers and brokers via SCOOP Python

129 views
Skip to first unread message

Arash Safari

unread,
Feb 21, 2016, 7:22:25 PM2/21/16
to scoop-users
Following simple Python code which uses SCOOP for multiprocessing across different virtual machines, does not seem to work at all:

from __future__ import print_function
from scoop import futures


def helloWorld(value):
   
return "Hello World from Future #{0}".format(value)


if __name__ == "__main__":
    returnValues
= list(futures.map(helloWorld, range(16)))
   
print("\n".join(returnValues))






FYI - I have identical VMs (192.168.0.13 and 192.168.0.12) and they SSH Tunnel with no password is already set up.

I launch the program using the following command:


user_a@vm12:~$ python -m scoop --hosts vm13 vm13 -vv --verbose --external-hostname 192.168.0.13 --path /home/user_a ./pycode/hw-01.py



And the only output I can see is:

[2016-02-18 09:09:25,222] launcher  INFO    SCOOP 0.7 1.1 on linux2 using Python 2.7.6 (default, Jun 22 2015, 18:00:18) [GCC 4.8.2], API: 1013
[2016-02-18 09:09:25,223] launcher  INFO    Deploying 2 worker(s) over 1 host(s).
[2016-02-18 09:09:25,223] launcher  DEBUG   Using hostname/ip: "192.168.0.13" as external broker reference.
[2016-02-18 09:09:25,223] launcher  DEBUG   The python executable to execute the program with is: /usr/bin/python.
[2016-02-18 09:09:25,223] launcher  INFO    Worker distribution:
[2016-02-18 09:09:25,223] launcher  INFO       vm13:    1 + origin
[2016-02-18 09:09:25,224] brokerLaunch DEBUG   Launching remote broker: ssh -x -n -oStrictHostKeyChecking=no vm13 /usr/bin/python -m scoop.broker.__main__ --echoGroup --echoPorts --backend ZMQ
[2016-02-18 09:09:29,252] brokerLaunch DEBUG   Foreign broker launched on ports 44950, 55874 of host vm13.
[2016-02-18 09:09:29,253] launcher  DEBUG   Initialising remote worker 2 [vm13].
[2016-02-18 09:09:29,253] launcher  DEBUG   Initialising remote origin worker 1 [vm13].
[2016-02-18 09:09:29,254] launcher  DEBUG   vm13: Launching '(/usr/bin/python -m scoop.bootstrap.__main__ --echoGroup  --size 2 --workingDirectory /home/user_a --brokerHostname 192.168.0.13 --externalBrokerHostname 192.168.0.13 --taskPort 44950 --metaPort 55874 --backend=ZMQ -v ./pycode/hw-01.py & ) && (/usr/bin/python -m scoop.bootstrap.__main__ --size 2 --workingDirectory /home/user_a --brokerHostname 192.168.0.13 --externalBrokerHostname 192.168.0.13 --taskPort 44950 --metaPort 55874 --origin --backend=ZMQ -v ./pycode/hw-01.py)'
[2016-02-18 09:09:29,536] __main__  INFO    Worker(s) launched using /bin/bash



It is strange because if I run it on 1 machine only, everything works smoothly:

python -m scoop ./pycode/hw-01.py
[2016-02-18 09:12:52,739] launcher  INFO    SCOOP 0.7 1.1 on linux2 using Python 2.7.6 (default, Jun 22 2015, 18:00:18) [GCC 4.8.2], API: 1013
[2016-02-18 09:12:52,739] launcher  INFO    Deploying 2 worker(s) over 1 host(s).
[2016-02-18 09:12:52,739] launcher  INFO    Worker distribution:
[2016-02-18 09:12:52,740] launcher  INFO       127.0.0.1:   1 + origin
Hello World from Future #0
Hello World from Future #1
Hello World from Future #2
Hello World from Future #3
Hello World from Future #4
Hello World from Future #5
Hello World from Future #6
Hello World from Future #7
Hello World from Future #8
Hello World from Future #9
Hello World from Future #10
Hello World from Future #11
Hello World from Future #12
Hello World from Future #13
Hello World from Future #14
Hello World from Future #15
[2016-02-18 09:12:53,174] launcher  (127.0.0.1:34747) INFO    Root process is done.
[2016-02-18 09:12:53,175] launcher  (127.0.0.1:34747) INFO    Finished cleaning spawned subprocesses.



Any thought or idea is much appreciated.

Yannick Hold-Geoffroy

unread,
Apr 5, 2016, 11:12:08 AM4/5/16
to scoop-users
Hello,

Quick question, is there a reason why you launch the computation from vm12 instead of vm13?
What happens if you add vm12 as the first parameter of the --hosts value?
I'm guessing an issue with the remote broker launch could be the problem, but I don't have enough information to confirm/infirm what I'm thinking as of right now.

Hope you the best,
Yannick Hold
Reply all
Reply to author
Forward
0 new messages