SSH Remote broker port numbers not fetched

[Anjum Sayed]

Hi,

When I run SCOOP in a terminal or PyCharm I get this error:

/usr/bin/python3.5 -m scoop --hostfile hosts /home/anjum/Python/genetic.py

[2016-05-08 23:40:24,239] launcher  INFO    SCOOP 0.7 2.0 on linux using Python 3.5.1+ (default, Mar 30 2016, 22:46:26) [GCC 5.3.1 20160330], API: 1013

[2016-05-08 23:40:24,240] launcher  INFO    Deploying 1 worker(s) over 1 host(s).

[2016-05-08 23:40:24,240] launcher  INFO    Worker d--istribution: 

[2016-05-08 23:40:24,240] launcher  INFO       192.168.0.48: 0 + origin

ERROR:root:Error while launching SCOOP subprocesses:

ERROR:root:Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/scoop/launch/brokerLaunch.py", line 135, in __init__

    self.brokerPort, self.infoPort = ports

ValueError: not enough values to unpack (expected 2, got 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/scoop/launcher.py", line 486, in main

    rootTaskExitCode = thisScoopApp.run()

  File "/usr/local/lib/python3.5/dist-packages/scoop/launcher.py", line 262, in run

    ssh_executable=self.ssh_executable,

  File "/usr/local/lib/python3.5/dist-packages/scoop/launch/brokerLaunch.py", line 149, in __init__

    "SSH process stderr:\n{stderr}".format(**locals()))

Exception: Could not successfully launch the remote broker.

Requested remote broker ports, received:

b''

Port number decoding error:

not enough values to unpack (expected 2, got 1)

SSH process stderr:

b"Warning: Permanently added '192.168.0.48' (ECDSA) to the list of known hosts.\r\nPermission denied (publickey,password).\r\n"

[2016-05-08 23:40:24,492] launcher  INFO    Finished cleaning spawned subprocesses.

INFO:launcherLogger:Finished cleaning spawned subprocesses.

Process finished with exit code 255

I'm using the dev version of SCOOP from GitHub due to an issue with DEAP, greenlet 0.4.9 and pyzmq-15.2.0.

Thanks,

Anjum

[Yannick Hold-Geoffroy]

Hello Anjum,

The issue is written in the error:

"Permission denied (publickey,password)."

That means that SCOOP tried to perform "ssh <your host>" and the remote SSH server told it "Permission denied".

Ensure that you can log into the second system from the first one using a key (without entering your password), that should do the trick.

Also, if you could reorder the rows in your "hosts" file to have the system you will issue the launch command on the first line, that would help SCOOP by using a local broker instead of a remote one.

Have a nice day,

Yannick

[Anjum Sayed]

Hi Yannick,

Thanks for the quick reply. I've set up the ssh key pairs on both machines, and in a bash terminal I can directly login just by typing "ssh 192.168.0.49" in both directions. However, I'm still getting the same issue.

This is my new host file edited after your comments:

192.168.0.50 2   (This is the machine launching SCOOP)
192.168.0.49 24  (This is a worker machine)

And this is what I'm seeing:

/usr/bin/python3.5 -m scoop --hostfile hosts /home/anjum/Python/genetic.py

[2016-05-09 19:43:49,197] launcher  INFO    SCOOP 0.7 2.0 on linux using Python 3.5.1+ (default, Mar 30 2016, 22:46:26) [GCC 5.3.1 20160330], API: 1013
[2016-05-09 19:43:49,197] launcher  INFO    Deploying 26 worker(s) over 2 host(s).
[2016-05-09 19:43:49,197] launcher  INFO    Worker d--istribution: 
[2016-05-09 19:43:49,197] launcher  INFO       192.168.0.50:	1 + origin
[2016-05-09 19:43:49,206] launcher  INFO       192.168.0.49:	24 

ERROR:root:Error while launching SCOOP subprocesses:
ERROR:root:Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/scoop/launch/brokerLaunch.py", line 135, in __init__
    self.brokerPort, self.infoPort = ports
ValueError: not enough values to unpack (expected 2, got 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/scoop/launcher.py", line 486, in main
    rootTaskExitCode = thisScoopApp.run()
  File "/usr/local/lib/python3.5/dist-packages/scoop/launcher.py", line 262, in run
    ssh_executable=self.ssh_executable,
  File "/usr/local/lib/python3.5/dist-packages/scoop/launch/brokerLaunch.py", line 149, in __init__
    "SSH process stderr:\n{stderr}".format(**locals()))
Exception: Could not successfully launch the remote broker.
Requested remote broker ports, received:
b''
Port number decoding error:
not enough values to unpack (expected 2, got 1)
SSH process stderr:

b"Warning: Permanently added '192.168.0.50' (ECDSA) to the list of known hosts.\r\nPermission denied (publickey,password).\r\n"

[2016-05-09 19:43:49,381] launcher  INFO    Finished cleaning spawned subprocesses.

INFO:launcherLogger:Finished cleaning spawned subprocesses.

Process finished with exit code 255

Thanks very much for your help!
Anjum

[Anjum Sayed]

Ok sorry for the immediate reply, but I got it going :)

For some reason I had to ssh my own machine from itself (Inception springs to mind) and then set up the keys. That seemed to get things moving. But now iIm seeing this:

/usr/bin/python3.5 -m scoop --hostfile hosts /home/anjum/genetic.py

[2016-05-09 20:07:46,973] launcher  INFO    SCOOP 0.7 2.0 on linux using Python 3.5.1+ (default, Mar 30 2016, 22:46:26) [GCC 5.3.1 20160330], API: 1013

[2016-05-09 20:07:46,973] launcher  INFO    Deploying 0 worker(s) over 2 host(s).

[2016-05-09 20:07:46,973] launcher  INFO    Worker d--istribution: 

[2016-05-09 20:07:46,973] launcher  INFO       192.168.0.50: -1 + origin

[2016-05-09 20:07:46,974] launcher  INFO       192.168.0.49: 0 

Warning: Permanently added '192.168.0.50' (ECDSA) to the list of known hosts.

Warning: Permanently added '192.168.0.49' (ECDSA) to the list of known hosts.

/usr/bin/python3.5: Error while finding spec for 'scoop.launch.__main__' (<class 'ImportError'>: No module named 'scoop')

Connection to 192.168.0.49 closed.

Launching 2 worker(s) using /bin/bash.

Any ideas as to what going on? Do both machines need to have the same version of Python installed? The worker machine has 3.4 (Ubuntu 15.10), but the origin machine has 3.5 (Ubuntu 16.04).

Thanks,

Anjum

[Yannick Hold-Geoffroy]

Hello,

"-1" worker launched? Never saw this one before...

The Python 3.5 executable crashes because SCOOP is not installed on it. While different Python versions among workers *should* work (provided they have the same highest pickling protocol, so 2 and 3 together won't work), but you must install SCOOP on every worker beforehand. Be sure to install in on the Python version that will be used to perform the paralellization.

Have a nice day,

Yannick

[Anjum Sayed]

Thanks Yannick - I updated the other machine to Python 3.5 and SCOOP started doing its thing. Both machines did have the dev version of SCOOP installed, but it seemed like it was looking for SCOOP in "/usr/bin/python3.5" which I presume didn't exist because that machine was running 3.4.

I think maybe the "-1" worker bit is because my host file now doesn't have the number of workers defined, so it should default to the number of CPUs?

Thanks for helping me to get things moving.

Anjum

--
Vous recevez ce message, car vous êtes abonné à un sujet dans le groupe Google Groupes "scoop-users".
Pour vous désabonner de ce sujet, visitez le site https://groups.google.com/d/topic/scoop-users/TWFrXTguDik/unsubscribe.
Pour vous désabonner de ce groupe et de tous ses sujets, envoyez un e-mail à l'adresse scoop-users...@googlegroups.com.
Pour obtenir davantage d'options, consultez la page https://groups.google.com/d/optout.