ssh to FreeBSD is hanging

[candlerb]

I'm stuck on this; I can't see the way forward.

I've installed FreeBSD 9.2 in a VM. I can ssh to it. I've also set "PermitRootLogins yes", so I can login directly either as a normal user or as root.

~~~

Brians-MacBook-Air:ansible $ ssh root@bsdtest echo ok

Password for root@freebsd92:

ok

Brians-MacBook-Air:ansible $ ssh brian@bsdtest echo ok

Password for brian@freebsd92:

ok

~~~

The client machine is OSX, and also has ansible 1.3.3. But it simply hangs whenever ansible tries to connect to the BSD machine:

~~~

Brians-MacBook-Air:ansible $ cat bsd.inv

bsdtest

Brians-MacBook-Air:ansible $ ansible bsdtest -i bsd.inv -u root -vvvv -m raw -a 'echo ok' -k

SSH password:

<bsdtest> ESTABLISH CONNECTION FOR USER: root

<bsdtest> EXEC ['sshpass', '-d7', 'ssh', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/brian/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', 'GSSAPIAuthentication=no', '-o', 'PubkeyAuthentication=no', '-o', 'User=root', '-o', 'ConnectTimeout=10', 'bsdtest', 'echo ok']

~~~

No further progress is displayed. It doesn't even time out: at least, I've waited several minutes and nothing more happens. Although ssh is being run with -vvv it appears that the output is being buffered so I can't see it.

If I hit ctrl-C then the FreeBSD box logs the following:

~~~

sshd[xxx]: Connection closed by x.x.x.x [preauth]

~~~

So definitely the TCP connection is being established. tcpdump at both sides shows the several packets exchanged, to the point of negotiating ciphers, and there are no retransmits at intervals - the packets just stop.

Version of ssh on the FreeBSD side is: SSH-2.0-OpenSSH_6.2_hpn13v11 FreeBSD-20130515

Version of sshpass on the client side is:

$ sshpass -V

sshpass 1.05 (C) 2006-2011 Lingnu Open Source Consulting Ltd.

This program is free software, and can be distributed under the terms of the GPL

See the COPYING file for more information.

Connecting to a Linux VM is fine.

Now, if I scp an ssh key across manually, it *does* work. So it looks to be something to do with sshpass.

After that:

~~~

$ ansible bsdtest -i bsd.inv -u brian -m ping

bsdtest | FAILED >> {

    "failed": true,

    "msg": "/usr/bin/python: not found\r\n",

    "parsed": false

}

~~~

Actually I had installed python (pkg_add -r python), but it has gone in /usr/local/bin/python. But that's a different problem (issue #205).

Thanks,

Brian.

[Scott Sturdivant]

Not sure how the inner workings of providing a password (and needing sshpass) go, but do make sure you set ansible_python_interpreter as suggested in #205 (and here:  http://www.ansibleworks.com/docs/faq.html#how-do-i-handle-python-pathing-not-having-a-python-2-x-in-usr-bin-python-on-a-remote-machine)  Perhaps once that's resolved, it'll be able to find sshpass.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[candlerb]

I found ansible_python_interpreter. However that doesn't make any difference to the sshpass problem.

If I kill the local-side sshpass process, then it terminates, but ansible doesn't show the output collected so far from ssh -vvv. This doesn't help. All it says is:

~~~

fatal: [bsdtest] => Authentication or permission failure.  In some cases, you may have been able to authenticate and did not have permissions on the remote directory. ... etc ...

~~~

So I replaced sshpass with a wrapper script:

~~~

#!/bin/sh -x

exec /usr/local/bin/sshpass.real "$@" 2>/tmp/sshpass.err

~~~

and I can see it hangs at the following point:

~~~

...

debug1: SSH2_MSG_NEWKEYS received

debug1: Roaming not allowed by server

debug1: SSH2_MSG_SERVICE_REQUEST sent

debug2: service_accept: ssh-userauth

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug2: key: /Users/brian/.ssh/id_dsa (0x10d324910)

debug2: key: /Users/brian/.ssh/id_rsa (0x0)

debug1: Authentications that can continue: publickey,keyboard-interactive

debug3: start over, passed a different list publickey,keyboard-interactive

debug3: preferred keyboard-interactive,password

debug3: authmethod_lookup keyboard-interactive

debug3: remaining preferred: password

debug3: authmethod_is_enabled keyboard-interactive

debug1: Next authentication method: keyboard-interactive

debug2: userauth_kbdint

debug2: we sent a keyboard-interactive packet, wait for reply

debug2: input_userauth_info_req

debug2: input_userauth_info_req: num_prompts 1

~~~

And removing the wrapper, I can replicate this without using ansible at all:

~~~

$ sshpass -f ert ssh -vvv brian@bsdtest echo hi

~~~

So this is clearly some incompatibility between sshpass and FreeBSD.

It would have been better if ansible could have collected and displayed the ssh -vvv output, but the workaround is to install ssh keys manually instead of using ansible.

[Brian Coca]

try using -c paramiko in the commandline, this will help narrow down the source of the issues.

--
Brian Coca
Stultorum infinitus est numerus
0110000101110010011001010110111000100111011101000010000001111001011011110111010100100000011100110110110101100001011100100111010000100001

Pedo mellon a minno

[jo...@globalgospel.info]

I had the same problem. -c paramiko worked for me the first time. But then when I switched from using root to the user, that seemed to hang to, but later in the process.

I had already uploaded SSH keys in my initial playbook, so I switched back to using the keys and dropped the -c paramiko, and everything worked. I wish I knew more to help diagnose the problem.

[Brian Coca]

@john, using -vvvv with the problem would give you good diagnostic output, just guessing here but it sounds like it was waiting for a prompt in your case, either for password or known_hosts.

[Phil Regnauld]

Hi,

I have the same problem as the original poster (Hi, bcandler!).

Same issue with sshpass hanging. Didn't try with paramiko.

-vvv doesn't help here, it just hangs, and ctrl-c shows this:

SSH password:
<some.machine.here> ESTABLISH CONNECTION FOR USER: admin
<some.machine.here> REMOTE_MODULE ping
<some.machine.here> EXEC ['sshpass', '-d7', 'ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/regnauld/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'StrictHostKeyChecking=no', '-o', 'Port=2222', '-o', 'GSSAPIAuthentication=no', '-o', 'PubkeyAuthentication=no', '-o', 'User=admin', '-o', 'ConnectTimeout=10', 'some.machine.here', "/bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1441958968.83-124161391615192 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1441958968.83-124161391615192 && echo $HOME/.ansible/tmp/ansible-tmp-1441958968.83-124161391615192'"]

CTRL-C here

^CTraceback (most recent call last):
  File "/opt/local/bin/ansible", line 226, in <module>
    (runner, results) = cli.run(options, args)
  File "/opt/local/bin/ansible", line 200, in run
    results = runner.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/__init__.py", line 1200, in run
    results = [ self._executor(h, None) for h in hosts ]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/__init__.py", line 532, in _executor
    exec_rc = self._executor_internal(host, new_stdin)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/__init__.py", line 629, in _executor_internal
    return self._executor_internal_inner(host, self.module_name, self.module_args, inject, port, complex_args=complex_args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/__init__.py", line 815, in _executor_internal_inner
    result = handler.run(conn, tmp, module_name, module_args, inject, complex_args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/action_plugins/normal.py", line 57, in run
    return self.runner._execute_module(conn, tmp, module_name, module_args, inject=inject, complex_args=complex_args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/__init__.py", line 411, in _execute_module
    tmp = self._make_tmp_path(conn)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/__init__.py", line 1020, in _make_tmp_path
    result = self._low_level_exec_command(conn, cmd, None, sudoable=False)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/__init__.py", line 948, in _low_level_exec_command
    in_data=in_data)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ansible/runner/connection_plugins/ssh.py", line 285, in exec_command
    rfd, wfd, efd = select.select(rpipes, [], rpipes, 1)
KeyboardInterrupt

[candlerb]

I was rarely able to get anything useful with `-vvvv`, especially if the far end gave some unexpected prompt or there was some problem with sshpass, because the stdout/stderr is lost.

Try following the debugging process I outlined earlier:

* assemble the full ssh command line from the ['pieces', 'shown', 'in', 'the', 'debug', 'output'], run it manually, and look for errors. Make sure you are including the sshpass part of the command line

* replace sshpass with a wrapper script which redirects stderr to a tempfile and execs the real sshpass; then run ansible

* make sure you have paramiko installed and used (I believe it's used by default now)

* try setting up ssh key authentication, and making ansible talk to the FreeBSD box with that instead of password authentication

If the last case works then it's a bit painful because it means you can't use ansible to bootstrap the ssh key authentication - but at least it may work after that step.

HTH, Brian.

[candlerb]

Now I think about it, I may never have gotten to the bottom of this, and just gave up using password authentication with FreeBSD. If you can replicate it like this:

echo "xyzzy" >ert

sshpass -f ert ssh -vvv brian@bsdtest echo hi

<hang>

then that shows that it's an sshpass problem, rather than an ansible one.

Regards, Brian.

[Phil Regnauld]

I did find it was an sshpass problem, and there's a bug registered
for it. It's a problem with PAM:

https://github.com/ansible/ansible/issues/11565