socket.timout() while connecting to a junos devices via network_cli

[vicky...@motadata.com]

I was exploring ansible and the first step i tried was getting some information from a router (a virtual gns3 image).

I am able to connect to the device using ssh, get into it's cli and run commands normally but not via ansible.

I ran asnible playbook with -vvvvv option( ansible-playbook -vvvvv test.yml -i inv ) and these were the logs.

ansible-playbook 2.7.10
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/vicky/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
/opt/workminuscode/ansiblePractice/try2/inv did not meet host_list requirements, check plugin documentation if this is unexpected
/opt/workminuscode/ansiblePractice/try2/inv did not meet script requirements, check plugin documentation if this is unexpected
Parsed /opt/workminuscode/ansiblePractice/try2/inv inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/dist-packages/ansible/plugins/callback/default.pyc

PLAYBOOK: test.yml *********************************************************************************************************************************************************************
1 plays in test.yml

PLAY [junosdev] ************************************************************************************************************************************************************************
META: ran handlers

TASK [collect default set of facts and configuration] **********************************************************************************************************************************
task path: /opt/workminuscode/ansiblePractice/try2/test.yml:5
<172.16.10.185> attempting to start connection
<172.16.10.185> using connection plugin network_cli
<172.16.10.185> local domain socket does not exist, starting it
<172.16.10.185> control socket path is /home/vicky/.ansible/pc/77de749e3e
<172.16.10.185>
The full traceback is:
Traceback (most recent call last):
  File "/usr/bin/ansible-connection", line 106, in start
    self.connection._connect()
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/connection/network_cli.py", line 333, in _connect
    newline=self._terminal.terminal_inital_prompt_newline)
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/connection/network_cli.py", line 408, in receive
    data = self._ssh_shell.recv(256)
  File "/home/vicky/.local/lib/python2.7/site-packages/paramiko/channel.py", line 701, in recv
    raise socket.timeout()
timeout

fatal: [172.16.10.185]: FAILED! => {
    "msg": ""
}
    to retry, use: --limit @/opt/workminuscode/ansiblePractice/try2/test.retry

PLAY RECAP *****************************************************************************************************************************************************************************
172.16.10.185              : ok=0    changed=0    unreachable=0    failed=1  

------------------------------------------------------------------------------------------------------------------------------------------------------------USEFUL INFORMATION--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The device which i am trying to manage is running JUNOS 17.1R1.8 Kernel 64-bit  JNPR-10.3-20170209.344539_build.

ansible version: 2.7.10

python version: 2.7.15rc1

The direcroty structure is

.
├── group_vars
│   └── junosdev.yml
├── inv
└── test.yml

Here is the playbook (test.ymll)content.

---
- hosts: junosdev
  gather_facts: no
  tasks:
   - name: collect default set of facts and configuration
     junos_facts:

content of inv-

[junosdev]
172.16.10.185

content of group_vars/junosdev -

ansible_connection: network_cli
ansible_network_os: junos
ansible_user: root
ansible_ssh_pass: "Mind@123"

I have modified few parameters of ansible.cfg file to ease the ssh connection but nothing worked. Here is the content of /etc/ansible/ansible.cfg file.

host_key_checking = False

log_path = /opt/log/ansible.log

look_for_keys = False

host_key_auto_add = True

I have only added the lines which are not a comment in the ansible.cfg file. I am at my wits end here. Any insight would be helpful.

[vicky...@motadata.com]

There was some additional information in the log file (/opt/log/ansible.log), so posting its content

2019-05-02 11:58:58,802 p=7595 u=vicky |  ansible-playbook 2.7.10

  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/vicky/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0]

2019-05-02 11:58:58,802 p=7595 u=vicky |  Using /etc/ansible/ansible.cfg as config file
2019-05-02 11:58:58,802 p=7595 u=vicky |  setting up inventory plugins
2019-05-02 11:58:58,836 p=7595 u=vicky |  /opt/workminuscode/ansiblePractice/try2/inv did not meet host_list requirements, check plugin documentation if this is unexpected
2019-05-02 11:58:58,836 p=7595 u=vicky |  /opt/workminuscode/ansiblePractice/try2/inv did not meet script requirements, check plugin documentation if this is unexpected
2019-05-02 11:58:58,839 p=7595 u=vicky |  Parsed /opt/workminuscode/ansiblePractice/try2/inv inventory source with ini plugin
2019-05-02 11:58:58,916 p=7595 u=vicky |  Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/dist-packages/ansible/plugins/callback/default.pyc
2019-05-02 11:58:58,995 p=7595 u=vicky |  PLAYBOOK: test.yml *********************************************************************************************************************************************************************
2019-05-02 11:58:58,996 p=7595 u=vicky |  1 plays in test.yml
2019-05-02 11:58:59,006 p=7595 u=vicky |  PLAY [junosdev] ************************************************************************************************************************************************************************
2019-05-02 11:58:59,010 p=7595 u=vicky |  META: ran handlers
2019-05-02 11:58:59,018 p=7595 u=vicky |  TASK [collect default set of facts and configuration] **********************************************************************************************************************************
2019-05-02 11:58:59,019 p=7595 u=vicky |  task path: /opt/workminuscode/ansiblePractice/try2/test.yml:5
2019-05-02 11:58:59,384 p=7603 u=vicky |  network_os is set to junos
2019-05-02 11:58:59,419 p=7610 u=vicky | paramiko [172.16.10.185] starting thread (client mode): 0x1b8acbd0L
2019-05-02 11:58:59,419 p=7610 u=vicky | paramiko [172.16.10.185] Local version/idstring: SSH-2.0-paramiko_2.4.2
2019-05-02 11:58:59,435 p=7610 u=vicky | paramiko [172.16.10.185] Remote version/idstring: SSH-2.0-OpenSSH_6.6.1
2019-05-02 11:58:59,435 p=7610 u=vicky | paramiko [172.16.10.185] Connected (version 2.0, client OpenSSH_6.6.1)
2019-05-02 11:58:59,443 p=7610 u=vicky | paramiko [172.16.10.185] kex algos:[u'curve255...@libssh.org', u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521', u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group-exchange-sha1', u'diffie-hellman-group14-sha1', u'diffie-hellman-group1-sha1'] server key:[u'ssh-rsa', u'ssh-dss', u'ecdsa-sha2-nistp256', u'ssh-ed25519'] client encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'arcfour256', u'arcfour128', u'aes12...@openssh.com', u'aes25...@openssh.com', u'chacha20...@openssh.com', u'aes128-cbc', u'3des-cbc', u'blowfish-cbc', u'cast128-cbc', u'aes192-cbc', u'aes256-cbc', u'arcfour', u'rijnda...@lysator.liu.se'] server encrypt:[u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'arcfour256', u'arcfour128', u'aes12...@openssh.com', u'aes25...@openssh.com', u'chacha20...@openssh.com', u'aes128-cbc', u'3des-cbc', u'blowfish-cbc', u'cast128-cbc', u'aes192-cbc', u'aes256-cbc', u'arcfour', u'rijnda...@lysator.liu.se'] client mac:[u'hmac-m...@openssh.com', u'hmac-s...@openssh.com', u'umac-...@openssh.com', u'umac-1...@openssh.com', u'hmac-sha...@openssh.com', u'hmac-sha...@openssh.com', u'hmac-ripe...@openssh.com', u'hmac-sha...@openssh.com', u'hmac-md...@openssh.com', u'hmac-md5', u'hmac-sha1', u'uma...@openssh.com', u'umac...@openssh.com', u'hmac-sha2-256', u'hmac-sha2-256-96', u'hmac-sha2-512', u'hmac-sha2-512-96', u'hmac-ripemd160', u'hmac-ri...@openssh.com', u'hmac-sha1-96', u'hmac-md5-96'] server mac:[u'hmac-m...@openssh.com', u'hmac-s...@openssh.com', u'umac-...@openssh.com', u'umac-1...@openssh.com', u'hmac-sha...@openssh.com', u'hmac-sha...@openssh.com', u'hmac-ripe...@openssh.com', u'hmac-sha...@openssh.com', u'hmac-md...@openssh.com', u'hmac-md5', u'hmac-sha1', u'uma...@openssh.com', u'umac...@openssh.com', u'hmac-sha2-256', u'hmac-sha2-256-96', u'hmac-sha2-512', u'hmac-sha2-512-96', u'hmac-ripemd160', u'hmac-ri...@openssh.com', u'hmac-sha1-96', u'hmac-md5-96'] client compress:[u'none', u'zl...@openssh.com'] server compress:[u'none', u'zl...@openssh.com'] client lang:[u''] server lang:[u''] kex follows?False
2019-05-02 11:58:59,444 p=7610 u=vicky | paramiko [172.16.10.185] Kex agreed: ecdh-sha2-nistp256
2019-05-02 11:58:59,444 p=7610 u=vicky | paramiko [172.16.10.185] HostKey agreed: ssh-ed25519
2019-05-02 11:58:59,444 p=7610 u=vicky | paramiko [172.16.10.185] Cipher agreed: aes128-ctr
2019-05-02 11:58:59,445 p=7610 u=vicky | paramiko [172.16.10.185] MAC agreed: hmac-sha2-256
2019-05-02 11:58:59,447 p=7610 u=vicky | paramiko [172.16.10.185] Compression agreed: none
2019-05-02 11:58:59,462 p=7610 u=vicky | paramiko [172.16.10.185] kex engine KexNistp256 specified hash_algo <built-in function openssl_sha256>
2019-05-02 11:58:59,462 p=7610 u=vicky | paramiko [172.16.10.185] Switch to new keys ...
2019-05-02 11:58:59,567 p=7610 u=vicky | paramiko [172.16.10.185] userauth is OK
2019-05-02 11:58:59,610 p=7610 u=vicky | paramiko [172.16.10.185] Authentication (password) successful!
2019-05-02 11:58:59,615 p=7610 u=vicky | paramiko [172.16.10.185] [chan 0] Max packet in: 32768 bytes
2019-05-02 11:58:59,618 p=7610 u=vicky | paramiko [172.16.10.185] [chan 0] Max packet out: 32768 bytes
2019-05-02 11:58:59,618 p=7610 u=vicky | paramiko [172.16.10.185] Secsh channel 0 opened.
2019-05-02 11:58:59,624 p=7610 u=vicky | paramiko [172.16.10.185] [chan 0] Sesch channel 0 request ok
2019-05-02 11:58:59,631 p=7610 u=vicky | paramiko [172.16.10.185] [chan 0] Sesch channel 0 request ok
2019-05-02 11:59:09,739 p=7595 u=vicky |  The full traceback is:

Traceback (most recent call last):
  File "/usr/bin/ansible-connection", line 106, in start
    self.connection._connect()
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/connection/network_cli.py", line 333, in _connect
    newline=self._terminal.terminal_inital_prompt_newline)
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/connection/network_cli.py", line 408, in receive
    data = self._ssh_shell.recv(256)
  File "/home/vicky/.local/lib/python2.7/site-packages/paramiko/channel.py", line 701, in recv
    raise socket.timeout()
timeout

2019-05-02 11:59:09,741 p=7595 u=vicky |  fatal: [172.16.10.185]: FAILED! => {
    "msg": ""
}
2019-05-02 11:59:09,742 p=7595 u=vicky |      to retry, use: --limit @/opt/workminuscode/ansiblePractice/try2/test.retry

2019-05-02 11:59:09,742 p=7595 u=vicky |  PLAY RECAP *****************************************************************************************************************************************************************************
2019-05-02 11:59:09,742 p=7595 u=vicky |  172.16.10.185              : ok=0    changed=0    unreachable=0    failed=1  
2019-05-02 11:59:09,808 p=7603 u=vicky |  shutdown complete
2019-05-02 11:59:09,811 p=7610 u=vicky | paramiko [172.16.10.185] EOF in transport thread

[vicky...@motadata.com]

For anyone going through a similar issue. Using paramiko_ssh helped me progess further but i was still facing issues. Then i changed the connection to netconf and everything worked smoothly.

On Thursday, May 2, 2019 at 11:51:01 AM UTC+5:30, vicky...@motadata.com wrote: