Strange behavior: ESTABLISH SSH CONNECTION FOR USER: None

[Diego Lista]

Hello group, 

I would like to share this strange behavior.

PLAY [Update pi-hole servers] ************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************
ok: [zeropi.morelos.lan]
ok: [opizero.morelos.lan]

TASK [Update pi-hole] ********************************************************************************************************
changed: [zeropi.morelos.lan]
changed: [opizero.morelos.lan]

PLAY [Reconfigure DHCP to give both DNS servers] *****************************************************************************

TASK [Gathering Facts] *******************************************************************************************************
fatal: [zeropi.morelos.lan]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ro...@zeropi.morelos.lan: Permission denied (publickey,password).", "unreachable": true}
        to retry, use: --limit @/etc/ansible/playbooks/pi-hole_update.retry

PLAY RECAP *******************************************************************************************************************
opizero.morelos.lan        : ok=2    changed=1    unreachable=0    failed=0
zeropi.morelos.lan         : ok=2    changed=1    unreachable=1    failed=0

This is the Playbook

root@monitor ~# cat /etc/ansible/playbooks/pi-hole_update.yml
---
  - name: Update pi-hole servers
    hosts: pihole
    become: yes
    vars_files:
      - /etc/ansible/vars/pi-hole.yml

    tasks:
      - name: Update pi-hole
        shell: "pihole -up"

  - name: Reconfigure DHCP to give both DNS servers
    hosts: zeropi.morelos.lan
    become: yes
    tasks:
      - name: Reconfigure DHCP
        blockinfile:
          path: /etc/dnsmasq.d/02-pihole-dhcp.conf
          block: "dhcp-option=6,192.168.0.6,192.168.0.7"
          insertafter: "dhcp-option=option:router,192.168.0.1"
          state: present

      - name: Restart service
        service:
          name: pihole-FTL
          state: restarted

And this is the error with more verbosity (scroll down and see text in red...Why?)

TASK [Update pi-hole] ********************************************************************************************************
task path: /etc/ansible/playbooks/pi-hole_update.yml:9
Read vars_file '/etc/ansible/vars/pi-hole.yml'
<zeropi.morelos.lan> ESTABLISH SSH CONNECTION FOR USER: billiken
<zeropi.morelos.lan> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=billiken -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/fb9d4ad691 zeropi.morelos.lan '/bin/sh -c '"'"'echo ~billiken && sleep 0'"'"''
<opizero.morelos.lan> ESTABLISH SSH CONNECTION FOR USER: billiken
<opizero.morelos.lan> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=billiken -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/d44456bb79 opizero.morelos.lan '/bin/sh -c '"'"'echo ~billiken && sleep 0'"'"''
<opizero.morelos.lan> (0, b'/home/billiken\n', b'')
<opizero.morelos.lan> ESTABLISH SSH CONNECTION FOR USER: billiken
<opizero.morelos.lan> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=billiken -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/d44456bb79 opizero.morelos.lan '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/billiken/.ansible/tmp `"&& mkdir /home/billiken/.ansible/tmp/ansible-tmp-1688098406.404819-76255-68933708303734 && echo ansible-tmp-1688098406.404819-76255-68933708303734="` echo /home/billiken/.ansible/tmp/ansible-tmp-1688098406.404819-76255-68933708303734 `" ) && sleep 0'"'"''
<zeropi.morelos.lan> (0, b'/home/billiken\n', b'')
<zeropi.morelos.lan> ESTABLISH SSH CONNECTION FOR USER: billiken
<zeropi.morelos.lan> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=billiken -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/fb9d4ad691 zeropi.morelos.lan '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/billiken/.ansible/tmp `"&& mkdir /home/billiken/.ansible/tmp/ansible-tmp-1688098406.4177988-76254-195830227082798 && echo ansible-tmp-1688098406.4177988-76254-195830227082798="` echo /home/billiken/.ansible/tmp/ansible-tmp-1688098406.4177988-76254-195830227082798 `" ) && sleep 0'"'"''
<opizero.morelos.lan> (0, b'ansible-tmp-1688098406.404819-76255-68933708303734=/home/billiken/.ansible/tmp/ansible-tmp-1688098406.404819-76255-68933708303734\n', b'')
<zeropi.morelos.lan> (0, b'ansible-tmp-1688098406.4177988-76254-195830227082798=/home/billiken/.ansible/tmp/ansible-tmp-1688098406.4177988-76254-195830227082798\n', b'')
Using module file /usr/lib/python3/dist-packages/ansible/modules/commands/command.py
<zeropi.morelos.lan> PUT /root/.ansible/tmp/ansible-local-76210ktuoy3p3/tmp5cuf2kyt TO /home/billiken/.ansible/tmp/ansible-tmp-1688098406.4177988-76254-195830227082798/AnsiballZ_command.py
<zeropi.morelos.lan> SSH: EXEC sshpass -d13 sftp -o BatchMode=no -b - -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=billiken -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/fb9d4ad691 '[zeropi.morelos.lan]'
Using module file /usr/lib/python3/dist-packages/ansible/modules/commands/command.py
<opizero.morelos.lan> PUT /root/.ansible/tmp/ansible-local-76210ktuoy3p3/tmpm9l5d42u TO /home/billiken/.ansible/tmp/ansible-tmp-1688098406.404819-76255-68933708303734/AnsiballZ_command.py
<opizero.morelos.lan> SSH: EXEC sshpass -d13 sftp -o BatchMode=no -b - -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=billiken -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/d44456bb79 '[opizero.morelos.lan]'
<opizero.morelos.lan> (0, b'sftp> put /root/.ansible/tmp/ansible-local-76210ktuoy3p3/tmpm9l5d42u /home/billiken/.ansible/tmp/ansible-tmp-1688098406.404819-76255-68933708303734/AnsiballZ_command.py\n', b'')
<opizero.morelos.lan> ESTABLISH SSH CONNECTION FOR USER: billiken
<opizero.morelos.lan> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=billiken -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/d44456bb79 opizero.morelos.lan '/bin/sh -c '"'"'chmod u+x /home/billiken/.ansible/tmp/ansible-tmp-1688098406.404819-76255-68933708303734/ /home/billiken/.ansible/tmp/ansible-tmp-1688098406.404819-76255-68933708303734/AnsiballZ_command.py && sleep 0'"'"''
<zeropi.morelos.lan> (0, b'sftp> put /root/.ansible/tmp/ansible-local-76210ktuoy3p3/tmp5cuf2kyt /home/billiken/.ansible/tmp/ansible-tmp-1688098406.4177988-76254-195830227082798/AnsiballZ_command.py\n', b'')
<zeropi.morelos.lan> ESTABLISH SSH CONNECTION FOR USER: billiken
<zeropi.morelos.lan> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=billiken -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/fb9d4ad691 zeropi.morelos.lan '/bin/sh -c '"'"'chmod u+x /home/billiken/.ansible/tmp/ansible-tmp-1688098406.4177988-76254-195830227082798/ /home/billiken/.ansible/tmp/ansible-tmp-1688098406.4177988-76254-195830227082798/AnsiballZ_command.py && sleep 0'"'"''
<opizero.morelos.lan> (0, b'', b'')
<opizero.morelos.lan> ESTABLISH SSH CONNECTION FOR USER: billiken
<opizero.morelos.lan> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=billiken -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/d44456bb79 -tt opizero.morelos.lan '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=lejftgkijmhqqjrkcwhdbnejbjtfzwfr] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-lejftgkijmhqqjrkcwhdbnejbjtfzwfr; /usr/bin/python /home/billiken/.ansible/tmp/ansible-tmp-1688098406.404819-76255-68933708303734/AnsiballZ_command.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<zeropi.morelos.lan> (0, b'', b'')
<zeropi.morelos.lan> ESTABLISH SSH CONNECTION FOR USER: billiken
<zeropi.morelos.lan> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=billiken -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/fb9d4ad691 -tt zeropi.morelos.lan '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=ytsjpcrcmqzrvqszlkvpdvescwjpxotk] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-ytsjpcrcmqzrvqszlkvpdvescwjpxotk; /usr/bin/python /home/billiken/.ansible/tmp/ansible-tmp-1688098406.4177988-76254-195830227082798/AnsiballZ_command.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
Escalation succeeded
<zeropi.morelos.lan> (0, b'\r\n/home/billiken/.ansible/tmp/ansible-tmp-1688098406.4177988-76254-195830227082798/AnsiballZ_command.py:17: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module\'s documentation for alternative uses\r\n  import imp\r\n\r\n{"cmd": "pihole -up", "stdout": "  [i] Update local cache of available packages...\\r\\u001b[K  [\\u2713] Update local cache of available packages\\n  [i] Existing PHP installation detected : PHP version 8.1.2-1ubuntu2.11\\n  [i] Checking for git...\\r\\u001b[K  [\\u2713] Checking for git\\n  [i] Checking for iproute2...\\r\\u001b[K  [\\u2713] Checking for iproute2\\n  [i] Checking for dialog...\\r\\u001b[K  [\\u2713] Checking for dialog\\n  [i] Checking for ca-certificates...\\r\\u001b[K  [\\u2713] Checking for ca-certificates\\n\\n  [i] Checking for updates...\\n  [i] Pi-hole Core:\\tup to date\\n  [i] Web Interface:\\tup to date\\n  [i] FTL:\\t\\tup to date\\n\\n  [\\u2713] Everything is up to date!", "stderr": "", "rc": 0, "start": "2023-06-30 01:13:28.436699", "end": "2023-06-30 01:13:50.145443", "delta": "0:00:21.708744", "changed": true, "invocation": {"module_args": {"_raw_params": "pihole -up", "_uses_shell": true, "warn": true, "argv": null, "chdir": null, "executable": null, "creates": null, "removes": null, "stdin": null}}}\r\n', b'Shared connection to zeropi.morelos.lan closed.\r\n')
<zeropi.morelos.lan> ESTABLISH SSH CONNECTION FOR USER: billiken
<zeropi.morelos.lan> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=billiken -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/fb9d4ad691 zeropi.morelos.lan '/bin/sh -c '"'"'rm -f -r /home/billiken/.ansible/tmp/ansible-tmp-1688098406.4177988-76254-195830227082798/ > /dev/null 2>&1 && sleep 0'"'"''
<zeropi.morelos.lan> (0, b'', b'')
changed: [zeropi.morelos.lan] => {
    "changed": true,
    "cmd": "pihole -up",
    "delta": "0:00:21.708744",
    "end": "2023-06-30 01:13:50.145443",
    "invocation": {
        "module_args": {
            "_raw_params": "pihole -up",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "rc": 0,
    "start": "2023-06-30 01:13:28.436699",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "  [i] Update local cache of available packages...\r\u001b[K  [✓] Update local cache of available packages\n  [i] Existing PHP installation detected : PHP version 8.1.2-1ubuntu2.11\n  [i] Checking for git...\r\u001b[K  [✓] Checking for git\n  [i] Checking for iproute2...\r\u001b[K  [✓] Checking for iproute2\n  [i] Checking for dialog...\r\u001b[K  [✓] Checking for dialog\n  [i] Checking for ca-certificates...\r\u001b[K  [✓] Checking for ca-certificates\n\n  [i] Checking for updates...\n  [i] Pi-hole Core:\tup to date\n  [i] Web Interface:\tup to date\n  [i] FTL:\t\tup to date\n\n  [✓] Everything is up to date!",
    "stdout_lines": [
        "  [i] Update local cache of available packages...",
        "\u001b[K  [✓] Update local cache of available packages",
        "  [i] Existing PHP installation detected : PHP version 8.1.2-1ubuntu2.11",
        "  [i] Checking for git...",
        "\u001b[K  [✓] Checking for git",
        "  [i] Checking for iproute2...",
        "\u001b[K  [✓] Checking for iproute2",
        "  [i] Checking for dialog...",
        "\u001b[K  [✓] Checking for dialog",
        "  [i] Checking for ca-certificates...",
        "\u001b[K  [✓] Checking for ca-certificates",
        "",
        "  [i] Checking for updates...",
        "  [i] Pi-hole Core:\tup to date",
        "  [i] Web Interface:\tup to date",
        "  [i] FTL:\t\tup to date",
        "",
        "  [✓] Everything is up to date!"
    ]
}
<opizero.morelos.lan> (0, b'\r\n/home/billiken/.ansible/tmp/ansible-tmp-1688098406.404819-76255-68933708303734/AnsiballZ_command.py:17: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module\'s documentation for alternative uses\r\n  import imp\r\n\r\n{"cmd": "pihole -up", "stdout": "  [i] Update local cache of available packages...\\r\\u001b[K  [\\u2713] Update local cache of available packages\\n  [i] Existing PHP installation detected : PHP version 8.1.2-1ubuntu2.11\\n  [i] Checking for git...\\r\\u001b[K  [\\u2713] Checking for git\\n  [i] Checking for iproute2...\\r\\u001b[K  [\\u2713] Checking for iproute2\\n  [i] Checking for dialog...\\r\\u001b[K  [\\u2713] Checking for dialog\\n  [i] Checking for ca-certificates...\\r\\u001b[K  [\\u2713] Checking for ca-certificates\\n\\n  [i] Checking for updates...\\n  [i] Pi-hole Core:\\tup to date\\n  [i] Web Interface:\\tup to date\\n  [i] FTL:\\t\\tup to date\\n\\n  [\\u2713] Everything is up to date!", "stderr": "", "rc": 0, "start": "2023-06-30 01:13:28.943499", "end": "2023-06-30 01:13:51.562793", "delta": "0:00:22.619294", "changed": true, "invocation": {"module_args": {"_raw_params": "pihole -up", "_uses_shell": true, "warn": true, "argv": null, "chdir": null, "executable": null, "creates": null, "removes": null, "stdin": null}}}\r\n', b'Shared connection to opizero.morelos.lan closed.\r\n')
<opizero.morelos.lan> ESTABLISH SSH CONNECTION FOR USER: billiken
<opizero.morelos.lan> SSH: EXEC sshpass -d13 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=billiken -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/d44456bb79 opizero.morelos.lan '/bin/sh -c '"'"'rm -f -r /home/billiken/.ansible/tmp/ansible-tmp-1688098406.404819-76255-68933708303734/ > /dev/null 2>&1 && sleep 0'"'"''
<opizero.morelos.lan> (0, b'', b'')
changed: [opizero.morelos.lan] => {
    "changed": true,
    "cmd": "pihole -up",
    "delta": "0:00:22.619294",
    "end": "2023-06-30 01:13:51.562793",
    "invocation": {
        "module_args": {
            "_raw_params": "pihole -up",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "rc": 0,
    "start": "2023-06-30 01:13:28.943499",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "  [i] Update local cache of available packages...\r\u001b[K  [✓] Update local cache of available packages\n  [i] Existing PHP installation detected : PHP version 8.1.2-1ubuntu2.11\n  [i] Checking for git...\r\u001b[K  [✓] Checking for git\n  [i] Checking for iproute2...\r\u001b[K  [✓] Checking for iproute2\n  [i] Checking for dialog...\r\u001b[K  [✓] Checking for dialog\n  [i] Checking for ca-certificates...\r\u001b[K  [✓] Checking for ca-certificates\n\n  [i] Checking for updates...\n  [i] Pi-hole Core:\tup to date\n  [i] Web Interface:\tup to date\n  [i] FTL:\t\tup to date\n\n  [✓] Everything is up to date!",
    "stdout_lines": [
        "  [i] Update local cache of available packages...",
        "\u001b[K  [✓] Update local cache of available packages",
        "  [i] Existing PHP installation detected : PHP version 8.1.2-1ubuntu2.11",
        "  [i] Checking for git...",
        "\u001b[K  [✓] Checking for git",
        "  [i] Checking for iproute2...",
        "\u001b[K  [✓] Checking for iproute2",
        "  [i] Checking for dialog...",
        "\u001b[K  [✓] Checking for dialog",
        "  [i] Checking for ca-certificates...",
        "\u001b[K  [✓] Checking for ca-certificates",
        "",
        "  [i] Checking for updates...",
        "  [i] Pi-hole Core:\tup to date",
        "  [i] Web Interface:\tup to date",
        "  [i] FTL:\t\tup to date",
        "",
        "  [✓] Everything is up to date!"
    ]
}
META: ran handlers
META: ran handlers

PLAY [Reconfigure DHCP to give both DNS servers] *****************************************************************************

TASK [Gathering Facts] *******************************************************************************************************
task path: /etc/ansible/playbooks/pi-hole_update.yml:12
<zeropi.morelos.lan> ESTABLISH SSH CONNECTION FOR USER: None
<zeropi.morelos.lan> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/d38546ae06 zeropi.morelos.lan '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<zeropi.morelos.lan> (255, b'', b'ro...@zeropi.morelos.lan: Permission denied (publickey,password).\r\n')
fatal: [zeropi.morelos.lan]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: ro...@zeropi.morelos.lan: Permission denied (publickey,password).",
    "unreachable": true
}
        to retry, use: --limit @/etc/ansible/playbooks/pi-hole_update.retry

PLAY RECAP *******************************************************************************************************************
opizero.morelos.lan        : ok=2    changed=1    unreachable=0    failed=0
zeropi.morelos.lan         : ok=2    changed=1    unreachable=1    failed=0

[Dick Visser]

On Wed, 5 Jul 2023 at 19:22, Diego Lista <dli...@gmail.com> wrote:
>
> Hello group,
>
> I would like to share this strange behavior.
> PLAY [Update pi-hole servers] ************************************************************************************************
>
> TASK [Gathering Facts] *******************************************************************************************************
> ok: [zeropi.morelos.lan]
> ok: [opizero.morelos.lan]
>
> TASK [Update pi-hole] ********************************************************************************************************
> changed: [zeropi.morelos.lan]
> changed: [opizero.morelos.lan]
>
> PLAY [Reconfigure DHCP to give both DNS servers] *****************************************************************************
>
> TASK [Gathering Facts] *******************************************************************************************************
> fatal: [zeropi.morelos.lan]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ro...@zeropi.morelos.lan: Permission denied (publickey,password).", "unreachable": true}
> to retry, use: --limit @/etc/ansible/playbooks/pi-hole_update.retry

Based on the above, some tasks succeed, and then after a task called
[Reconfigure DHCP to give both DNS servers], things break.
That would point to some problem there, where you may have locked
yourself out in some way that caused the
What that is, is for you to know - you haven't posted any playbook.

[Diego Lista]

It´s a very small playbook.

---
  - name: Update pi-hole servers
    hosts: pihole
    become: yes
    vars_files:
      - /etc/ansible/vars/pi-hole.yml

    tasks:
      - name: Update pi-hole
        shell: "pihole -up"

  - name: Reconfigure DHCP to give both DNS servers
    hosts: zeropi.morelos.lan
    become: yes
    tasks:
      - name: Reconfigure DHCP
        blockinfile:
          path: /etc/dnsmasq.d/02-pihole-dhcp.conf
          block: "dhcp-option=6,192.168.0.6,192.168.0.7"
          insertafter: "dhcp-option=option:router,192.168.0.1"
          state: present

      - name: Restart service
        service:
          name: pihole-FTL
          state: restarted

And this is the file that I want to change every time the pi-hole is updated.

/etc/dnsmasq.d/02-pihole-dhcp.conf

###############################################################################
#  DHCP SERVER CONFIG FILE AUTOMATICALLY POPULATED BY PI-HOLE WEB INTERFACE.  #
#            ANY CHANGES MADE TO THIS FILE WILL BE LOST ON CHANGE             #
###############################################################################
dhcp-authoritative
dhcp-range=192.168.0.50,192.168.0.199,6h
dhcp-option=option:router,192.168.0.1
# BEGIN ANSIBLE MANAGED BLOCK
dhcp-option=6,192.168.0.6,192.168.0.7
# END ANSIBLE MANAGED BLOCK
dhcp-leasefile=/etc/pihole/dhcp.leases
#quiet-dhcp

domain=morelos.lan
local=/morelos.lan/