Become_user whose shell is /usr/sbin/nologin in Ansible 2.4

[Mathias Ettinger]

I've been using become:yes and become_user:logstash in a playbook to install extra plugins. It all works great with ansible 2.3.1.0. I mistakenly ran the playbook using the devel branch (2.4) and the shell module suddently broke.

I still consider it a bug as it's a backward incompatible change and the shell module is flagged as stableinterface. But my issue (https://github.com/ansible/ansible/issues/26741#issuecomment-315177251) got redirected here...

So I guess I should ask where to get documentation on this new behaviour and how can I fix my playbook. Full details behind the provided link.

[Brian Coca]

so this is the command:

ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o User=user
-o ConnectTimeout=10 -o ControlPath=/home/kniyl/.ansible/cp/c01e80a299
-tt 172.20.34.42 '/bin/sh -c '"'"'sudo -H -S -p "[sudo via ansible,
key=igumwenfzxlgcmexugwnmhdyisluhuet] password: " -u logstash /bin/sh
-c '"'"'"'"'"'"'"'"'echo
BECOME-SUCCESS-igumwenfzxlgcmexugwnmhdyisluhuet; /usr/bin/python
/tmp/ansible-tmp-1499936409.552182-155553290806161/command.py'"'"'"'"'"'"'"'"'
&& sleep 0'"'"''

And this is the response ... which seems to indicate sudo is failing
due to DNS resolution:

<172.20.34.42> (1, b'sudo: unable to resolve host
auditorium-valid\r\n\r\n\r\n ....

This does not seem to be an Ansible error, but a configuration one on
the remote.

----------
Brian Coca

[Mathias Ettinger]

This error is harmless. Hosts are openstack VMs configured from a single instance snapshot (so they are identical each time). The error is due to /etc/hosts not containing a reference to the content of /etc/hostname. I can provide an execution stack with this error removed on monday if need be.

The real issue is that the shell module returns "This account is currently not available." which is excatly what is returned by the nologin shell. But only with ansible 2.4. Again, it all works well with 2.3.1.0, with the same host cloned from the same initial state, having the same sudo warning and all.

[Brian Coca]

Can you show the verbose version of the play succeeding?

----------
Brian Coca

[Mathias Ettinger]

Sure, but on monday only :/. I don't have access to those machines from home.

[Mathias Ettinger]

Here is the task executing with ansible 2.3.1.0 and a broken /etc/hosts. I also updated the issue on github to provide these informations after fixing /etc/hosts (and thus removing the warning from sudo).

TASK [Install logstash-output-influxdb] ****************************************
task path: /home/kniyl/debug.yml:21
Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
<172.20.34.42> ESTABLISH SSH CONNECTION FOR USER: user
<172.20.34.42> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o User=user -o ConnectTimeout=10 -o ControlPath=/home/kniyl/.ansible/cp/c01e80a299 172.20.34.42 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1500277934.32-200214434533419 `" && echo ansible-tmp-1500277934.32-200214434533419="` echo /tmp/ansible-tmp-1500277934.32-200214434533419 `" ) && sleep 0'"'"''
<172.20.34.42> (0, 'ansible-tmp-1500277934.32-200214434533419=/tmp/ansible-tmp-1500277934.32-200214434533419\n', 'OpenSSH_7.5p1, OpenSSL 1.1.0f  25 May 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 1110\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<172.20.34.42> PUT /tmp/tmp98A7s7 TO /tmp/ansible-tmp-1500277934.32-200214434533419/command.py
<172.20.34.42> SSH: EXEC sshpass -d12 sftp -o BatchMode=no -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o User=user -o ConnectTimeout=10 -o ControlPath=/home/kniyl/.ansible/cp/c01e80a299 '[172.20.34.42]'
<172.20.34.42> (0, 'sftp> put /tmp/tmp98A7s7 /tmp/ansible-tmp-1500277934.32-200214434533419/command.py\n', 'OpenSSH_7.5p1, OpenSSL 1.1.0f  25 May 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 1110\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-...@openssh.com" revision 1\r\ndebug2: Server supports extension "sta...@openssh.com" revision 2\r\ndebug2: Server supports extension "fsta...@openssh.com" revision 2\r\ndebug2: Server supports extension "hard...@openssh.com" revision 1\r\ndebug2: Server supports extension "fs...@openssh.com" revision 1\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/user size 0\r\ndebug3: Looking up /tmp/tmp98A7s7\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/tmp/ansible-tmp-1500277934.32-200214434533419/command.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:25707\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 25707 bytes at 32768\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<172.20.34.42> ESTABLISH SSH CONNECTION FOR USER: user
<172.20.34.42> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o User=user -o ConnectTimeout=10 -o ControlPath=/home/kniyl/.ansible/cp/c01e80a299 172.20.34.42 '/bin/sh -c '"'"'setfacl -m u:logstash:r-x /tmp/ansible-tmp-1500277934.32-200214434533419/ /tmp/ansible-tmp-1500277934.32-200214434533419/command.py && sleep 0'"'"''
<172.20.34.42> (0, '', 'OpenSSH_7.5p1, OpenSSL 1.1.0f  25 May 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 1110\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<172.20.34.42> ESTABLISH SSH CONNECTION FOR USER: user
<172.20.34.42> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o User=user -o ConnectTimeout=10 -o ControlPath=/home/kniyl/.ansible/cp/c01e80a299 -tt 172.20.34.42 '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=nqhzejbvxpwezqffhcggmfwdkonfycpo] password: " -u logstash /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-nqhzejbvxpwezqffhcggmfwdkonfycpo; /usr/bin/python /tmp/ansible-tmp-1500277934.32-200214434533419/command.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<172.20.34.42> (0, 'sudo: unable to resolve host auditorium-valid\r\n\r\n\r\n{"changed": true, "end": "2017-07-17 07:54:42.950209", "stdout": "Validating logstash-output-influxdb\\nInstalling logstash-output-influxdb\\nInstallation successful", "cmd": "bin/logstash-plugin install logstash-output-influxdb", "rc": 0, "start": "2017-07-17 07:51:12.640828", "stderr": "", "delta": "0:03:30.309381", "invocation": {"module_args": {"creates": "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-influxdb-5.0.1", "executable": null, "chdir": "/usr/share/logstash", "_raw_params": "bin/logstash-plugin install logstash-output-influxdb", "removes": null, "warn": true, "_uses_shell": true}}, "warnings": []}\r\n', 'OpenSSH_7.5p1, OpenSSL 1.1.0f  25 May 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 1110\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to 172.20.34.42 closed.\r\n')
<172.20.34.42> ESTABLISH SSH CONNECTION FOR USER: user
<172.20.34.42> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o User=user -o ConnectTimeout=10 -o ControlPath=/home/kniyl/.ansible/cp/c01e80a299 172.20.34.42 '/bin/sh -c '"'"'rm -f -r /tmp/ansible-tmp-1500277934.32-200214434533419/ > /dev/null 2>&1 && sleep 0'"'"''
<172.20.34.42> (0, '', 'OpenSSH_7.5p1, OpenSSL 1.1.0f  25 May 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 1110\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
changed: [172.20.34.42] => {
    "changed": true,
    "cmd": "bin/logstash-plugin install logstash-output-influxdb",
    "delta": "0:03:30.309381",
    "end": "2017-07-17 07:54:42.950209",
    "invocation": {
        "module_args": {
            "_raw_params": "bin/logstash-plugin install logstash-output-influxdb",
            "_uses_shell": true,
            "chdir": "/usr/share/logstash",
            "creates": "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-influxdb-5.0.1",
            "executable": null,
            "removes": null,
            "warn": true
        }
    },
    "rc": 0,
    "start": "2017-07-17 07:51:12.640828",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "Validating logstash-output-influxdb\nInstalling logstash-output-influxdb\nInstallation successful",
    "stdout_lines": [
        "Validating logstash-output-influxdb",
        "Installing logstash-output-influxdb",
        "Installation successful"
    ]
}

[Mathias Ettinger]

Code block seems to have messed things up. Here is the result verbatim:

[Mathias Ettinger]

I'm still interested into knowing how to make this task work with ansible 2.4.

Since the github issue is still closed, this doesn't seem to be considered a bug, so I need advices about fixing my playbook.

Thanks.

[Mathias Ettinger]

I thought I'd stick to ansible 2.3 for a while and ignore this problem, but my system upgrade this morning pulled in ansible 2.4.0.0

Now my playbook is unusable.

I still tried to add       become_flags: '-s /bin/bash'     but to no avail.


So asking the question again, how can I execute the shell module and "become: logstash" (knowing that the logstash user has been configured with the /usr/sbin/nologin shell) in ansible 2.4?

Thanks

[Mathias Ettinger]

For anyone hitting this issue as well, the proper fix is available in the Github issue and boils down to "use the executable argument of the shell module".

The shell module appears to have changed from using `/bin/sh` blindly to using the shell defined in the `$SHELL` environment variable of the connected user on the remote end; thus breaking the behaviour when the configured shell of the user cannot really execute commands (git-shell, nologin, /bin/false…).

This was totally overlooked because the default shell made it work in ansible prior to 2.4.0.0 when it theoretically shouldn't have.