ansible python interpreter discovery failing with what appears to be a SIGPIPE return code

[Joanne Martel]

I am seeing occasional problems where ansible-playbook fails during python interpreter discovery, and would appreciate any and all advice.
The only diagnostics I can find in ansible debug output is what appears to be a SIGPIPE return code.

Background: I am running rocky8, (Rocky Linux release 8.4 (Green Obsidian)), and using ansible-7.2.0-1.el8.noarch, ansible-core-2.14.2-3.el8.x86_64
with python 3.11.2 on one node and deploying to many other nodes also running rocky8 with python 3.6 installed. I have an application
the is installing and configuring software on all the nodes using ansible playbooks. Usually this runs without a problem but occasionally
I get a failures in a playbook when it is attempting python interpreter discovery. The playbooks are run multiple times on each node to
install/configure different services so the falling playbook is executed 1 or more times before an execution instance fails. If I execute
the playbook again after the failure it runs ok. I verified that /usr/bin/python3.6, /usr/bin/python3 and /usr/libexec/platform-python
all exist on each node before I start the application  


Debug output:

This is the output, with -vvvvv, when one of the playbooks fails.

 Attempting python interpreter discovery
<10.42.3.199> ESTABLISH SSH CONNECTION FOR USER: root
<10.42.3.199> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<10.42.3.199> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<10.42.3.199> SSH: ANSIBLE_REMOTE_PORT/remote_port/ansible_port set: (-o)(Port=22)
<10.42.3.199> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/root/.ssh/id_rsa")
<10.42.3.199> SSH: ansible_password/ansible_ssh_password not set:
(-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<10.42.3.199> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="root")
<10.42.3.199> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<10.42.3.199> SSH: Set ssh_common_args: ()
<10.42.3.199> SSH: Set ssh_extra_args: ()
<10.42.3.199> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/root/.ansible/cp/1a125e8154")
<10.42.3.199> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/root/.ssh/id_rsa"' -o
KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o
'ControlPath="/root/.ansible/cp/1a125e8154"' 10.42.3.199 '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'python3.10'"'"'"'"'"'"'"'"'; command -v
'"'"'"'"'"'"'"'"'python3.9'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.8'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v
'"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v
'"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v
'"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<10.42.3.199> (-13, b'', b"OpenSSH_8.0p1, OpenSSL 1.1.1k  FIPS 25 Mar 2021

My reading of the ansible source code indicates that the output of the last line is (rcode=-13, stdout, stderr).

The return code is from the subprocess.Popen call in the run_cmd. A return of a negative number indicates a the process was terminated by
a signal: "13 SIGPIPE    Terminate   Write to pipe with no readers".


Here is the output from a successful run of the playbook with the same arguments:

Attempting python interpreter discovery
<10.42.3.199> ESTABLISH SSH CONNECTION FOR USER: root
<10.42.3.199> SSH: ansible.cfg set ssh_args: (-C)(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<10.42.3.199> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<10.42.3.199> SSH: ANSIBLE_REMOTE_PORT/remote_port/ansible_port set: (-o)(Port=22)
<10.42.3.199> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile="/root/.ssh/id_rsa")
<10.42.3.199> SSH: ansible_password/ansible_ssh_password not set:
(-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<10.42.3.199> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User="root")
<10.42.3.199> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<10.42.3.199> SSH: Set ssh_common_args: ()
<10.42.3.199> SSH: Set ssh_extra_args: ()
<10.42.3.199> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath="/root/.ansible/cp/7dfa49af55")
<10.42.3.199> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/root/.ssh/id_rsa"' -o
KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o
'ControlPath="/root/.ansible/cp/1a125e8154"' 10.42.3.199 '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'python3.10'"'"'"'"'"'"'"'"'; command -v
'"'"'"'"'"'"'"'"'python3.9'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.8'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v
'"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v
'"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v
'"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<10.42.3.199> (0, b'PLATFORM\nLinux\nFOUND\n/usr/bin/python3.6\n/usr/bin/python3\n/usr/libexec/platform-python\nENDFOUND\n', b"OpenSSH_8.0p1, OpenSSL 1.1.1k  FIPS 25 Mar
2021