remote_user on task is not working, how can I get troubleshooting information?

[Gordon Messmer]

I have a task defined that sets "remote_user" which has mysteriously started connecting as "root" instead of my own user account.  I've been using this task for some time, and haven't made any changes that appear relevant, recently.  I'm also using it in a test environment where it is working normally, and where I can't find any relevant differences.

When I run the playbook containing this task with -vvv, I see ansible connecting as root, the relevant output is below.

Ansible version:
$ rpm -q ansible
ansible-2.4.1.0-1.el7.noarch

Where can I look in the code to try to determine why remote_user isn't being set properly?

The task is defined:

- name: check for kerberos ticket
  shell: "klist | egrep -q 'Default principal: ({{ \"|\".join(admin_users) }})@'"
  register: has_kerberos_admin
  ignore_errors: True
  delegate_to: "{{ ipa_server }}"
  remote_user: "{{ lookup('env', 'USER') }}"
  tags: configuration

Output from -vvv:

TASK [ipa-admin-command : check for kerberos ticket] *****************************************************************************************************************************************************************************************
task path: /home/gordon/ansible-example/roles/ipa-admin-command/tasks/main.yml:1
Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
<ds-20170921.private.example.net> ESTABLISH SSH CONNECTION FOR USER: root
<ds-20170921.private.example.net> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=60s -o GSSAPIAuthentication=yes -o GSSAPIDelegateCredentials=yes -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/gordon/.ansible/cp/923a4e9819 ds-20170921.private.example.net '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
<ds-20170921.private.example.net> (1, '\n{"changed": true, "end": "2017-12-02 19:26:10.012527", "stdout": "", "cmd": "klist | egrep -q \'Default principal: (gordon)@\'", "failed": true, "delta": "0:00:00.012965", "stderr": "klist: Credentials cache keyring \'persistent:0:0\' not found", "rc": 1, "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": true, "_raw_params": "klist | egrep -q \'Default principal: (gordon)@\'", "removes": null, "creates": null, "chdir": null, "stdin": null}}, "start": "2017-12-02 19:26:09.999562", "msg": "non-zero return code"}\n', '')

[Gordon Messmer]

Setting "remote_user" statically does not change the outcome.  With "remote_user: gordon", ansible still connects as "root".

[Gordon Messmer]

ansible.cfg does contain:

[defaults]
remote_user=root

but that file has the same contents on the test system, where the same playbook works, and ansible connects as the user named in the task's "remote_user" setting.

[Gordon Messmer]

If I set "strategy: debug" for the play and "p vars" there are a number of settings in the broken system which aren't set to any value on the system that works.

 'vars': {...
          'ansible_delegated_vars': {u'ds-20170921.private.example.net': {...
                                                                              'ansible_connection': u'smart',
                                                                              'ansible_delegated_host': u'ds-20170921.private.example.net',
                                                                              'ansible_host': u'ds-20170921.private.example.net',
....
                                                                              'ansible_port': None,
                                                                              'ansible_user': u'root',

Those 5 values in ansible_delegated_vars[ the delegated_to host ] don't have any value on the working system.  However, the working system has at least one value that the broken one does not:

{...
 'ansible_current_hosts': [u'network-2017120101.tutorial.example.net'],

[Gordon Messmer]

SOLVED:

This problem appears to have been caused by the introduction of dynamic inventory.  I pull a list of hosts from libvirt (virsh list --name), and on the broken system, one VM has a name with a typo.  It does not match the host's name in DNS, and the name given in the delegate_to setting (the ipa_server variable).

The problem appears to be that ansible allows you to delegate_to a host that doesn't appear in your inventory, but that connection doesn't behave like a connection to a host that *is* in the inventory.  Among other things, you can't set "remote_user" for a task that is delegated to a host that doesn't appear in the inventory.

I'll file a bug report later.