Intermittent error "failed to resolve remote temporary directory from"

[Pradeep Reddy]

I keep getting this error once in every 4/5 times. 

fatal: [xxx]: FAILED! => {"failed": true, "msg": "ERROR! failed to resolve remote temporary directory from 
ansible-tmp-1451292385.49-145367138177040: 
`( umask 22 && mkdir -p \"$( echo $HOME/.ansible/tmp/ansible-tmp-1451292385.49-145367138177040 )\" && 
echo \"$( echo $HOME/.ansible/tmp/ansible-tmp-1451292385.49-145367138177040 )\" )` returned empty string"}

This is happening for one of the tasks randomly, following are those tasks

- name: update journald.conf to ForwardToSyslog
  lineinfile:
    dest: /etc/systemd/journald.conf
    regexp: ForwardToSyslog
    line: ForwardToSyslog=yes
  sudo: yes

- name: create config directory
  file: name={{config_dir}} state=directory
        owner=core group=core mode=0755

I am not sure how to debug this as this doesn't happen always.

Target host is CoreOS with pypy setup, on AWS EC2.
ansible 2.1.0 (devel d5446f9804)

[Brad Murray]

Happens to me too.  Running Ubuntu 14.04 boxes via Vagrant, both Virtualbox and KVM.  OSX control machine

Ansible is version 2.0.0.2

Is really making development painful.

[Andrea Sorbini]

I'm also experiencing the same annoying issue running Ansible from Ubuntu 15.04 (from source: ansible 2.1.0 (devel ccbc7d217b) last updated 2016/02/07 19:18:50 (GMT -700)) on Virtualbox VMs running boot2docker (python 2.7.10).

It happens on random topics. Having a play that simply pings all hosts and fails on any error helps a little bit to mitigate, ie. most of the time, if that passes, the error won't show up. Black magic (sort of).

Any help in identifying/solving the issue is much appreciated.

[Jonas Kongslund]

I also experience the same thing on Ubuntu 14.04.3 LTS using Ansible 2.0.0.2-1ppa~trusty.

sysadm@controller:~$ ansible -vvv host1 -m command -a 'ls -l /'
Using /etc/ansible/ansible.cfg as config file
<10.10.10.51> ESTABLISH SSH CONNECTION FOR USER: None
<10.10.10.51> SSH: EXEC ssh -C -q -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/sysadm/.ansible/cp/ansible-ssh-%h-%p-%r -tt 10.10.10.51 '( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1455084832.33-184704261198709 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1455084832.33-184704261198709 )" )'
10.10.10.51 | FAILED | rc=0 >>
ERROR! failed to resolve remote temporary directory from ansible-tmp-1455084832.33-184704261198709: `( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1455084832.33-184704261198709 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1455084832.33-184704261198709 )" )` returned empty string

sysadm@controller:~$ ansible -vvv host1 -m command -a 'ls -l /'
Using /etc/ansible/ansible.cfg as config file
<10.10.10.51> ESTABLISH SSH CONNECTION FOR USER: None
<10.10.10.51> SSH: EXEC ssh -C -q -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/sysadm/.ansible/cp/ansible-ssh-%h-%p-%r -tt 10.10.10.51 '( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1455085172.76-47799942152084 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1455085172.76-47799942152084 )" )'
<10.10.10.51> PUT /tmp/tmpMAqVn7 TO /home/sysadm/.ansible/tmp/ansible-tmp-1455085172.76-47799942152084/command
<10.10.10.51> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/sysadm/.ansible/cp/ansible-ssh-%h-%p-%r '[10.10.10.51]'
<10.10.10.51> ESTABLISH SSH CONNECTION FOR USER: None
<10.10.10.51> SSH: EXEC ssh -C -q -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/sysadm/.ansible/cp/ansible-ssh-%h-%p-%r -tt 10.10.10.51 'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/sysadm/.ansible/tmp/ansible-tmp-1455085172.76-47799942152084/command; rm -rf "/home/sysadm/.ansible/tmp/ansible-tmp-1455085172.76-47799942152084/" > /dev/null 2>&1'
10.10.10.51 | SUCCESS | rc=0 >>
total 84
drwxr-xr-x  2 root   root    4096 Jan 18 04:41 bin
drwxr-xr-x  3 root   root    4096 Jan 18 04:41 boot
drwxr-xr-x 15 root   root    4040 Feb 10 06:06 dev
...

Any idea why this is happening?

[Jonas Kongslund]

After further investigations I have come to believe that it relates to SSH connection reuse.

After disabling any connection reuse I no longer encounter the error:

ansible all -i hosts -m ping --ssh-extra-args="-o ControlMaster=no -o ControlPath=none -o ControlPersist=no"

Doing this will of course impact performance so consider it a workaround until the cause of the error has been identified and fixed.

You can also add the extra parameters in your inventory file using ansible_ssh_extra_args.