delegate_to: with a different user from the target hosts

[Kesten Broughton]

We are operating a private openstack cloud.  We would like to host a single local mirror for data and package repositories per cloud, with many tenants for a given cloud.

Each tenant/project is on a private subnet with a single "pivot" vm with a floating ip.   All traffic is routed through the tenant's pivot.

Each time a tenant is created we create an ssh key for it and load it into all vms on creation.  The admin gets the private/public keys for their .ssh/identities/ folder.

For rapid prototyping, we are sharing keys as needed.  Eventually we would replace this with web-services.

The goal is to use jump host settings in .ssh/config to make a call like this

 - name: Get the jdk from local_mirror_host

   synchronize: src="/opt/java/jdks/{{jdk_rpm}}"

                      dest="{{_ansible_downloads}}/{{jdk_rpm}}"

   delegate_to: "{{local_mirror_host.name}}"

Currently, running ansible playbooks from a laptop via the pivot is working fine using settings like this

[mac ansible controller] /Users/kbroughton/.ssh/config

Host 10.x.y.z  # tenant pivot

    User cloud-user

    HostName app-dev1-pivot

    IdentityFile ~/.ssh/identities/app_dev1_key

Host app-dev1-*

    User cloud-user

    ProxyCommand ssh 10.x.y.z -W %h:%p    # -W instead of nc

    IdentityFile ~/.ssh/identities/app_dev1_key

However, tasks with delegation fail with an error like this

failed: [21ct-dev1-pivot] => {"cmd": "rsync --delay-updates -FF --compress --timeout=10 --archive --rsh 'ssh -i /Users/kbroughton/.ssh/identities/app_dev1_key -o StrictHostKeyChecking=no' --out-format='<<CHANGED>>%i %n%L' /opt/java/jdks/testjdk cloud-user@app-dev1-pivot:/tmp/ansible/jdk1.7.0_25.rpm", "failed": true, "item": "", "rc": 23}

msg: Warning: Identity file /Users/kbroughton/.ssh/identities/app_dev1_key not accessible: No such file or directory.

rsync: mkstemp "/tmp/ansible/.jdk1.7.0_25.rpm.NYd0r7" failed: Permission denied (13)

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]

Killed by signal 1.

It appears there are several problems:

1.  For the pivot vm the delegate is trying to find the key in /Users/kbroughton/.ssh rather than /root/.ssh as specified in the delegate's .ssh/config

2.  For the lynx01 vm, the delegate is trying to ssh as root.  Cloud vms have no root user, only cloud-user.  Note the connection from delegate to pivot correctly attempts to use cloud-user.

It appears the delegate .ssh/config is not being applied.  

[delegate /local_mirror_host] /root/.ssh/config

Host 21ct-dev1-*

        ProxyCommand ssh 10.x.y.z -W %h:%p

        User cloud-user

        IdentityFile /root/.ssh/identities/app_dev1_key

Host 10.1.30.186

    User cloud-user

    HostName app-dev1-pivot

    IdentityFile ~/.ssh/identities/app_dev1_key

Note that with the above settings it is possible to do

[root@dev1-pivot] ssh app-pivot or app-lynx01 

just fine.  

Also not that "localhost" is among the target hosts of the task and when applied via the delegate, ansible changes the meaning of localhost from the laptop running the task to the delegate.  The file is copied locally from the delegate to itself on the localhost play.

I would expect that ansible would attempt to apply the .ssh/config settings on the delegate host, or at least allow that as an option.

COMPLETE ERROR TRACE

TASK: [centos_common | Get the jdk from local_mirror_host] ******************** 

<dev1-pivot> ESTABLISH CONNECTION FOR USER: root

<dev1-pivot> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/kbroughton/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=root', '-o', 'ConnectTimeout=10', u'dev1-pivot', "/bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1397489133.67-55633491203993 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1397489133.67-55633491203993 && echo $HOME/.ansible/tmp/ansible-tmp-1397489133.67-55633491203993'"]

<dev1-pivot> ESTABLISH CONNECTION FOR USER: root

<dev1-pivot> ESTABLISH CONNECTION FOR USER: root

<dev1-pivot> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/kbroughton/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=root', '-o', 'ConnectTimeout=10', u'dev1-pivot', "/bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1397489133.68-9986*****848 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-139748****48848 && echo $HOME/.ansible/tmp/ansible-tmp-13974***97748848'"]

<dev1-pivot> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/kbroughton/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=root', '-o', 'ConnectTimeout=10', u'dev1-pivot', "/bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1397489***24490 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-13974***924490 && echo $HOME/.ansible/tmp/ansible-tmp-1397489133.68-74531171924490'"]

<dev1-pivot> PUT /var/folders/t2/h2233****s8_088c/T/tmpqymjL0 TO /root/.ansible/tmp/ansible-tmp-1397***748848/synchronize

<dev1-pivot> PUT /var/folders/t2/h223***fk7s8_088c/T/tmpZmzUty TO /root/.ansible/tmp/ansible-tmp-139748***3993/synchronize

<dev1-pivot> PUT /var/folders/t2/h2233***fk7s8_088c/T/tmpUm0OXD TO /root/.ansible/tmp/ansible-tmp-1397489***24490/synchronize

<dev1-pivot> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/kbroughton/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=root', '-o', 'ConnectTimeout=10', u'dev1-pivot', "/bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-13974***993/synchronize; rm -rf /root/.ansible/tmp/ansible-tmp-139***1203993/ >/dev/null 2>&1'"]

<dev1-pivot> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/kbroughton/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=root', '-o', 'ConnectTimeout=10', u'dev1-pivot', "/bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-13974***4490/synchronize; rm -rf /root/.ansible/tmp/ansible-tmp-1397489133.68-74531171924490/ >/dev/null 2>&1'"]

<dev1-pivot> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/kbroughton/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=root', '-o', 'ConnectTimeout=10', u'dev1-pivot', "/bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-13974***748848/synchronize; rm -rf /root/.ansible/tmp/ansible-tmp-13974***48/ >/dev/null 2>&1'"]

ok: [localhost] => {"changed": false, "cmd": "rsync --delay-updates -FF --compress --timeout=10 --archive --rsh 'ssh  -o StrictHostKeyChecking=no' --out-format='<<CHANGED>>%i %n%L' /opt/java/jdks/testjdk /tmp/ansible/jdk1.7.0_25.rpm", "item": "", "msg": "", "rc": 0}

failed: [21ct-dev1-lynx01] => {"cmd": "rsync --delay-updates -FF --compress --timeout=10 --archive --rsh 'ssh  -o StrictHostKeyChecking=no' --out-format='<<CHANGED>>%i %n%L' /opt/java/jdks/testjdk root@21ct-dev1-lynx01:/tmp/ansible/jdk1.7.0_25.rpm", "failed": true, "item": "", "rc": 2}

msg: Warning: Permanently added '21ct-dev1-lynx01' (RSA) to the list of known hosts.

protocol version mismatch -- is your shell clean?

(see the rsync man page for an explanation)

rsync error: protocol incompatibility (code 2) at compat.c(171) [sender=3.0.6]

failed: [21ct-dev1-pivot] => {"cmd": "rsync --delay-updates -FF --compress --timeout=10 --archive --rsh 'ssh -i /Users/kbroughton/.ssh/identities/21ct_dev1_key -o StrictHostKeyChecking=no' --out-format='<<CHANGED>>%i %n%L' /opt/java/jdks/testjdk cloud-user@21ct-dev1-pivot:/tmp/ansible/jdk1.7.0_25.rpm", "failed": true, "item": "", "rc": 23}

msg: Warning: Identity file /Users/kbroughton/.ssh/identities/21ct_dev1_key not accessible: No such file or directory.

rsync: mkstemp "/tmp/ansible/.jdk1.7.0_25.rpm.NYd0r7" failed: Permission denied (13)

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]

Killed by signal 1.

--

Kesten Broughton
512 701 4209

[kesten broughton]

Discovered that i had ansible_ssh_user=cloud-user for the pivot host and not the lynx01 host in my ansible hosts file.

That's why in the above the delegate tried to connect as cloud-user to pivot and root to lynx01.

I'd rather not have to use the hosts file because we have one per tenant (dozens) and it's less maintainable than a single .ssh/config.

But the larger problem is the key being in different locations on my ansible controller (/User/kbroughton) vs the delegate (/roor).

Is there any way to force ansible to respect .ssh/config both locally and on delegates?

For the -i path to keys, the ansible_ssh_private_key_file won't work as the location is different for controller vs delegate.

The only fix would be to place all tenant keys in /etc/ssh/identities/ on all hosts which seems pretty non-standard.

[Michael DeHaan]

"Is there any way to force ansible to respect .ssh/config both locally and on delegates?"

When ansible is using the SSH connection type (read: not paramiko, the default on CentOS and RHEL) it will/should pick all this up automatically.   

If you're using paramiko, that's a bit of a different story.   

It will be a very nice day when RHEL7 comes out with a newer SSH and everyone can use -c openssh by default.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/1eaf296e-aa4b-4c79-b1ee-cbbac6e32a5c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[kesten broughton]

After some more digging, it appears the problem is at the rsync level, so no matter what ansible synchronize does to wrap rsyn, it can't work.

using rsync on the commandline with -e "ssh -F /Users/kbroughton/.ssh/config" to force it to use the config file, it picks up the IdentityFile, but not the jump-host stuff.

I've submitted a bug report 10557

https://bugzilla.samba.org/show_bug.cgi?id=10557

And posted on stackexchange

http://unix.stackexchange.com/questions/94421/how-to-use-ssh-config-setting-for-each-server-by-rsync/125251#125251

kesten