Issue with git module and ssh agent forwarding

[tamakisquare]

My setup is a vagrant box whose provisioning is provided by ansible. I have set up ssh agent forwarding from my local/host machine to the vagrant box so that the user 'vagrant' can perform `git clone` on a private bitbucket repo, which is configured to perform ssh key authentication.

To verify that the ssh agent forwarding works, I ssh'ed to the vagrant box with the user 'vagrant' and I was able to perform `git clone` on the mentioned remote repo, without setting up the required private key in the vagrant box itself.

However, when ansible performs the same task via the git module it fails with "Permission denied (publickey)" error

The play/task:

- name: Set up source
  hosts: appserver
  tasks:
    - name: Git | Clone private repo from bitbucket
      git:
        repo="g...@bitbucket.org:someuser/somerepo.git"
        dest=/home/vagrant/example

The verbose output for the task:

TASK: [Git | Clone private repo from bitbucket] ****************
<192.168.55.139> ESTABLISH CONNECTION FOR USER: vagrant
<192.168.55.139> REMOTE_MODULE git repo="g...@bitbucket.org:someuser/somerepo.git" dest=/home/vagrant/example
<192.168.55.139> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ForwardAgent=yes', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/windbottle/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'StrictHostKeyChecking=no', '-o', 'Port=22', '-o', 'IdentityFile=/Users/windbottle/.vagrant.d/insecure_private_key', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=vagrant', '-o', 'ConnectTimeout=10', '192.168.55.139', "/bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1399490355.47-128414001438969 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1399490355.47-128414001438969 && echo $HOME/.ansible/tmp/ansible-tmp-1399490355.47-128414001438969'"]
<192.168.55.139> PUT /var/folders/l2/22zwjkz106vdwz1846jp7d0w0000gn/T/tmpp_JWyh TO /home/vagrant/.ansible/tmp/ansible-tmp-1399490355.47-128414001438969/git
<192.168.55.139> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ForwardAgent=yes', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/Users/windbottle/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'StrictHostKeyChecking=no', '-o', 'Port=22', '-o', 'IdentityFile=/Users/windbottle/.vagrant.d/insecure_private_key', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=vagrant', '-o', 'ConnectTimeout=10', '192.168.55.139', "/bin/sh -c '/usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1399490355.47-128414001438969/git; rm -rf /home/vagrant/.ansible/tmp/ansible-tmp-1399490355.47-128414001438969/ >/dev/null 2>&1'"]
failed: [B612] => {"cmd": ["/usr/bin/git", "ls-remote", "g...@bitbucket.org:someuser/somerepo.git", "-h", "refs/heads/HEAD"], "failed": true, "item": "", "rc": 128}
stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

According to the above output, 'ForwardAgent=yes' is indeed there, so I can't comprehend why the error. 

Calling on experts to shed some light on this issue. Thanks.

[Gilles Cornu]

Hi,

Your play snippet and execution log excerpt do not allow to know if the `git` task is executed by the remote user (in this example `vagrant`) and I suspect that this task is in fact executed by root, due to some `sudo: yes` (enabled either in Vagrant settings or in your playbook file). In this case, you might have a look at following SO questions:

- http://stackoverflow.com/a/22768453/2951970

- http://stackoverflow.com/a/23524170/2951970

If you are in such similar situation, I would recommend to organize your play so that the git module is executed by the remote user (vagrant).

Hope it helps, Gilles

[Gilles Cornu]

Your play snippet and execution log excerpt do not allow to know if the `git` task is executed by the remote user (in this example `vagrant`) and I suspect that this task is in fact executed by root, due to some `sudo: yes` (enabled either in Vagrant settings or in your playbook file). 

Oops sorry, just a little correction on what I wrote before: Your play is complete and does not include any `sudo: yes`, so I suppose that you are using the `ansible.sudo = true` option in the vagrant provisioner settings. Could confirm? 

[tamakisquare]

Gilles, thanks for the reply.

I have double checked that the `git` task is indeed executed by the intended remote user, vagrant, not root. Sudo is off in both Vagrant and Ansible configuration.

I have a new discovery to the problem though. The reported issue happened with Ubuntu 14.04 ("phusion/ubuntu-14.04-amd64" vagrant image) being the target box.  Using the exact same Vagrant configuration and Ansible playbook, I tried Ubuntu 12.04 ("hashicorp/precise64" vagrant image) and, surprisingly, the `git` task was successfully executed.  How odd.

My very first thought of the different result was that, perhaps, `AllowAgentForwarding` was disabled in `sshd_config` in this particular Ubuntu 14.04 image. I checked it but that wasn't the case. 

Now, I am left with a big question mark on what's causing the issue for the Ubuntu 14.04 box. It could be something to do with the particular image that I use, or something to do with Ubuntu 14.04 in general. I don't know what else I can do to diagnose the issue. Any thoughts?   

[tamakisquare]

I should mention that the verbose output from Ansible for the `git` task is also exactly the same in both cases.

[Michael DeHaan]

I've heard this claimed before.

Doesn't the vagrant provisioner do something weird like SSH in and then run Ansible, so the issue is with vagrant not doing the forwarding, regardless if your SSH config is so set?

Either way, be sure you are using "-c ssh".  If you are on Enterprise Linux, Paramiko won't forward things.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/0b11710d-ce31-4ce8-96df-ccb650593b71%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gilles Cornu]

tamakisquare,

Thanks for feedback, and sorry for bad suspicion on potential root abuse ;-)

For me everything behaves the same on Ubuntu 12.04 and 14.04 flavors. Do you still observe this problem on 14.04?
I suspect (again!) that you might experienced some "transient" troubles, maybe due to ControlMaster delays (in my experience, setting ControlMaster=no during debugging phase can help).

For completeness, I should mention that during my tests, I faced another problem (on both ubuntu boxes):  `accept_hostkey=yes` or `ssh_opts='-o StrictHostKeyChecking=no'` options of the git module do not work for me, and the task hung. So far I could fix it by filling ~/.ssh/known_hosts via other means (see also https://github.com/ansible/ansible/issues/6358 by the way). As I did not find any github issues nor mailing list topic referring to similar problem, I assume so far that I am doing something wrong on my end and will further investigate (and maybe report in a separate thread). Note that based on https://groups.google.com/d/topic/ansible-project/d5OVhIWQ8AI/discussion, I also compared ansible v1.6.1 and v1.5.3, but I got exactly the same blocking issue with the git task.


Michael,

Here some details current status of Vagrant support for Ansible provisioning:

As of Vagrant 1.5.0, the Ansible provisioner supports config.ssh.forward_agent from Vagrant settings to auto-configure ansible-playbook command with  --connection=ssh and ANSIBLE_SSH_ARGS='-o ForwardAgent=yes ...'. With older Vagrant versions, it was indeed necessary to manually tweak these parameters (via ansible.cfg, ANSIBLE_SSH_ARGS, etc.) but I hope the usages will change overtime and less people get confused. The answer from tamakisquare on http://stackoverflow.com/a/23554663 also confirms that this change makes things quite simpler.

More generally, I also would like to mention that since Vagrant 1.6.0, the provisioner shows the complete ansible-playbook command in vagrant terminal log, for example:

ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false PYTHONUNBUFFERED=1 ANSIBLE_SSH_ARGS='-o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s' ansible-playbook --private-key=/.../.vagrant.d/insecure_private_key --user=vagrant --connection=ssh --inventory-file=/.../.vagrant/provisioners/ansible/inventory -vv --limit='machine*' playbook.yml

I think that this simple addition is quite handy for support/debugging as it makes very easy to copy-paste and tweak in a simple shell, without having the Vagrant pipeline involved. In the future, I would propose as a general guideline for Vagrant+Ansible users, to first check that `vagrant provision` and pure `ansible-playbook` commands behave the same before reporting issues to Ansible mailing list / GH issue tracker.

Best,
Gilles

[tamakisquare]

Gilles,

I have magically resolved the blocking issue by adding the option '-o UserKnownHostsFile=/dev/null', together with the option '-o StrictHostKeyChecking=no'. I tried this solution with both Ubuntu 12.04 and 14.04 and didn't get any issue at all. I, however, have yet to find the rationale behind this solution as why would it resolve the blocking issue. I believe it has something to do with the default known hosts file used by Ansible, either that it's having trouble writing to the file or the file it writes to is different from the known hosts file it reads from. Just a wild guess from me and it's pending investigation. Anyhow, please give the mentioned solution a try and share your result with us.