Troubleshooting ssh proxy failure during ansible provisioning on centos/7 virtualbox-ovf build

[Andrew Feller]

Hey,

I would appreciate any help available on figuring out why the ansible (remote) provisioner is failing to reach the guest.  Since the box was built with the vagrant user setup, I decided to use the pre-existing user and later wipe it once ansible was working.

Thanks,
Andy


Software:

• OS X 10.11.6
• Ansible 2.1.0.0
• Packer 0.10.1
• VirtualBox 5.0.20 r106931

Packer file (packer/example.json):

{
    "builders": [{
        "communicator": "ssh",
        "guest_additions_mode": "disable",
        "headless": true,
        "shutdown_command": "echo 'packer' | sudo -S shutdown -P now",
        "source_path": "boxes/centos-VAGRANTSLASH-7/1607.01/virtualbox/box.ovf",
        "ssh_private_key_file": "insecure_private_key",
        "ssh_username": "vagrant",
        "ssh_wait_timeout": "1m",
        "type": "virtualbox-ovf"
    }],
    "provisioners": [{
        "type": "ansible",
        "playbook_file": "packer/example.yml",
        "groups": [
            "example-group"
        ],
        "user": "vagrant",
        "ssh_authorized_key_file": "authorized_keys",
        "extra_arguments": [
            "--private-key", "insecure_private_key"
        ]
    }],
    "post-processors": [{
        "type": "vagrant"
    }]
}

Ansible playbook (packer/example.yml:

---
- hosts: example-group
  tasks:
    - name: Debugging
      debug:
        msg: Hello world!

Setup:

me@localhost tmp $ VAGRANT_HOME=. vagrant box add --provider virtualbox --box-version 1607.01 centos/7
==> box: Loading metadata for box 'centos/7'
    box: URL: https://atlas.hashicorp.com/centos/7
==> box: Adding box 'centos/7' (v1607.01) for provider: virtualbox
    box: Downloading: https://atlas.hashicorp.com/centos/boxes/7/versions/1607.01/providers/virtualbox.box
==> box: Successfully added box 'centos/7' (v1607.01) for 'virtualbox'!
me@localhost tmp $ ssh-keygen -y -f insecure_private_key > authorized_keys
me@localhost tmp $ packer build packer/example.json

Output:

virtualbox-ovf output will be in this color.

==> virtualbox-ovf: Importing VM: boxes/centos-VAGRANTSLASH-7/1607.01/virtualbox/box.ovf
==> virtualbox-ovf: Creating forwarded port mapping for communicator (SSH, WinRM, etc) (host port 2874)
==> virtualbox-ovf: Starting the virtual machine...
    virtualbox-ovf: The VM will be run headless, without a GUI. If you want to
    virtualbox-ovf: view the screen of the VM, connect via VRDP without a password to
    virtualbox-ovf: 127.0.0.1:5947
==> virtualbox-ovf: Waiting 10s for boot...
==> virtualbox-ovf: Typing the boot command...
==> virtualbox-ovf: Waiting for SSH to become available...
==> virtualbox-ovf: Connected to SSH!
==> virtualbox-ovf: Uploading VirtualBox version info (5.0.20)
==> virtualbox-ovf: Provisioning with Ansible...
==> virtualbox-ovf: SSH proxy: serving on 127.0.0.1:51593
==> virtualbox-ovf: Executing Ansible: ansible-playbook /Users/me/tmp/packer/example.yml -i /var/folders/5s/y2lv1n2j00g62yzk28sjs_300000gn/T/packer-provisioner-ansible615319840 --private-key insecure_private_key
    virtualbox-ovf:
    virtualbox-ovf: PLAY [example-group] ***********************************************************
    virtualbox-ovf:
    virtualbox-ovf: TASK [setup] *******************************************************************
    virtualbox-ovf: SSH proxy: accepted connection
==> virtualbox-ovf: authentication attempt from 127.0.0.1:51594 to 127.0.0.1:51593 as vagrant using none
==> virtualbox-ovf: authentication attempt from 127.0.0.1:51594 to 127.0.0.1:51593 as vagrant using publickey
    virtualbox-ovf: rejecting auth-ag...@openssh.com request
==> virtualbox-ovf: starting sftp subsystem
    virtualbox-ovf: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh", "unreachable": true}
    virtualbox-ovf:     to retry, use: --limit @/Users/me/tmp/packer/example.retry
    virtualbox-ovf:
    virtualbox-ovf: PLAY RECAP *********************************************************************
    virtualbox-ovf: default                    : ok=0    changed=0    unreachable=1    failed=0
    virtualbox-ovf:
==> virtualbox-ovf: shutting down the SSH proxy
==> virtualbox-ovf: Unregistering and deleting imported VM...
==> virtualbox-ovf: Deleting output directory...
Build 'virtualbox-ovf' errored: Error executing Ansible: Non-zero exit status: exit status 3

==> Some builds didn't complete successfully and had errors:
--> virtualbox-ovf: Error executing Ansible: Non-zero exit status: exit status 3

==> Builds finished but no artifacts were created.

[Andrew Feller]

So after doing some work, it appears my problem was the same as https://github.com/mitchellh/packer/issues/3529 where the output around rejecting auth-ag...@openssh.com request was confusing the issue as I'm not familiar with SSH proxying to realize the default sftp-server command isn't standard on CentOS builds.  After setting "sftp_command": "/usr/libexec/openssh/sftp-server -e", it worked fine.

I'm going to submit a pull request and cite this within ansible provisioner's limitations to help some other soul not waste time.