Can't get remote ansible provisioner to assign to existing group during provisioning

[Stephen Nesbitt]

All:

Fundamental question is how do I assign a newly minted lxd container to an ansible group when using the remote ansible provisioner? Here's what I have so far:

Packer Template:

{
  "builders": [
    {
      "type": "lxd",
      "name": "consulserver",
      "image": "AWTAlpine37",
      "output_image": "consulserver",
      "publish_properties": {
      "description": "Consul Server"
      }
    }
  ],
  "provisioners": [
  {
    "type": "shell",
    "inline": [ "sleep 10; apk update && apk add python2" ]
  },
  {
    "type": "ansible",
    "groups": ["consul_servers"],
    "ansible_env_vars": [ "ANSIBLE_CONFIG=/home/FUZZBUTT/snesbitt/projects/ansible/fuzzbutt.awt_ansible/ansible.cfg" ],
    "inventory_directory": "/home/FUZZBUTT/snesbitt/projects/ansible/fuzzbutt.awt_ansible/inventories/prod",
    "playbook_file": "/home/FUZZBUTT/snesbitt/projects/ansible/fuzzbutt.awt_ansible/playbooks/domain-server.yml",
    "extra_arguments": [ "-c", "lxd", "-i", "packer-consulserver,", "-l", "packer-consulserver" ]
  }
  ]
}

Ansible output:

==> consulserver: Executing Ansible: ansible-playbook --extra-vars packer_build_name=consulserver packer_builder_type=lxd -i /home/FUZZBUTT/snesbitt/projects/ansible/fuzzbutt.awt_ansible/i
nventories/prod/packer-provisioner-ansible904637113 /home/FUZZBUTT/snesbitt/projects/ansible/fuzzbutt.awt_ansible/playbooks/domain-server.yml -e ansible_ssh_private_key_file=/tmp/ansible-k
ey529167426 -c lxd -i packer-consulserver, -l packer-consulserver                                                                                                                            
    consulserver:
    consulserver: PLAY [all] *********************************************************************
    consulserver:
    consulserver: TASK [Gathering Facts] *********************************************************
    consulserver:  [WARNING]: lxd does not support remote_user, using container default: root
    consulserver: ok: [packer-consulserver]
    consulserver:
    consulserver: TASK [Play hosts] **************************************************************
    consulserver:  [WARNING]: lxd does not support remote_user, using container default: root
    consulserver: ok: [packer-consulserver] => {
    consulserver:     "msg": "play_hosts=[u'packer-consulserver']"
    consulserver: }
    consulserver:
    consulserver: TASK [Groups] ******************************************************************
    consulserver:  [WARNING]: lxd does not support remote_user, using container default: root
    consulserver: ok: [packer-consulserver] => {
    consulserver:     "msg": "groups=['ungrouped']"
    consulserver: }
    consulserver:
    consulserver: TASK [Test] ********************************************************************
    consulserver: skipping: [packer-consulserver]
    consulserver:
    consulserver: PLAY RECAP *********************************************************************
    consulserver: packer-consulserver        : ok=3    changed=0    unreachable=0    failed=0

If I've got this correct, I've told Packer in the template to assign the container to group "consul_servers". But when I query Ansible for the list of groups, the only group recognized is 'ungrouped'.

Any assistance would be much appreciated.

-steve

[Stephen Nesbitt]

Ok - take two. After a lot of poking and prodding and head banging I think I've identified the issue as the provisioner not incorporating my existing group_vars definition.

I put a debug statement into the provisioner.go file which dumps the generated inventory file and it looks like this:

[consul_instances]
2018/03/20 23:16:52 packer: default ansible_host=127.0.0.1 ansible_user=snesbitt@fuzzbutt.aussieswithtails.com ansible_port=37147

So the container is being assigned to a group.

What doesn't seem to be happening is that despite setting the "inventory_directory" to the directory containing my host file and my group_vars directory, the provisioner is not grabbing the variables defines in my group_vars/consul_instances.yml file.

Here's my test setup:

template.json:

{

  "builders": [

    {

      "type": "lxd",

      "name": "consulserver",

      "image": "AWTAlpine37",

      "output_image": "consulserver",

      "publish_properties": {

      "description": "Consul Server"

      }

    }

  ],

  "provisioners": [

  {

    "type": "shell",

    "inline": [ "sleep 10; apk update && apk add python2" ]

  },

  {

    "type": "ansible",

    "groups": ["consul_instances"],

    "ansible_env_vars": [ "ANSIBLE_CONFIG=/home/FUZZBUTT/snesbitt/projects/ansible/fuzzbutt.awt_ansible/ansible.cfg" ],

    "inventory_directory": "/home/FUZZBUTT/snesbitt/projects/ansible/fuzzbutt.awt_ansible/inventories/prod",

    "playbook_file": "/home/FUZZBUTT/snesbitt/projects/ansible/fuzzbutt.awt_ansible/playbooks/domain-server.yml",

    "extra_arguments": [ "-c", "lxd", "-i", "packer-consulserver,",  "-l", "packer-consulserver" ]

  }

  ]

}

Playbook:

---
- hosts: all


  tasks:
    - name: Play hosts
      debug: msg="play_hosts={{play_hosts}}"
      run_once: true


    - name: Dump consul_instance group
      debug: msg="consul instances {{ groups['consul_instances'] | to_nice_yaml }}"


    - name: Dump workstations group
      debug: var=groups['workstations']


    - name: Dump consul_instances group var
      debug: var=consul_node_role

Output from packer run:

 consulserver: Provisioning with Ansible...

==> consulserver: Executing Ansible: ansible-playbook --extra-vars packer_build_name=consulserver packer_builder_type=lxd -i /home/FUZZBUTT/snesbitt/projects/ansible/fuzzbutt.awt_ansible/inventories/prod/packer-provisioner-ansible074578073 /home/FUZZBUTT/snesbitt/projects/ansible/fuzzbutt.awt_ansible/playbooks/domain-server.yml -e ansible_ssh_private_key_file=/tmp/ansible-key628749218 -c lxd -i packer-consulserver, -l packer-consulserver --extra-vars remote_addr=packer-consulserver

    consulserver:

    consulserver: PLAY [all] *********************************************************************

    consulserver:

    consulserver: TASK [Gathering Facts] *********************************************************

    consulserver:  [WARNING]: lxd does not support remote_user, using container default: root

    consulserver: ok: [packer-consulserver]

    consulserver:

    consulserver: TASK [Play hosts] **************************************************************

    consulserver:  [WARNING]: lxd does not support remote_user, using container default: root

    consulserver: ok: [packer-consulserver] => {

    consulserver:     "msg": "play_hosts=[u'packer-consulserver']"

    consulserver: }

    consulserver:

    consulserver: TASK [Dump consul_instance group] **********************************************

    consulserver:  [WARNING]: lxd does not support remote_user, using container default: root

    consulserver: ok: [packer-consulserver] => {

    consulserver:     "msg": "consul instances - default\n"

    consulserver: }

    consulserver:

    consulserver: TASK [Dump workstations group] *************************************************

    consulserver:  [WARNING]: lxd does not support remote_user, using container default: root

    consulserver: ok: [packer-consulserver] => {

    consulserver:     "groups['workstations']": "VARIABLE IS NOT DEFINED!"

    consulserver: }

    consulserver:

    consulserver: TASK [Dump consul_instances group var] *****************************************

    consulserver:  [WARNING]: lxd does not support remote_user, using container default: root

    consulserver: ok: [packer-consulserver] => {

    consulserver:     "consul_node_role": "VARIABLE IS NOT DEFINED!"

    consulserver: }

    consulserver:

    consulserver: PLAY RECAP *********************************************************************

    consulserver: packer-consulserver        : ok=5    changed=0    unreachable=0    failed=0

    consulserver:

consul_instances.yml group-var file:

---
consul_node_role: server
consul_bootstrap_expect: true

Directory structure of inventory_directory:

/home/FUZZBUTT/snesbitt/projects/ansible/fuzzbutt.awt_ansible/inventories/prod
├── group_vars
│   ├── all.yml
│   ├── consul_hosts.yml
│   ├── consul_instances.yml
│   └── consul_servers.yml
├── hosts
└── host_vars

God I hope this is a bug and not something obviously stupid on my part ;-)

-stefve

[Stephen Nesbitt]

Ok - take 3

This morning I dumped the hostvars variable to see what I could find. What I found was 2 hosts defined - a "default" host and a "packer-consulserver host". And lo and behold, in the default host were my groups and my group variables. These entries are not defined in the packer-consulserver host.

This is a problem because the ip address assigned to the default host is 127.0.0.1, and a container named default does not exist so any attempt to issue lxc commands fails.

The packer-consulserver host reflects the name of the container and has a 10.125.175.188 ip address meaning that the lxd connector can make the connection. The only problem being is that host is missing my group vars and groups.

Bottom line, if I don't add the -i packer-consulserver -l packer-consulserver parameters to my extra variables, ansible uses the default host and can't connect.

If I do add the  -i packer-consulserver -l packer-consulserver parameters to my extra variables then ansible uses packer-consulserver as the inventory host, but my group variables aren't visible.

I apologize for the lengthy posts, but I've spent a number of hours on this, and hope that by recording my efforts I might save someone else the pain and suffering :-)

And if someone has a solution, I'd love to hear it.

-steve

[Rickard von Essen]

I think this is a problem of how lxc/LXD works with Ansible and packer. 

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/7cf7d077-2397-49bf-b8d1-6d9b0d7afaf7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stephen Nesbitt]

 I've looked into this a bit more, and I believe the core of the problem is in the generation of the temporary host file. I hacked line 276 in the ansible/provisioner.go file (the host definition line) and hardcoded in "packer-consulserver" in place of p.config.HostAlias. This worked for me and my group vars were associated with the packer-consulserver ansible host.

Off hand, I wonder if a configuration could be added to the ansible provisioner that would allow the default host generation behavior to be modified. One idea - though I don't know if it will work -would be to allow the user to specify an existing hosts file to use rather than to generate one. If doable, this could potentially allow me to use my test/stage/production ansible hosts file.

There is one work around that I've discovered. One can add the group variables and their values into the variables section of the packer template. Really non-DRY and really ugly IMO.

What I'd like to do, is to enter an issue on this. Even if the choice is not to make the change or not to commit the resources, the problem should be filed, and the documentation updated to make users aware of the issue and potential work around.

I would consider submitting a patch but I would want to have a clear sense of how it should be implemented. Also in the way is my limited knowledge of Golang and the overall architecture of the ansible provisioner and its relationship to the lxd builder.

-steve

 

[Megan Marsh]

Sounds good to me -- open an issue, and link your hack.