Emulating host_vars and group_vars for files

[Anand Buddhdev]

Dear Ansible developers and users,

I really like the groups_vars and host_vars concept of Ansible, and I was thinking of a way to emulate this for files. So here's an example playbook:

- hosts: all
  tasks:
    - debug: var=item
      with_first_found:
        - paths:
            - host_files/{{ inventory_hostname }}
          files:
            - myfile
        - paths: "{{ group_names }}"

          files:
            - myfile

This works just fine for a host-specific file, because there can only ever be one directory for a host. it also sort-of works for groups, but there are 2 complications:

1. the order of the names in group_names does not match the inventory parent-child order; and

2. I don't know how to prefix "group_files" to each group name, in order to keep my group files in their own directory.

Can anyone suggest how I can achieve this? Or does this require a new custom plugin?

Anand

[Serge van Ginderachter]

On 18 April 2015 at 15:51, Anand Buddhdev <arh...@gmail.com> wrote:

1. the order of the names in group_names does not match the inventory parent-child order; and

​There is no way of knowing that, as the inventory does not export that information to other ansible parts​

 

2. I don't know how to prefix "group_files" to each group name, in order to keep my group files in their own directory.

perhaps I miss your point but


  - paths: "group_files/{{ group_names }}"

    files:
            - myfile

 ?​

[Anand Buddhdev]

On Saturday, 18 April 2015 17:01:16 UTC+2, Serge van Ginderachter wrote:

Hi Serge,

On 18 April 2015 at 15:51, Anand Buddhdev <arh...@gmail.com> wrote:

1. the order of the names in group_names does not match the inventory parent-child order; and

​There is no way of knowing that, as the inventory does not export that information to other ansible parts​

Ah, that's a bummer. Is there any way to get the inventory to expose this information for lookup plugins to use?

 

2. I don't know how to prefix "group_files" to each group name, in order to keep my group files in their own directory.

perhaps I miss your point but


  - paths: "group_files/{{ group_names }}"

    files:
            - myfile

This can't work. The "paths" attribute expects a list. By passing it "group_files/{{ group_names }}", it turns into the string:

"group_files/['group1', 'group2', 'group3']" 

This is a weird, but valid path, so ansible will look in there, but not find anything.

[Anand Buddhdev]

On Saturday, 18 April 2015 17:19:09 UTC+2, Anand Buddhdev wrote:

This can't work. The "paths" attribute expects a list. By passing it "group_files/{{ group_names }}", it turns into the string:

"group_files/['group1', 'group2', 'group3']" 

This is a weird, but valid path, so ansible will look in there, but not find anything.

I should have added that given the list a list ['group1', 'group2'], I want to pass it to ansible as:

['group_files/group1', 'group_files/group2']

[Serge van Ginderachter]

On 18 April 2015 at 17:19, Anand Buddhdev <arh...@gmail.com> wrote:

On Saturday, 18 April 2015 17:01:16 UTC+2, Serge van Ginderachter wrote:

Hi Serge,

On 18 April 2015 at 15:51, Anand Buddhdev <arh...@gmail.com> wrote:

1. the order of the names in group_names does not match the inventory parent-child order; and

​There is no way of knowing that, as the inventory does not export that information to other ansible parts​

Ah, that's a bummer. Is there any way to get the inventory to expose this information for lookup plugins to use?

​No. There is nothing in the api that allows this without an update to th​e core code.

It is though something to consider, when we have a look at the inventory after 2.0.

 

2. I don't know how to prefix "group_files" to each group name, in order to keep my group files in their own directory.

perhaps I miss your point but


  - paths: "group_files/{{ group_names }}"

    files:
            - myfile

This can't work. The "paths" attribute expects a list. By passing it "group_files/{{ group_names }}", it turns into the string:

"group_files/['group1', 'group2', 'group3']" 

This is a weird, but valid path, so ansible will look in there, but not find anything.

​Yes, sorry, I missed that obviously.​

 

​Slow Saturday here :)

This would require the possibility to redifine lists​. Allowing set_fact to work with loops. Which is not possible either now, but will also hopefully become a thing post 2.0

Serge

[Anand Buddhdev]

On Saturday, 18 April 2015 19:20:32 UTC+2, Serge van Ginderachter wrote:

Hi Serge,

​No. There is nothing in the api that allows this without an update to th​e core code.

It is though something to consider, when we have a look at the inventory after 2.0.

Okay, thanks for confirming this.

The reason I have asked this question is that we're using ansible in pull mode, with ansible running locally. This has the downside that all our roles, group_vars and host_vars are downloaded onto every host, thereby exposing some sensitive data to hosts that shouldn't see it. Naturally, we cannot use the vault, because it requires password entry. And trying to figure out the roles that a host needs also appears undoable.

I have figured out a simple way to built rsync filters, that ensure that a pull-mode host only gets its own host_vars file, and group_vars files of the groups it belongs to. This has solved the issue of variables.

However, roles are a different story. We have several roles for different things, and many of these roles contain files that are only needed on one or a small subset of hosts. I would like to distribute these files only to the hosts that need them. This is the reason I asked my question.

For now, I have a work-around, that works, but is ugly. In my roles, wherever I need to copy a file, I'm doing this:

copy: content='{{ file_content }}' dest=/path/to/dest

I'm keeping the entire file's content in a variable called "file_content" in that host's host_vars file, or in a group_vars file if appropriate. For lines of a few lines, this isn't bad. But files that are bigger look ugly in a variable.

So if Ansible had the concept of host_files and group_files, with some kind of "first found" logic, it would be really nice.

Regards,

Anand

[Don Harper]

Hum...could you make use of git or svn, and just checkout host specific branches?  Might lead to a bit a bloat on the server side, but would keep the client side cleaner.

Don 

[Anand Buddhdev]

On 19 April 2015 at 16:54, Don Harper <duck...@gmail.com> wrote:

Hi Don,

Hum...could you make use of git or svn, and just checkout host specific branches?  Might lead to a bit a bloat on the server side, but would keep the client side cleaner.

We floated this idea early on in our design, and our entire team felt that a per-host branch would add too much overhead for us. Imagine managing a few hundred hosts, and having to switch branches for each host to make changes...

I gave this some more thought last night, and it turns out that many of the files we distribute are scripts or secret-less config files, and it's not too bad if they are exposed to all our hosts. The number of files that contain actual secrets are few, and the secrets are things like licence files, so my trick of keeping them in variables works well.

While Ansible was designed to be used in "push" mode, so that no state or sensitive data needs to be kept on managed servers, it also has the "pull" mode, which I really like, so we don't have to push anything. Clients check in when they can and reconfigure themselves. Therefore, something that would allow selective exposure of files and variables would be great!

Anand

[Krzysztof Zarzycki]

Hi! Has anyone found any solution to this problem? Or a good workaround?

I'm hitting the same issue, that I would like to have host/group or inventory (like dev,prod) specific files, maybe also templates.

Storing them in variables would not work well, when e.g. you care about indentation, or when the data is binary...

Thanks for any help with that!

Krzysztof