request: group_vars loading strategy to handle deploy-specific roles variables

[Kesten Broughton]

In a previous thread, it was discussed how one could apply deployment specific vars to a play.  My solution at the time was to pass in a master config "pointer" file via the commandline with the -e "@filename" idiom containing file names that could be loaded by vars_files or include_vars.

ansible-playbook -i hosts site.yml -e "@cluster_config.yml"

Then in site.yml (or a tasks file with include_vars: )

 - hosts: hadoop_cluster

   vars_files:

     - ["{{network_config}}]

     - ["{{environment_config}}]

     - ["{{hadoop_config}}]

Michael suggested this was an anti pattern and I did find a maintenance problem because of all those vars_files lines sprinkled about the playbook.

My preferred method is now to modify the hosts file to add additional deployment specific groups to the top of the hosts file and have the vars loaded automatically via group_vars.

Currently, the group_vars loader does what I would call "Immediate folder match".  If a top level folder in group_vars matches any of the groups, all descended files and folders ending in .yml will be loaded.

This prevents hierarchical groupings of deployments.  I would like to organize my deployments like this

[kbroughton@mb-kbroughton:lynx-ansible/dev-ansible + (develop)] tree ../21ct-ansible/group_vars/

../21ct-ansible/group_vars/

├── all

└── datacenter

    ├── datacenter.yml

    ├── production

     |       ├── production.yml  # vars common to all prod systems

     │      └── client1_prod

     │               ├── client1_app.yml

     │      └── client2_prod

     │               ├── client2_app.yml

     └── staging

        ├── staging.yml

        ├── client1_stag

        │   ├── client1_stag.yml

        │   ├── client1_app.yml

        └── client2_stag

Each step of the descent, files in a matching folder are loaded but only matching subfolders will be descended into.

I'm writing this to open discussion again on how to best accomplish deployment-specific vars.  At the moment, my solution is to modify the signature of the _load_vars and related methods in

ansible/lib/ansible/inventory/vars_pluginsgroup_vars.py

to pass in groups information.

Then I add the checking that only descends into a directory if it is a in groups[host].

The default behavior could remain as it is with a new variable in .ansible.cfg

group_vars_load_strategy: { one of HIERARCHICAL_FOLDER_MATCH,

IMMEDIATE_FOLDER_MATCH - the current behavior

}

Would a pull request along these lines be considered, or is there an alternate preferred method that achieves the deployment-specific vars management we need?

kesten

 

--

Kesten Broughton
512 701 4209

[Michael DeHaan]

"Would a pull request along these lines be considered..."

It wouldn't, as I don't want too many ways to do this and wouldn't want to maintain the code for a fringe case -- however this is a great example of a case where you could write  your own external inventory script to implement any behavior you like, and this is why these exist -- they aren't just for talking to a database, can also be for "I want to organize my information differently and have these preferences..."

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAO2fFsVtAEsnDC_JsE5Z5qsyrfYDOKHLhYwT_6NZWsiUbyvF%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Serge van Ginderachter]

Another option might also be to write your own vars_plugin.

Though working with an inventory script, as Michael put it, is the better way to do it, and will also be a lot more performant.

--

[Michael DeHaan]

Yep, the vars plugin will (currently) be evaluated once per host.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAEhzMJA80L0Q3Bou0hkRV_Zzmq2-%3De6mCUu98hDn4EN0-P_ePA%40mail.gmail.com.

[Kesten Broughton]

Thanks serge and michael.

I'm not opposed to writing my own plugin, though I'm basically happy with group_vars.py.  My change is just the 5 lines here.

    # directory

    if stat.S_ISDIR(pathstat.st_mode):

        # support organizing variables across multiple files in a directory

        if GROUP_VARS_STRATEGY == 'HIERARCHICAL_FOLDER_MATCH':

            if os.path.basename(path) in groups:

                return True, _load_vars_from_folder(groups, path, results)

            else:

                return True, results

        else:

            return True, _load_vars_from_folder(groups, path, results)  # the original group_vars.py line

So what is the porper way to get ansible to use my_group_vars.py as the plugin rather than group_vars.py?  I would like to keep pulling in future changes to group_vars.py and just run off a local branch with my patch applied.  Should i just locally over-write group_vars.py or is there a proper plugin switch?  I will have to distribute my plugin across multiple users.

kesten

[Serge van Ginderachter]

On 1 April 2014 16:50, Kesten Broughton <solarmobi...@gmail.com> wrote:

So what is the porper way to get ansible to use my_group_vars.py as the plugin rather than group_vars.py?

​Just put your plugin in ./vars_plugins/ where . = the dir where your inventory (hosts file) is.

Update your plugin to only handle the subdirs you need, the original plugin will continue to work as intended (but in practice not loading any vars in your setup.)

​

[Kesten Broughton]

Sorry, i missed the "distributing plugins" section at the bottom

http://docs.ansible.com/developing_plugins.html#distributing-plugins

However, i'm not clear if dropping my group_vars into ansible/plugins/vars_plugins/group_vars.py will run before, after or instead of lib/ansible/inventory/vars_plugins/group_vars.py?

Is there a way to specify ordering / precedence of files in a plugin directory?  Maybe just point me to the plugin loader code?

k

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/oIQzp6nCs7k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.

To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/8c38baf0-f7c0-47de-ac42-3257ab91890a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

[Serge van Ginderachter]

On 1 April 2014 19:33, Kesten Broughton <solarmobi...@gmail.com> wrote:

However, i'm not clear if dropping my group_vars into ansible/plugins/vars_plugins/group_vars.py will run before, after or instead of lib/ansible/inventory/vars_plugins/group_vars.py?

Is there a way to specify ordering / precedence of files in a plugin directory?  Maybe just point me to the plugin loader code?

​They will all run, and the order is not really defined, nor configureable, though i *think* the core plugin will run first.

Plugin loader code is in ​lib/ansible/utils/plugins.py.

The vars plugin get initialized in lib/ansible/inventory/__init__.py (at the end of the Inventory constructor)

[kesten broughton]

Serge, thanks for pointing me to the help.

It seems that group_vars.py plugin runs after my plugin which gives the wrong precedence for what i need.

Just to close the loop on this thread, my solution was to add self.priority = 1 in the constructor of VarsModule for my deploy_vars.py plugin and then this in

 lib/ansible/inventory/__init__.py

       #self._vars_plugins = [ x for x in utils.plugins.vars_loader.all(self) ]  # original line replaced by the lines below

        # 21ct KB add priority to all deploy_vars to have higher precedence than group_vars

        # higher priority value = higher precedence

        unsorted_plugins = [ x for x in utils.plugins.vars_loader.all(self) if not hasattr(x,'priority')]

        sorted_plugins = [ x for x in utils.plugins.vars_loader.all(self) if hasattr(x,'priority')]

        sorted_plugins = sorted(sorted_plugins, key=lambda t: t.__getattribute__('priority'))

        # last in list has highest precedence 

        self._vars_plugins = (unsorted_plugins or []) + (sorted_plugins or [])

thanks for your assistance on this.

kesten

[Michael DeHaan]

Removing the core plugin is unsupported and would create a legion of issues if you were to file bugs on a version of ansible that ran different code than everyone else is running.

I would recommend using it as implemented.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/fb1c6e10-1342-4868-9048-17433bc92540%40googlegroups.com.