Feature Addition: Argument to Dynamic Inventory Scripts for Initial Parsing

[Justin Seubert]

Currently, all scripts for dynamic inventory are expected to return all hosts that are only wanted using two passed arguments: --list and --host. There is no way from ansible, ansible-playbook, the base API or others to specify the criteria for which the host of the inventory are to be made dynamically against.

Use cases and examples.

• For cloud based inventories, we have to find a way to tell the scripts what regions to pull servers from. This is done through the script configuration and environmental variables.
  
• If you run a large enterprise where you want to deploy one controller for many organizational units that you run, separated into customer or account numbers and separate devices in each configuration.
• Managing thousands of hosts, in these same ways, concepts of configuration of the script have to be accounted to cut down on the number of API calls we make.

Instead, we should be able to also pass a --script-search argument, allowing this extra information to be sent to and parsed by the end script. This is meant to be ambiguous, optional, and left up to the developer of the resulting dynamic inventory script to do with as they please. As it stands, we either have to account for this through outside script and Ansible interactions, or rely on returning all the data possible from a script, and parsing our host within Ansible using patterns and subsets. Adding the additional argument, we could do the following.

• For cloud inventories we could now then pass a geographic region that the script will only retrieve the hosts needed: ansible -i cloud.py --script-search US -m ping web
• For a large enterprise with multiple business units, customers and devices, this will allow the script only to call and retrieve the needed host: ansible -i MyCompany.py --script-search "customer=1234 BU=small" -m ping mysql-server
• For managing thousands of host, we can cut the API cost down by specifying the criteria -before- delivering to Ansible: ansible -i company.py --script-search stagingEast -m ping webheads

This allows for easier maintainability of our dynamic inventories, cuts down on the cost they cause on the external systems and makes for a better fit in existing automated systems or configurations. We already have completed changes in the inventory code, ansible and ansible-playbook for this, and in testing in our environment the results are exactly as expected. I am seeking interest to have this feature and code review to submit the pull request if wanted.

Also any thoughts or questions are welcome!

[Matt Martz]

I have a number of dynamic inventory scripts that allow this.  We use environment variables for it, and have the inventory script look for those environment variables.  The rax.py inventory script included with ansible does this too:

RAX_REGION=DFW ansible all -i /path/to/rax.py -m ping

That restricts the rax.py inventory plugin, to only look at the DFW region, by inspecting os.getenv('RAX_REGION') or similar.

-- 
Matt Martz
ma...@sivel.net

--
You received this message because you are subscribed to the Google Groups "Ansible Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-deve...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Michael DeHaan]

Yes, environment variables (or reading a config file) are your friends.

--

Michael DeHaan <mic...@ansibleworks.com>
CTO, AnsibleWorks, Inc.
http://www.ansibleworks.com/

[Justin Seubert]

Right, I am trying to solve and prevent from having to use something like environmental variables. When working in our case, with an automation piece on top of the Ansible API, it gets difficult having to be required to do that. If I am simply running ansible by command line, there are not any issues with doing that.

[Kai Wilken]

Hi,

we are currently evaluating ansible for our deployments. Currently we have homebrew scripts to do all this stuff, and one option would be to reeengeneer this using ansible. We have a central database / webapp, where all the deployment parameters (hosts, artifacts, configuretion) is stored. It seem very easy to write an ansible adapter for this datastore.

One of the parameters for a deployment is which envrionment we want to deploy to. The same scripts from the same host work for every envrionment (DEV, Testing, Prod; multiple prod instances for different countries) the same at the moment. An environment variable to limit the scope of the exportet environments would be working. But a parameter passed from the command line call of ansible to the datastore woul be much more helpful.

Another option would be to pass this information to the playbooks, to run on all webservers in group envrionment1 or envrionment2 or ... or  environmentN.

In the documentation, a seperation of staging and producton envrionments is recommended (I would agree), but there is no internal way (command line option, etc) of passing such an information to the inventory scripts. From my expierience, an explicit command line option is somewhat safer than an environment variable.

We are still at the beginning of eploxring all the options of ansible. But this already popped up.

[Michael DeHaan]

You're just going to get the environment variable.

While you don't have to set it in your environment and could just prefix the command, you could also use a config file and use OS permissions to control access.

Either way.

There won't be a command line option as there is no where to put it and we don't want to break compatibility with any of the other external inventory scripts by feeding them an option they aren't expecting.

--

[Brian Coca]

I do this by having several inventory scripts, each with their own ini file so using -i inventory/prod.py has that script query the DB (vmware in my case) and only return the production hosts.

In my case I wrote the script so each instance can be a link to the original but will look for an ini file with the name it was invoked with.

[Michael DeHaan]

The $0 trick is pretty clever.

Another easy way might be

INVENTORY_SCRIPT_CFG=/path/to/config.cfg

Which doesn't surface anything in the environment that is sensitive and can still use OS perms.

On Wed, Feb 26, 2014 at 9:07 AM, Brian Coca <bria...@gmail.com> wrote:

I do this by having several inventory scripts, each with their own ini file so using -i inventory/prod.py has that script query the DB (vmware in my case) and only return the production hosts.

In my case I wrote the script so each instance can be a link to the original but will look for an ini file with the name it was invoked with.

--

[Robert Thomson]

Hi Justin,

I hate to revive old threads, but this one is directly relevant to me.

My inventory system is key/value-based, rather than Ansible's group-based.  It's also huge and can't be cleanly separated into sub-sets (I could make some generalisations, but there'd always be exceptions).

The approach that I've used so far has been to use wrapper-scripts to ansible and ansible-playbook which set some environment variables, and then call ansible* with my inventory script.  The inventory script parses the playbook to evaluate the host patterns and then only includes relevant hosts.  https://github.com/ansible/ansible/issues/7622 (Closed, Not Implemented) explains my approach.

Using environment variables and wrapper scripts does seem very hacky to me (although it works fine), but I'm hoping for an alternative solution at some point.

In the ideal world, we'd be able to write our own internal inventory plugin for Ansible, instead of using a script.  It would then be the inventory plugins's responsibility to produce a stable inventory based on all the available information.

I think one of the concerns from the core team is that alternative internally pluggable inventories wouldn't be stable, which could cause issues with logging, group membership changing halfway through a playbook, etc.

I don't see that as a good reason to not support an internally pluggable inventory, but it'd be a good reason to make the internal API's requirements extremely clear.  A well defined internal-inventory API and a clean default implementation would also be its own reward.

Given that I have something working and limited personal time these days, I'm unable to attack this myself.  I'd certainly get involved if the core team picked this up, however.

Best Regards,

  Robert Thomson

[Michael DeHaan]

"Using environment variables and wrapper scripts does seem very hacky to me (although it works fine)"

This seems like a mental block that is easy to get over :)

You can even make an environment variable set a path to a config file.

"I don't see that as a good reason to not support an internally pluggable inventory"

Sorry, I don't know what this phrase means.

"I think one of the concerns from the core team is that alternative internally pluggable inventories wouldn't be stable"

Nor this.

If we can, let's step back, ignore the old thread, and talk about what you are trying to model, and we can arrive at understanding what you are trying to do more.

--

You received this message because you are subscribed to the Google Groups "Ansible Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-deve...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Robert Thomson]

Hi Michael,

In my current position, I have to think about process scaling in a larger enterprise.  I work toward providing standard solutions that can be used and supported across the entire organisation, which will reduce chaos and unsupportable services/products along the way.

We have lots of teams full of talented people with varying backgrounds and skill-sets, and if the organisation doesn't provide standard solutions then many teams will create their own (which is usually fun and rewarding for them in the short term), but in the long term we could end up with some costly problems.

For Ansible to have a chance of becoming an official offering in my organisation, We'd need:

1. good integration with our CMDB
2. integration with our existing deployment and management tools
3. a standard means to create, edit, version, audit, and execute plays or ad-hoc commands, supporting context-aware custom authorization, resource locking etc.
4. internal documentation, training, etc.

I realise that everything above, with the possible exception of #3, would have to be done in-house. The result would also have to be compelling enough to have folks prefer it over custom solutions in order to justify continued support.

I think that Ansible could be a good fit for many of our tasks. But without a very careful approach, it's just another tool adding to long-term chaos.

I recently saw a talk by someone from Twitter about their use of Ansible, and it looks like they've invested a lot of effort in solving the above problems for their organisation.  I'm looking forward to seeing more from them, and I believe that they intend to release some patches also, which might be interesting for my use-case.

So, the crux of it: Before I invest too much effort, I'd like to know if the Ansible team understands and supports goals and requirements like this and will keep them in mind during future development.

A few comments are in-line below also.

Cheers,

    Rob

On Friday, June 6, 2014 2:09:18 PM UTC+2, michael wrote:

"Using environment variables and wrapper scripts does seem very hacky to me (although it works fine)"

This seems like a mental block that is easy to get over :)

You can even make an environment variable set a path to a config file.

It's hard to tell if you're trying to be helpful or condescending with this obvious example.  It's a good idea to assume the best intentions though, so thanks for the example.

"I don't see that as a good reason to not support an internally pluggable inventory"

Sorry, I don't know what this phrase means.

Right now the inventory functionality is only pluggable via an external script or files.  An internally pluggable inventory would run in the same process as Ansible.

It would have benefits over external inventory scripts if it were also tasked with all the group logic (ie. if it converted frontend:&production etc. into the hosts by itself, rather than being queried separately for the frontend & production groups), because this would allow for more efficient integration with external inventory sources.  The more context that's available, the more efficient it could be.

I realise that this idea may conflict with current understanding of what the inventory is in Ansible today, so please suggest a better name for what I described and I'll switch my terminology.

Wrapper/custom scripts, environment variables, and parsing command line arguments & playbooks also solve this problem for me. I still feel it's a hacky approach, although as you say, it's a relatively small mental hurdle.

"I think one of the concerns from the core team is that alternative internally pluggable inventories wouldn't be stable"

Nor this.

I must be mis-remembering a conversation from a year or so ago, sorry.

[Michael DeHaan]

Check out ansible.com and the team page for some of our background, and I think you can look at Ansible's history over the last 2+ years for an example about how we make language decisions as a project.

(For other business questions, shoot us an email at in...@ansible.com -- we can help with services integrations)

--