Using Spacewalk or Red Hat Satellite - new inventory script

[Michael DeHaan]

I just merged a pull request for an Ansible inventory source.

Testing (and patches) quite welcome.

Thanks to Jon Miller for the inventory script!

As a reminder, inventory scripts are ways to pull ansible inventory (host names, group memberships, and variables) from external software systems as opposed to the default INI files.   Just point to what inventory source you are using with -i.  Other examples include Cobbler, AWS or Eucalyptus, and OpenStack Nova.  If a file is marked executable, it returns JSON and accepts certain parameters, it is treated as a script not a static inventory INI file.   If you have a directory full of inventory files and scripts, just specify "-i" and the directory name to use multiple data sources at once.  (This works best in 1.2 otherwise ansible doesn't like seeing config files for the inventory scripts in the same directory).   You can read more about them here:

http://ansible.cc/docs/api.html#external-inventory-scripts

--

Michael DeHaan <mic...@ansibleworks.com>
CTO, AnsibleWorks, Inc.
http://www.ansibleworks.com/

[Vincent Van der Kussen]

Hi,

This sounds great and would be really useful in conjunction with the
rhn_channel module to fill in the sysname variable.

I had a look at this, but from what i see, this can only be used on a
Satellite server?

Regards,
Vincent

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Dag Wieers]

On Fri, 24 May 2013, Vincent Van der Kussen wrote:

> This sounds great and would be really useful in conjunction with the
> rhn_channel module to fill in the sysname variable.
>
> I had a look at this, but from what i see, this can only be used on a
> Satellite server?

I have an inventory script that connects through the RHN API and it is
dead slow. Just querying each system to get even the most basic
information is slow and if you have 2000 servers...

And while you can cache the information, the moment the cache needs to be
renewed you're waiting for the inventory to finish.

(The fact that I add the IP address from RHN into the inventory doesn't
help in speed, but it's the only way to know how to get to them)

--
-- dag wieers, d...@wieers.com, http://dag.wieers.com/
-- dagit linux solutions, con...@dagit.net, http://dagit.net/

[Any errors in spelling, tact or fact are transmission errors]

[Dag Wieers]

On Fri, 24 May 2013, Dag Wieers wrote:

> On Fri, 24 May 2013, Vincent Van der Kussen wrote:
>
>> This sounds great and would be really useful in conjunction with the
>> rhn_channel module to fill in the sysname variable.
>>
>> I had a look at this, but from what i see, this can only be used on a
>> Satellite server?
>
> I have an inventory script that connects through the RHN API and it is dead
> slow. Just querying each system to get even the most basic information is
> slow and if you have 2000 servers...
>
> And while you can cache the information, the moment the cache needs to be
> renewed you're waiting for the inventory to finish.
>
> (The fact that I add the IP address from RHN into the inventory doesn't help
> in speed, but it's the only way to know how to get to them)

Looking at the implemented solution for spacewalk.py, it will fail to work
on any current RHN Satellite since spacewalk-report does not support the
"system-groups-systems" report.

The "inventory" report on our RHN Satellite with about 2000 active systems
takes more than 10 minutes to complete (which probably matches with what
we do directly using the RHN API).

Not sure if there's anything we can do to speed up the RHN Satellite.

[Vincent Van der Kussen]

Dag, do you have a link to your script?
I'm working on something myself...

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.

[Dag Wieers]

On Fri, 24 May 2013, Vincent Van der Kussen wrote:

> Dag, do you have a link to your script?
> I'm working on something myself...

Nope, it's unfinished and too slow to be useful.

I wanted to add the various resources found in RHN to the inventory. We
have groups for patch-management and additional info per server that I
thought could be useful, but since it's even unbearable to get a list of
hosts and (only!) query to get the IP address for each host to use as
ansible_ssh_host, I didn't even bother going the whole nine yards...

I'll probably turn to the internal Linux CMDB (Mysql based) instead, if I
find the time :)

[Michael DeHaan]

As someone who used to work 3 cube rows over from the Satellite team, I'm definitely interested in seeing it work with Satellite proper as well.

I suspect they will bring the groups support into a future release.

Meanwhile, if the module can detect what version of software it is talking to and not use the groups API unless it's recent, that makes sense.

The EC2 module caches on it's end, so I think a similar approach of caching is the logical way to deal with the API speed here.

Are you saying talking to RHN is slow or Satellite?  I'd think you would mean RHN, and yeah, I would assume that to be slow just because it's SaaS, but would also hope it could take advantage of similar caching.

--Michael

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Dag Wieers]

On Fri, 24 May 2013, Dag Wieers wrote:

> On Fri, 24 May 2013, Vincent Van der Kussen wrote:
>
>> Dag, do you have a link to your script?
>> I'm working on something myself...
>
> Nope, it's unfinished and too slow to be useful.
>
> I wanted to add the various resources found in RHN to the inventory. We have
> groups for patch-management and additional info per server that I thought
> could be useful, but since it's even unbearable to get a list of hosts and
> (only!) query to get the IP address for each host to use as ansible_ssh_host,
> I didn't even bother going the whole nine yards...
>
> I'll probably turn to the internal Linux CMDB (Mysql based) instead, if I
> find the time :)

Here's my current script.

The things I wanted to add was support for custom info, adding automatic
groups (e.g. based on base channel), adding more host-specific
information.

Since it's based on a previous CMDB script, it does include support for
DNS-based inventory as well as a network inventory (based on network
ranges).

Have fun with it :)

[Vincent Van der Kussen]

Hi,

I just had a look at retrieving hostnames and ip's from RHN. This is
indeed dead slow. Getting a list with dics from all hosts on the other
hand is quite fast.

Retrieving details from hosts comes with a penalty is seems...

I'm going to have a look at the caching options in Python. Never used
that before.. If someone has pointers (maybe someone from the
satellite/Sapcewalk dev team? )

@Dag thanks for the script, might come in handy

Vincent

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ansible-proje...@googlegroups.com.

[Michael DeHaan]

Great!

The EC2 module just caches to disk, but I'd probably just use the python shelve module:

http://docs.python.org/2/library/shelve.html

Having it take the invalidation option like the ec2 module seems smart.

[Dag Wieers]

On Sun, 26 May 2013, Vincent Van der Kussen wrote:

> I just had a look at retrieving hostnames and ip's from RHN. This is
> indeed dead slow. Getting a list with dics from all hosts on the other
> hand is quite fast.
>
> Retrieving details from hosts comes with a penalty is seems...
>
> I'm going to have a look at the caching options in Python. Never used
> that before.. If someone has pointers (maybe someone from the
> satellite/Sapcewalk dev team? )
>
> @Dag thanks for the script, might come in handy

My script does caching already, caching is not a problem. The problem is
that when the cache expires (and we don't want to cache for too long) that
Ansible is delayed. (Check out '-r' for renewing the cache)

Imagine your cache expires every day, it impacts your workflow every day
(more than once, depending on the set of systems that need to be renewed).

In my case I _only_ get group-information and IP addresses, but my
intention was to get much more from RHN, which unfortunately would make it
quite painful. 10 minutes for 2000 systems with only IP/group and DNS
query.

A solution could be to renew the cache in the background (in an atomic
way) so that it should not impact any concurrent Ansible. However
currently '-r' only renews a --list or --host lookup (mostly for testing
purposes).

[Michael DeHaan]

Yeah there's a standard way the ec2 module replaces the cache for all hosts.

It does seem like running it on cron is reasonable, though it might need a way to lock.

Send me a pull request if you would like to submit any upgrades or mergers of your two script ideas.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Vincent Van der Kussen]

I think caching will only be helpfull if you need something from you
inventory. Imho building/updating your inventory will always take
time.

What I plan to use for now is an inventory with just the groupnames
and the hosts in it. SInce we register our machines with their
hostname in RHN anyway we can use them to connect to.

One thing I would like to add as a variable is each host's id in rhn
so I can use it with the rhn_channel mod.

I'm no expert in developing or Python so it can take some time.

Regards,
Vincent

[Dag Wieers]

The system id is already part of that script I send you. Since it's needed
for any communication with the RHN API I figured it's best to include it
as a host fact.

BTW If you have multiple entries for a given hostname (duplicate RHN
entries) the inventory script will use the one that was last connected.
(so best to avoid this situation altogether)

[Vincent Van der Kussen]

On Mon, May 27, 2013 at 9:34 AM, Dag Wieers <d...@wieers.com> wrote:
> On Mon, 27 May 2013, Vincent Van der Kussen wrote:
>
>> I think caching will only be helpfull if you need something from you
>> inventory. Imho building/updating your inventory will always take
>> time.
>>
>> What I plan to use for now is an inventory with just the groupnames
>> and the hosts in it. SInce we register our machines with their
>> hostname in RHN anyway we can use them to connect to.
>>
>> One thing I would like to add as a variable is each host's id in rhn
>> so I can use it with the rhn_channel mod.
>>
>> I'm no expert in developing or Python so it can take some time.
>
>
> The system id is already part of that script I send you. Since it's needed
> for any communication with the RHN API I figured it's best to include it as
> a host fact.

I added a function to the rhn_channel mod to retrieve the sysid from
the host itself, but it's not used since i think that the one true
source is rhn itself ( i've seen cases where the local sysid is
different than the one in RHN.

>
> BTW If you have multiple entries for a given hostname (duplicate RHN
> entries) the inventory script will use the one that was last connected. (so
> best to avoid this situation altogether)
>

Nice feature :-)

I'm currently working on a small script that suits our needs. I'll
mail it when it's more or less finished...

Vincent

>
> --
> -- dag wieers, d...@wieers.com, http://dag.wieers.com/
> -- dagit linux solutions, con...@dagit.net, http://dagit.net/
>
> [Any errors in spelling, tact or fact are transmission errors]
>

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ansible-proje...@googlegroups.com.

[Dag Wieers]

On Mon, 27 May 2013, Vincent Van der Kussen wrote:

> On Mon, May 27, 2013 at 9:34 AM, Dag Wieers <d...@wieers.com> wrote:
>
>> BTW If you have multiple entries for a given hostname (duplicate RHN
>> entries) the inventory script will use the one that was last connected. (so
>> best to avoid this situation altogether)
>
> Nice feature :-)
>
> I'm currently working on a small script that suits our needs. I'll
> mail it when it's more or less finished...

I guess it can't hurt to put my script in the ansible-provisioning tree,
so we can send pull-requests.

[Dag Wieers]

On Sun, 26 May 2013, Michael DeHaan wrote:

> Yeah there's a standard way the ec2 module replaces the cache for all hosts.

Well, I use the same option in my script, however it will only replace the
cache for the query that is executed. And that is the problem. Even the
ec2 module will not refresh the host caches in this case.

If you want to do the same for each individual host query, you basically
have to script this yourself. It would be useful to standarize something
like:

-h, --host HOST
-l, --list
-r, --refresh-cache
-R, --refresh-all-caches

where --refresh-all-caches would iterate over each host to update its
cache as well.

Another possibility is to make --refresh-cache act differently depending
on whether it is executed together with --host, --list or no query. In the
latter case it could refresh everything, rather than just the query.

> It does seem like running it on cron is reasonable, though it might need a
> way to lock.

Just an atomic move of the cache should be sufficient. I don't think we
necessarily want to add locking-complexities to all inventory-scripts.


Something else that needs standardizing is the way to influence
inventory-scripts so that one does not need to modify the script itself to
add server-name, credentials or cache ttl.

I usually read out a specific file (e.g. hpam.ini, rhn.ini) however if we
have multiple sources we may want to read everything inventory-related
from a single file from different sections ? And standardize things like
cache ttl, etc...

[Michael DeHaan]

Someone was nice enough to send a contributing to the project, so pull requests should be send to ansible's main project.

There is no reason to have a seperate repo for this when we already have lots of great inventory modules in core.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Michael DeHaan]

You should submit upgrades to this one:

https://github.com/ansible/ansible/blob/devel/plugins/inventory/spacewalk.py

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Dag Wieers]

On Mon, 27 May 2013, Michael DeHaan wrote:

> Someone was nice enough to send a contributing to the project, so pull
> requests should be send to ansible's main project.
>
> There is no reason to have a seperate repo for this when we already have
> lots of great inventory modules in core.

The difference is that the original inventory script does not require any
configuration (as it uses spacewalk-report) whereas if I would add my
stuff to it, it requires configuration directives to work.

So it would break existing users.

[Vincent Van der Kussen]

I also think that this script can only used on a spacewalk server.
I would prefer to use the native rhn api instead.

Vincent

[Michael DeHaan]

We merged it in last week, so there's hardly anyone to break.

I think it's fine if it requires configuration, in fact, it seems it should require an INI for the spacewalk server anyway :)

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Michael DeHaan]

Yes, definitely.

It should at least degrade nicely if there is some new hotness that is not yet in a released Satellite server.

[Dag Wieers]

On Mon, 27 May 2013, Michael DeHaan wrote:

> We merged it in last week, so there's hardly anyone to break.
>
> I think it's fine if it requires configuration, in fact, it seems it should
> require an INI for the spacewalk server anyway :)

I'll rewrite it (remove some of the items I never finished) and do a new
pull-request.

What about having an option to regenerate the whole cache (all host info)
in the standard options of inventories ? The aim is to have a single
option to invalidate and renew the cache from a cronjob.

PS Could we not use the ansible.cfg for inventory options as a default ?

[Michael DeHaan]

Cache invalidation can more or less be a convention, yes.   We should stick with the parameter that the ec2 module adds now, and add more if need be.

There's nothing in ansible that needs to know about those flags to make it a requirement, but obviously calling it with shell is ok.

I don't think ansible.cfg is necc. ideal in the sense that you might want to use multiple inventory sources (like two different ec2 INI's) at different times (say stage vs prod?  or different customers in a MSP or hosted arrangement?), so in that case, you might have two different INI files.

--Michael

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Dag Wieers]

On Mon, 27 May 2013, Michael DeHaan wrote:

> Cache invalidation can more or less be a convention, yes. We should stick
> with the parameter that the ec2 module adds now, and add more if need be.
>
> There's nothing in ansible that needs to know about those flags to make it
> a requirement, but obviously calling it with shell is ok.
>
> I don't think ansible.cfg is necc. ideal in the sense that you might want
> to use multiple inventory sources (like two different ec2 INI's) at
> different times (say stage vs prod? or different customers in a MSP or
> hosted arrangement?), so in that case, you might have two different INI
> files.

A solution could be that the inventory-script's name should match the
section name in the config, e.g. spacewalk.py will look for
section [inventory:spacewalk] and rhn.py will look for section
[inventory:rhn].

Just an idea to avoid having to modify inventory scripts (even if only
to change the config-file to use).

[Michael DeHaan]

Yeah, I don't really care for that, I don't think.  

Renaming the script would be somewhat awkward and then the script would have to look at argv[0] to decide the section name, which would be confusing to have to explain to everyone I think.

Maybe I'm wrong though, but I'd like things, because they do not already do this, to remain consistent.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Vincent Van der Kussen]

On Mon, May 27, 2013 at 5:54 PM, Dag Wieers <d...@wieers.com> wrote:
> On Mon, 27 May 2013, Michael DeHaan wrote:
>
>> Someone was nice enough to send a contributing to the project, so pull
>> requests should be send to ansible's main project.
>>
>> There is no reason to have a seperate repo for this when we already have
>> lots of great inventory modules in core.
>
>
> The difference is that the original inventory script does not require any
> configuration (as it uses spacewalk-report) whereas if I would add my stuff
> to it, it requires configuration directives to work.

You mean adding rhn? How are you going to make the distinction between
spacewalk-report and using the rhn-api?

Vincent

>
> So it would break existing users.
>
>
> --
> -- dag wieers, d...@wieers.com, http://dag.wieers.com/
> -- dagit linux solutions, con...@dagit.net, http://dagit.net/
>
> [Any errors in spelling, tact or fact are transmission errors]
>

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to ansible-proje...@googlegroups.com.

[Michael DeHaan]

Well, I really meant Satellite.  But if it works with RHN too, great.

One should be able to talk to the API endpoint and figure out what you are talking to.

I'd hope.

That all being said, if it just uses the RHN API, isn't that fine?  I think it would be.

[Vincent Van der Kussen]

On Tue, May 28, 2013 at 2:45 PM, Michael DeHaan
<mic...@ansibleworks.com> wrote:
> Well, I really meant Satellite. But if it works with RHN too, great.
>
> One should be able to talk to the API endpoint and figure out what you are
> talking to.
> I'd hope.
>
> That all being said, if it just uses the RHN API, isn't that fine? I think
> it would be.

As far as I understand, the current inventory script uses a wrapper script
to get info from a satellite server, but can only be run on a satellite server
(correct me if i'm wrong).
The inventory script is not talking to the api itself...

Vincent

[Michael DeHaan]

Yes, I'm suggesting to merge dag's ideas and this one, or maybe we should just replace it with Dag's, if that gets us RHN and deployed Satellite support.

[Vincent Van der Kussen]

Ok.

I'm going to test Dag's script

[Vincent Van der Kussen]

Just tested Dag's script but ran into some problems

- I think it's not compatible with the latest RHN api. ( listSystems
-> listUserSystems )
- sysid must be integer and not a string ( int(system['id'] )
- all returned groups contain the same (all) hosts. haven't looked into this.

Vincent

On Tue, May 28, 2013 at 8:47 PM, Vincent Van der Kussen

[Michael DeHaan]

Since the RHN API is XMLRPC you should be able to do listMethods on the top level to see what functions it supports.

Unless it's not Python XMLRPC.  That's a non-standard method.

Though this does tend to imply it should be importing into a database instead, so it's easier to lookup later, or at least caching very very smartly :)

[Dag Wieers]

On Tue, 28 May 2013, Vincent Van der Kussen wrote:

> Just tested Dag's script but ran into some problems
>
> - I think it's not compatible with the latest RHN api. ( listSystems
> -> listUserSystems )
> - sysid must be integer and not a string ( int(system['id'] )
> - all returned groups contain the same (all) hosts. haven't looked into this.

Vincent,

It works in our infrastructure using RHN Satellite 5.5.0 (the latest IMO).

[Vincent Van der Kussen]

On Wed, May 29, 2013 at 2:09 PM, Dag Wieers <d...@wieers.com> wrote:
> On Tue, 28 May 2013, Vincent Van der Kussen wrote:
>
>> Just tested Dag's script but ran into some problems
>>
>> - I think it's not compatible with the latest RHN api. ( listSystems
>> -> listUserSystems )
>> - sysid must be integer and not a string ( int(system['id'] )
>> - all returned groups contain the same (all) hosts. haven't looked into
>> this.
>
>
> Vincent,
>
> It works in our infrastructure using RHN Satellite 5.5.0 (the latest IMO).

Dag,

I'm connecting to RHN (rhn.redhat.com) and yields the same errors I
had with my own
scripts connecting to it.
Weird....

Anyway, see you at the meetup. Maybe I can show you

>
>
> --
> -- dag wieers, d...@wieers.com, http://dag.wieers.com/
> -- dagit linux solutions, con...@dagit.net, http://dagit.net/
>
> [Any errors in spelling, tact or fact are transmission errors]
>

[Tracy Phillips]

On Wednesday, May 22, 2013 9:21:43 AM UTC-4, Michael DeHaan wrote:

I just merged a pull request for an Ansible inventory source.

Testing (and patches) quite welcome.

Thanks to Jon Miller for the inventory script!

Is there a script available that will connect to our Spacewalk server from our Ansible server (using an api) to get the host/group information?

Thanks,

Tracy

[James Cammarata]

Yes, in the source tar (or from github) there is plugins/inventory/spacewalk.py. 

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/df83e144-76bd-476c-8f4c-07eb0a75b27d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tracy Phillips]

Hi James,

Unless I am missing something spacewalk.py needs to be executed on the spacewalk server, not the ansible server, correct?

Thanks,

Tracy

[Michael DeHaan]

It does use the spacewalk-reports package.  I honestly don't know if you can configure it to point at a remote server.

It presumes you are running Ansible from the spacewalk server if you can't - but there's also a REST API for Satellite/Spacewalk, so I imagine alternatives could also be built fairly easily.

The documentation on the Spacewalk/Satellite API was sufficient for me to get a authentication plugin for Cobbler going with a live server, and I'm sure Red Hat might also be able to help with questions.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/054d35b4-ff7b-48d5-8942-09e91a168e71%40googlegroups.com.