Puppet report aggregation

[Luke Bigum]

Hi all,

Can anyone recommend any tools for Puppet report aggregation? I'm interested in something that can take a given set of Puppet reports and summarise to me what resources have changed across all hosts.

If nothing exists I will look to write one myself. In that case, is Puppet report format 3 valid for Puppet 3.0?

http://projects.puppetlabs.com/projects/puppet/wiki/Report_Format_3

Thanks,

-Luke

[Ohad Levy]

You could use foreman for that? filtering the hosts via search should allow you to find the exact resources you are looking for?

Ohad

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/Fcx6zByYGPQJ.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Luke Bigum]

On Wednesday, December 5, 2012 12:50:43 PM UTC, ohad wrote:

You could use foreman for that? filtering the hosts via search should allow you to find the exact resources you are looking for?

Ohad

Hi Ohad,

I haven't looked at The Foreman in a while but in my mind it's more like Puppet Dashboard - correct me if I'm wrong. What I'm aiming for is a tool that can aid change / release management where we run Puppet --noop across the estate, gather all the reports, then summarise what changes will be applied (resolv.conf changes on all hosts, fstab changes on 20 hosts, service X refreshes on Y hosts).

I don't really want to be searching for explicit resources changing across hosts, it's the resources I don't know about that worry me ;-) Is the foreman worth a look in this case?

-Luke

[Ohad Levy]

On Thu, Dec 6, 2012 at 11:29 AM, Luke Bigum <Luke....@lmax.com> wrote:

On Wednesday, December 5, 2012 12:50:43 PM UTC, ohad wrote:

You could use foreman for that? filtering the hosts via search should allow you to find the exact resources you are looking for?

Ohad

Hi Ohad,

I haven't looked at The Foreman in a while but in my mind it's more like Puppet Dashboard - correct me if I'm wrong. What I'm aiming for is a tool that can aid change / release management where we run Puppet --noop across the estate, gather all the reports, then summarise what changes will be applied (resolv.conf changes on all hosts, fstab changes on 20 hosts, service X refreshes on Y hosts).

You should be able to setup a search term that would match your requirement 

it might be best to use the API for that (http://bit.ly/Uq033I), since you would get all of your reports already parsed and you could cross reference the info with a puppet env, fact etc.

Ohad

I don't really want to be searching for explicit resources changing across hosts, it's the resources I don't know about that worry me ;-) Is the foreman worth a look in this case?

-Luke

 

On Tue, Dec 4, 2012 at 11:00 PM, Luke Bigum <Luke....@lmax.com> wrote:

Hi all,

Can anyone recommend any tools for Puppet report aggregation? I'm interested in something that can take a given set of Puppet reports and summarise to me what resources have changed across all hosts.

If nothing exists I will look to write one myself. In that case, is Puppet report format 3 valid for Puppet 3.0?

http://projects.puppetlabs.com/projects/puppet/wiki/Report_Format_3

Thanks,

-Luke

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/Fcx6zByYGPQJ.

To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/zk_ospPVmYkJ.

[John Warburton]

On 6 December 2012 20:29, Luke Bigum <Luke....@lmax.com> wrote:

I haven't looked at The Foreman in a while but in my mind it's more like Puppet Dashboard - correct me if I'm wrong. What I'm aiming for is a tool that can aid change / release management where we run Puppet --noop across the estate, gather all the reports, then summarise what changes will be applied (resolv.conf changes on all hosts, fstab changes on 20 hosts, service X refreshes on Y hosts).

I don't really want to be searching for explicit resources changing across hosts, it's the resources I don't know about that worry me ;-) Is the foreman worth a look in this case?

Luke, we use the puppet dashboard which aggregates all the reports and then lets us suck down a CSV ("Export nodes as CSV" on front page) which contains a status of all resources on all machine reporting. We run puppet in noop all the time, so need similar reports you are requesting. It is just a matter of slicing & dicing the csv to get what you want

% wget http://localhost:3000/nodes.csv

# Omit all the resources in sync
% egrep -v ',0,0,false,false' nodes.csv | wc -l
146057
% wc -l nodes.csv
612762 nodes.csv
% echo "scale=2; 146057*100/612762" | bc
23.83

Almost 24% of all resources (600K mind you) out of sync

We can check our security module compliance:
% egrep -v ',0,0,false,false' nodes.csv | grep modules/security|wc -l
6992
% grep modules/security nodes.csv|wc -l
30433
% echo "scale=2; 6992*100/130433" | bc
5.36

5% *resources* out of sync from our security module. Not bad. We can dice it to give percentages of hostnames too

Top 10 resources out of sync:
% egrep -v ',0,0,false,false' nodes.csv | awk -F, '{print $7, $8}' | sort | uniq -c | sort -rn | head
    898 File /etc/sudoers
    801 File /opt/local/sbin/run-puppet.sh
    792 File ntp.conf
    754 File vposend-test.pl
    752 File /opt/local/sbin/puppet-register.sh
    751 File /var/puppet/etc/local_puppet_info
    745 File /etc/default/passwd
    711 File /local/apps
    710 File /etc/default/nfs
    705 File /etc/ssh/sshd_config

John

[Luke Bigum]

On Thursday, December 6, 2012 10:07:43 PM UTC, John Warburton wrote:

On 6 December 2012 20:29, Luke Bigum <Luke....@lmax.com> wrote:

I haven't looked at The Foreman in a while but in my mind it's more like Puppet Dashboard - correct me if I'm wrong. What I'm aiming for is a tool that can aid change / release management where we run Puppet --noop across the estate, gather all the reports, then summarise what changes will be applied (resolv.conf changes on all hosts, fstab changes on 20 hosts, service X refreshes on Y hosts).

I don't really want to be searching for explicit resources changing across hosts, it's the resources I don't know about that worry me ;-) Is the foreman worth a look in this case?

Luke, we use the puppet dashboard which aggregates all the reports and then lets us suck down a CSV ("Export nodes as CSV" on front page) which contains a status of all resources on all machine reporting. We run puppet in noop all the time, so need similar reports you are requesting. It is just a matter of slicing & dicing the csv to get what you want

% wget http://localhost:3000/nodes.csv

Thanks John and Ohad,

I use Puppet Dashboard but I've never tried that control before ;-) That should do as a very good start.

Cheers,

-Luke