get a *structured* version of the puppet agent output

[David Portabella]

puppet agent --verbose shows a verbose output of the changes done by puppet, such as:

notice: /Stage[main]/Logstash::Config/Logstash::Configdir[agent]/File[/etc/logstash/agent/config]/owner: owner changed 'root' to 'logstash'
notice: /Stage[main]/Varnish/Service[varnish]/ensure: ensure changed 'stopped' to 'running'

I'd need to make some analysis about these changes, but it is not trivial to correctly parse this output.
is there an option to get an structured version of this output (such as in xml or yaml)?
so that I can make queries such as:
- list all the files that have changed
- list all the services that changed from stopped to running.
...

yes, i can use sed and the like as a temporal workaround, but I'd need to have a robust approach.

option1: maybe there is a tool that correctly parses this output?
option2: or maybe there is a "puppet agent" equivalent command that directly produces a structured output?

otherwise, how much time would it required to implement option2 (i am a software developer, and a puppet-user (not currently a puppet-developer)?

Regards,

David

[R.I.Pienaar]

----- Original Message -----
> From: "David Portabella" <david.po...@gmail.com>
> To: puppet...@googlegroups.com
> Sent: Tuesday, November 26, 2013 4:54:03 PM
> Subject: [Puppet Users] get a *structured* version of the puppet agent output
>
>
>
> puppet agent --verbose shows a verbose output of the changes done by
> puppet, such as:
>
> notice:
> /Stage[main]/Logstash::Config/Logstash::Configdir[agent]/File[/etc/logstash/agent/config]/owner:
> owner changed 'root' to 'logstash'
> notice: /Stage[main]/Varnish/Service[varnish]/ensure: ensure changed
> 'stopped' to 'running'
>
> I'd need to make some analysis about these changes, but it is not trivial
> to correctly parse this output.

The same data is in the reports, a bit better there.

>> require 'puppet'
>> YAML.load_file("/var/lib/puppet/state/last_run_report.yaml").logs.first
=> #<Puppet::Util::Log:0x7f2e3dcbcc68 @source="Puppet", @level=:notice, @message="Finished catalog run in 0.18 seconds", @time=Thu Nov 21 16:38:42 +0000 2013, @tags=["notice"]>

> is there an option to get an structured version of this output (such as in
> xml or yaml)?
> so that I can make queries such as:
> - list all the files that have changed
> - list all the services that changed from stopped to running.
> ...
>
> yes, i can use sed and the like as a temporal workaround, but I'd need to
> have a robust approach.
>
> option1: maybe there is a tool that correctly parses this output?
> option2: or maybe there is a "puppet agent" equivalent command that
> directly produces a structured output?
>
> otherwise, how much time would it required to implement option2 (i am a
> software developer, and a puppet-user (not currently a puppet-developer)?
>
> Regards,
> David
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/7e4a2aa2-2bd2-4311-81c2-8f7e4bd12a5b%40googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>

[Dan White]

I am not sure if this will answer your question, but have you looked at the yaml files to be found under /var/lib/puppet/reports/  on the Puppet Master ?

Changes can be found by searching for "changed: true" , "out_of_sync: true", and/or "change_count: ?" where the "?" is NOT zero.

“Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.”
Bill Waterson (Calvin & Hobbes)

---

From: "David Portabella" <david.po...@gmail.com>
To: puppet...@googlegroups.com

Sent: Tuesday, November 26, 2013 11:54:03 AM

Subject: [Puppet Users] get a *structured* version of the puppet agent output

--

[Stuart Cracraft]

We have this exact same requirement.

Enforce a non-changing policy on the cloud, avoid automatic drift-prevention, await errors and reports,

track down cause, fix root cause, apply puppet manifests to right things back, etc.

Using Puppet to bandaid things without root-cause drill-down/post-mortem, is unwise!!!!

We need a simple report-formatting mechanism like David says!

[Stuart Cracraft]

What we want is not more complexity, but more simplicity!

I could go into

  puppet config print reportdir

and then to its

  /var/lib/puppet/reports

then to the host directories of interest and grep out message.

But that seems a very sorry state of affairs.

Puppetmasters speak UP!

[R.I.Pienaar]

----- Original Message -----
> From: "Stuart Cracraft" <smcra...@gmail.com>
> To: puppet...@googlegroups.com
> Sent: Tuesday, November 26, 2013 7:02:42 PM
> Subject: Re: [Puppet Users] get a *structured* version of the puppet agent output
>
> What we want is not more complexity, but more simplicity!
>
> I could go into
>
> puppet config print reportdir
>
> and then to its
>
> /var/lib/puppet/reports
>
> then to the host directories of interest and grep out message.
>
> But that seems a very sorry state of affairs.
>
> Puppetmasters speak UP!

you do not need to grep out anything, as I showed you there is a ruby API
for accessing this data.

If you use PuppetDB it will also be able to store this information and it has
APIs for extracting these logs in a structured manner.

API access is about as much as you can hope for I think. Your alternatives are
to write a logger plugin for Puppet that outputs JSON, I've done this and it
was not clear sailing.

[Stuart Cracraft]

Who will share a report generator for the yaml reports generated by puppet so that we do not have to reinvent wheel after wheel after wheel!!!!!!

> --
> You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/cHpZlKkPmr4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1024951816.609.1385492893475.JavaMail.zimbra%40devco.net.

[R.I.Pienaar]

----- Original Message -----
> From: "Stuart Cracraft" <smcra...@me.com>
> To: puppet...@googlegroups.com
> Sent: Tuesday, November 26, 2013 7:10:53 PM
> Subject: Re: [Puppet Users] get a *structured* version of the puppet agent output
>

> Who will share a report generator for the yaml reports generated by puppet so
> that we do not have to reinvent wheel after wheel after wheel!!!!!!

not everyone shares the same goals so everyone tend to write them, but here you go this is one I wrote

http://www.devco.net/archives/2013/10/10/cli-report-viewer-for-puppet.php

[Stuart Cracraft]

Puppet Labs:

What report generators do you
have and provide at your website
so that we can report intelligently?

Stuart

> --
> You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/cHpZlKkPmr4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/216012641.610.1385493314105.JavaMail.zimbra%40devco.net.

[Dan White]

Reference: http://docs.puppetlabs.com/puppet/3/reference/format_report.html

Start with this:
----------------------------------------------------------------------------------------
#!/usr/bin/ruby

require 'puppet'
if defined?(ARGV)
filename = ARGV.first

if defined?(filename)
report = YAML.load_file(filename)
print "Report for : "
puts report.host
print "Started : "
puts report.time
print "Log count: "
puts report.logs.size
print "Metrics count: "
puts report.metrics.size
print "Resource Status count: "
puts report.resource_statuses.size
print "Status: "
puts report.status

report.logs.each do |logg|
puts "LOG ---------------------------------------------"
if logg.file
print "File: "
puts logg.file
end

if logg.line
print "Line: "
puts logg.line
end

print "Level: "
puts logg.level
print "Message: "
puts logg.message
print "Source: "
puts logg.source
print "Tags: "
puts logg.tags.join(" ")
print "Time: "
puts logg.time
end

report.resource_statuses.keys.each do |kk|
if report.resource_statuses["#{kk}"].change_count > 0
puts "RESOURCE STATUS ===================================="
puts "#{kk} "
report.resource_statuses["#{kk}"].events.each do |line|
print "property: "
puts line.property
print "message: "
puts line.message
print "name: "
puts line.name
print "status: "
puts line.status
print "when: "
puts line.time
end
end
end
puts "-------------------------------------------------"
end
end
------------------------------------------------------------------------------------------

Good luck.

“Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.”
Bill Waterson (Calvin & Hobbes)

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/6E1AA223-0401-485C-8E4C-AABF12FB2291%40me.com.

[Stuart Cracraft]

I'll try it after exiting this meeting.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/877739462.4751612.1385494040183.JavaMail.root%40sz0126a.westchester.pa.mail.comcast.net.

[Christopher Wood]

(inline)

On Tue, Nov 26, 2013 at 10:43:31AM -0800, Stuart Cracraft wrote:
> We have this exact same requirement.
> Enforce a non-changing policy on the cloud, avoid automatic
> drift-prevention, await errors and reports,
> track down cause, fix root cause, apply puppet manifests to right things
> back, etc.
> Using Puppet to bandaid things without root-cause drill-down/post-mortem,
> is unwise!!!!

I don't see how it's wise to let configuration deviations loiter in your systems until a human has the time to manually fix them. This applies even more if you're working at scale. Surely the ticket generated from (whatever is monitoring agent runs that unpredictably change things) will still be there once your configs are back in conformance?

Short form would be that I think you'll experience fewer pain points if you let puppet enforce your configuration like it's designed to.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit

> [1]https://groups.google.com/d/msgid/puppet-users/5907f644-2285-4298-9a07-3d5e8de63a24%40googlegroups.com.
> For more options, visit [2]https://groups.google.com/groups/opt_out.
>
> References
>
> Visible links
> 1. https://groups.google.com/d/msgid/puppet-users/5907f644-2285-4298-9a07-3d5e8de63a24%40googlegroups.com
> 2. https://groups.google.com/groups/opt_out

[Stuart Cracraft]

Understood.

But, we have some real good troubleshooters here.

And we want to get at core fixes.

Not just log-examination, esoteric
thought and bandaids.

> You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/cHpZlKkPmr4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/20131126204924.GA4749%40iniquitous.heresiarch.ca.

[David Portabella]

Thanks Ygor and R.I. Pienaar.

that's exactly what i was looking for!

Regards,

David

[Felix Frank]

Hi,

On 11/26/2013 07:43 PM, Stuart Cracraft wrote:
>
> Using Puppet to bandaid things without root-cause
> drill-down/post-mortem, is unwise!!!!

you are entitled to your opinion, but it's not exactly common in the
community and, frankly, I believe it's not a good fit with puppet's
philosophy. In other words, puppet may in fact not be the best tool for
the job.

But then, you can probably bend it towards doing what you need. Please
keep in mind that the ruby code that RI Pienaar supplied is just a very,
very basic example. The YAML in puppet reports grants you great power
and flexibility, but do not step into either of the traps of

a) grepping through and manually parsing the YAML
or
b) loading the YAML using ruby or irb, but then just converting all key
datapoints to string representations to grep through these

Instead, find out which data structures are of interest to you (i.e.,
which attributes denote a resource you want to highlight from the
report) and write a ruby script that uses the common library tools (e.g.
http://www.ruby-doc.org/core-1.8.7/Array.html#method-i-select) to fetch
those objects and generate output that is suited to your auditing needs.

HTH,
Felix

[Felix Frank]

...aaaand then I saw that Dan had already shared it in what my mail
client considers a different thread for some unfathomable reason.

Sorry for the noise.

[David Portabella]

Thanks again! It is great to use the puppet report yaml files!

here there is an example script that prints all resources statuses, 

then it filters them by taking only the services,

then it filters them by taking only the services changed to running.

test.ruby

#!/usr/bin/ruby 

require 'puppet' 

filename = ARGV.first 

report = YAML.load_file(filename) 

rs = report.resource_statuses.values.sort_by{|r| r.resource}

puts "PRINT ALL RESOURCES STATUSES."

rs.each do |r| 

  puts "#{r.change_count} #{r.events.map {|e| e.previous_value}} #{r.events.map {|e| e.desired_value}} #{r.resource}"

end 

puts "PRINT ALL SERVICES."

services = rs.select{|r| r.resource_type == 'Service'}

services.each do |r| 

  puts "#{r.change_count} #{r.events.map {|e| e.previous_value}} #{r.events.map {|e| e.desired_value}} #{r.resource}"

end 

puts "PRINT ALL SERVICES CHANGED TO RUNNING."

services_changed_to_running = rs.select{|r| r.resource_type == 'Service' && r.change_count > 0 && r.events.index { |e| e.desired_value == :running } != nil}

services_changed_to_running.each do |r| 

  puts "#{r.change_count} #{r.events.map {|e| e.previous_value}} #{r.events.map {|e| e.desired_value}} #{r.resource}"

end 

 

example output:

$ test.ruby /var/opt/lib/pe-puppet/reports/test/201311261645.yaml 

PRINT ALL RESOURCES STATUSES.

1 [:absent] [:directory] File[/var/app]

1 [:absent] [:file] File[/var/app/app.conf]

2 [0, "755"] [91, "775"] File[/var/log/tomcat6]

0 [] [] Filebucket[puppet]

1 [:absent] [:present] Group[release]

1 [:absent] [:present] Package[varnish]

0 [] [] Schedule[weekly]

1 [:stopped] [:running] Service[varnish]

0 [] [] Tidy[/etc/collectd.d/]

1 ["/sbin/nologin"] ["/bin/bash"] User[tomcat]

...

PRINT ALL SERVICES.

1 [:stopped] [:running] Service[collectd]

1 [:stopped] [:running] Service[logstash-agent]

1 [:true] [:false] Service[logstash]

1 [:stopped] [:running] Service[nginx]

0 [] [] Service[rsyslog]

1 [:stopped] [:running] Service[statsd]

1 [:false] [:true] Service[supervisord]

1 [:stopped] [:running] Service[varnish]

PRINT ALL SERVICES CHANGED TO RUNNING

1 [:stopped] [:running] Service[collectd]

1 [:stopped] [:running] Service[logstash-agent]

1 [:stopped] [:running] Service[nginx]

1 [:stopped] [:running] Service[statsd]

1 [:stopped] [:running] Service[varnish]

[Stuart Cracraft]

Thanks for stating the obvious Felix. It's always good and I don't take it
any way except in which it was intended: educational.

Puppet has been running here in the desired scenario since last night near COB
which means simple monitoring and reporting of all actions which would be done,
but not done (noop = true and reports = true on node with classes enabled on
puppetmaster in puppet.conf) with daemon on node being --verbose in the /etc/init.d/puppet
script.

While I would still like more central control of the nodes from the Puppet Master, the above
does fit the requirement here so we are good on PM!

> --
> You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/cHpZlKkPmr4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/5295CC7D.8080902%40alumni.tu-berlin.de.

[Dan White]

Very nice.
I am still learning Ruby and this will help

Thanks.

“Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.”
Bill Waterson (Calvin & Hobbes)

From: "David Portabella" <david.po...@gmail.com>
To: puppet...@googlegroups.com

Sent: Wednesday, November 27, 2013 10:12:48 AM

Subject: Re: [Puppet Users] get a *structured* version of the puppet agent output

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/c70db1f6-b133-4290-a8e1-827ea239028b%40googlegroups.com.

[Jeffrey Watts]

+1

I've been using this script for the last month to view the "last run" details, and it works very well.  Here at my shop I am not worried about changes, but I am worried about errors.  I wrote a small Python script that queries PuppetDB (if you're not using it, use it) and reports if there were any errors on any nodes during their last run of the night (we do not run Puppet during trading hours).  I also use Puppet Dashboard for at-a-glance sanity checking and as an easy way to drill down on unresponsive hosts and to easily view reports.

The combination of these three tools work very well for me.

Jeffrey.

[Stuart Cracraft]

Get the book Learning Ruby. 

Worth it. 

You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/cHpZlKkPmr4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1253465309.4764508.1385567092577.JavaMail.root%40sz0126a.westchester.pa.mail.comcast.net.

[Dan White]

This one: http://shop.oreilly.com/product/9780596529864.do  ?

“Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.”
Bill Waterson (Calvin & Hobbes)

---

From: "Stuart Cracraft" <smcra...@me.com>
To: puppet...@googlegroups.com
Sent: Wednesday, November 27, 2013 11:00:00 AM

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/36085646-13B6-4457-A36B-B2C8F97A603E%40me.com.

[David Portabella]

Given this puppet manifest (test.pp):

$dir = '/tmp'

file {'myfile':

  path => "$dir/myfile.txt",

  content => 'hello'

}

puppet produces this report:

[...]

    File[myfile]: !ruby/object:Puppet::Resource::Status

      resource: File[myfile]

      file: /Users/david/test.pp

      line: 4

      evaluation_time: 0.001354

      change_count: 1

      out_of_sync_count: 1

      tags: 

        - file

        - myfile

        - class

      time: 2013-12-13 11:44:59.092716 +01:00

      events: 

        - !ruby/object:Puppet::Transaction::Event

          audited: false

          property: ensure

          previous_value: !ruby/sym absent

          desired_value: !ruby/sym file

          historical_value: 

          message: "defined content as '{md5}5d41402abc4b2a76b9719d911017c592'"

          name: !ruby/sym file_created

          status: success

          time: 2013-12-13 11:44:59.093067 +01:00

      out_of_sync: true

      changed: true

      resource_type: File

      title: myfile

      skipped: false

      failed: false

      containment_path: 

        - Stage[main]

        - ""

        - File[myfile]

[...]

so, the report tells that it has created File[myfile],

but it does not contain the path of the file /tmp/myfile.txt.

that's a pity, because it means that I cannot get a list of all the files updated by puppet.

is there a way to get the final path of all File resources created by puppet?

(not "$dir/myfile.txt", but "/tmp/myfile.txt")

[Stuart Cracraft]

You ask a great question.

Also, it would be great if we could get the actual Unix cli equivalent of what Puppet is proposing to do.

[Charlie Sharpsteen]

On Friday, December 13, 2013 5:30:19 AM UTC-8, David Portabella wrote:

The report is focused on describing the changes that occurred during that run. To that end, only the title is included as that information is sufficient to uniquely identify the reporting resource when combined with resource_type. Information concerning properties that were not changed during the run is omitted as this data can be obtained from the catalog that the agent was processing.

In this specific situation, you can modify your file resource such that the path is used as the title:

$dir = '/tmp'
file { "$dir/myfile.txt":
  content => 'hello'
}

However, the general problem of matching changes to resources can be solved by loading the report and then loading the catalog and joining the two datasets using the resource_type and title. However, this particular task is one of the problems PuppetDB was designed to solve. PuppetDB stores both the catalogs and reports and provides an API that can be used to query the data.

For example, the changes related to myfile can be retrieved by querying the events endpoint of the PuppetDB API:

curl -G 'http://localhost:8080/v3/events' --data-urlencode query=\
'["and", ["=", "resource-type", "File"],
  ["=", "containing-class", "Testfile"],
  ["=", "certname", "pe-310-agent.puppetdebug.vlan"]]'

Which gives results similar to the following:

[ {

  "status" : "success",
  "timestamp" : "2013-12-13T17:11:39.144Z",
  "certname" : "pe-310-agent.puppetdebug.vlan",
  "containing-class" : "Testfile",
  "containment-path" : [ "Stage[main]", "Testfile", "File[myfile]" ],
  "report" : "6bed5163b50b5857921b5ec27d9147b428c684f8",
  "run-start-time" : "2013-12-13T17:11:29.382Z",
  "resource-title" : "myfile",
  "configuration-version" : "1386954691",
  "run-end-time" : "2013-12-13T17:11:36.527Z",
  "property" : "ensure",
  "message" : "defined content as '{md5}5d41402abc4b2a76b9719d911017c592'",
  "new-value" : "file",
  "old-value" : "absent",
  "line" : 44,
  "file" : "/etc/puppetlabs/puppet/modules/testfile/manifests/init.pp",
  "report-receive-time" : "2013-12-13T17:11:41.334Z",
  "resource-type" : "File"
} ]

The resource contained in the catalog delivered to that node, which contains properties such as the path, can be retrieved by passing the resource Type and title to the resources endpoint:

curl -G 'http://localhost:8080/v3/resources/File/myfile' --data-urlencode 'query=["=", "certname", "pe-310-agent.puppetdebug.vlan"]'

This returns all the parameters of interest:

[ {
  "parameters" : {
    "path" : "/tmp/myfile.txt",
    "mode" : "0755",
    "content" : "hello",
    "backup" : "main",
    "alias" : [ "/tmp/myfile.txt" ]
  },
  "line" : 45,
  "file" : "/etc/puppetlabs/puppet/modules/testfile/manifests/init.pp",
  "exported" : false,
  "tags" : [ "default", "node", "myfile", "testfile", "class", "file" ],
  "title" : "myfile",
  "type" : "File",
  "resource" : "93f90701c8f54a485246a9e3725040f1992fd90b",
  "certname" : "pe-310-agent.puppetdebug.vlan"
} ]

The power of PuppetDB in this situation is that your reporting script can now focus on analyzing the data instead of finding and then filtering it.

Hope this helps!

-- 

Charlie Sharpsteen

Open Source Support Engineer

Puppet Labs

[David Portabella]

wow, thanks for this very exhaustive answer!

is there a way to get the list of resources created when using a puppet apply (instead of puppet agent)?

the point is that we need this when refactoring puppet modules, in order to test the modules in a vagrant machine and check that there are not regression issues.

so, i run our current puppet modules in a vagrant machine, get the list of all puppet resources created,

then i refactor the puppet modules, i run again the refactored puppet modules in a new vagrant machine,

and i compare all the files and other resources.

[Charlie Sharpsteen]

On Friday, December 13, 2013 2:43:41 PM UTC-8, David Portabella wrote:

is there a way to get the list of resources created when using a puppet apply (instead of puppet agent)?

the point is that we need this when refactoring puppet modules, in order to test the modules in a vagrant machine and check that there are not regression issues.

so, i run our current puppet modules in a vagrant machine, get the list of all puppet resources created,

then i refactor the puppet modules, i run again the refactored puppet modules in a new vagrant machine,

and i compare all the files and other resources.

If you are looking to keep track of resources while refactoring, then there are a few approaches you can take:

1. Add another VM to your Vagrant environment that runs a Puppet Master and do the refactoring there. This gives you access to tools such as PuppetDB and the Dashboard while also simulating how your refactor will affect an agent/master setup.
2. Set up your VM such that puppet apply submits the catalog and reports to PuppetDB: https://docs.puppetlabs.com/puppetdb/1.5/connect_puppet_apply.html
3. Configure configure YAML caching for puppet apply in /etc/puppet/routes.yaml:

---
apply:
  catalog:
    cache: yaml

This will cause puppet apply to save a copy of the last compiled catalog to /var/lib/puppet/state/client_yaml/catalog/<certname>.yaml. The information in this file can be combined with the report in /var/lib/puppet/state/last_run_report.yaml to create a list of applied resources along with their properties and resulting changes.

The first two approaches will cut down on the amount of custom code you need to write as the task of parsing and storing the reports and catalogs is handled by PuppetDB or the dashboard. Approach 3 works as well and offers a lot of control, but you will need to write a bit of code that extracts data from the catalog and report files before you can focus on the analysis.

[David Portabella]

Hi, thanks again for the info.

This seems an overcomplicated issue. Just to step back and look at the global picture:

we are just asking to get the list of all resources updated when executing "puppet apply" in a non master/agent puppet environment.

this should be a basic functionality; we are not asking something weird.

after Christmas I'll take a look at installing PuppetDB in the same virtual machine, connecting puppet apply to it, and try to get the list of resources updated by puppet apply.

Regards,

David

[Felix Frank]

Hi,

this thread has been going on for a while, and you may have already
commented on this, in this case please humor me:

Couldn't you collect and interpret the output of puppet apply?

[David Portabella]

hi,

yes, this script works quite well! :)

https://groups.google.com/d/msg/puppet-users/cHpZlKkPmr4/RRQ6VWsGdMQJ

However, we are currently using the "Oracle WebLogic / Fusion Middleware" puppet module,

http://forge.puppetlabs.com/biemond/wls/1.3.3

which declares some file resources with the path parameter, so the script no longer works because

the file's $name and $path differ, and the report does not contain the information about the path of the resource.

example: orawls/manifests/domain.pp:

# the domain.py used by the wlst
file { "domain.py ${domain_name} ${title}":
  path    => "${download_dir}/domain_${domain_name}.py",
  ...

[Erik Dalén]

If that is all you want, run it with --write-catalog-summary and examine the resourcefile (puppet apply --configprint resourcefile)

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/6d2793ce-0b9f-468b-bc35-76b149f31f46%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Erik Dalén

[Erik Dalén]

Well, that is actually all resources managed rather than all resources changed. The changed resources are in the report. In my case we use puppetdb for report querying.

--
Erik Dalén

[Stuart Cracraft]

What we are looking for is a Ruby program which takes the contents of

  /var/lib/puppet/reports/*/*.yaml

and reports in detail on everything changed or proposed for change if in noop mode

(file permissions, modes, user creates, etc.)

Stuart

[Jason Slagle]

Hi Stuart,

Puppet Labs has a large professional service department that you might
want to engage with these very specific requests. I'm sure they can
give you a hand with whatever you need done.

Jason

[Stuart Cracraft]

thanks.

who is your contact?

I am not getting the help I need.

> --
> You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/cHpZlKkPmr4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/52B1E2AB.7060909%40tacorp.net.

[Dan White]

Try this link:

http://puppetlabs.com/services/consulting

“Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.”
Bill Waterson (Calvin & Hobbes)

---

From: "Stuart Cracraft" <smcra...@me.com>
To: puppet...@googlegroups.com

Sent: Wednesday, December 18, 2013 1:01:42 PM

Subject: Re: [Puppet Users] get a *structured* version of the puppet agent output

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/36C6B660-92B8-4056-B82D-789C1B0AE7ED%40me.com.

[Stuart Cracraft]

It's too expensive given our small company.

[Jason Antman]

Stuart,

You have *very* specific requirements here. In these situations, there are generally three options:
1) find someone else who's done it and use theirs
2) learn how to do it yourself
3) pay someone else to do it
4) don't do it

You posted your question (which I assume you researched before posting, and did not find an appropriate solution) and nobody has yet stepped forward to say "I have that," though you have been given many examples, some very detailed, and haven't given back any indication that you tried them. If the prices that PL charges for consulting are too expensive, and the same is true for hiring some random ruby coder, I think either you (or someone at your company) need to learn ruby, or reassess how you are using Puppet and your requirements. From your many previous posts, it seems like you're asking things of puppet that are relatively uncommon.

There have been some very good responses in this thread, including the (IMO) best, which is to use PuppetDB and its API, in which case you also wouldn't be restricted to Ruby. What have you tried so far? What issues do you have with the various options?

This list has over 6,600 members, and you alone are responsible for a whopping 10% (yes, one in ten) of the posts in the last month. If you feel that the responses you receive are not adequate, perhaps you should be more judicious in how quickly you appeal to the community for an answer, how much detail you give (specifically "I tried this <link to source code> and seem to be experiencing this <very specific problem>"), and give a better indication that you've actually tried the suggested solutions, and what the results were.

On 12/18/2013 02:46 PM, Stuart Cracraft wrote:

It's too expensive given our small company.

On Wednesday, December 18, 2013 11:25:31 AM UTC-8, Ygor wrote:

Try this link:

http://puppetlabs.com/services/consulting

�Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.�

Bill Waterson (Calvin & Hobbes)

---

From: "Stuart Cracraft" <smcra...@me.com>
To: puppet...@googlegroups.com
Sent: Wednesday, December 18, 2013 1:01:42 PM
Subject: Re: [Puppet Users] get a *structured* version of the puppet agent output


thanks.

who is your contact?

I am not getting the help I need.

> On Dec 18, 2013, at 10:00 AM, Jason Slagle <rais...@tacorp.net> wrote:
>
> Hi Stuart,
>

> Puppet Labs has a large professional service department that you might want to engage with these very specific requests. �I'm sure they can give you a hand with whatever you need done.

>
> Jason
>
>> On 12/18/2013 12:55 PM, Stuart Cracraft wrote:
>> What we are looking for is a Ruby program which takes the contents of
>>

>> � /var/lib/puppet/reports/*/*.yaml

>>
>> and reports in detail on everything changed or proposed for change if in
>> noop mode
>> (file permissions, modes, user creates, etc.)
>>
>> Stuart
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/cHpZlKkPmr4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/52B1E2AB.7060909%40tacorp.net.
> For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/36C6B660-92B8-4056-B82D-789C1B0AE7ED%40me.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/65902161-3831-4b46-8828-d80f2f3355f1%40googlegroups.com.