yamlyzer

[Stuart Cracraft]

Hi everybody,

Is there a command-line program which parses:

  puppetmasters*:/var/lib/puppet/reports/*/*.yaml

into simple a simple (though lengthy) report perhaps with

column/field/etc. selection based on a straightforward method.

Holler if you know of any. I know this was queried by me

on this list recently and all sorts of stuff was posted. If

I don't get it here, I'll need to go over to puppet-developers

and ask there.

Surely, someone has written something they're proud of and

will share it forward.

Stuart

[Dan White]

On Nov 26, 2013, at 2:27 PM, Dan White <yg...@comcast.net> wrote:

> Reference: http://docs.puppetlabs.com/puppet/3/reference/format_report.html
>
> Start with this:
> ----------------------------------------------------------------------------------------
> #!/usr/bin/ruby
>
> require 'puppet'
> if defined?(ARGV)
> filename = ARGV.first
>
> if defined?(filename)
> report = YAML.load_file(filename)
> print "Report for : "
> puts report.host
> print "Started : "
> puts report.time
> print "Log count: "
> puts report.logs.size
> print "Metrics count: "
> puts report.metrics.size
> print "Resource Status count: "
> puts report.resource_statuses.size
> print "Status: "
> puts report.status
>
> report.logs.each do |logg|
> puts "LOG ---------------------------------------------"
> if logg.file
> print "File: "
> puts logg.file
> end
>
> if logg.line
> print "Line: "
> puts logg.line
> end
>
> print "Level: "
> puts logg.level
> print "Message: "
> puts logg.message
> print "Source: "
> puts logg.source
> print "Tags: "
> puts logg.tags.join(" ")
> print "Time: "
> puts logg.time
> end
>
> report.resource_statuses.keys.each do |kk|
> if report.resource_statuses["#{kk}"].change_count > 0
> puts "RESOURCE STATUS ===================================="
> puts "#{kk} "
> report.resource_statuses["#{kk}"].events.each do |line|
> print "property: "
> puts line.property
> print "message: "
> puts line.message
> print "name: "
> puts line.name
> print "status: "
> puts line.status
> print "when: "
> puts line.time
> end
> end
> end
> puts "-------------------------------------------------"
> end
> end
> ------------------------------------------------------------------------------------------
>
> Good luck.
>
>
> “Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.”
> Bill Waterson (Calvin & Hobbes)
>
> ----- Original Message -----
> From: "Stuart Cracraft" <smcra...@me.com>
> To: puppet...@googlegroups.com
> Sent: Tuesday, November 26, 2013 2:10:53 PM
> Subject: Re: [Puppet Users] get a *structured* version of the puppet agent output
>
> Who will share a report generator for the yaml reports generated by puppet so that we do not have to reinvent wheel after wheel after wheel!!!!!!
>
>> On Nov 26, 2013, at 11:08 AM, "R.I.Pienaar" <r...@devco.net> wrote:
>>
>>
>>
>> ----- Original Message -----
>>> From: "Stuart Cracraft" <smcra...@gmail.com>
>>> To: puppet...@googlegroups.com
>>> Sent: Tuesday, November 26, 2013 7:02:42 PM
>>> Subject: Re: [Puppet Users] get a *structured* version of the puppet agent output
>>>
>>> What we want is not more complexity, but more simplicity!
>>>
>>> I could go into
>>>
>>> puppet config print reportdir
>>>
>>> and then to its
>>>
>>> /var/lib/puppet/reports
>>>
>>> then to the host directories of interest and grep out message.
>>>
>>> But that seems a very sorry state of affairs.
>>>
>>> Puppetmasters speak UP!
>>
>> you do not need to grep out anything, as I showed you there is a ruby API
>> for accessing this data.
>>
>> If you use PuppetDB it will also be able to store this information and it has
>> APIs for extracting these logs in a structured manner.
>>
>> API access is about as much as you can hope for I think. Your alternatives are
>> to write a logger plugin for Puppet that outputs JSON, I've done this and it
>> was not clear sailing.


“Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us.”
Bill Waterson (Calvin & Hobbes)

[Stuart Cracraft]

Do you personally use something
more comprehensive with all fields
enabled?

> --
> You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/GxeE-URziu8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/F5C8FE40-A9A0-4A8C-84B6-4BFDDD937ADD%40comcast.net.
> For more options, visit https://groups.google.com/groups/opt_out.

[Dan White]

No.
I whipped that sample up in about 10 minutes.

It provides a starting point for you to develop what you want.

On Dec 12, 2013, at 8:09 PM, Stuart Cracraft <smcra...@me.com> wrote:

> Do you personally use something
> more comprehensive with all fields
> enabled?

[Stuart Cracraft]

Great.

Okay - anyone additionally, how would you extend Dan's script to dump the entirety of a Puppet YAML
to a plain text report.

I know, I know. It sounds boring, painful even but the rationale is to know exactly
what the YAML says even though it is voluminous.

The reason?

Our Security Group would like to have complete visibility to logs in simple formats they can parse
and understand and socialize it to the C-suite with. Also, it can be used to analyze Puppet's proposed
operations in noop mode prior to an eventual turnup in full 7x24x365 mode for selected subsets of
our cloud.

> --
> You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/GxeE-URziu8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/0D6FF77B-2D05-43A4-B42B-C1268179D221%40comcast.net.

[John Warburton]

On 13 December 2013 15:07, Stuart Cracraft <smcra...@me.com> wrote:

Our Security Group would like to  have complete visibility to logs in simple formats they can parse
and understand and socialize it to the C-suite with. Also, it can be used to analyze Puppet's proposed
operations in noop mode prior to an eventual turnup in full 7x24x365 mode for selected subsets of
our cloud.

Are you using puppet dashboard? You can get the same information in CSV format from http://localhost:3000/nodes.csv. It has the same information as the yaml reports, but in "easier" (familiar) to manage CSV format, and always up to date when you suck it from the dashboard. Here's a line of an out of sync resource:

hostname,pending,736,34,0,702,File,/etc/sudoers,0.483092,XXXX/modules/sudoers/manifests/init.pp,22,2013-12-11 23:16 UTC,0,1,false,false

You don't get everything (like a diff on the file),and it is a good idea to omit in sync resources (!~ /,0,0,[a-z]*,false/)

And why hasn't someone written exactly what you want? Because probably no-one has the exact requirements of your organisation. So, at some stage you're going to have to do some custom work.

Or even better, engage puppet labs to do what you want, and ensure it gets rolled back into the product for the community. That's where the totting up the # resources and states and the CSV dump of the dashboard came from...

John

[Dan White]

Extending the script depends on what yaml files you plan to parse and what serialized objects are contained therein.

http://www.skorks.com/2010/04/serializing-and-deserializing-objects-with-ruby/

On Dec 12, 2013, at 11:07 PM, Stuart Cracraft <smcra...@me.com> wrote:

> Great.
>
> Okay - anyone additionally, how would you extend Dan's script to dump the entirety of a Puppet YAML
> to a plain text report.
>
> I know, I know. It sounds boring, painful even but the rationale is to know exactly
> what the YAML says even though it is voluminous.
>
> The reason?
>
> Our Security Group would like to have complete visibility to logs in simple formats they can parse
> and understand and socialize it to the C-suite with. Also, it can be used to analyze Puppet's proposed
> operations in noop mode prior to an eventual turnup in full 7x24x365 mode for selected subsets of
> our cloud.
>
> On Dec 12, 2013, at 8:03 PM, Dan White <yg...@comcast.net> wrote:
>
>> No.
>> I whipped that sample up in about 10 minutes.
>>
>> It provides a starting point for you to develop what you want.

[Stuart Cracraft]

John,

I will look into this tomorrow.

It sounds like the solution.

Thanks very much.

Stuart

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/GxeE-URziu8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAAJLFxVfb8A9TwsEng%2B2VK_eFarRa7a%3DkRMMGYu9hu%2BTvmBUGw%40mail.gmail.com.

[Stuart Cracraft]

Thank-you Dan.

Stuart

> --
> You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/GxeE-URziu8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/A5133F3F-CAAC-45D3-BD84-8D8CA5A174DF%40comcast.net.

[Stuart Cracraft]

Is the back-end piece of Puppet Dashboard which implements the CSV-capability

of the YAML interpretation available as a standalone?

[Jeff Bachtel]

Have you tried a yaml to csv converter like https://github.com/tokland/yaml2csv ? Note that it seems to require the csv module syntax from ruby 1.9.3.

Jeff

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/d333f68c-ad66-40bb-b969-9248114cd766%40googlegroups.com.

[Stuart Cracraft]

This is very interesting Jeff. 

I will play with it. 

Thanks!

You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/GxeE-URziu8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/52B0CA5A.2010505%40bericotechnologies.com.

[Stuart Cracraft]

Alas, it was all for naught.

Anyone know about the uninitialized constant Yaml2csv error?

[root@ca-sna-pm01 bin]# /usr/local/bin/yaml2csv.rb foobar.yml

./yaml2csv.rb:43: uninitialized constant Yaml2csv (NameError)

        from /usr/lib/ruby/site_ruby/1.8/rubygems/core_ext/kernel_require.rb:45:in `gem_original_require'

        from /usr/lib/ruby/site_ruby/1.8/rubygems/core_ext/kernel_require.rb:45:in `require'

        from /usr/local/bin/yaml2csv.rb:7

[root@ca-sna-pm01 bin]#

Stuart

On Tuesday, December 17, 2013 2:20:47 PM UTC-8, Stuart Cracraft wrote:

This is very interesting Jeff. 

I will play with it. 

Thanks!

On Dec 17, 2013, at 2:04 PM, Jeff Bachtel <jbachtel@bericotechnologies.com> wrote:

Have you tried a yaml to csv converter like https://github.com/tokland/yaml2csv ? Note that it seems to require the csv module syntax from ruby 1.9.3.

Jeff

On 12/16/2013 03:29 PM, Stuart Cracraft wrote:

Is the back-end piece of Puppet Dashboard which implements the CSV-capability

of the YAML interpretation available as a standalone?

On Thursday, December 12, 2013 3:20:41 PM UTC-8, Stuart Cracraft wrote:

Hi everybody,

Is there a command-line program which parses:

  puppetmasters*:/var/lib/puppet/reports/*/*.yaml

into simple a simple (though lengthy) report perhaps with

column/field/etc. selection based on a straightforward method.

Holler if you know of any. I know this was queried by me

on this list recently and all sorts of stuff was posted. If

I don't get it here, I'll need to go over to puppet-developers

and ask there.

Surely, someone has written something they're proud of and

will share it forward.

Stuart

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/d333f68c-ad66-40bb-b969-9248114cd766%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/GxeE-URziu8/unsubscribe.

To unsubscribe from this group and all its topics, send an email to puppet-users+unsubscribe@googlegroups.com.

[Stuart Cracraft]

Not just placing all trust in Puppet. That's the point of this tiny program now named

  yaml2csv.rb

If you revise/improve, you must post back your improved version.

And remember, by ensuring you noop your Puppet cloud and review it first before actions,

you will keep the human in the loop before you un-noop and puppet agent -t from a far,

rerun on all the YAML's, and see what you did in an actual human-readable format.

Imagine that?

The fear, of course, is that some kind of whacko code gets into Puppet (I know, open

source and all that), but, frankly, I can't take that chance so verifying and proofing

mass-cloud changers is what I'm into.

Happy Hacking,

Stuart

#!/usr/bin/ruby

require 'puppet'

 

output = ""

 

# Are there args?

if defined?(ARGV)

 

   # First argument needs to be the yaml file

   filename = ARGV.first

 

   # Has to be defined

   if defined?(filename)

 

     # Make a real version we can work on that is not frozen

     fileName = filename.dup

 

     # Get the report object

     report = YAML.load_file(filename)

 

     # Get the base name of the file to the far right

     fileName = File.basename(filename)

 

     # Formulate the timestamp

     dateStamp = fileName.split /\./

 

     # Loop through the logs and print appropriately

     report.logs.each do |logg|

      if logg.line or logg.line == 0 then

        output = sprintf("%s, %s, %s, %s, (line %s), %s, %s, %s",

                 report.host,

                 dateStamp[0],

                 logg.time,

                 logg.file,

                 logg.line,

                 logg.source,

                 logg.level,

                 logg.message

                  )

        searchResult = output.grep(/changed|noop/)

        if searchResult.any? then

          printf("%s\n",output)

        end

      else

        output = sprintf("%s, %s, %s, %s, (line %s), %s, %s, %s",

                 report.host,

                 dateStamp[0],

                 logg.time,

                 logg.file,

                 "none",

                 logg.source,

                 logg.level,

                 logg.message

                  )

        searchResult = output.grep(/changed|noop/)

        if searchResult.any? then

          printf("%s\n",output)

        end

      end

    end

  end

end

# Bless Ygor. Merry Christmas everybody!!!!

On Thursday, December 12, 2013 3:20:41 PM UTC-8, Stuart Cracraft wrote: