obtaining stdout information from client configuration executions

[Tim Lank]

puppet-users:

I'm trying to find a way to obtain the stdout (and stderr) output from individual puppet client configuration runs.   I'd like a central storage of this information if possible.

In short, I'd like to know what the client exactly did with the configuration that it was sent.  Ideally I'd like for this to be in one location so that I can write scripts to report on the results.

Thanks in advance for your assistance.

[Darren Chamberlain]

We're not using a puppetmaster, so we're collecting info on runs via
syslog and puppet -v:

puppet -v /path/to/manifest.pp | logger -t puppet

And then we're using a splunk-alike to run ad-hon reports on the
logs.

* Tim Lank <timlank at timlank.com> [2010/09/24 06:52]:

> I'm trying to find a way to obtain the stdout (and stderr) output
> from individual puppet client configuration runs. I'd like a
> central storage of this information if possible.

--
Patrick's Law:
The permanence of a temporary solution rises in proportion with the
urgency of its initial implementation.

[Patrick]

Why not use puppet's reports or a remote syslog?

[Tim Lank]

Thanks.  Can you elaborate on the puppet's reports?  All I see are here: http://docs.puppetlabs.com/references/latest/report.html

Is there one that reports on the stdout&stderr of the tasks the client performs? (and not in YAML)

The remote syslog would take some coding to cull together in a coherent and readable manner for each individual client.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Tim Lank]

A sample report that is sent back in yaml format to the puppetmaster in $vardir/puppet/reports/<puppetdclient.example.org> looks like this:

--- !ruby/object:Puppet::Transaction::Report
  host: puppetdclient.example.org
  logs:
    - !ruby/object:Puppet::Util::Log
      file: /etc/puppet/manifests/templates.pp
      level: !ruby/sym notice
      line: 37
      message: executed successfully
      source: "//Node[default]/Exec[/bin/ps aux | /bin/grep puppet | /bin/grep -v grep]/returns"
      tags:
        - exec
        - node
        - default
        - class
        - main
        - returns
        - notice
      time: 2010-09-24 13:53:17.465847 -04:00
      version: 1285350797

I see that the source command string is there, but the stdout (and err) of the ps command does not appear in the yaml.  Is there a way to turn this on so it ends up in here?

[Patrick]

On Sep 24, 2010, at 10:08 AM, Tim Lank wrote:

Thanks.  Can you elaborate on the puppet's reports?  All I see are here: http://docs.puppetlabs.com/references/latest/report.html

Is there one that reports on the stdout&stderr of the tasks the client performs? (and not in YAML)

I'm almost sure the answer it no.

The remote syslog would take some coding to cull together in a coherent and readable manner for each individual client.

Your solution feels like reinventing the wheel to me, but I'll admit that I've never used remote syslog before.  It might be easiest to put something like this in a script called by cron (after redirecting the output):

puppetd --verbose --onetime --no-deamonize

[Nigel Kersten]

On Fri, Sep 24, 2010 at 10:08 AM, Tim Lank <tim...@timlank.com> wrote:
> Thanks.  Can you elaborate on the puppet's reports?  All I see are here:
> http://docs.puppetlabs.com/references/latest/report.html
>
> Is there one that reports on the stdout&stderr of the tasks the client
> performs? (and not in YAML)

I believe we have a patch that should show the stderr output when an
exec task fails, but I'm curious as to why you want the stdout?

--
nigel

[Tim Lank]

It would be great to have it configurable, so that it could be off by default for those that don't want it. 

I may be in the minority here, but having a surety of exactly what was executed and a knowledge of the output would be a good thing to me.  Seems that the ability to turn this on would be a very desirable thing.

Please provide the patch# you are referring to for the on_failure stderr.

[Nigel Kersten]

On Fri, Sep 24, 2010 at 11:29 AM, Tim Lank <tim...@timlank.com> wrote:
> It would be great to have it configurable, so that it could be off by
> default for those that don't want it.
>
> I may be in the minority here, but having a surety of exactly what was
> executed and a knowledge of the output would be a good thing to me.  Seems
> that the ability to turn this on would be a very desirable thing.

I understand wanting to know why an exec failed, but what are some
actual use cases where you care about the stdout of an exec resource
in Puppet?

>
> Please provide the patch# you are referring to for the on_failure stderr.

http://projects.puppetlabs.com/issues/2359 covers it.

[Tim Lank]

debugging your exec's for one.  The exec may return something that is functionally correct, but not what was intended.

Granted, I am brand new to puppet within the last month or so.  Conceptually, I'm sure that I don't have a very good handle on the "puppet way" of doing things yet so I may not be able to provide a good solid set of use cases for you.

[Tim Lank]

It looks like adding the line "logoutput => true" in the exec definition ends up sending this stdout to the centralized yaml log in the "message:" field along a separate log with the standard "executed successfully" summary.

--- !ruby/object:Puppet::Transaction::Report
  host: puppetdclient.example.org
  logs:
    - !ruby/object:Puppet::Util::Log

      file: &id001 /etc/puppet/manifests/templates.pp
      level: !ruby/sym notice
      line: 38
      message: root     27164  0.0  2.0 130292 41976 ?        Ssl  13:53   0:02 /usr/bin/ruby /usr/sbin/puppetd

      source: "//Node[default]/Exec[/bin/ps aux | /bin/grep puppet | /bin/grep -v grep]/returns"
      tags:
        - exec
        - node
        - default
        - class
        - main
        - returns
        - notice

      time: 2010-09-24 15:23:21.562537 -04:00
      version: 1285356077

    - !ruby/object:Puppet::Util::Log

      file: *id001
      level: !ruby/sym notice
      line: 38

      message: executed successfully
      source: "//Node[default]/Exec[/bin/ps aux | /bin/grep puppet | /bin/grep -v grep]/returns"
      tags:
        - exec
        - node
        - default
        - class
        - main
        - returns
        - notice

      time: 2010-09-24 15:23:21.563504 -04:00
      version: 1285356077

[Tim Lank]

puppet-users:

in case this will help someone else besides me....

I've developed a simple mini-script system that can be setup on a
puppetmaster to send the stdout of the last reports for each client to
a set of email addresses. I find it convenient to get these results
in my inbox so that I can view them at a glance, review at leisure,
forward on, cut & paste elsewhere, etc...

recommendations, suggestions, comments welcome. Please improve this
and let me know what you did. If I've unknowingly re-invented
something that exists, please let me know that also.

1. /usr/local/bin/mailpuprpt (takes as a parameter a comma
separated list of email addresses & calls /usr/local/bin/getpupoutput
- performs mailing of the final summary report)
2. /usr/local/bin/getpupoutput (processes the last report file for
each client, pulls the line-oriented "message:" fields out except the
confirmation, replaces ecaped tabs with actual tabs, and calls
/usr/local/bin/reconstructstdout.pl)
3. /usr/local/bin/reconstructstdout.pl (returns the cleaned stdout -
not including puppet-added quotation marks, and replacing
puppet-escaped quotations with actual quotation marks)

1. /usr/local/bin/mailpuprpt

#!/bin/bash

die () {
echo -e "\nusage: mailpuprpt [comma separated email list]"
echo "example: mailpuprpt some...@somewhere.com,some...@somewhere.com"
exit 1
}

[ "$#" -eq 1 ] || die

/usr/local/bin/getpupoutput | mailx -s "puppet report `date`" $1

2. /usr/local/bin/getpupoutput

#!/bin/bash

REPORTDIR=/var/lib/puppet/reports
RECONSTRUCTSCRIPT=/usr/local/bin/reconstructstdout.pl
RAWPUPOUTPUT=/tmp/rawpupoutput.tmp
RECONSTRUCTEDSTDOUT=/tmp/reconstructedstdout.tmp

for i in `ls $REPORTDIR`
do
LASTREPORT=`ls -tr $REPORTDIR/$i | tail -1`
RUNTIME=`grep "time: " $REPORTDIR/$i/$LASTREPORT | tail -1 | cut -c8-`
echo -e "______________________________"
echo -e "$i $LASTREPORT $RUNTIME\n"
grep "message: " $REPORTDIR/$i/$LASTREPORT | grep -v "executed
successfully" | cut -c16- > $RAWPUPOUTPUT
$RECONSTRUCTSCRIPT
/bin/sed 's:\\t:\t:g' $RECONSTRUCTEDSTDOUT
done

3. /usr/local/bin/reconstructstdout.pl

#!/usr/bin/perl

open(RAW, "/tmp/rawpupoutput.tmp");
open(RECONSTRUCTED, ">/tmp/reconstructedstdout.tmp");

while (<RAW>) {
$line = $_;
@column = split //, $line;
$lastcol = ($#column - 1);

if (("$column[0]" eq "\"") && ("$column[$lastcol]" eq "\"")) {
my $colcounter = 1;
while ($colcounter < $lastcol) {
if (("$column[$colcounter]" eq "\\") &&
("$column[$colcounter+1]" eq "\"")) {
$colcounter++;
}
else {
print RECONSTRUCTED "$column[$colcounter]";
$colcounter++;
}
}
print RECONSTRUCTED "\n";
}
else {
print RECONSTRUCTED "$line";
}
}
close RAW;
close RECONSTRUCTED;

[Daniel Pittman]

Tim Lank <tim...@timlank.com> writes:

> I've developed a simple mini-script system that can be setup on a
> puppetmaster to send the stdout of the last reports for each client to a set
> of email addresses. I find it convenient to get these results in my inbox
> so that I can view them at a glance, review at leisure, forward on, cut &
> paste elsewhere, etc...

Isn't this what the tagmail system does?

http://docs.puppetlabs.com/references/stable/report.html

> recommendations, suggestions, comments welcome.

The main comment is a general sysadmin one: make sure you only generate noise
when something goes wrong, not when it goes right. Otherwise puppet will
rapidly overwhelm you with reports and you will never notice the problems. :)

Regards,
Daniel
--
✣ Daniel Pittman ✉ dan...@rimspace.net ☎ +61 401 155 707
♽ made with 100 percent post-consumer electrons

[Peter Meier]

>> recommendations, suggestions, comments welcome.
>
> The main comment is a general sysadmin one: make sure you only generate noise
> when something goes wrong, not when it goes right. Otherwise puppet will
> rapidly overwhelm you with reports and you will never notice the problems. :)

a huge +1!

~pete

[Tim Lank]

Thanks Daniel. I appreciate your commentary.

The main difference it would appear from this and tagmail is that
tagmail constantly sends numerous emails. This script is run
on-demand or out of cron and harvests the stdout of all the last run
reports into one summary email/report. One email when you want it and
how you want it vs. hundreds or thousands every hour. I agree, this
would be way too much noise and why I didn't like the idea of tagmail.
I only want a summary once in a while, not a constant flow of them.

[Ohad Levy]

On Tue, Oct 5, 2010 at 12:23 PM, Tim Lank <tim...@timlank.com> wrote:

Thanks Daniel.  I appreciate your commentary.

The main difference it would appear from this and tagmail is that
tagmail constantly sends numerous emails.  This script is run
on-demand or out of cron and harvests the stdout of all the last run
reports into one summary email/report.  One email when you want it and
how you want it vs. hundreds or thousands every hour.  I agree, this
would be way too much noise and why I didn't like the idea of tagmail.
 I only want a summary once in a while, not a constant flow of them.

just my two cents - this is why Foreman daily (or what ever interval) emails exists - they provide a summary of all puppet activities (with links to see more of course).

another benefit is to see which hosts are not reporting...

Ohad