capturing logs from ansible/ansible-playbook runs

[seth vidal]

I wanted something like sfromm's ansible-report system but for a
variety of reasons I was concerned about the single sqlite db.

So I started hacking up a solution for lots of simultaneous runs of
ansible and capturing logs of the whole thing.

this is a callback plugin:

http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/callback_plugins/logdetail.py

and this is a viewer for those logs:
http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/scripts/logview

If you put the callback_plugin in a path for callback plugins specified
in your /etc/ansible/ansible.cfg then it will also capture data on
tasks run via ansible - not just ansible-playbook.

This is an example of the output from running

logview -d yesterday -p mirrorlist

mirrorlist
20.50.54 host1 Jul 03 2013 20:54:04 54 CHANGED /etc/nagios/nrpe.cfg
20.50.54 host1 Jul 03 2013 20:55:29 75 CHANGED restart nrpe
20.50.54 host2 Jul 03 2013 20:54:06 54 CHANGED /etc/nagios/nrpe.cfg
20.50.54 host2 Jul 03 2013 20:55:31 75 CHANGED restart nrpe
20.50.54 host3 Jul 03 2013 20:54:05 54 CHANGED /etc/nagios/nrpe.cfg
20.50.54 host3 Jul 03 2013 20:55:29 75 CHANGED restart nrpe


I thought I'd post it here in case it is helpful to anyone.
-sv

[Jan-Piet Mens]

Seth,

> So I started hacking up a solution for lots of simultaneous runs of
> ansible and capturing logs of the whole thing.

that's lovely! I'd like to propose the following patch, which avoids a
traceback when the process doesn't have a controlling tty (happens here,
dunno why).

Regards,

-JP

[seth vidal]

You are absolutely correct. Thank you for that. I had not tested it
outside of a controlling TTY.

-sv

[seth vidal]

On Thu, 4 Jul 2013 10:35:16 +0200
Jan-Piet Mens <jpm...@gmail.com> wrote:

I found one bug - if I sudo -i to root - geteuid() doesn't return who I
_was_ - only 'root' which is unfortunate - I'll have to see if I can
get a best possible situation which uses getlogin() if possible

What's the exception if you don't have a controlling tty with
os.getlogin() - I can just catch it and use geteuid()

-sv

[Jan-Piet Mens]

> I found one bug - if I sudo -i to root - geteuid() doesn't return who I
> _was_ - only 'root' which is unfortunate - I'll have to see if I can
> get a best possible situation which uses getlogin() if possible

maybe
os.getenv('HOME', os.getenv('LOGNAME', 'unknown'))


(Sorry, I'm away from $cust at the moment: can't check on the exception
I got.)

-JP

[Michael DeHaan]

BTW, if folks are interested in some really nice views into logging, I'd recommend signing up for the AWX webinar here:

http://ansibleworks.enterthemeeting.com/m/DBVI6YDJ

This is coming out August 8th, hopefully about a week after the AWX release.   You'll be able to try it out free for a small number of nodes.

http://www.ansibleworks.com/

It's very "lickable".

[Michael DeHaan]

Slight correction, webinar is August 8th -- logging is out as part of the release :)

--

Michael DeHaan <mic...@ansibleworks.com>
CTO, AnsibleWorks, Inc.
http://www.ansibleworks.com/

[Michael DeHaan]

Am I terrible with words today or what :)

"out" means "is part of the release" :)

[seth vidal]

I ended up doing

def getlogin():
try:
user = os.getlogin()
except OSError, e:
user = pwd.getpwuid(os.geteuid())[0]
return user


which seems to step around it just fine

-sv

[Stephen Fromm]

On Wed, Jul 3, 2013 at 11:22 PM, seth vidal <skv...@fedoraproject.org> wrote:

I wanted something like sfromm's ansible-report system but for a
variety of reasons I was concerned about the single sqlite db.

One comment on the above.  I plan to devote some time in the coming weeks to work on the concurrency problem when using a sqlite database.  One can still choose to use another database that doesn't have the limitations of sqlite -- whatever sqlalchemy supports should work with ansible-report.  Of course, that does introduce dependencies that sqlite doesn't have.  :-)

 

So I started hacking up a solution for lots of simultaneous runs of
ansible and capturing logs of the whole thing.

this is a callback plugin:

http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/callback_plugins/logdetail.py

 and this is a viewer for those logs:
http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/scripts/logview

If you put the callback_plugin in a path for callback plugins specified
in your /etc/ansible/ansible.cfg then it will also capture data on
tasks run via ansible - not just ansible-playbook.

This is an example of the output from running

logview -d yesterday -p mirrorlist

mirrorlist
20.50.54 host1 Jul 03 2013 20:54:04 54 CHANGED /etc/nagios/nrpe.cfg
20.50.54 host1 Jul 03 2013 20:55:29 75 CHANGED restart nrpe
20.50.54 host2 Jul 03 2013 20:54:06 54 CHANGED /etc/nagios/nrpe.cfg
20.50.54 host2 Jul 03 2013 20:55:31 75 CHANGED restart nrpe
20.50.54 host3 Jul 03 2013 20:54:05 54 CHANGED /etc/nagios/nrpe.cfg
20.50.54 host3 Jul 03 2013 20:55:29 75 CHANGED restart nrpe


I thought I'd post it here in case it is helpful to anyone.

Looking at logdetail.py, how does it handle the case where you have a playbook with the same name, but in different paths?

I like the minimal dependencies of this approach and that it is easy to prune with something like logrotate.

--
Stephen Fromm

[Jan-Piet Mens]

Seth,

> What's the exception if you don't have a controlling tty with
> os.getlogin() - I can just catch it and use geteuid()

The exception is
OSError: [Errno 2] No such file or directory

I remember now, that the message stumped me. ;) But you've fixed it all,
so ignore.

-JP

[seth vidal]

On Thu, 4 Jul 2013 13:11:02 -0700
Stephen Fromm <sfr...@gmail.com> wrote:

> One comment on the above. I plan to devote some time in the coming
> weeks to work on the concurrency problem when using a sqlite
> database. One can still choose to use another database that doesn't
> have the limitations of sqlite -- whatever sqlalchemy supports should
> work with ansible-report. Of course, that does introduce
> dependencies that sqlite doesn't have. :-)

cool!

> > logview -d yesterday -p mirrorlist
> >
> > mirrorlist
> > 20.50.54 host1 Jul 03 2013 20:54:04 54 CHANGED /etc/nagios/nrpe.cfg
> > 20.50.54 host1 Jul 03 2013 20:55:29 75 CHANGED restart nrpe
> > 20.50.54 host2 Jul 03 2013 20:54:06 54 CHANGED /etc/nagios/nrpe.cfg
> > 20.50.54 host2 Jul 03 2013 20:55:31 75 CHANGED restart nrpe
> > 20.50.54 host3 Jul 03 2013 20:54:05 54 CHANGED /etc/nagios/nrpe.cfg
> > 20.50.54 host3 Jul 03 2013 20:55:29 75 CHANGED restart nrpe
> >
> >
> > I thought I'd post it here in case it is helpful to anyone.
> >
>
> Looking at logdetail.py, how does it handle the case where you have a
> playbook with the same name, but in different paths?

It doesn't. I realized that when I wrote it and just kinda said "meh".

If anyone would like to fix it I'm happy to accept it - but in our
usecase the likelihood of having such a situation is extremely low.

> I like the minimal dependencies of this approach and that it is easy
> to prune with something like logrotate.

I like that your sqlite db is a million times easier to search :)

-sv

[Brian Coca]

idk, i find grep and awk much easier to use than sql when looking at events.

I like that your sqlite db is a million times easier to search :)

--
Brian Coca
Stultorum infinitus est numerus
0110000101110010011001010110111000100111011101000010000001111001011011110111010100100000011100110110110101100001011100100111010000100001

Pedo mellon a minno