Parsing Nagios Log files with Fluentd

[Rhys Campbell]

Anyone have any experience of parsing Nagios log files with fluentd? They're in the following format...

[1404239939] SERVICE ALERT: servername;PING;OK;SOFT;2;PING OK - Packet loss = 16%, RTA = 72.18 ms

I can't find a specification for the Nagios log but I think it may be best to interpret it something like...

[unix_timestamp] <LOG ENTRY TYPE>: <hostname>;<CHECK>;<STATUS>;<CHECK>;<STATUS>;<CHECK>;<STATUS>; - <MESSAGE>

There are a few variations in the <LOG ENTRY TYPE> which then affects the subsequent format of the log (i.e. it varies a lot). Has anyone had a stab at this previously

Rhys

[Kiyoto Tamura]

Hy Rhys,

A good timing. I just released fluent-plugin-grok-parser, which lets you extend Fluentd's parser beyond hand-coded regexp.

In particular, this is all you need to do:

1. Install fluent-plugin-grok-parser

If you are using vanilla fluentd, this is "gem install fluent-plugin-grok-parser". If you are using td-agent, it's /usr/lib/fluent/ruby/bin/fluent-gem fluent-plugin-grok-parser".

2. Configure

Use the following configuration with in_tail:

<source>

  type tail

  path /path/to/nagios/log

  format grok

  grok_pattern %{NAGIOSLOGLINE}

  time_format %s

  tag nagios

  # other tail options

</source>

For example, it parses your example Nagios Line like this:

2014-07-01 18:38:59 +0000 foo: {"nagios_message":"PING OK - Packet loss = 16%, RTA = 72.18 ms","nagios_type":"SERVICE ALERT","nagios_hostname":"servername","nagios_service":"PING","nagios_state":"OK","nagios_statelevel":"SOFT","nagios_attempt":"2"}

Do remember though: I wrote this parser plugin last Sunday, and there's a lot of room for improvements.

Kiyoto

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Check out Fluentd, the open source data collector for high-volume data streams

[Rhys Campbell]

Hi Kiyoto,

Apologies for the late reply. I've been abroad. 

Thanks for that. I also found this...

http://grokdebug.herokuapp.com/patterns#

So I'm going to see if I can get it working with your plugin. 

Cheers,

Rhys

[Rhys Campbell]

Got this working well. Details here http://www.youdidwhatwithtsql.com/parsing-nagios-log-files-fluentd/1926/

Is it possible to use multiple match patterns in a single source or do you need to setup a source for each one?

Rhys

On Tuesday, July 1, 2014 10:30:00 PM UTC+1, kiyoto wrote:

[ramy...@avekshaa.com]

i followed below link and configured as mentioned in that link to parse nagios log file using td-agent grok parser but getting an error saying that "config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="no grok patterns. Check configuration, e.g. typo, configuration syntax, etc"  

 below is the configuration of my td-agent 

/etc/tdagent/td-agent.conf

<source>

  @type tail

  format grok

  grok_pattern %{NAGIOS_SERVICE_ALERT}

  custom_pattern_path /usr/bin/scripts/nagios_grok_patterns.txt 

  path /usr/local/nagios/var/nagios.log

  pos_file /var/log/td-agent/nagios_log.pos

  tag nagios

</source>

<match nagios>

@type stdout

tag nagios.source

source nagios

</match>

custom pattern is added to this file - /usr/bin/nagios_grok_patterns.txt 

NAGIOS_SERVICE_ALERT %{NAGIOS_TYPE_SERVICE_ALERT:nagios_type}: %{DATA:nagios_hostname};%{DATA:nagios_service};%{DATA:nagios_state};%{DATA:nagios_statelevel};%{NUMBER:nagios_attempt};%{GREEDYDATA:nagios_message}

Kindly help me

[Awesomely]

Hi Ramya,

inside your source just change it to this..

<source>

  @type tail

  <parse>

    @type grok

    grok_pattern %{NAGIOS_SERVICE_ALERT}

    custom_pattern_path /usr/bin/scripts/nagios_grok_patterns.txt 

  </parse>

  path /usr/local/nagios/var/nagios.log

  pos_file /var/log/td-agent/nagios_log.pos

  tag nagios

</source>