Unordered multiline log parsing with in_tail?

[lu...@di-pentima.com.ar]

Hello all! 

I'm just getting my feet wet with fluentd and I don't know how can I solve this kind of problem:

I need to parse postfix's mail.log files, and need to extract data from every single session, so I have to merge multiple log lines. The typical postfix log session is like this:

Aug  3 08:25:25 mailserver postfix/smtpd[8614]: 1FE1111B694F: client=mailserver.domain.com[11.22.33.44]

Aug  3 08:25:25 mailserver postfix/cleanup[8823]: 1FE1111B694F: message-id=<some-random-message-id>

Aug  3 08:25:25 mailserver postfix/qmgr[18500]: 1FE1111B694F: from=<sen...@email.com>, size=5486, nrcpt=1 (queue active)

Aug  3 08:25:27 mailserver postfix/smtp[8824]: 1FE1111B694F: to=<reci...@otheremail.com>, relay=mx.otheremail.com[55.66.77.88]:25, delay=2, delays=0.05/0/0.15/1.7, dsn=2.0.0, status=sent (250  OK status message)

Aug  3 08:25:27 mailserver postfix/qmgr[18500]: 1FE1111B694F: removed

The problem is that these lines are not necessarily one after the other on the log file, so I should track the session-id (1FE1111B694F in this case) to put every bit of information on the right "session bucket" before sending the record through the rest of the pipeline.

I already have a ruby script that takes the entire log from stdin and populates a temporary hash until every record has all the required data, then outputs them to stdout. Can I use this as some kind of plugin within fluentd? The script has to be "statefull", and shouldn't have to be run every input line.

Thanks in advance!

[Mr. Fiber]

Fluentd has parser plugin mechanizm used by in_tail and other plugins.

Implementing your parser with your script resolve this problem.

http://docs.fluentd.org/articles/parser-plugin-overview

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mario Fetka]

do you already have created an plugin that is working or could you publisch your script ?

[Divij Sehgal]

Hi.
I am also working on postfix and need to accomplish the same task that you mentioned.
However, I am using the fluentd_tail plugin and I need to parse Log Files just the same way as you do.
So, it would be really helpful if you could share your script for others to iterate upon

P.S, Running low on time :-(

Thanks in advance :)