syslog, the messages field, with Elastic and Kibana

[Tim Middleton]

Hi Guys,

I have done hours of searching on this, tried numerous methods, I hope I'm just being dumb

So after a small amount of work once I got the firewall sending syslogs > fluentd > elastic > kibana working, I quickly realised that the data being presented to elastic needs to be better, specifically the syslog message field, as this has the interesting stuff

So I then spent some time learning and understanding a bit better the fluentd side of things. 

I successfully wrote a regexp to handle inbound firewall syslog messages (well 99% of them, as some broke my regexp parser)

But when I tried to fix the last 1% and then especially when I added outbound rules, the subtle differences in the syslog message format, escaped my current skills

So

1. Am I right that I need to get the data batter formatted in fluentd, before it hits elastic/kibana

2. what is the best method to break out the syslog message into individual fields, all I need is a parser, that just works thought it, based on the delimiter... so far every on I have looked at either requires the regexp, or for me to in advance specify each field

3. can I write a regexp that finds say SRC= and lets me just return that value?

Sorry for what must be a dumb set of questions, as surely this is what everybody is wanting to do ?

Regards

;-)

[Mr. Fiber]

> 2

If your logs can be parsed by multiple regex pattern, https://github.com/repeatedly/fluent-plugin-multi-format-parser plugin may help.

> 3

regexp doesn't fit for conditional like matching.

if you need to parse log by complex condition, writing own parser is better for performance and maintenace.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.