How can i use your grok fluent plugin correctly

[Daniel Mack]

Hello,

i use fluentd Elasticsearch and Kibana as Logging like Splunk.

Our Routers send his logs to our rsyslog deamon over Port 514. And then Transport that to the Fluentd. With the Fluentd elasticsearch plugin came the logs to the Elasticsearch and our Kibana  pick up this from the Elasticsearch over port 9200.

How can i parse the cisco router logs? I had installed the fluent-plugin-grok-parser plugin and have found this Patterns: http://grokdebug.herokuapp.com/patterns  with the correct Firewall pattern.

But where should i say that the router logs should parsed?

Can anybody help me?

Thanks

Best Regards

Daniel

[Kiyoto Tamura]

Hey Daniel-

Currently, my Grok parser isn't designed to handle multiple Grok patterns (as in if CISCOFW106001 does not match, try CISCOFW106006_106007_106010 and so forth).

Let me update fluent-plugin-grok-parser to support this "or"-like matching. at that point, i think CISCO logs can be parsed cleanly.

Kiyoto

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Check out Fluentd, the open source data collector to unify log management.

[Dany]

Hello Kiyoto,

thank you for your answer. When do you think came the new release which support multiple Grok Patterns?

After that, can you tell me then where i can paste the firewall patterns and which file should i change with which code to use this pattern and your plugin?

Dany

[Kiyoto Tamura]

Dany-

I just prototyped it on a local branch and need to write tests for it before releasing it. Do you have a particular deadline you need to meet with this project?

--

You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dany]

Hello Kiyoto,

no, i haven't a real deadline, but i want to work on this project as soon as may be.

Dany

[Mr. Fiber]

https://github.com/repeatedly/fluent-plugin-multi-format-parser

Using grok parser with multi-format-parser is alternative approach.

But supporting by grok parser is more efficient.

--

[Dany]

 i will wait  until Kiyoto has finished his new grok Parser release which support multiple Grok patterns.

Dany

[Kiyoto Tamura]

This is implemented now in the master branch of github.com/kiyoto/fluent-plugin-grok-parser.

That said, I am starting to think perhaps it is better to implement Grok as filter and not parser. For example, for the use case described in http://everythingshouldbevirtual.com/cisco-asa-logstash-parsing, "grok-patterns as filters" sounds like a better approach (since filters are applied one by one, it requires no support to apply grok filters successively. In such cases, the input will use NoneParser and delegate the task of grokking to grok filters.

@repeatedly, what do you think?

Kiyoto.

On Thu, Jan 8, 2015 at 5:49 AM, Dany <daniel...@googlemail.com> wrote:

 i will wait  until Kiyoto has finished his new grok Parser release which support multiple Grok patterns.

Dany

--

You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dany]

Hello Kiyoto,

i don't understand your last written answer really. I should use better the Grok as filter and not the parser, but if i use the howto which you have posted in the link i don't need the grok filter? 

Your link have i found too before you have post it. But i have seen that the implementation is with logstash and not with fluentd which i use.

Dany 

[Mr. Fiber]

No need filter version because we have a fluent-plugin-parser.

If fluent-plugin-parser provide a filter version, all parser plugins become a filter. 

Masahiro

[Kiyoto Tamura]

Masa-

That's a better approach. Didn't think about it :-)

[Dany]

ok, and what do you think Kiyoto, when can i use your new grok plugin with i can parse cisco logs?

Dany
 

[Viet Nguyen Chan]

Hi kiyoto,

How about this new release which support multiple Grok Patterns? Is it already release ?

Best Regards,

VietNC

[Kiyoto Tamura]

Hi Viet,

The better person to answer this question is @okkez https://github.com/okkez He has largely taken over the development and maintenance of the project.

The best path forward is to create an issue on GitHub https://github.com/kiyoto/fluent-plugin-grok-parser

Thank you for your interest in Fluentd and the grok parser!

Kiyoto

[Mr. Fiber]

Already supported > https://github.com/kiyoto/fluent-plugin-grok-parser#multiline-support

[Viet Nguyen Chan]

Thanks to all for your reply :)

Best Regards,

VietNC

[Kiyoto Tamura]

Lol. RTFM, including the author!

--