Query on parser plugin limitation ?

[RAJKUMAR selvaraj]

Dear Team,

We are using fluentd in our development environment and we have some specific use-case, and like to know whether its possible using parser plugin or any other plugin.

We are reading our logs using in-tail plugin and we know if we use in-tail then  parser or format is mandatory to  use.

The following is our sample log from our application

Sample log:

timestamp="2018-05-25 13:36:38.802" application="xyz" category="MessagingEvent" messagingEventType="MessageReceived"  messageType="Demand" barcode="323200023884776648484"   

Regex used for parsing:

^timestamp=(?<timestamp>[^ ]* [^ ]*) application=(?<application>(...)*) *category=(?<Category>\S*) *messagingEventType=(?<messagingEventType>\S*) *messageType=(?<messageType>\S*) *barcode=(?<barcode>\S*)

output:

We are able to get the following key, value records.

timestamp "2018-05-25 13:36:38.802"

application "xyz"

Category "MessagingEvent"

messagingEventType "MessageReceived"

messageType "Demand"

barcode "323200023884776648484"

Query:

Mostly of our application logs format will be similar only the key  names might change, so is there a way to parse the log with existing plugin using generic regex(/((\w+)="([^"]+)"+)/) to get key/value records without writing a custom parser plugin.

Regards,

Raj

[Mr. Fiber]

>  so is there a way to parse the log with existing plugin using generic regex(/((\w+)="([^"]+)"+)/) to get key/value records without writing a custom parser plugin.

Fluentd's build-in regular expression parser requires key name for patterns,

If your field delimiar is tab, you can use ltsv parser: https://docs.fluentd.org/v1.0/articles/parser_ltsv

I'm not sure 3rd party parsers meet your requirement. 

If not, writing own parser is needed.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[RAJKUMAR T S]

Thanks for the quick reply Fiber.

Actually we tried couple of third party plugins

https://github.com/mosuka/fluent-plugin-kvp-parser   ---> its not getting installed

https://github.com/arunmj/fluent-plugin-keyvalue-parser  ---> not working after installation failing in code.

So we are writing our own plugin. But before taking this to environment, we are trying to test in windows local machine. We placed the ruby file in couple of places, but fluentd is not detecting the plugin.

Error:

2018-06-20 15:27:59 +0530 [error]: config error file="etc\\td-agent\\td-agent.conf" error_class=Fluent::ConfigError error="Unknown parser plugin 'rkv'. Run 'gem search -rd fluent-plugin' to find plugins"

C:\opt\td-agent\embedded\lib\ruby\2.4.0

C:\opt\td-agent\etc\td-agent\fluent\plugin

So could you advise us, where to place the ruby script so that it will be identified by fluentd during startup.

Attached the ruby file.

Regards,

Raj

[RAJKUMAR T S]

Dear Fiber,

Any update on how to test the custom plugin in windows local will be helpful 

Regards,

Raj

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

How about following path?

C:\etc\fluent\plugin

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.