Filtering multiple format from same log file

[Leo Graells]

Hello,

I'm currently filtering multiple log format from same log file.

The problem I have is that I have a lot of parser error when the log for example match the regex 1 but not the regex 2.

I would like if possible to parse one format log that match the regex and directly go to the server without trying to match other filter.

<source>

@type tail

tag tag.x

</source>

<filter tag.x>

@type parser

format /regex 1.../

</filter>

<filter tag.x>

@type parser

format /regex 2.../

</filter>

<filter tag.x>

@type parser

format /regex 3.../

</filter>

...

<match tag.x>

server xxx

</match>

Do I need to create multiple tag ?

Thanks for your help :]

[Mr. Fiber]

You can use multi-format-parser for it

https://github.com/repeatedly/fluent-plugin-multi-format-parser

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Leo Graells]

Hello,

I've installed multi format parser and it's working well !

Thank you,

[ayush singh rathore]

Can you please help me in configuring the multi_format parser plugin. I am having issues same as you.

[ayush singh rathore]

This is my configuration file and I am facing errors using this please help.

<source>

  @type tail

  pos_file /var/log/td-agent/magento-server3000.pos

  path /home/ayush/elastic-stack/log/debug3.log

  tag debug_logs

</source>

<filter debug_logs>

 @type parser

 key_name log

 <parse>

  @type regexp

  expression /(?<timestamp>\[([0-9-]+ [0-9:]+)]) (?<@Log_level>[a-z]+.[A-Z:]+) (?<message>.+)/

 </parse>

</filter>

<filter debug_logs>

 @type parser

 key_name log

 <parse>

  @type regexp

  expression /(?<timestamp>\[([0-9-]+ [0-9:]+)]) (?<@Log_level>[a-z]+.[A-Z:]+)(?<message_type>[^"]*)[^:]*:(?<method>[^"]*("[A-Z]+")),[^:]*:(?<URL>(http[s]?:\/\/)?([^\/\s]+\/)[^"]*)",(?<"message">"invalidateInfo":.+)/

 </parse>

</filter>

On Wednesday, October 10, 2018 at 2:11:43 PM UTC+5:30, Leo Graells wrote:

[Leo Graells]

Hello

If you already installed multiparser plugin, you should try  this conf :

<source>

  @type tail

  pos_file /var/log/td-agent/magento-server3000.pos

  path /home/ayush/elastic-stack/log/debug3.log

  tag debug_logs

</source>

<filter debug_logs>

 @type parser

 

 <parse>

 @type multi_format

 

 <pattern>

  format regexp

  key_name log

  expression /(?<timestamp>\[([0-9-]+ [0-9:]+)]) (?<@Log_level>[a-z]+.[A-Z:]+) (?<message>.+)/

 </pattern>

 

 <pattern>

  format regexp

  key_name log

  expression /(?<timestamp>\[([0-9-]+ [0-9:]+)]) (?<@Log_level>[a-z]+.[A-Z:]+)(?<message_type>[^"]*)[^:]*:(?<method>[^"]*("[A-Z]+")),[^:]*:(?<URL>(http[s]?:\/\/)?([^\/\s]+\/)[^"]*)",(?<"message">"invalidateInfo":.+)/

 </pattern>

 

 </parse>

 

</filter>