how to forward unmatched input in filter plugin parser

[starship]

Hi,

<filter docker.**>

  @type parser

  format json

  key_name log

</filter>

I have this configuration in td-agent.conf. Some of my containers log in json but some do not. 

For containers that log using json format, this configuration works. 

But for container that do not use json format, it drops the log messages getting forwarded to Elasticsearch.

How do I configure parser such that if there is a match do json flatten but if there is no match just forgive

and forward it to the next element in the chain?

thanks

Starship

[starship]

I see this directive in documents

emit_invalid_record_to_error

Emit invalid record to @ERROR label. Default is false. Invalid cases are

key not exist

format is not matched

unexpected error

You can rescue unexpected format logs in @ERROR label.

Could someone explain how to use @ERROR label ?

thanks

Starship

[Mr. Fiber]

Same as normal label. Just name is `@ERROR`.

http://docs.fluentd.org/v0.12/articles/config-file#5-group-filter-and-output-the-ldquolabelrdquo-directive

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[starship]

Hi Masahiro,

In parser filter plugin, will it be possible to introduce a new feature to label or retag events

that did match the format?

I have several containers running using docker. Our home grown containers log in

JSON format but other 3 rd party containers like nginx, mongodb etc log in simple 

text ( non-json ). 

We are using docker logging driver and orchestration is done by kubernetes. 

It is difficult to separate log streams of our containers ( in JSON ) vs 3rd party 

containers ( non-JSON ) and use different routing logic. 

Actually all the logs are going to AWS ES. We don't even want different 

routing logic for JSON vs non-JSON. 

If parser filter plugin has a simple flag (default value as false ) that says

"allow events that fail format (i.e non JSON) to use the same routing as events that 

follow the format (i.e JSON formatted events", then it will be useful. 

Now I am actually using @ERROR label to achieve this requirement.

regards,

Starship

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

How about this parser plugin?

https://github.com/repeatedly/fluent-plugin-multi-format-parser

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[starship]

Thanks a lot Masahiro for the recommendation. I was not aware of this plugin.