How can I discard unmatched logs?

[Waldemar Neto]

Hey folks, I'm using fluentd with k8s and using filter_parser to parse json. The problem is that the log file is not only json.
How can I remove those that don't unmatched? I saw this property suppress_parse_error_log the documentation says: "This parameter is useful for parsing mixed logs and you want to ignore non target lines."
That's exactly what I want, how can I use it?

My configurations are:

<source>

  type tail

  path /var/log/containers/node-kube-*.log

  pos_file /var/log/fluentd-containers.log.pos

  time_format %Y-%m-%dT%H:%M:%S.%NZ

  tag node-kube-*

  format json

  read_from_head true

</source>

<filter node-kube-**>

  @type parser

  key_name log

  format json

  reserve_data true

  emit_invalid_record_to_error true

  time_parse false

  ignore_key_not_exist true

</filter>

Thanks!

[Mr. Fiber]

Hi, 

> How can I remove those that don't unmatched? 

parser filter discards unmatched lines.

> suppress_parser_error_log

parser filter puts unmatches lines with error information to fluentd log.

This option suppress such logs for like your case.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Waldemar Neto]

Thank you Masahiro for the reply

So I'm using version 14.14 and using this option emit_invalid_record_to_error but unparsed logs (that are not json) are still sent to elasticsearch.
I would like to send only filtered data to elasticsearch, how can I do that?

Cheers

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

>  unparsed logs (that are not json) are still sent to elasticsearch.

It seems the another problem.

I tested following configuration and it works.

<source>
  @type forward
</source>

<filter test.**>
  @type parser
  key_name log

  <parse>
    @type json
  </parse>
  emit_invalid_record_to_error
</filter>

<filter test.**>
  @type record_transformer
  <record>
    category normal
  </record>
</filter>

<match test.**>
  @type stdout
</match>

<label @ERROR>
  <filter **>
    @type record_transformer
    <record>
      category error
    </record>
  </filter>
  <match **>
    @type stdout
  </match>
</label>

- fluent-cat

$ echo '{"log":"foo"}' |  ~/dev/fluentd/fluentd/bin/fluent-cat test.foo
$ echo '{"log":"{\"k\":\"v\"}"}' |  ~/dev/fluentd/fluentd/bin/fluent-cat test.foo

- Output

2017-06-23 04:34:36.261342000 +0900 test.foo: {"log":"foo","category":"error"}
2017-06-23 04:34:55.953569000 +0900 test.foo: {"k":"v","category":"normal"}

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.