Conditional JSON extraction from a "log" field

[Fernando Valverde]

Hello everyone, I'm starting with my Fluentd experimentation and I'm using a tail type source.

I'm actually making use of a sample EFK stack that scrapes lots of logs from a Kubernetes cluster (found here: https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch)

This uses the Kubernetes metadata filter: https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter

So I'm looking for a way to get a JSON extracted from the "log" field when a certain condition is met (for example the payload contains a certain label like "kubernetes.labels.app == 'rails-app'")

More detailed explanation and discussion can be found in this StackOverflow question: https://stackoverflow.com/questions/52415159/fluentd-in-kubernetes-daemonset-selectively-parsing-different-logs

I appreciate your time and any suggestion you can give me.

Thanks,

Fernando.

[Rich Megginson]

On 9/20/18 12:01 PM, Fernando Valverde wrote:
> Hello everyone, I'm starting with my Fluentd experimentation and I'm using a /tail /type source.

>
> I'm actually making use of a sample EFK stack that scrapes lots of logs from a Kubernetes cluster (found
> here: https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch)
>
> This uses the Kubernetes metadata filter: https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter

I'm not sure, but _do not use the Kubernetes metadata filter to parse the JSON valued "log" field_.

It was a mistake to ever have that functionality in that plugin - that plugin should do Kubernetes metadata annotation of log records _only_, and nothing else (and really, its knowledge of
kubernetes log file naming and CONTAINER_NAME formats is not ideal either . . .)

https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter#configuration

> *NOTE:* As of the release 2.1.x of this plugin, it no longer supports parsing the source message into JSON and attaching it to the payload. The following configuration options are removed:
>
> * |merge_json_log|
> * |preserve_json_log|
>

Instead, do the JSON parsing separately, in another filter, or something like this which we are using in openshift origin-aggregated-logging:

https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/configs.d/filter-parse-json-field.conf

and the implementation of the parse_json_field plugin is here: https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/lib/filter_parse_json_field/lib/filter_parse_json_field.rb

We had to do that for 0.12/ruby 2.0 - but there is probably a better way to do that in fluentd 1.x

>
> So I'm looking for a way to get a JSON extracted from the "log" field when a certain condition is met (for example the payload contains a certain label like "kubernetes.labels.app ==
> 'rails-app'")
>
> More detailed explanation and discussion can be found in this StackOverflow
> question: https://stackoverflow.com/questions/52415159/fluentd-in-kubernetes-daemonset-selectively-parsing-different-logs
>
> I appreciate your time and any suggestion you can give me.
>
> Thanks,
> Fernando.

> --
> You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com <mailto:fluentd+u...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.