rewrite_tag_filter ; forest ; kubernetes_metadata ; event routing based on custom kubernetes label

[Bryan Stenson]

Hi all -

I'm trying to route messages to different fluentd instances based on kubernetes labels.  A basic architecture diagram is below:

kubernetes pods -> docker logs -> fluentd docker parser -> then route based on "logparsingservice" kubernetes label.

If a "logparsingservice" label exists on a pod, logs will be shipped to the specified URL (kubernetes.labels.logparsingservice="myfancylogparser" would get routed to "myfancylogparser") ...if one doesn't exist, the logs will be shipped to a "default" log collector.

Here's my fluentd.conf: https://gist.github.com/stensonb/c1f6c96a09e83d2856c3c55d97e5654a

I'm seeing the kubernetes metadata correctly attached to logs from kubernetes pods, but all logs, even those from pods with the "logparsingservice" label are going to the "default" log collector.

I have "proven" the rewrite_tag_filter works correctly when running locally (with 3 fluentd instances) without kubernetes...when I pass data with a json structure containing "kubernetes.labels.logparsingservice" to my local test, the routing works as expected...

So, perhaps the kube metadata added by the kubernetes_metadata plugin isn't being passed to the <match> stanza here: https://gist.github.com/stensonb/c1f6c96a09e83d2856c3c55d97e5654a#file-gistfile1-txt-L26

Any ideas on where to look here, or what to look for?

Thanks.

Bryan

[Bryan Stenson]

Here's a sample log structure:

{

  "log":"some info here",

    "kubernetes": {

      "labels": {

        "logparsingservice":"foobar",

        "app":"monkey"

      }

    }
}

how do (or can?) I use rewrite_tag_filter to reference kubernetes["labels"]["logparsingservice"] as the attribute I want to use when looking for the rewrite regex?  dot notation ("kubernetes.labels.logparsingservice") doesn't appear to work.

[Bryan Stenson]

Looks like I'm running into this: https://github.com/fluent/fluent-plugin-rewrite-tag-filter/issues/13

Anybody have a workaround?