Filter logs in Kubernetes Only From Specific Namespaces

[Jacob Levertov]

Hi,

I set up EFK Cluster in my Kubernetes. The fluent version is 0.12. And I have a question regarding configuring log filters/matches to parse the logs.

I wish to parse logs that I collect from my services in Kuebernetes before fluentd sends them to the elasticsearch. I want to parse only logs that come from my services that are in a specific Kubernetes namespaces and not others. I found that I can apply filter only on tags and that there are Wild Cards that I can use, but, after some research I found that the tag that my logs get looks like this:

"kubernetes.var.log.containers.<name_of_pod>_<name_of_namespace>_... .log"
I wish to write a filter that will choose tags of specific namespace for example:
<fliter my_namespace>

  ...

</filter>

Is this possible?

[anis ben hassine]

Hi Jacob,

Did you tried for example to exclude the undesired NS like this:


fluentd-inputs.conf: |
           ....... REMOVED ........

</filter>

      <match kubernetes.var.log.containers.**kube-system**.log>

        @type null

      </match>

      <match kubernetes.var.log.containers.**logging**.log>

        @type null

      </match>

      <match kubernetes.var.log.containers.**monitoring**.log>

        @type null

      </match>

      <match kubernetes.var.log.containers.*nginx-ingress**.log>

        @type null

      </match>

Another question please Why not using Fluend v1 instead of v0.12

Best,

Anis

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/7f793a0d-8c75-4e1d-b362-ecfb9ddd90e3n%40googlegroups.com.

[Jacob Levertov]

I don't want to exclude them because I still want them. but, your suggestion gave me an idea, can I do this?

<filter kubernetes.var.log.containers.**<my_namespace>**.log>

  @do my parsing

</filter>

So it will filter logs from my namespace and parse only them before sending to elastic

[Jacob Levertov]

And as for the version, It is a long story, but the short version is this is what I can get for now, with hopes that in the near future it will be updated.