Condition on tag

[Nicolas Pepinster]

Hello,

I'm using forest plugin and I would like to output on a specfic directory following a certain tag and output to an another directory following an another tag.

Ex : my client produces logs with foo.bar.#{ENV['NAMESPACE']}.#{ENV['HOSTNAME']} as tag.

My namespace can take many value like, zzz-test-mynamespace or zzz-prd-mynamespace

I would write logs where test is part of the tag in /var/public/logs/ and logs where prod is part of the tag in /var/private/logs.

So, the condition must be done on a subset of tag_parts[2].

Do you have an idea how can I do that ?

If you think it's possible, I can split the NAMESPACE in more tag like foo.bar.#{ENV['TENANT']}.#{ENV['ENVIRONMENT']}.#{ENV['NAMESPACE_NAME']}.#{ENV['HOSTNAME']} at my client side and after, write the correct match at my server side like :

  <case foo.bar.zzz.test.*>

   path /var/public/logs/mylog

  </case>

and

  <case foo.bar.zzz.prod.*>

   path /var/private/logs/mylog

  </case>

but I should be done in the fluentd config because NAMESPACE is an environment variable setted by my client and I can't change that.

[Satoshi Tagomori]

> but I should be done in the fluentd config because NAMESPACE is an environment variable setted by my client and I can't change that.

Is it the main problem for you?

If fluentd process also has its environment variable NAMESPACE, you can use it in your configuration file for rewriting tags by embedding ruby code.

https://groups.google.com/forum/#!topic/fluentd/Ta_MtMzM29s

And then, you can use it in forest plugin like you show an example.

2016年6月2日木曜日 0時02分57秒 UTC+9 Nicolas Pepinster:

[Nicolas Pepinster]

Hello,

I tried to use rewrite_tag_filter at my server level. Here is my config :

  <source>

    @type forward

    @log_level debug

    port 24224

  </source>

  <match logback.security.**>

    @type rewrite_tag_filter

    @log_level debug

    rewriterule1 publictest  ^\(-test-\).*$  public.logback.security.${tag_parts[2]}.${tag_parts[3]}

    rewriterule2 privateall  ^\(!-test-\).*$  private.logback.security.${tag_parts[2]}.${tag_parts[3]}

  </match>

  <match {jbossserver.**,public.logback.security.**,private.logback.security.**}>

    @type forest

    @log_level debug

    subtype file

    <template>

      time_slice_format %Y%m%d

      time_slice_wait 1m

      time_format %Y%m%dT%H%M%S%z

      compress gzip

      utc

      format single_value

    </template>

    <case jbossserver.**>

      path /var/public/log/${tag_parts[1]}/${tag_parts[2]}/jbossserver

    </case>

    <case public.logback.security.**>

      path /var/public/log/${tag_parts[2]}/${tag_parts[3]}/security-logback

    </case>

    <case private.logback.security.**>

      path /var/private/log/${tag_parts[2]}/${tag_parts[3]}/security-logback

    </case>

  </match>

The idea is : 

- my client send a log tagged logback.security.tenant-test-001-foo.myhostname

- my server match the tag logback.security.**

- (if my rewrite rule work) the rewriterule rewrites logback.security.tenant-test-001-foo.myhostname to public.logback.security.tenant-test-001-foo.myhostname

- the new tag public.logback.security.** is matched by the other match bloc

- the forest plugin produces the log under /var/public/log/tenant-test-001-foo/myhostname/security-logback

but it doesn't work, probably something wrong with my config.

I only saw in /var/log/td-agent/td-agent.log :

2016-06-03 11:26:48 +0200 [info]: out_forest plants new output: file for tag 'jbossserver.tenant-test-001-foo.myhostname' 

but nothing for public.logback.security or private.logback.security.

Do you have an idea of where my config is wrong ?

[Nicolas Pepinster]

nobody ? :(

[Mr. Fiber]

Paste your sample log is better for problem investigation.

Do all your logs contain privateall or publictest field?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nicolas Pepinster]

Hello,

No, my logs doesn't contain privateall or publictest field and this is the problem.

I didn't understand well the synthax, sorry.

The problem is that the content of my message is not sufficient to determine if the log must be writed in the public directory or in the private directory.

The only info enabling to make the decision is a subset of tag_parts[2].

Maybe the rewrite_tag_filter plugin is not the good one to do that (or maybe I don't use it in a correct way...).

To summarize : 

- if the log is tagged with LOGBACK.SECURITY.TENANT-TEST-ID.APPNAME, I would like to write it in /var/public/log/${tag_parts[2]}/${tag_parts[3]}/security-logback

- if the log is tagged with LOGBACK.SECURITY.TENANT-PROD-ID.APPNAME, I would like to write it in /var/private/log/${tag_parts[2]}/${tag_parts[3]}/security-logback

Is it something wrong with my approach ?

The log is tagged by my fluentd client and I write in the correct folder in my fluentd server.

[Mr. Fiber]

rewrite-tag-filter rewrites tags based on event record, not a tag.

See the README: https://github.com/fluent/fluent-plugin-rewrite-tag-filter#syntax

[Nicolas Pepinster]

Ok :(

Unfortunately, I can't add information in the log.

Last try (I don't believe in but ...) :

Is it possible to add a regex in a case or a match directive ?

Ex : 

<match logback.security.*test*>

    path /var/public/log/${tag_parts[2]}/${tag_parts[3]}/security-logback

  </match>

[Mr. Fiber]

<match logback.security.*test*> matches logback.security.tenant-test-001-foo.

If you want to match with hostname, use <match logback.security.*test*.*> instead.

[Nicolas Pepinster]

Thank you ! It works as desired. In fact, the most simple answer is often the better.