Fluentd replace field conditionally

[Eugene Bolshakoff]

Hello,

I have the problem.

I use Fluentd (td-agent, but I think it doesn't matter so much) for analyzing Nginx logs.

I have field "response_time" and I need to replace "dash" (-) to number (e.g. 0.000), because I need to use float field in Elasticsearch.

I use this:

<filter td.logstash.nginx.log>

  @type record_transformer

  enable_ruby

  <record>

    response_time == - ? 0.000 : ${response_time} 

  </record>

</filter>

<source>

  type tail

  path /var/log/nginx/access.log

  pos_file /var/log/td-agent/access.pos

  tag td.logstash.nginx.log

    format /^(?<code>[^ ]*) (?<vhost>[^ ]*) (?<ip>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<timestamp>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<size>[^ ]*) (?<processing_time>[^ ]*) (?<scheme>[^ ]*) (?<response_time>[^ ]*) (?<upstream_addr>[^ ]*)?$/

</source>

It doesn't work. I tested also rewrite_plugin, but no success too.

Of course, I can use Unix tools (sed, for example), but wouldn't it be better to do this in Fluentd itself? I thought that this task should be easy... only change one field to other one...

Best regards,

Eugene

[Mr. Fiber]

> response_time == - ? 0.000 : ${response_time} 

Read record_transformer's article first. Your config doesn't follow record_transformer's syntax.

http://docs.fluentd.org/articles/filter_record_transformer

http://docs.fluentd.org/articles/filter_record_transformer#enableruby-optional

<filter pattern>

  @type record_transformer

  enable_ruby true

  auto_typecast true

  <record>

    response_time ${  you cord here }

  </record>

</filter>

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eugene Bolshakoff]

Hello Masahiro,

Thanks a lot. So I tried this construction:

   response_time ${ response_time == '-' ? 0.000 : response_time }

and it still doesn't work. I still see dashes in my Elasticsearch indices.

I am asking because I googled a lot and couldn't find suitable solution for this easy task. The same question on StackOverflow is still unanswered.

(it looks like to use something like grep/sed before sending to ES is much easier).

пятница, 20 января 2017 г., 0:05:14 UTC+1 пользователь repeatedly написал:

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

> response_time ${ response_time == '-' ? 0.000 : response_time }

response_time ${record["response_time"] == '-' ? 0.000 : record["response_time"]}

How about this?

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Eugene Bolshakoff]

Thanks!

Tested, but I still see dashes in Elasticsearch. (Now I had to make field "string", because insert failed every time when line with dash had come).

пятница, 20 января 2017 г., 16:27:58 UTC+1 пользователь repeatedly написал:

[Mr. Fiber]

I'm not sure what happen in your environment.

But it works with my test conf.

- fluentd log

2017-01-25 07:28:32 +0900 [info]: reading config file path="eugene.conf"
2017-01-25 07:28:32 +0900 [info]: starting fluentd-0.14.11 pid=49346
[...snip...]
2017-01-25 07:28:32 +0900 [info]: adding filter pattern="test.**" type="record_transformer"
2017-01-25 07:28:33 +0900 [info]: adding match pattern="test.**" type="stdout"
2017-01-25 07:28:33 +0900 [info]: adding source type="forward"
2017-01-25 07:28:33 +0900 [info]: using configuration file: <ROOT>
  <source>
    @type forward
  </source>
  <filter test.**>

    @type record_transformer
    enable_ruby true
    auto_typecast true
    <record>

      response_time ${record["response_time"] == '-' ? 0.000 : record["response_time"]}

    </record>
  </filter>
  <match test.**>
    @type stdout
  </match>
</ROOT>
2017-01-25 07:28:33 +0900 [info]: #0 starting fluentd worker pid=49374 ppid=49346 worker=0
2017-01-25 07:28:33 +0900 [info]: #0 listening a tcp port port=24224 bind="0.0.0.0"
2017-01-25 07:28:33 +0900 [info]: #0 fluentd worker is now running worker=0
2017-01-25 07:28:47.872725000 +0900 test.foo: {"response_time":0.0}
2017-01-25 07:29:03.395371000 +0900 test.foo: {"response_time":0.9}

- input

% echo '{"response_time":"-"}' | fluent-cat test.foo
% echo '{"response_time":0.9}' | fluent-cat test.foo