Accessing nested Json fields in record_transformer

[Mohit Gupta]

Hi,

Is there a support for accessing and modifying the nested Json fields inside the record_transformer of fluentd config?

I am looking for accessing this timestamp to be used as event's timestamp as well as use it for creating new fields.

my log is : {"metadata":{"timestamp":"2016-05-23T15:49:31","level":"INFO"},"serviceName":"myService"

fluentd config : didn't work!

source>

  @type tail

  path test.log

  read_from_head true

  pos_file myfile.pos # This is where you record file position

  tag abc.json #fluentd tag!

  format json # one JSON per line

  time_key metadata.timestamp 

  time_format %FT%T

  keep_time_key true

</source>

<filter abc.json>

  @type record_transformer

  enable_ruby

  <record>

    new_time ${metadata.timestamp}

  </record>

</filter>

Does anyone has any idea on how to do this or is it not supported? Am using td-agent.

Thanks!

[Mr. Fiber]

${record["metadata"]["timestamp"]} doesn't work?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shuya Tsukamoto]

Hello.

I have a json record with nested fields. It has an event time field, but it could not be access using time_key.
fluentd is supported nested field?

Also, I tried using record_transformer, but it did not go well.
Could you please help me?

% cat conf/fluent.conf
<source>
  @type       tail
  @label      @mainstream
  tag         record.*
  format      json
  # Didn't work
  # time_key    metadata.timestamp
  # time_key    ${record["metadata"]["timestamp"]}
  path        /fluentd/log/in
  pos_file    /fluentd/log/in.pos
</source>

<label @mainstream>
  <filter **>
    @type record_transformer
    enable_ruby
    # Didn't work
    renew_time_key ${record["metadata"]["timestamp"]}
    <record>
      additional field123
    </record>
  </filter>
  <match **>
    @type stdout
  </match>
</label>

% cat data/in

% docker run -v $(pwd)/data:/fluentd/log -v $(pwd)/conf:/fluentd/etc fluent/fluentd:v0.14.12
2017-09-28 10:02:03 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2017-09-28 10:02:03 +0000 [info]: starting fluentd-0.14.12 pid=1
2017-09-28 10:02:03 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2017-09-28 10:02:03 +0000 [info]: gem 'fluentd' version '0.14.12'
2017-09-28 10:02:03 +0000 [info]: adding filter in @mainstream pattern="**" type="record_transformer"
2017-09-28 10:02:03 +0000 [info]: adding match in @mainstream pattern="**" type="stdout"
2017-09-28 10:02:03 +0000 [info]: adding source type="tail"
2017-09-28 10:02:03 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    @label @mainstream
    tag "record.*"
    format json
    path "/fluentd/log/in"
    pos_file "/fluentd/log/in.pos"
    <parse>
      @type json
    </parse>
  </source>
  <label @mainstream>
    <filter **>
      @type record_transformer
      enable_ruby
      renew_time_key "${record[\"metadata\"][\"timestamp\"]}"
      <record>
        additional field123
      </record>
    </filter>
    <match **>
      @type stdout
    </match>
  </label>
</ROOT>
2017-09-28 10:02:03 +0000 [info]: #0 starting fluentd worker pid=10 ppid=1 worker=0
2017-09-28 10:02:03 +0000 [info]: #0 following tail of /fluentd/log/in
2017-09-28 10:02:03 +0000 [info]: #0 fluentd worker is now running worker=0

# Another terminal
% echo '{"metadata": {"timestamp":1506581107}}' >> data/in

# Main terminal(docker run)
# Unix time 1506581107 is 2017-09-28T15:45:07+09:00
2017-09-28 10:02:12.953487760 +0000 record.fluentd.log.in: {"metadata":{"timestamp":1506581107},"additional":"field123"}

Thank you.


2016年5月24日火曜日 0時05分00秒 UTC+9 repeatedly:

[Mr. Fiber]

> fluentd is supported nested field?

Currently, not yet in v0.14.

We now have record_accessor so we may support it via record_accessor helper.

> Also, I tried using record_transformer, but it did not go well.

You can use following workaround for it.

  <filter **>
    @type record_transformer
    enable_ruby
    # Didn't work

    renew_time_key t

    remove_keys t # removed keys after apply <record> and renew_time_key
    <record>
      additional field123

      t ${record["metadata"]["timestamp"]}
    </record>
  </filter>

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

[Shuya Tsukamoto]

Got it! Thank you Mr. repeatedly.

2017年9月29日金曜日 11時51分30秒 UTC+9 repeatedly:

2017<span style="color:#660" c