fluentd's filter "record_transform" : How to add complete json msg as another json field

[Mohit Gupta]

Hi,

I have a scenario wherein am trying to transform a Json record to add another field to it and assign this complete Json record as the value of this new field. 

The scenario is that some Json fields may get removed or transformed down the pipeline and so if I want to fetch the raw record down the pipeline, I can use this new field.

From what I got from documentation and my own hands-on, I could access the diff fields using their key-name but I could not find any placeholder that may return me the complete Json record. Workaround was to fetch all Json fields individually and then put in csv like format as shown below but that is undesirable. 

My json :

{"name":"abc","hostname":"xyz","msg":"Searching..","time":"2016-04-12T16:16:33.202Z"}

I tried :

<filter tag.json>

  @type record_transformer

  <record>

    new_field1 ${record}          # This doesn't work and fetches me nothing as this key does not exist in my Json schema

    new_field2  ${name},${hostname},${msg},${time}     # This works fine but cannot use it as this includes dependency on knowing number of fields pre-hand or their field-names

  </record>

</filter>

Is there any existing placeholder for achieving this or any other way that does not being in any dependency of knowing about the fields pre-hand?

Thanks for your suggetsions.

- Mohit

[Mr. Fiber]

Use `enable_ruby true`.

https://groups.google.com/d/msg/fluentd/HxW2SipudMQ/HaH8azthAwAJ

I will document it.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mohit Gupta]

Thanks Masahiro, it worked!

[Mohit Gupta]

Hi Masahiro,

${record} had worked fine last time when we discussed but now when I have again installed a recent version of td-agent, it is not returning any value as u can see here :

[from logs]

2016-09-02 02:31:27 +0530 [info]: using configuration file: <ROOT>

  <source>

    @type tail

    path /home/test/td-agent/td.%Y%m%d.log

    pos_file /var/log/td-agent/td.pos

    tag json.*

    format json

    read_from_head true

  </source>

  <filter json.**>

    @type record_transformer

    renew_record true

    enable_ruby true

    <record>

      new_field ${record}

      logfile   ${tag.partition("td-agent.").last}

    </record>

  </filter>

  <match json.**>

    @type stdout

  </match>

</ROOT>

2016-09-02 02:31:27 +0530 [info]: following tail of /home/test/td-agent/td.20160902.log

2016-09-02 02:31:27 +0530 json.home.test.td-agent.td.20160902.log: {"new_field":"","logfile":"td.20160902.log"}

$cat  /home/test/td-agent/td.20160902.log

{"name":"abc","hostname":"xyz","msg":"Searching..","time":"2016-04-12T16:16:33.202Z"}

Any idea if anything changed in latest distribution? I am not sure which version I had used last time.

On Friday, April 15, 2016 at 5:01:54 PM UTC+5:30, Mohit Gupta wrote:

Thanks Masahiro, it worked!

[Naotoshi Seo]

Latest td-agent version is v2.3.2, and its fluentd is v0.12.26 https://docs.treasuredata.com/articles/td-agent-changelog

I checked with fluentd v0.12.26, and it seems working fine.

2016-09-02 12:18:42 +0900 [info]: reading config file path="fluent.conf"

2016-09-02 12:18:42 +0900 [info]: starting fluentd-0.12.26

2016-09-02 12:18:42 +0900 [info]: gem 'fluentd' version '0.12.26'

2016-09-02 12:18:42 +0900 [info]: adding filter pattern="*.*" type="record_transformer"

2016-09-02 12:18:42 +0900 [info]: adding match pattern="*.*" type="stdout"

2016-09-02 12:18:42 +0900 [info]: adding source type="dummy"

2016-09-02 12:18:42 +0900 [info]: using configuration file: <ROOT>

  <source>

    @type dummy

    tag foo.bar

    dummy {"foo":"foo"}

  </source>

  <filter *.*>

    @type record_transformer

    enable_ruby yes

    auto_typecast yes

    <record>

      foo ${record}

    </record>

  </filter>

  <match *.*>

    @type stdout

  </match>

</ROOT>

2016-09-02 12:18:43 +0900 foo.bar: {"foo":{"foo":"foo"}}

2016-09-02 12:18:44 +0900 foo.bar: {"foo":{"foo":"foo"}}

2016-09-02 12:18:45 +0900 foo.bar: {"foo":{"foo":"foo"}}

2016-09-02 12:18:46 +0900 foo.bar: {"foo":{"foo":"foo"}}

BTW, you should add `auto_typecast yes` for your purpose.

Regards,

Naotoshi a.k.a. sonots