Need help with record_transformer

[Akshat Sharma]

Hi, 

I am feeding few logs to fluentd using tail plugin and sending them to stdout as below:

      <source>

        @type tail

        path /var/log/test/test.log

        tag apache

        pos_file /var/log/td-agent.pos

        read_from_head true

        format none

      </source>

      <match apache**>

        @type stdout

      </match>

example of the logs present in /var/log/test/test.log are:

{"type":"response","@timestamp":"2021-03-04T06:13:26Z","tags":["api"],"pid":10,"method":"get","statusCode":200,"req":{"url":"/api/status?pretty=","method":"get","headers":{"user-agent":"curl/7.29.0","host":"abcxyz","accept":"*/*"},"remoteAddress":"xxx.yyy.xx.zzz","userAgent":" xxx.yyy.xx.zzz "},"res":{"statusCode":200,"responseTime":5,"contentLength":9},"message":"GET /api/status?pretty= 200 5ms - 9.0B"}

Now I want to use a filter to put this entire log in a message field and add few extra fields like type, times etc. I want to use the @timestamp filed's value from the above logs as a value for the added "time" field.

Also, after the logs are read from /var/log/test/test.log and sent to stdout, there are a lot of "\"(backslashes) added in the stdout log messages which I want to avoid. 

for example, the above logs is shown on stdout as:

{"message":"{\"type\":\"response\",\"@timestamp\":\" 2021-03-04T06:13:26Z  \",\"tags\":[\"api\"],\"pid\":1,\"method\":\"get\",\"statusCode\":200,\"req\":{\"url\":\"/api/status?pretty=\",\"method\":\"get\",\"headers\":{\"host\":\"abcxyz\",\"user-agent\":\"curl/7.29.0\",\"accept\":\"*/*\"},\"remoteAddress\":\" xxx.yyy.xx.zzz \",\"userAgent\":\" xxx.yyy.xx.zzz \"},\"res\":{\"statusCode\":200,\"responseTime\":5,\"contentLength\":9},\"message\":\"GET /api/status?pretty= 200 5ms - 9.0B\"}","type":"log","timestamp":null}

Can anyone help me with this configuration?

Thanks

[Akshat Sharma]

note that I have used to basic filter in my configuration to add type and timestamp field.

      <filter apache**>

        @type record_transformer

        <record>

          type "log"

          timestamp 

        </record>

      </filter>