Re: How to filter only child element of json?

[Mr. Fiber]

How about using fluent-plugin-map like below?

<filter test.**>
  @type map
  time time
  record (record["log"])
</filter>

On Tue, Jun 9, 2020 at 4:46 AM Leonardo Fuzeto <leonard...@wspot.com.br> wrote:

I have the following message log:

{"log":{"Prefix":"NXFILTER","Time":"2020-05-29 19:30:43","Blocked":"N","Domain":"qwert.com","User":"10.8184882","ClientIp":"0.0.0.0","Policy":"Teste","Category":"Download","Reason":"","Type":1,"Group":"test"}}

But I want that result to be just the content inside the "log" parent, e.g:

{"Prefix":"NXFILTER","Time":"2020-05-29 19:30:43","Blocked":"N","Domain":"qwert.com","User":"10.8184882","ClientIp":"0.0.0.0","Policy":"Teste","Category":"Download","Reason":"","Type":1,"Group":"test"}

Any idea?

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/6a0791e4-1805-4e71-9e94-ac1d412c952co%40googlegroups.com.

[Ranajit Jana]

This is solution which I was searching but when I put this in my configuration the logs has stopped flowing into the es.**

This is my configuration. Hope it will be easy for you to detect the error.

When i remove this map filter things starts flowing again.

This configuration is picked from the fluentd logs so this is what got applied.

  <match fluent.**>
    @type null
  </match>
  <source>
    @type tail
    @id in_tail_abc_container_logs_dd
    path "/var/log/containers/abc-*.log"
    pos_file "/var/log/abc-fluentd-containers-dd.pos"
    tag "datadog"
    read_from_head true
    emit_unmatched_lines false
    <parse>
      @type "json"
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      unmatched_lines false
      time_type string
    </parse>
  </source>
  <source>
    @type tail
    @id in_tail_abc_container_logs_es
    path "/var/log/containers/abc-*.log"
    pos_file "/var/log/abc-fluentd-containers-es.pos"
    tag "es"
    read_from_head true
    emit_unmatched_lines false
    <parse>
      @type "json"
      time_format "%Y-%m-%dT%H:%M:%S.%NZ"
      unmatched_lines false
      time_type string
    </parse>
  </source>
  <filter datadog.**>
    @type grep
    <regexp>
      key "log"
      pattern /(\s|^|[[:punct:]])(error|fail|failed|critical|fatal)/i
    </regexp>
  </filter>
  <filter es>

    @type map
    time "time"
    record "(record[\"log\"])"
  </filter>

  <match datadog.**>
    @type datadog
    @id abc_datadog_agent
    api_key xxxxxx
    <buffer>
      @type "memory"
      flush_thread_count 4
      flush_interval 3s
      chunk_limit_size 5m
      chunk_limit_records 500
    </buffer>
  </match>
  <match es.**>
    @type aws-elasticsearch-service
    type_name "access_log"
    logstash_format true
    include_tag_key true
    tag_key "@log_name"
    flush_interval 1s
    logstash_prefix "abcdev"
    <endpoint>
      url "https://vpc-abc-application-logsxxxxxxxxxxxx.es.amazonaws.com"
      region "us-east-1"
      access_key_id "xxxxxxx"
      secret_access_key xxxxxx
    </endpoint>
    <buffer>
      flush_interval 1s
    </buffer>
  </match>

Regards,

Ranajit

On Wednesday, June 10, 2020 at 3:40:37 PM UTC+5:30, repeatedly wrote:

How about using fluent-plugin-map like below?

<filter test.**>
  @type map
  time time
  record (record["log"])
</filter>

On Tue, Jun 9, 2020 at 4:46 AM Leonardo Fuzeto <leonard...@wspot.com.br> wrote:

I have the following message log:

{"log":{"Prefix":"NXFILTER","Time":"2020-05-29 19:30:43","Blocked":"N","Domain":"qwert.com","User":"10.8184882","ClientIp":"0.0.0.0","Policy":"Teste","Category":"Download","Reason":"","Type":1,"Group":"test"}}

But I want that result to be just the content inside the "log" parent, e.g:

{"Prefix":"NXFILTER","Time":"2020-05-29 19:30:43","Blocked":"N","Domain":"qwert.com","User":"10.8184882","ClientIp":"0.0.0.0","Policy":"Teste","Category":"Download","Reason":"","Type":1,"Group":"test"}

Any idea?

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to flu...@googlegroups.com.

[Ranajit Jana]

To add, I have used es.**, es , ** but all of them gives same output.

The logs are not forwarded further.

  <filter es>

    @type map
    time "time"
    record "(record[\"log\"])"
  </filter>