Help with input fluentd-plugin-s3

75 views
Skip to first unread message

Zahirul Islam

unread,
Nov 23, 2021, 4:49:50 AM11/23/21
to Fluentd Google Group
Hi,

I am new with Fluentd. And I want to read log file stored in s3 in json format and then write into Grafana-Loki. While reading the file from s3, fluentd reads each line of json file into json instead of taking whole file as a single log stream. My fluentd configuration is look like below:

<source>
  @type s3
  @log_level debug      
  aws_key_id XXXXXXXXXX
  aws_sec_key XXXXXXXXXXXX
  s3_bucket XXXXXXXXXXXXXX
  s3_region XXXXXXXXXXXXX
  add_object_metadata true
  
  <sqs>
queue_name XXXXXXX
  </sqs>

  tag siem.log
  @label @siem
  store_as json

  <parse>
@type json
  </parse>
</source>
    
<label @siem>
  <match siem.log>     
@type loki
endpoint_url          "http://loki.monitoring:3100"              
labels                {"log":"siem"}   
buffered              true

# @type stdout

  </match>
</label>




I would appreciate if anyone help me to figure out how to read entire json file as single log stream.

Many thanks.
Reply all
Reply to author
Forward
0 new messages