Help with input fluentd-plugin-s3

[Zahirul Islam]

Hi,

I am new with Fluentd. And I want to read log file stored in s3 in json format and then write into Grafana-Loki. While reading the file from s3, fluentd reads each line of json file into json instead of taking whole file as a single log stream. My fluentd configuration is look like below:

<source>

  @type s3

  @log_level debug      

  aws_key_id XXXXXXXXXX

  aws_sec_key XXXXXXXXXXXX

  s3_bucket XXXXXXXXXXXXXX

  s3_region XXXXXXXXXXXXX

  add_object_metadata true

  

  <sqs>

queue_name XXXXXXX

  </sqs>

  tag siem.log

  @label @siem

  store_as json

  <parse>

@type json

  </parse>

</source>

    

<label @siem>

  <match siem.log>     

@type loki

endpoint_url          "http://loki.monitoring:3100"              

labels                {"log":"siem"}   

buffered              true

# @type stdout

  </match>

</label>

I would appreciate if anyone help me to figure out how to read entire json file as single log stream.

Many thanks.