Input fluent-plugin-s3 is not fetching objects from s3

[Chen]

Hi, I'm trying to run fluentd on kubernetes to fetch logs from S3 and output them to elastic.

I raised the log level to debug, and I see SQS messages are read by fluentd. However, it doesn't download or parse the objects from there.

I don't see them in elastic nor in stdout (tried to put it there for debugging).

I put here the full configurations, but I only changed the input and output filters. Any idea what am I doing wrong?

  <label @FLUENT_LOG>

    <match **>

      @type null

      @id ignore_fluent_logs

    </match>

  </label>

  <system>

    log_level debug

  </system>

  <source>

    @type s3

    @label @DISPATCH

    aws_key_id xxxxxx

    aws_sec_key xxxxxx

    s3_bucket "my-bucket"

    s3_region "us-east-1"

    add_object_metadata false

    store_as "text"

    <parse>

      @type "json"

    </parse>

    <sqs>

      queue_name "chen-queue"

    </sqs>

  </source>

  <label @DISPATCH>

    <filter **>

      @type prometheus

      <metric>

        name fluentd_input_status_num_records_total

        type counter

        desc The total number of incoming records

        <labels>

          tag ${tag}

          hostname ${hostname}

        </labels>

      </metric>

    </filter>

    <filter **>

      @type stdout

    </filter>

    <match **>

      @type relabel

      @label @OUTPUT

    </match>

  </label>

  <label @OUTPUT>

    <match **>

      @type copy

      <store>

        @type "stdout"

      </store>

      <store>

        @type "elasticsearch"

        host "elasticsearch-master"

        port 9200

        scheme http

        index_name "logs-fluentd-%Y%m%d"

      </store>

    </match>

  </label>

  <source>

    @type prometheus

    @id in_prometheus

    bind "0.0.0.0"

    port 24231

    metrics_path "/metrics"

  </source>

  <source>

    @type prometheus_monitor

    @id in_prometheus_monitor

  </source>

  <source>

    @type prometheus_output_monitor

    @id in_prometheus_output_monitor

  </source>

Thanks