fluentd s3 output plugin configuration

[Das]

I am trying to get the out_s3 for fluentd working from the past 2 days, I am unable to see the logs on my s3

This is my current config:

 

<match web.all>
        type s3
        aws_key_id ......
        as_sec_key ......
        s3_bucket ......
        path logs/
        buffer_path /var/log/td-agent/s3
        s3_region ap-southeast-1
        time_slice_format %Y%m%d%H%M
        time_slice_wait 1M
        utc

    buffer_chunk_limit 256m
</match>

If I try to match the 'web.all' and store it to a file, it works properly 

<match web.all>
  type file
  path /var/log/td-agent/web-all.log
</match>

After finding out how to read td-agent logs I was able to find the problem
2015-10-18 22:46:51 -0400 [error]: unexpected error error_class=RuntimeError error=#<RuntimeError: can't call S3 API. Please check your aws_key_id / aws_sec_key or s3_region configuration. error = #<AWS::S3::Errors::InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.>>
But the key provided is valid. I am able to see change in access key "last used time" update every time I restart td-agent.
I'm thinking the problem is on the s3 side of things. Since I'm new to both, would be happy if I can get directions.

[Mr. Fiber]

> as_sec_key ......

Is this a typo in this thread or configuration also has a typo?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lance N.]

For reference, here is my working S3 configuration (part of a forest plugin). Notice the AWS keys are inside double-quoted strings. I supply them from the environment to avoid checking them into our source control.
Good luck!

   type s3
    aws_key_id "#{ENV['AWS_KEY_ID']}"
    aws_sec_key "#{ENV['AWS_SECRET_KEY']}"
    s3_bucket name_withheld
    s3_region us-west-2
    s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
    path "${tag}/year=%Y/month=%m/day=%d/hour=%H/#{ENV['IP']}_"
    buffer_path /var/log/td-agent/s3_buffer/${tag}/
    time_slice_format %Y-%m-%d_%H
    flush_interval  10m
    flush_at_shutdown true   
    store_as gzip
    format json
    utc