Fluentd S3 plugin and access key / secret key

[Sebastien Stormacq]

Hello,

I am new to fluentd but I know AWS very well.  I am trying to configure the S3 plugin on an EC2 instance.  The EC2 instance has a profile linked to a Role with proper authorisation to write on my S3 bucket.

As per GitHub's Readme, aws_key_id and aws_secret_id are optional then.

This conflicts with Fluents'd doc that says the parameters are required.

At runtime, I receive this error message.  Lending me to think Fluentd's doc is correct while GitHub is not.

2014-04-27 19:42:00 +0000 [error]: unexpected error error_class=RuntimeError error=#<RuntimeError: aws_key_id or aws_sec_key is invalid. Please check your configuration>

I really do not want to store an access key / secret key on my instance as this is considered as poor practice in terms of security.

How can use Fluentd's S3 plugin with a Amazon IAM Role and an EC2 Instance profile ?

Thanks

Seb

[Sebastien Stormacq]

Looks like I found the solution (RTFM) 

Adding this line in the config

check_apikey_on_start true

solves the issue 

Seb

[Sebastien Stormacq]

Not solved :-(  The option below just prevent fluentd to verify the keys at startup.

It still can not flush to S3

2014-04-27 20:49:18 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2014-04-27 20:49:25 +0000 error_class="AWS::S3::Errors::Forbidden" error="AWS::S3::Errors::Forbidden" instance=70013167046360

Any help ?

Thanks

[Masahiro Nakagawa]

Hi Sebastien,

> AWS::S3::Errors::Forbidden

Similar problem of this post? Could you check your permissions?

https://groups.google.com/forum/#!topic/fluentd/Vvok7FTVuq4

> As per GitHub's Readme, aws_key_id and aws_secret_id are optional then.
This conflicts with Fluents'd doc that says the parameters are required.

Thanks for pointing out. I will update docs.fluentd.org later.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sebastien Stormacq]

Hello Masahiro-san

Mmmmmh, indeed.  I opened my S3 permissions to 

{
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::bucket_name/logs/*", "arn:aws:s3:::bucket_name"
      ]
    }

and it works.

As I would like to give minimum permissions to fluentd, what is the minimum IAM policy required ?  Can this be added to the doc ?

Many thanks for your help

Seb

[Michael H. Oshita]

Hi,

I've been using IAM roles with the s3 plugin for some time now.

Here is the minimum policy that will work, provided that my-bucket is created beforehand (separately from fluentd).

{ "Statement": [

 { "Effect":"Allow",

   "Action":"s3:*",

   "Resource":"arn:aws:s3:::my-s3bucket*"

  } ]

}

I usually add the s3:ListAllMyBuckets action in addition to this, because I have a worker that obtains the s3 files on the same server running fluentd.

Hope this helps.

-Michael

[Rama S]

Hi Sebastian , I am new to FluentD and I hit the exact issue. Using the permissions mentioned by Michael and you works, but I was wondering if you were able to find a more restrictive policy. 

Thanks

[Sebastien Stormacq]

Hello,

I am using :

{

      "Effect": "Allow",

      "Action": [

        "s3:Get*", "s3:List*","s3:Put*", "s3:Post*"

      ],

      "Resource": [

        "arn:aws:s3:::bucket_name/prefix/logs/*", "arn:aws:s3:::bucket_name"

      ]

    }

Seb

You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/_853gJrs7uY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.