Does anyone have fluent-plugin-s3 working with AWS S3 VPC endpoint?

[jota.sanch]

I'm getting the below error.  I have an EC2 instance in a VPC and have configured an S3 Endpoint (private, direct access to S3) for the VPC.  The server has no internet access (no NAT instance or gateway).  I get the below error when trying to access an S3 bucket configured with the S3 Endpoint.

The 'no internet' instance can do everything normally to S3 using the awscli.  So there must be something specific with this plugin an it needing direct access to the internet.

I have tested this plugin with an instance that can get to the internet, and everything works as expected.  I'm avoiding allowing these servers access to the internet, so I'm hoping for a solution where that isn't needed.

Config:

  <source>

   log_level debug

   type syslog

   port 42185

   bind 0.0.0.0

   tag rsyslog

   with_priority true

 </source>

  <match *.**>

   type copy

   <store>

     type stdout

   </store>

   <store>

     type s3

     aws_key_id xxxxxx

     aws_sec_key xxxxxx

     s3_bucket s3_bucket-name

     s3_region us-west-2

     s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}

     path logs/

     buffer_path /var/log/td-agent/buffer

     check_apikey_on_start false

     flush_interval 5s

     time_slice_format %Y%m%d-%H

     utc

   </store>

 </match>

gem versions:

/opt/td-agent/embedded/bin/gem list | grep aws

aws-sdk (2.2.14)

aws-sdk-core (2.2.14)

aws-sdk-resources (2.2.14)

aws-sdk-v1 (1.64.0)

/opt/td-agent/embedded/bin/gem list | grep fluent

fluent-logger (0.4.10)

fluent-mixin-config-placeholders (0.3.0)

fluent-mixin-plaintextformatter (0.2.6)

fluent-plugin-mongo (0.7.10)

fluent-plugin-rewrite-tag-filter (1.4.1)

fluent-plugin-s3 (0.6.5)

fluent-plugin-scribe (0.10.14)

fluent-plugin-td (0.10.27)

fluent-plugin-td-monitoring (0.2.1)

fluent-plugin-webhdfs (0.4.1)

fluentd (0.12.12)

fluentd-ui (0.4.2)

Error:

2016-02-02 12:59:48 -0800 [error]: unexpected error error_class=Seahorse::Client::NetworkingError error=#<Seahorse::Client::NetworkingError: execution expired>

 2016-02-02 12:59:48 -0800 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:879:in `initialize'

 2016-02-02 12:59:48 -0800 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:879:in `open'

 2016-02-02 12:59:48 -0800 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:879:in `block in connect'

 2016-02-02 12:59:48 -0800 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/timeout.rb:91:in `block in timeout'

 2016-02-02 12:59:48 -0800 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/timeout.rb:101:in `call'

 2016-02-02 12:59:48 -0800 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/timeout.rb:101:in `timeout'

 2016-02-02 12:59:48 -0800 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:878:in `connect'

 2016-02-02 12:59:48 -0800 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:863:in `do_start'

 2016-02-02 12:59:48 -0800 [error]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:858:in `start'

[Mr. Fiber]

Do you test it with only ruby aws sdk?

I want to know this problem is by ruby aws sdk or S3 plugin.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.