Error while sending logs to aws kinesis firehose

114 views
Skip to first unread message

Dhirendra Rawal

unread,
Feb 11, 2021, 9:21:31 PM2/11/21
to Fluentd Google Group
Hi All,
I am getting below error while sending logs to aws kinesis firehose.

2021-02-12 01:09:30 +0000 [warn]: #0 failed to flush the buffer. retry_time=2 next_retry_seconds=2021-02-12 01:09:32 +0000 chunk="5bb193af6adf1d26828aa3e418c1820f" error_class=Seahorse::Client::NetworkingError error="execution expired"

Few details:
td-agent v3
OS: amazon linux1 
working fine with streams issue with firehose(datastreams)
IAM role is fine

Dhirendra Rawal

unread,
Feb 11, 2021, 9:27:59 PM2/11/21
to Fluentd Google Group
more logs 
 2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/net/http.rb:904:in `initialize'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/net/http.rb:904:in `open'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/net/http.rb:904:in `block in connect'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/timeout.rb:103:in `timeout'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/net/http.rb:902:in `connect'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/net/http.rb:887:in `do_start'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/net/http.rb:882:in `start'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/2.4.0/delegate.rb:83:in `method_missing'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/seahorse/client/net_http/connection_pool.rb:300:in `start_session'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/seahorse/client/net_http/connection_pool.rb:99:in `session_for'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/seahorse/client/net_http/handler.rb:124:in `session'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/seahorse/client/net_http/handler.rb:76:in `transmit'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/seahorse/client/net_http/handler.rb:50:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/seahorse/client/plugins/content_length.rb:17:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/seahorse/client/plugins/request_callback.rb:85:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/json/error_handler.rb:10:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/signature_v4.rb:68:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/transfer_encoding.rb:26:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/retry_errors.rb:349:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/retry_errors.rb:383:in `retry_request'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/retry_errors.rb:371:in `retry_if_possible'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/retry_errors.rb:360:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/retry_errors.rb:383:in `retry_request'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/retry_errors.rb:371:in `retry_if_possible'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/retry_errors.rb:360:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/retry_errors.rb:383:in `retry_request'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/retry_errors.rb:371:in `retry_if_possible'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/retry_errors.rb:360:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/http_checksum.rb:18:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/endpoint_pattern.rb:31:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/json/handler.rb:13:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/endpoint_discovery.rb:80:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/seahorse/client/plugins/endpoint.rb:47:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/logging.rb:41:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/seahorse/client/plugins/raise_response_errors.rb:16:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/seahorse/client/plugins/request_callback.rb:71:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/seahorse/client/plugins/response_target.rb:24:in `call'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.111.2/lib/seahorse/client/request.rb:72:in `send_request'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/aws-sdk-firehose-1.35.0/lib/aws-sdk-firehose/client.rb:1533:in `put_record_batch'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.3.0/lib/fluent/plugin/out_kinesis_firehose.rb:52:in `block in write'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.3.0/lib/fluent/plugin/kinesis_helper/api.rb:91:in `batch_request_with_retry'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.3.0/lib/fluent/plugin/kinesis.rb:154:in `block in write_records_batch'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.3.0/lib/fluent/plugin/kinesis_helper/api.rb:86:in `split_to_batches'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.3.0/lib/fluent/plugin/kinesis.rb:152:in `write_records_batch'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-3.3.0/lib/fluent/plugin/out_kinesis_firehose.rb:48:in `write'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/output.rb:1123:in `try_flush'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/output.rb:1423:in `flush_thread_run'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/output.rb:452:in `block (2 levels) in start'
  2021-02-12 01:07:24 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'


Mr. Fiber

unread,
Feb 15, 2021, 10:21:59 AM2/15/21
to Fluentd Google Group
Hmm...
If this error randomly happens, the problem seems temporal network error.
If this error always happens, the setting or network environment is something wrong.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/8442238f-35bf-4a7e-ac0d-f11a2874c1dcn%40googlegroups.com.

Dhirendra Rawal

unread,
Feb 15, 2021, 10:29:16 AM2/15/21
to Fluentd Google Group

Thanks 
I am looking network setting like security groups and vpcs . 
is td-agent use different port to connect data stream and firehose?
Reply all
Reply to author
Forward
0 new messages