Cannot output messages with tag 'ad-be-cd.hybris.access'

77 views

Skip to first unread message

Riz vaughan

unread,

Apr 26, 2021, 12:18:30 PM4/26/21

to Fluentd Google Group

Hello,

We have few nodes in autoscaling in aws environment. When new nodes gets created, td-agent doesn't send logs and give us the following error untill we restart td-agent.

2021-04-26 16:07:48 +0000 [error]: #0 Cannot output messages with tag 'ad-be-cd.hybris.access'

2021-04-26 16:07:58 +0000 [error]: #0 failed to configure sub output s3: Other 's3' plugin already use same buffer path: type = s3, buffer path = /var/log/td-agent/buffer/s3/ad-be-cd.hybris.access

2021-04-26 16:07:58 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/buf_file.rb:74:in `configure'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin.rb:178:in `configure'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/output.rb:342:in `configure'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin_helper/inject.rb:104:in `configure'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin_helper/formatter.rb:83:in `configure'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-s3-1.5.1/lib/fluent/plugin/out_s3.rb:179:in `configure'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin.rb:178:in `configure'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-forest-0.3.3/lib/fluent/plugin/out_forest.rb:132:in `block in plant'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-forest-0.3.3/lib/fluent/plugin/out_forest.rb:128:in `synchronize'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-forest-0.3.3/lib/fluent/plugin/out_forest.rb:128:in `plant'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-forest-0.3.3/lib/fluent/plugin/out_forest.rb:169:in `emit'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/compat/output.rb:201:in `process'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/bare_output.rb:53:in `emit_sync'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/out_copy.rb:61:in `block in process'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/out_copy.rb:59:in `each'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/out_copy.rb:59:in `each_with_index'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/out_copy.rb:59:in `process'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/multi_output.rb:148:in `emit_sync'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/event_router.rb:160:in `emit_events'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/event_router.rb:97:in `emit_stream'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/in_tail.rb:533:in `receive_lines'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/in_tail.rb:913:in `block in handle_notify'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/in_tail.rb:945:in `with_io'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/in_tail.rb:893:in `handle_notify'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/in_tail.rb:876:in `block in on_notify'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/in_tail.rb:876:in `synchronize'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/in_tail.rb:876:in `on_notify'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/in_tail.rb:711:in `on_notify'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/in_tail.rb:361:in `block in setup_watcher'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin/in_tail.rb:652:in `on_timer'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run_once'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:in `run'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.12.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

After restart of td-agent, the errors goes away and everything works normal.

I have noticed that the first time when it runs, it doesn't create the 's3' folder in buffer path (/var/log/td-agent/buffer/s3/ad-be-cd.hybris.access) and upon restart it does.

In my config, I have various store directives where I create folders on the fly and I do not face any issue at all. In fact, the same configuration is running on 4 environements and none of them has this issue.

Here is my config for s3 part.

<store>

type forest

subtype s3

s3_bucket "my-bucket"

s3_region "eu-west-1"

path archive/

s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}

time_slice_format ${tag}/YEAR=%Y/MONTH=%m/DAY=%d/HOSTNAME=${hostname}/HOUR=%H/%M

@type json

</format>

store_as gzip

timekey 30

@type file

path /var/log/td-agent/buffer/s3/${tag}

timekey_wait 1m

chunk_limit_size 50m

flush_at_shutdown true

</buffer>

</template>

</store>

Could someone help me find out what the issue is?

Thank you

Riz vaughan

unread,

May 2, 2021, 7:21:20 PM5/2/21

to Fluentd Google Group

I had missed some logs and here are they.

2021-05-02 20:24:48 +0000 [error]: #0 failed to configure/start sub output s3: can't call S3 API. Please check your credentials or s3_region configuration. error = #<Seahorse::Client::NetworkingError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)>

2021-05-02 20:24:48 +0000 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-s3-1.5.1/lib/fluent/plugin/out_s3.rb:469:in `rescue in check_apikeys'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-s3-1.5.1/lib/fluent/plugin/out_s3.rb:464:in `check_apikeys'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-s3-1.5.1/lib/fluent/plugin/out_s3.rb:254:in `start'

/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-forest-0.3.3/lib/fluent/plugin/out_forest.rb:133:in `block in plant'