docker fluentd logging driver - can't send logs client is reconnecting

[starship]

Hi,

I am using docker's fluentd logging driver to forward all logs to fluentd running on the host. 

This stopped working on some of the nodes and I see this error message from docker in /var/log/messages

Aug  7 17:45:11 ip-10-0-60-181 docker[65207]: time="2017-08-07T17:45:11.863947825-07:00" level=error msg="Failed to log msg \"<FULL TEXT OF THE MESSAGE>\" for logger fluentd: fluent#send: can't send logs, client is reconnecting"

I am using td-agent-2

[root@ip-10-0-38-39 ansible]# cat /var/log/td-agent/td-agent.log | grep -i VERSION 
2017-08-08 12:34:59 -0700 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2017-08-08 12:34:59 -0700 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '0.1.6'
2017-08-08 12:34:59 -0700 [info]: gem 'fluent-plugin-aws-elasticsearch-service-hotfix' version '0.1.6.pre.f1'
2017-08-08 12:34:59 -0700 [info]: gem 'fluent-plugin-elasticsearch' version '1.9.5'
2017-08-08 12:34:59 -0700 [info]: gem 'fluent-plugin-kafka' version '0.5.5'
2017-08-08 12:34:59 -0700 [info]: gem 'fluent-plugin-mongo' version '0.8.0'
2017-08-08 12:34:59 -0700 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2017-08-08 12:34:59 -0700 [info]: gem 'fluent-plugin-s3' version '0.8.2'
2017-08-08 12:34:59 -0700 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2017-08-08 12:34:59 -0700 [info]: gem 'fluent-plugin-td' version '0.10.29'
2017-08-08 12:34:59 -0700 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2017-08-08 12:34:59 -0700 [info]: gem 'fluent-plugin-webhdfs' version '0.7.0'
2017-08-08 12:34:59 -0700 [info]: gem 'fluentd' version '0.12.36'

[root@ip-10-0-38-39 ansible]# docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:20:01 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:20:01 2017
 OS/Arch:      linux/amd64

Could someone help ? Is this related to https://github.com/fluent/fluent-logger-golang/issues/38 ?

regards

starship

[Mr. Fiber]

From the link, the problem in fluent-logger is resolved.

So it seems the other problem.

Is this problem reproducible?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[starship]

We do not know a way to reproduce this issue. I restarted docker but still the problem 

is not resolved. 

Do you have any debugging tips:? 

Masahiro

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

You need to check this problem is docker loggind driver or fluent-logger-golang.

Here is an example of golang logger: https://github.com/fluent/fluent-logger-golang#example

If no problem, docker logging driver or your network setting has a problem.

Masahiro

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.