Periodic connection timeouts

[Rocket Spam]

Hello,

We've been experiencing this issue for a while now when using the fluent java logger across a few services. I can't seem to reproduce this issue by killing the td-agent process. I'm also not seeing anything in the td-agent logs that match up with these errors. There don't seem to be any network hiccups happening either during this time. Has anyone seen this before or have any thoughts on what could be happening? 

We have 2 forwarders that forward to 4 agents which then index our logs into elasticsearch. There are about 1000 clients that are sending messages to the forwards which translates to about 2MB of logs per second for each forwarder.

Here is the (very generic) error message:

[ERROR] [2016-02-25 11:07:38,078] [default-akka.actor.default-dispatcher-7] o.f.l.s.RawSocketSender: org.fluentd.logger.sender.RawSocketSender

java.net.SocketTimeoutException: connect timed out

        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.7.0_91]

        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[na:1.7.0_91]

        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[na:1.7.0_91]

        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[na:1.7.0_91]

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.7.0_91]

        at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_91]

        at org.fluentd.logger.sender.RawSocketSender.connect(RawSocketSender.java:78) [services.jar:1.36]

        at org.fluentd.logger.sender.RawSocketSender.reconnect(RawSocketSender.java:87) [services.jar:1.36]

        at org.fluentd.logger.sender.RawSocketSender.flush(RawSocketSender.java:181) [services.jar:1.36]

        at org.fluentd.logger.sender.RawSocketSender.send(RawSocketSender.java:172) [services.jar:1.36]

        at org.fluentd.logger.sender.RawSocketSender.emit(RawSocketSender.java:142) [services.jar:1.

36]

        at org.fluentd.logger.sender.RawSocketSender.emit(RawSocketSender.java:124) [services.jar:1.36]

        at org.fluentd.logger.sender.RawSocketSender.emit(RawSocketSender.java:119) [services.jar:1.36]

        at org.fluentd.logger.FluentLogger.log(FluentLogger.java:100) [services.jar:1.36]        at org.fluentd.logger.FluentLogger.log(FluentLogger.java:85) [services.jar:1.36]

[Mr. Fiber]

> There are about 1000 clients that are sending messages to the forwards

Lots of TCP connections are in backlog queue?

Could you try setup more fluentd process or longer flush time to mitigate this problem?

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.