Forwarding Detaches Every Time

[igor Finkelshteyn]

Hi Folks,
I'm evaluating FluentD for use at a company I work for, and I'm having some trouble setting up forwarding properly. I'm doing a test setup right now with two dev web servers attempting to forward to one server that does aggregation and then sends logs over to s3. I've gotten the s3 part to work without any issues, but the forwarding will not work no matter what I do. I'll start fluentd on each of the web servers and always receive the following in logs:

…
adding forwarding server 'chat-server' host="x.x.x.x" port=24224 weight=60
2012-08-29 02:19:25 +0000: listening fluent socket on 0.0.0.0:24224
2012-08-29 02:19:43 +0000: detached forwarding server 'chat-server' host="x.x.x.x" port=24224 phi=8.191940320901058
…

That detached message always comes about 20 seconds after the agent starts up, which makes it sound like the aggregation server is never sending back a heart beat. I've verified that I can telnet to the aggregation server from each of the web servers on port 24224, and the aggregation server looks like it has fluentD listening on the appropriate port:

2012-08-29 02:17:10 +0000: listening fluent socket on 0.0.0.0:24224

My relevant config file for the web servers is as follows:

<match debug.forward.**>
type forward
send_timeout 60s
recover_wait 10s
heartbeat_interval 1s
phi_threshold 8
hard_timeout 60s

<server>
name chat-server
host x.x.x.x
port 24224
weight 60
</server>
<secondary>
type file
path /var/log/fluent/forward-failed
</secondary>

My relevant config for the aggregation server is:

<source>
type forward
port 24224
bind 0.0.0.0
</source>

Am I missing something in my config? Any insight on what I'm doing wrong? I'd love to roll out FluentD on our production servers eventually if I can get it to work and get more familiar with it.

Igor

[Eli]

Nevermind. Only had TCP open on the port in the firewall. UDP was closed so heartbeats weren't getting through.

[Waldemar Quevedo]

Hello Igor,

One thing to consider is that the Fluentd forwarding plugin forwards
the logs using UDP,
while Telnet uses TCP so maybe UDP packets are not being sent.

Hope this helps,

- Wally

[Eli]

Yup. That was the problem exactly. See my last post. Thanks for the help!

[Waldemar Quevedo]

Glad to hear you solved the problem, I've run into this one sometimes too.

Cheers

[Eli]

Thanks, Waldemar. While on the subject: I'm still not getting any logs to output on the receiving server. When I telnet, I can get it to output stuff to td-agent.log, but from the same server it still doesn't seem to send anything over by match pattern. For example on the receiving end when I telnet send, I get:

tail -F /var/log/td-agent/td-agent.log

2012-08-29 02:56:59 +0000: adding source type="forward"

2012-08-29 02:56:59 +0000: adding source type="http"

2012-08-29 02:56:59 +0000: adding match pattern="td.*.*" type="tdlog"

2012-08-29 02:56:59 +0000: adding match pattern="debug.**" type="stdout"

2012-08-29 02:56:59 +0000: listening fluent socket on 0.0.0.0:24224

2012-08-29 03:11:32 +0000: no patterns matched tag="1"

2012-08-29 03:11:32 +0000: no patterns matched tag="1"

2012-08-29 03:11:32 +0000: no patterns matched tag="0"

2012-08-29 03:11:34 +0000: no patterns matched tag="0"

2012-08-29 03:11:44 +0000: no patterns matched tag="1"

2012-08-29 03:11:44 +0000: no patterns matched tag="0"

2012-08-29 03:11:49 +0000: no patterns matched tag="1"

But when I send stuff that matches the forwarding regexp, it just seems to disappear. No error messages, no "no patterns matched tag." It just doesn't say anything. Any idea what that might be? Is it buffering the logs in some undisclosed location until they get long enough to send over? My settings are the same as what I listed in my first message. I was hoping to just have the receiving server output messages to stout for now to test it works.

[Waldemar Quevedo]

# (Resending to the list)

I wonder what is happening.. I recommend you to try setting
flush_interval 0s in your forwarding settings to help you out debug
the issue without having to wait for the forwarding plugin to flush
the logs (around 60 secs by default, I think).

<match debug.forward.**>
type forward
send_timeout 60s
recover_wait 10s
heartbeat_interval 1s
phi_threshold 8
hard_timeout 60s

flush_interval 0s

[Eli]

Yup, that fixed it. It just wasn't flushing, and I couldn't find the information saved in a buffer anywhere. Forcing it to flush makes everything go through no problem. Thanks, Waldemar!