Having some difficulty with fluent-plugin-ping-message plugin.

[Stuart Cracraft]

On a test pair of fluent boxes, B sending to A, I am getting this in the latter's td-agent.log:

2014-06-23 21:53:36 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2014-06-23 21:57:52 +0000 error_class="RuntimeError" error="no nodes are available" instance=70324300280040

  2014-06-23 21:53:36 +0000 [warn]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/plugin/out_forward.rb:153:in `write_objects'

  2014-06-23 21:53:36 +0000 [warn]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/output.rb:449:in `write'

  2014-06-23 21:53:36 +0000 [warn]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/buffer.rb:296:in `write_chunk'

  2014-06-23 21:53:36 +0000 [warn]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/buffer.rb:276:in `pop'

  2014-06-23 21:53:36 +0000 [warn]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/output.rb:310:in `try_flush'

  2014-06-23 21:53:36 +0000 [warn]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.45/lib/fluent/output.rb:132:in `run'

The td-agent.conf on A:

include conf.d/*.conf

<match debug.**>

  type stdout

</match>

<source>

  type debug_agent

  bind 127.0.0.1

  port 24230

</source>

<source>

  type forward

  port 24224

</source>

<match ping.**>

  type ping_message_checker

  tag missing.ping

  check_interval  60   # 1hour by default (3600 seconds)

  notification_times 999   # 3 times by default

</match>

On B, td-agent.conf is:

include conf.d/*.conf

<match debug.**>

  type stdout

</match>

<source>

  type debug_agent

  bind 127.0.0.1

  port 24230

</source>

<source>

  type forward

  port 24224

</source>

<source>

  type ping_message

  tag ping.webserver

  interval 30

  data ping message from ${hostname}

</source>

The conf.d/forwarder.conf file on the B points to A.

I've tried it both with and without the forwarder.conf file and a td-agent restart.

pings are blocked on the network so I am hoping that the plugin does its ping over its port

and not, as its name would imply (instead of being fluent-plugin-heartbeat-message)

or on the other hand that I've hopefully overlooked something.

[Christian Hedegaard]

Shut down fluentd entirely.

 

Use netcat to test connectivity between those two boxes on 24230. That’s a weird port number, so I doubt it’s open. The default port for fluentd is 24224

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Masahiro Nakagawa]

And you should also open UDP port for heartbeat when use udp heartbeat.

http://docs.fluentd.org/articles/out_forward#troubleshooting

http://docs.fluentd.org/articles/out_forward#heartbeattype

[Stuart Cracraft]

Okay, the firewall ports for tcp/udp for the port mentioned in the <match **> stanza for the two test hosts have been corrected and opened.

While fluent traffic is flowing and logger -p user.error message logs quickly,  fluent-plugin-ping-message doesn't seem to generate

any regular messages indicating the connection is up. The source/matches are as described in the first post in the thread.

I'd expect the message "ping message from hostname" to appear once every interval seconds in the destination log file.

I've tried different ideas of ordering of the matches and sources but followed:

  https://github.com/tagomoris/fluent-plugin-ping-message

  http://docs.fluentd.org/articles/config-file

Anyone have the fluent plugin running and can send your fluent configuration files?

[Masahiro Nakagawa]

Your pasted configuration doesn't have out_file or similar plugin.

Does the configuration under conf.d/* have such plugins?

BTW, I tried your configurations on my one Mac and ping_message_checker emits events correctly.

So your configuration seems correct.

2014-06-25 11:55:29 +0900 [info]: starting fluentd-0.10.50
2014-06-25 11:55:29 +0900 [info]: reading config file path="a.conf"
2014-06-25 11:55:29 +0900 [info]: gem 'fluent-mixin-config-placeholders' version '0.2.4'
2014-06-25 11:55:29 +0900 [info]: gem 'fluent-plugin-eval-filter' version '0.0.5'
2014-06-25 11:55:29 +0900 [info]: gem 'fluent-plugin-ping-message' version '0.1.1'
2014-06-25 11:55:29 +0900 [info]: gem 'fluentd' version '0.10.46'
2014-06-25 11:55:29 +0900 [info]: using configuration file: <ROOT>
  <match missing.**>  # changed from debug to missing for ping_message_checker

    type stdout
  </match>
  <source>
    type debug_agent
    bind 127.0.0.1
    port 24230
  </source>
  <source>
    type forward
    port 24224
  </source>
  <match ping.**>
    type ping_message_checker
    tag missing.ping

    check_interval 20  # changed to short interval for testing
    notification_times 2
  </match>
</ROOT>
2014-06-25 11:55:29 +0900 [info]: adding source type="debug_agent"
2014-06-25 11:55:29 +0900 [info]: adding source type="forward"
2014-06-25 11:55:29 +0900 [info]: adding match pattern="missing.**" type="stdout"
2014-06-25 11:55:29 +0900 [info]: adding match pattern="ping.**" type="ping_message_checker"
2014-06-25 11:55:29 +0900 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"
2014-06-25 11:55:29 +0900 [info]: listening fluent socket on 0.0.0.0:24224
2014-06-25 11:56:49 +0900 missing.ping: {"data":"ping message from MacBook-Pro.local"}
2014-06-25 11:57:09 +0900 missing.ping: {"data":"ping message from MacBook-Pro.local"}
2014-06-25 11:57:43 +0900 [info]: shutting down fluentd
2014-06-25 11:57:43 +0900 [info]: process finished code=0

From above log, after passed 20 seconds, ping_message_checker emits events to 'missing.*' stdout.

For testing, could you try 

<match ping.**>

  type stdout

</match>

instead of ping_message_checker on A node?

Your configuration, check_interval  60, needs 60 seconds interval.

[Stuart Cracraft]

Great! - out_file was the clue we needed. Pardon, as I am very

new to fluent (but getting more fluent!)

As well, my test was overly complex/indirect, lacking simplicity:

  B forwarding to A and A forwarding to L

I had started the thread only mentioning A and B and failed to check all the logfiles

repeatedly on L during tests. My bad.

The ping messages from the plugin were arriving, unmatched, on L in td-agent.log

since L's td-agent.conf and forwarders were left unchanged during the initial testing.

When I switched A to use an out_file and local log on A instead of a forward

to L, the match started working and it sees the ping on A every half-minute,

locally.

Strange though: for testing it by denying port 24224 (tcp+udp) on B, the message

still comes through. If I disable B with td-agent stop, the message stops.

So that's an unsolved mystery here. I would have thought disabling 24224 tcp/udp

at B would stop the ping.

My hope was that the ping plugin would notice when the pings weren't coming in

for the interval, emit then a missing, and only then the match would catch, 

instead of the opposite. I know it is possible to code around it. Perhaps

some capability in the fluent configuration language permits finding the opposite?

[Masahiro Nakagawa]

> I had started the thread only mentioning A and B and failed to check all the logfiles

> repeatedly on L during tests. My bad.

> The ping messages from the plugin were arriving, unmatched, on L in td-agent.log

> since L's td-agent.conf and forwarders were left unchanged during the initial testing.

So after changed the td-agent.conf on L, logfiles contain ping message, right?

> Strange though: for testing it by denying port 24224 (tcp+udp) on B, the message

still comes through.

Hmm... that's strange but in/out_forward uses normal tcp / udp socket.

If you can't deny an access, maybe you may lack some settings.

> the ping plugin would notice when the pings weren't coming in for the interval, emit then a missing,

In ping_message_checker do this behaviour.

If ping_message_checker doesn't receive any ping message during check_interval,

this plugin emits the message to tag, missing.ping in this thread.

[Stuart Cracraft]

Solved earlier today. 

Very straightforward based

on interval adjustments. 

The tests passed for the

first implementation phase 

about 3-4 hours ago. 

I'll be back when/if any major

new questions arise. 

Have a great day.

You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/YbOomcPhfv4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

[Masahiro Nakagawa]

> Solved earlier today. 

Glad to hear that :)

> I'll be back when/if any major new questions arise. 

Okay.

Please don't hesitate to ask here if you have any questions.

Masahiro