FluentD - Routing events to InfluxDB

[Steve Merchant]

Hello,
Could anyone please advice me on routing events from FluentD to InfluxDB?

I have tried two techniques so far, and both fail.

I'm not really worried which technique eventually works.

My Environment

• UBUNTU 14.04 (running under Oracle VM Virtual Box) within Windows 8.
• td-agent version 2.1.3
• InfluxDB version 0.8.8
• Submitting JSON events from a Java client via TCP sockets. Test JSON message sent: ["test.metrics", [1308466941, {"name":"foo1","columns":["val"],"points":[[24]]}]]
• FluentD and InfluxDB are installed on the same host.
  

TECHNIQUE #1 : USING OUTSOURCED FLUENTD INFLUXDB (OUTPUT) PLUGIN-IN

Using the https://github.com/fangli/fluent-plugin-influxdb plugin.

Detail

My FluentD route configuration :
   <store>
    type influxdb
    host 127.0.0.1
    port 8086
    dbname test_database
    user root
    password root
    time_precision m
    flush_interval 10s
   </store>



Result

Snippet from td-agent.log :

2015-02-10 09:00:43 +0000 [info]: adding source type="forward"
2015-02-10 09:00:43 +0000 [info]: adding match pattern="test.*" type="copy"
2015-02-10 09:00:43 +0000 [debug]: adding store type="influxdb"
2015-02-10 09:00:43 +0000 [info]: listening fluent socket on 0.0.0.0:24224
2015-02-10 09:00:43 +0000 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"
2015-02-10 09:00:56 +0000 [trace]: accepted fluent socket from '172.28.147.67:55391': object_id=70147932543540
2015-02-10 09:00:56 +0000 [trace]: closed fluent socket object_id=70147932543540
2015-02-10 09:01:05 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-02-10 09:01:05 +0000 error_class="NoMethodError" error="undefined method `keys' for 0:Fixnum" plugin_id="object:3fcc9698e14c"
  2015-02-10 09:01:05 +0000 [warn]: /etc/td-agent/plugin/out_influxdb.rb:42:in `block in write'
  2015-02-10 09:01:05 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/buf_memory.rb:62:in `feed_each'
  2015-02-10 09:01:05 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/buf_memory.rb:62:in `msgpack_each'
  2015-02-10 09:01:05 +0000 [warn]: /etc/td-agent/plugin/out_influxdb.rb:39:in `write'
  2015-02-10 09:01:05 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/buffer.rb:300:in `write_chunk'
  2015-02-10 09:01:05 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/buffer.rb:280:in `pop'
  2015-02-10 09:01:05 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/output.rb:315:in `try_flush'
  2015-02-10 09:01:05 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/output.rb:136:in `run'
2015-02-10 09:01:06 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-02-10 09:01:07 +0000 error_class="NoMethodError" error="undefined method `keys' for 0:Fixnum" plugin_id="object:3fcc9698e14c"
  2015-02-10 09:01:06 +0000 [warn]: /etc/td-agent/plugin/out_influxdb.rb:42:in `block in write'

Question
Any idea what might be causing this issue?

(In view of this I tried technique #2, summarised below).

=======================================================================================================================

TECHNIQUE #2 : USING THE FLUENTD BUILT-IN  'out_forward' PLUGIN.

Using the FluentD 'out_forward' FluentD (output) plugin to deliver JSON into InfluxDB via UDP.

Detail

Before testing I open UDP port 5551 :

sudo iptables -A INPUT -p udp --dport 5551 -j ACCEPT


My FluentD route configuration :
   <store>
      type forward
      send_timeout 60s
      recover_wait 10s
      phi_threshold 16
      hard_timeout 60s
      buffer_type memory
      heartbeat_type tcp
      heartbeat_interval 20
      <server>
        name test.udp
        host 127.0.0.1
        port 5551
        weight 60
     </server>
   </store>

Result

Snippet from td-agent.log.

Why does the forwarding server get detached?

2015-02-10 09:25:15 +0000 [info]: adding match pattern="test.*" type="copy"
2015-02-10 09:25:15 +0000 [debug]: adding store type="forward"
2015-02-10 09:25:15 +0000 [info]: adding forwarding server 'test.udp' host="127.0.0.1" port=5551 weight=60 plugin_id="object:3ff43311e470"
2015-02-10 09:25:15 +0000 [info]: listening fluent socket on 0.0.0.0:24224
2015-02-10 09:25:15 +0000 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"
2015-02-10 09:25:26 +0000 [trace]: accepted fluent socket from '172.28.147.67:55800': object_id=70318060276300
2015-02-10 09:25:26 +0000 [trace]: closed fluent socket object_id=70318060276300
2015-02-10 09:26:15 +0000 [warn]: detached forwarding server 'test.udp' host="127.0.0.1" port=5551 hard_timeout=true
2015-02-10 09:26:16 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-02-10 09:26:17 +0000 error_class="RuntimeError" error="no nodes are available" plugin_id="object:3ff43311e470"
  2015-02-10 09:26:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/plugin/out_forward.rb:156:in `write_objects'
  2015-02-10 09:26:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/output.rb:454:in `write'
  2015-02-10 09:26:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/buffer.rb:300:in `write_chunk'
  2015-02-10 09:26:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/buffer.rb:280:in `pop'
  2015-02-10 09:26:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/output.rb:315:in `try_flush'
  2015-02-10 09:26:16 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.58/lib/fluent/output.rb:136:in `run'
2015-02-10 09:26:18 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-02-10 09:26:19 +0000 error_class="RuntimeError" error="no nodes are available" plugin_id="object:3ff43311e470"

Question
Both FluentD and InfluxDB are installed ion the same host, and I've opened UDP port 5551.

Again, any ideas?

Many Thanks

[Lance N.]

#1 has a simple problem that is common to many Fluentd plugins: it expects all fields to be string, and you have a number in there. I use the 'monitor' plugin to create per-second rollups, and it creates a fixnum field called "num". I have to add a 'reformer' plugin after it with a ruby dropin that expresses the field as a string called "count", and remove the "num" field.

<match foo.**>
  type record_reformer
  renew_record false
  enable_ruby true
  tag set.a.new.tag
  count "${num}"
  remove_keys num
</match>

Cheers,

Lance Norskog

[Mr. Fiber]

> TECHNIQUE #2 : USING THE FLUENTD BUILT-IN  'out_forward' PLUGIN.

out_forward is not general output plugin. Please check the document.

http://docs.fluentd.org/articles/out_forward

    The out_forward Buffered Output plugin forwards events to other fluentd nodes.

So this can't be used for inserting logs into InfluxDB.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mr. Fiber]

> Submitting JSON events from a Java client via TCP sockets. Test JSON message sent: ["test.metrics", [1308466941, {"name":"foo1","columns":["val"],"points":[[24]]}]]

in_forward can't accept this format correctly.

This is why InfluxDB raises an error. See in_forward document:

http://docs.fluentd.org/articles/in_forward#protocol