Errors trying to send dstat data to influxdb

Clarence H

unread,

Sep 14, 2014, 1:51:36 AM9/14/14

to flu...@googlegroups.com

Hi,

I'm just getting started and I cant seem to get the fluentd process to pass the dstat data to the database. It looks like its a HTTP problem but the firewall is disabled and I can update the database using curl.

The Error Message in the logs:

2014-09-14 01:13:58 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2014-09-14 01:14:13 -0400 error_class="Net::HTTPServerException" error="400 \"Bad Request\"" instance=-612354068

2014-09-14 01:13:58 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in `error!'

2014-09-14 01:13:58 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http/response.rb:128:in `value'

2014-09-14 01:13:58 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-influxdb-0.1.2/lib/fluent/plugin/out_influxdb.rb:50:in `write'

2014-09-14 01:13:58 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/buffer.rb:296:in `write_chunk'

2014-09-14 01:13:58 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/buffer.rb:276:in `pop'

2014-09-14 01:13:58 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/output.rb:311:in `try_flush'

2014-09-14 01:13:58 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/output.rb:132:in `run'

My setup:

type tdlog

apikey YOUR_API_KEY

auto_create_table

buffer_type file

buffer_path /var/log/td-agent/buffer/td

</match>

type stdout

</match>

type forward

</source>

type http

port 8888

</source>

type debug_agent

bind 127.0.0.1

port 24230

</source>

type dstat

tag input.dstat

option -c

delay 5

</source>

#option -cdmsy

type influxdb

host 192.168.1.6

port 8086

dbname dstat

user xxxx

password xxxx

time_precision s

</match>

-----------------

Wireshare Packet Trace:

POST /db/dstat/series?u=xxxx&p=xxxx&time_precision=s HTTP/1.1

Content-Type: application/json; charset=utf-8

Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3

Accept: */*

User-Agent: Ruby

Connection: close

Host: 192.168.1.6:8086

Content-Length: 2571

[{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"1.148","sys":"0.863","idl":"97.556","wai":"0.135","hiq":"0.049","siq":"0.249"}},1410673009]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"3.441","sys":"0.810","idl":"95.749","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673009]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.202","sys":"0.202","idl":"99.194","wai":"0.0","hiq":"0.0","siq":"0.403"}},1410673014]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.202","idl":"99.798","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673019]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.202","sys":"0.0","idl":"99.798","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673024]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.202","idl":"99.597","wai":"0.0","hiq":"0.202","siq":"0.0"}},1410673029]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.0","idl":"100.0","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673034]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.203","idl":"99.797","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673039]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.202","idl":"99.798","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673044]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.0","idl":"100.0","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673049]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.0","idl":"100.0","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673054]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.201","sys":"0.201","idl":"99.598","wai":"0.0","hiq":"0.0","siq":"0.0"}},1410673059]]},{"name":"input.dstat","columns":["hostname","dstat","time"],"points":[["new-host-3",{"total cpu usage":{"usr":"0.0","sys":"0.0","idl":"99.798","wai":"0.0","hiq":"0.202","siq":"0.0"}},1410673064]]}]

====>

HTTP/1.1 400 Bad Request

Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept

Access-Control-Allow-Methods: GET, POST, PUT, DELETE

Access-Control-Allow-Origin: *

Access-Control-Max-Age: 2592000

Content-Type: text/plain

Date: Sun, 14 Sep 2014 05:37:45 GMT

Content-Length: 36

Connection: close

Unknown type map[string]interface {}

====

This command works:

curl -X POST -d '[{"name":"foo","columns":["val"],"points":[[23]]}]' 'http://localhost:8086/db/dstat/series?u=xxxx&p=xxxx'

Kiyoto Tamura

unread,

Sep 14, 2014, 9:33:54 AM9/14/14

to flu...@googlegroups.com

Hi Clarence,

Thanks for trying Fluentd =)

>2014-09-14 01:13:58 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2014-09-14 01:14:13 -0400 error_class="Net::

HTTPServerException" error="400 \"Bad Request\"" instance=-612354068

This message shows up because curl is hitting the in_http endpoint with an unexpected data format. in_http expects a JSON or MessagePack payload in either the URL parameter or HTTP message body. See docs.fluentd.org/articles/in_http

in_dstat is independent of in_http, and it should be able to send data to InfluxDB with your current configuration.

Kiyoto

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Check out Fluentd, the open source data collector to unify log management.

Clarence H

unread,

Sep 14, 2014, 2:48:59 PM9/14/14

to flu...@googlegroups.com

I confused as to how to get dstat to send the data to influxdb. I think your telling me that it has to go through the in_http module so Im trying to do this.

dstat --> in_http --> influxdb

So am I trying to get this to work : (it doesnt though)

# --------------------- Start Here

type http

port 8887

tag input.dstat

</source>

type dstat

tag input.json

option -c

delay 5

</source>

type forward

port 8887

host localhost

body_size_limit 32m

keepalive_timeout 10s

</match>

#option -cdmsy

type influxdb

host localhost

port 8086

dbname dstat

user root

password root

time_precision s

</match>

# --------------------- End Here

Kiyoto Tamura

unread,

Sep 14, 2014, 3:54:46 PM9/14/14

to flu...@googlegroups.com

Hi Clarence-

>I confused as to how to get dstat to send the data to influxdb. I think your telling me that it has to go through the in_http module so Im trying to do this.

Sorry if I confused you. To be clear, you do NOT need to use in_http to get data from dstat to influxdb.

type dstat

tag input.dstat

option -c

delay 5

</source>

#option -cdmsy

type influxdb

host 192.168.1.6

port 8086

dbname dstat

user xxxx

password xxxx

time_precision s

</match>

That should work, as long as 192.168.1.6:8086 is reachable from the machine you are running Fluentd on. If the data is not showing up in InfluxDB, can you run Fluentd with "-vv" and share your Fluentd logs?

Kiyoto

Clarence H

unread,

Sep 14, 2014, 4:32:55 PM9/14/14

to flu...@googlegroups.com

Okay.. Gotcha.

I ran the daemon with the -vv option and I really didnt see much ... I think the dstat plugin is not formatting the data properly.

[root@new-host-3 td-agent]# tail -f /var/log/td-agent/td-agent.log

2014-09-14 16:16:27 -0400 [info]: fluent/engine.rb:107:block in configure: adding source type="dstat"

2014-09-14 16:16:27 -0400 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'dstat'

2014-09-14 16:16:27 -0400 [info]: fluent/engine.rb:124:block in configure: adding match pattern="td.*.*" type="tdlog"

2014-09-14 16:16:27 -0400 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'tdlog'

2014-09-14 16:16:27 -0400 [info]: fluent/engine.rb:124:block in configure: adding match pattern="debug.**" type="stdout"

2014-09-14 16:16:27 -0400 [info]: fluent/engine.rb:124:block in configure: adding match pattern="input.dstat" type="influxdb"

2014-09-14 16:16:27 -0400 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'influxdb'

2014-09-14 16:16:27 -0400 [info]: plugin/in_forward.rb:75:listen: listening fluent socket on 0.0.0.0:24224

2014-09-14 16:16:27 -0400 [debug]: plugin/in_http.rb:81:start: listening http on 0.0.0.0:8888

2014-09-14 16:16:27 -0400 [info]: plugin/in_debug_agent.rb:44:start: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"

2014-09-14 16:17:28 -0400 [warn]: fluent/output.rb:344:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2014-09-14 16:17:28 -0400 error_class="Net::HTTPServerException" error="400 \"Bad Request\"" instance=-608640758

2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in `error!'

2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http/response.rb:128:in `value'

2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-influxdb-0.1.2/lib/fluent/plugin/out_influxdb.rb:50:in `write'

2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/buffer.rb:296:in `write_chunk'

2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/buffer.rb:276:in `pop'

2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/output.rb:311:in `try_flush'

2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/output.rb:132:in `run'

2014-09-14 16:17:29 -0400 [warn]: fluent/output.rb:344:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2014-09-14 16:17:30 -0400 error_class="Net::HTTPServerException" error="400 \"Bad Request\"" instance=-608640758

If I take the string that in the wireshark capture file and try to send it via curl is fails. I think the dstat module is supposed to format so the it can be forwarded by the influxdb plugin. I don't think the influxdb is supposed to really do much with that data except place the database name and user credentials onto the request.

Soooooo.. I guess I need to to find out if I'm using the right version of the dstat module or see if there are some extra options to help with the formatting.

Clarence

Reply all

Reply to author

Forward