Influxdb 0.9 and riemann configuration

Tomeu Capó Capó

unread,

Aug 11, 2015, 8:21:33 AM8/11/15

to Riemann Users

Hi

I using Riemann and InfluxDB and update recently both to Rieman 0.2.10 and Influx 0.9.2, and need change configuration for using new version of InfluxDB into Riemann:

Before updating have a 0.8 influxdb version and this function for send asynchonously events to Influx:

(require 'capacitor.core)
(require 'capacitor.async)
(require 'clojure.core.async)

(defn make-async-influxdb-client [opts]
(let [client (capacitor.core/make-client opts)

;; Make a channel to buffer influxdb events
events-in (capacitor.async/make-chan)

;; Make a channel to collect influxdb responses (ignored, really)
resp-out (capacitor.async/make-chan)]

;; Start the run loop with a batch size of max 100 events and max 10
;; seconds

(capacitor.async/run! events-in resp-out client 100 10000)

(fn [series payload]
(let [p (merge payload {
:series series
:time (* 1000 (:time payload)) ;; s → ms
:value (:value payload)
})]
(clojure.core.async/put! events-in p))
)
)
)

;; Connexio a la bbdd InfluxDB

(def influx (make-async-influxdb-client {
:host "localhost"
:port 8086
:username "usr"
:password "pass"
:db "devel_metrics"
}))

And now, using a client of influxdb:

(require 'capacitor.core)

(require 'capacitor.async)

(require 'clojure.core.async)

(require '[riemann.influxdb :as influxdb])

(defn make-async-influxdb-client [opts]

(let [client (influxdb/influxdb opts)

events-in (capacitor.async/make-chan)

resp-out (capacitor.async/make-chan)]

;; Start the run loop with a batch size of max 100 events and max 10

;; seconds

(capacitor.async/run! events-in resp-out client 100 10000)

(fn [series payload]

(let [p (merge payload {

:series series

:time (* 1000 (:time payload)) ;; s → ms

:value (:value payload)

})]

(clojure.core.async/put! events-in p)

(info (format "Sending %s" (str p)))

)

(def influx (make-async-influxdb-client {

:host "inxlux-extra.env"

:port 8086

:version :0.9

:username "x"

:password "x"

:db "devel_metrics"

}))

But now, riemann recevie envents normaly from different sources (such as riemann-tools) but don't save anything into InfluxDB 0.9, put DEBUG level of riemann but don't show any errors.

Can help me?

Thanks for advance.

Brendan Tobolaski

unread,

Aug 11, 2015, 1:23:35 PM8/11/15

to Riemann Users

Hi,

I have to say, I'm a bit puzzled. From what I gather, you're 'influx is a function with two arguments, 'series and 'payload, so you must be using something to prep riemann events before passing them to it. Perhaps if you gave an example of how you use 'influx, I could be of more help.

On a related topic, why are you using capacitor directly? Riemann has influxdb wrapped into a more convenient function. Now to toot my own horn a bit, I have a similar function based around Riemann's built in 'influxdb function that uses batched asynchronous sends, https://ruin.io/2015/capturing-influxdb-send-errors-riemann/ . Its a bit more complicated due to it also capturing and alerting on any exceptions generated by sending events to InfluxDB.

Tomeu Capó Capó

unread,

Aug 11, 2015, 1:37:25 PM8/11/15

to Riemann Users

Hi and thanks :)

4 months ago I'm using the Riemann and components are not yet fully mastered. I'm reading the API documentation and examples github, so I consider this new aspect. Regarding use of asynchronous capacitor to ship to influxdb, had examples like: https://github.com/aphyr/riemann/issues/411 and working with version 0.8 of Influxdb.

Anyway the problem has arisen with version 0.9, which is likely if the client use influxdb directly together with the batch function is not necessary to use the capacitor.

The solution you mention I met her and change a little, but fails to run me. Another problem is that I do not know if you have problems sending batches to Influx or has a connection problem with Influx, and logs Influxdb riemann or not output any error. I find it hard to detect the problem.

Any idea about the role you mention me using batch?

Thanks!

Tomeu Capó Capó

unread,

Aug 11, 2015, 1:58:26 PM8/11/15

to Riemann Users

Hi, again.

Now I have a riemann server and influxdb cluster. Applications and hosts send events to Riemann, with tag (environment tag) for example, riemann receive this event:

{:host myhost.domain, :service MyService, :state ok, :metric 1.0, :tags [devel], :time 1439315649, :ttl 30}

In my configuration have a stream processing and gave first tag for define "environment" and influxdb destination database, have a database for each environment and save metric into series like:

myhost.domain.MyService

Until now, using a capacitor working fine with InfluxDB 0.8. But when upgraded to 0.9 doesn't work. And researching way for solve this issue.

Thanks again.

On Tuesday, August 11, 2015 at 7:23:35 PM UTC+2, Brendan Tobolaski wrote:

Kyle Kingsbury

unread,

Aug 11, 2015, 2:00:13 PM8/11/15

to rieman...@googlegroups.com

On 08/11/2015 10:58 AM, Tomeu Capó Capó wrote:
> Hi, again.
>
> Now I have a riemann server and influxdb cluster. Applications and hosts
> send events to Riemann, with tag (environment tag) for example, riemann
> receive this event:
>

> *{:host myhost.domain, :service MyService, :state ok, :metric 1.0, :tags
> [devel], :time 1439315649, :ttl 30}*

>
> In my configuration have a stream processing and gave first tag for
> define "environment" and influxdb destination database, have a database
> for each environment and save metric into series like:
>

> *myhost.domain.MyService*

>
> Until now, using a capacitor working fine with InfluxDB 0.8. But when
> upgraded to 0.9 doesn't work. And researching way for solve this issue.

Influx's API changed (twice!) in the shift to 0.9. The latest Riemann
has support for the new 0.9 API; just pass :version :0.9.

https://github.com/aphyr/riemann/blob/master/src/riemann
/influxdb.clj#L217

--Kyle

Tomeu Capó Capó

unread,

Aug 12, 2015, 2:49:27 AM8/12/15

to Riemann Users

Hi

At first, before upgrade influxdb 0.9, upgraded riemann to 0.2.10 because I readed all change logs and blogs about this: influxdb 0.9 protocol changed!! But, 0.2.10 riemann version still doesn't support 0.9 influxdb api?? In the changelog says YES. Or need make patch over my 0.2.10 riemann version???

I do something wrong into my riemann configuration using influxdb client, I think. But at the past, using a Influx 0.8 with capacitor async client and works, and now, doesn't needs and change to combine batch with influxdb new functions.

Still researching for my error on my riemann.config...

Any ideas?

Thanks for advance

Tomeu Capó Capó

unread,

Aug 12, 2015, 10:15:23 AM8/12/15

to Riemann Users

Hi, again.

Now put to DEBUG and use different configuration for InfluxDB riemann client, and the output is:

DEBUG [2015-08-12 16:13:02,652] pool-1-thread-1 - org.apache.http.impl.client.DefaultHttpClient - Attempt 1 to execute request

DEBUG [2015-08-12 16:13:02,652] pool-1-thread-1 - org.apache.http.impl.conn.DefaultClientConnection - Sending request: POST /db/bedsbank_test_metrics/series?u=riemann&p=riemann&time_precision=s HTTP/1.1

DEBUG [2015-08-12 16:13:02,652] pool-1-thread-1 - org.apache.http.wire - >> "POST /db/bedsbank_test_metrics/series?u=riemann&p=riemann&time_precision=s HTTP/1.1[\r][\n]"

DEBUG [2015-08-12 16:13:02,652] pool-1-thread-1 - org.apache.http.wire - >> "Connection: close[\r][\n]"

DEBUG [2015-08-12 16:13:02,652] pool-1-thread-1 - org.apache.http.wire - >> "content-type: application/json[\r][\n]"

DEBUG [2015-08-12 16:13:02,652] pool-1-thread-1 - org.apache.http.wire - >> "accept-encoding: gzip, deflate[\r][\n]"

DEBUG [2015-08-12 16:13:02,652] pool-1-thread-1 - org.apache.http.wire - >> "Content-Length: 172[\r][\n]"

DEBUG [2015-08-12 16:13:02,652] pool-1-thread-1 - org.apache.http.wire - >> "Host: inxlux-extra.env:8086[\r][\n]"

DEBUG [2015-08-12 16:13:02,652] pool-1-thread-1 - org.apache.http.wire - >> "User-Agent: Apache-HttpClient/4.3.5 (java 1.5)[\r][\n]"

Using this influxdb client configuration, force 0.9 version but http scheme...

(influxdb {:host "inxlux-extra.env"

:version :0.9

:scheme "http"

:port "8086"

:db "bedsbank_test_metrics"

:username "riemann"

:password "riemann"

:tag-fields #{:host :environment}})

Tomeu Capó Capó

unread,

Aug 12, 2015, 10:36:50 AM8/12/15

to Riemann Users

Sorry at all ...

URL for push metrics, changed /db/bedsbank_test_metrics/series?u=riemann&p=riemann&time_precision=s to /write?db=bedsbank_test_metrics

Now using Riemann 0.2.10, and I downloaded latest snapshot version (0.2.11) and influx.clj supports new URL ...

!!! ...WTF... InfluxDB protocol !!!

Thanks and sorry, Compile and test with riemann 0.2.11-snapshot if it works ... :P

Tomeu Capó Capó

unread,

Aug 12, 2015, 2:08:16 PM8/12/15

to Riemann Users

Compiled and install new version of riemann-0.2.11.20150812.192729-1, installs well and startups well. Configure batch sender and influxdb connection, and simple stream:

(def influxBatchSender

(batch 100 1/10

(async-queue! :agg {:queue-size 1000

:core-pool-size 1

:max-pool-size 4

:keep-alive-time 60000}

(let [send-influx (influxdb {:host "inxlux-extra.env"

:version :0.9

:scheme "http"

:port "8086"

:db "bedsbank_test_metrics"

:username "riemann"

:password "riemann"

})]

(fn influx-sending [event]

(send-influx event)

)

)))

(let [index (index)]

(streams

(default :ttl 60

index

#(info %)

influxBatchSender

)

But now appears new error:

WARN [2015-08-12 19:55:02,678] pool-1-thread-1 - riemann.streams - riemann.config$fn__20052$influx_sending__20053@3800875b threw

java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.CharSequence

at clojure.string$escape.invoke(string.clj:307)

at riemann.influxdb$replace_disallowed_9.invoke(influxdb.clj:19)

at riemann.influxdb$kv_encode_9$fn__18869.invoke(influxdb.clj:23)

at clojure.core$map$fn__481.invoke(core.clj:2559)

at clojure.lang.LazySeq.sval(LazySeq.java:40)

at clojure.lang.LazySeq.seq(LazySeq.java:49)

at clojure.lang.LazySeq.first(LazySeq.java:71)

at clojure.lang.RT.first(RT.java:577)

at clojure.core$first.invoke(core.clj:55)

at clojure.string$join.invoke(string.clj:185)

at riemann.influxdb$kv_encode_9.invoke(influxdb.clj:22)

at riemann.influxdb$lineprotocol_encode_9.invoke(influxdb.clj:26)

at clojure.core$map$fn__481.invoke(core.clj:2557)

at clojure.lang.LazySeq.sval(LazySeq.java:40)

at clojure.lang.LazySeq.seq(LazySeq.java:49)

at clojure.lang.RT.seq(RT.java:484)

at clojure.core$seq.invoke(core.clj:133)

at clojure.core$apply.invoke(core.clj:624)

at clojure.string$join.invoke(string.clj:183)

at riemann.influxdb$lineprotocol_encode_list_9.invoke(influxdb.clj:36)

at riemann.influxdb$influxdb_9$stream__18910.invoke(influxdb.clj:196)

at riemann.config$fn__20052$influx_sending__20053.invoke(influxdb.clj:25)

at riemann.streams$execute_on$stream__10081$runner__10082$fn__10087.invoke(streams.clj:268)

at riemann.streams$execute_on$stream__10081$runner__10082.invoke(streams.clj:268)

at clojure.lang.AFn.applyToHelper(AFn.java:152)

at clojure.lang.AFn.applyTo(AFn.java:144)

at clojure.core$apply.invoke(core.clj:624)

at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1862)

at clojure.lang.RestFn.invoke(RestFn.java:425)

at clojure.lang.AFn.applyToHelper(AFn.java:156)

at clojure.lang.RestFn.applyTo(RestFn.java:132)

at clojure.core$apply.invoke(core.clj:628)

at clojure.core$bound_fn_STAR_$fn__360.doInvoke(core.clj:1884)

at clojure.lang.RestFn.invoke(RestFn.java:397)

at clojure.lang.AFn.run(AFn.java:22)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Maybe send's incorrect event format? have simple stream: index, log and send, still research ... Any ideas, still learning about Riemann and Clojoure I'm newby ... but learning ... ;)

excuse my state of stupidity ...

Thanks for advance

Lele Long

unread,

Dec 20, 2015, 7:41:14 AM12/20/15

to Riemann Users

riemann 0.2.10, influxdb 0.9.6.1, the following config is fine with me.

(def influxBatchSender
        (batch 100 1/10
               (async-queue! :agg {:queue-size 1000
                                   :core-pool-size 1
                                   :max-pool-size 4
                                   :keep-alive-time 60000}


                  (influxdb {:host "localhost"
                             :version :0.9
                             :port "8086"
                             :db "riemann"
                             :username "root"
                             :password "root"}))))

Reply all

Reply to author

Forward