Riemann and Kakfa

[The Gun]

Since Riemann processes at an impressive high rate, has anone tried or written a kafka queue lient before it actually goes to other systems like graphite or elastic search?

Basically, is a riemann-kafka client available? I do not see it on the site.

If not kafka, any other queuing system ?

Thanks

Gunjan

[Aphyr]

On 04/02/2014 08:22 PM, The Gun wrote:
> Since Riemann processes at an impressive high rate, has anone tried or
> written a kafka queue lient before it actually goes to other systems
> like graphite or elastic search?
> Basically, is a riemann-kafka client available? I do not see it on the site.

Drop https://github.com/pingles/clj-kafka/ on your classpath, require it
from the config, and write a stream like

(streams
(fn [event]
(send-message my-kafka-producer
(message "event"
(.getBytes
(riemann.common/event-to-json event))))))

--Kyle

[Pierre-Yves Ritschard]

or you could just use the https://github.com/pyr/riemann-kafka plugin !

--
You received this message because you are subscribed to the Google Groups "Riemann Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to riemann-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pierre-Yves Ritschard]

I'll add that you want to make sure kafka makes sense in your infrastructure, since it may mean reading old messages. It doesn't make sense for a kafka messages to end up in time windows for instance.

[The Gun]

Great we have plugin(s) like thiese.

I am still to finalize on whether I will want to use Kafka.

Kyle, Pierre

Do we have something similar for simply pumping data into elastic search.

Note; I do not want to use kiries. It has been a pain making it work.

Thanks

Gunjan

To unsubscribe from this group and stop receiving emails from it, send an email to riemann-user...@googlegroups.com.

[Aphyr]

On 04/03/2014 10:30 PM, The Gun wrote:
> Great we have plugin(s) like thiese.
>
> I am still to finalize on whether I will want to use Kafka.
>
> Kyle, Pierre
> Do we have something similar for simply pumping data into elastic search.
> Note; I do not want to use kiries. It has been a pain making it work.

You can use any Clojure or Java library in Riemann. I suggest
https://github.com/clojurewerkz/elastisch

--Kyle

[Norman Richards]

On Fri, Apr 4, 2014 at 12:30 AM, The Gun <thapliy...@gmail.com> wrote:

Note; I do not want to use kiries. It has been a pain making it work.

Could you give some feedback on what isn't working well for you with Kiries.  We're using it quite successfully for our use case, but without feedback it's hard to know what troubles other people are having so we can improve it.

[The Gun]

Kyle

Using elasticsh

I have the following in my config

(load-plugins)

; Keep events in the index for 5 minutes by default.

(let [index (default :ttl 300 (update-index (index)))]

  ; Inbound events will be passed to these streams:

  (streams

    ; Index all events immediately.

    index

    ; Calculate an overall rate of events.

    (with {:metric 1 :host nil :state "ok" :service "events/sec"}

      (rate 5 index))

    ; Log expired events.

    (expired

      (fn [event] (info "expired" event)))

      (fn [event]

    (send-message elastisch        

                 (message "event"

                          (.getBytes 

                            (riemann.common/event-to-json event))))))

))

I get this clojure error

ERROR [2014-04-05 11:49:39,045] main - riemann.bin - Couldn't start

java.lang.RuntimeException: Unable to resolve symbol: send-message in this context, compiling:(/home/lab/riemann-0.2.4/bin/../etc/riemann.config:36:5)

at clojure.lang.Compiler.analyze(Compiler.java:6380)

at clojure.lang.Compiler.analyze(Compiler.java:6322)

at clojure.lang.Compiler$InvokeExpr.parse(Compiler.java:3573)

at clojure.lang.Compiler.analyzeSeq(Compiler.java:6562)

Also looks like the plugin hasnt loaded.I have checked out the elastisch project and added this to my .bash_profile

PATH=$PATH:/my/esplugin/elastisch/trunk

PATH=$PATH:/my/esplugin/elastisch/trunk/src

export PATH

Thanks

Gunjan

[The Gun]

Norman , please refer to this for the errors I am seeing

https://groups.google.com/forum/#!topic/riemann-users/EriBWSNdp3Y

Thanks

Gunjan

[Aphyr]

On 04/04/2014 11:23 PM, The Gun wrote:
> Kyle

> I get this clojure error

> /ERROR [2014-04-05 11:49:39,045] main - riemann.bin - Couldn't start/
> /java.lang.RuntimeException: Unable to resolve symbol: send-message in
> this context,

You need to require the library in order to use its functions in the
config namespace. Try

(require '[clojurewerkz.elastisch.native :as es])

and see

http://clojureelasticsearch.info/articles/getting_started.html

for details about how to work with elastisch.

--Kyle

[The Gun]

While I was looking to get through, reached that page but yet to make those changes. Excuse me for my poor knowledge of clojure.

Thanks

Gunjan

[The Gun]

I get this on doing a require

Couldn't start

java.lang.ClassNotFoundException: clojurewerkz.elastisch.native, compiling:(/home/lab/riemann-0.2.4/bin/../etc/riemann.config:1:1)

Note I have added the elastisch to the PATH variable. Also I have copied the clojurewerkz folder to from where I am executing riemann.

Thanks

Gunjan

[Aphyr]

On 04/05/2014 10:15 PM, The Gun wrote:
> I get this on doing a require
> Couldn't start
> java.lang.ClassNotFoundException: clojurewerkz.elastisch.native,
> compiling:(/home/lab/riemann-0.2.4/bin/../etc/riemann.config:1:1)
>
> Note I have added the elastisch to the PATH variable. Also I have copied
> the clojurewerkz folder to from where I am executing riemann.

PATH is for unix commands; classpath is for Java libraries. I usually
use -cp as an argument to java.

--Kyle

[The Gun]

Yes, I had also set CLASSPATH in the profile. When executing riemann, how can I add classpaths on executing riemann? Does the command bin/riemann follow with arguments also?

I can find good number of clojure plugins that riemann can use, but still looking for step by step of using a simple plugin in riemann.

Thanks

Gunjan

[Aphyr]

On 04/05/2014 11:31 PM, The Gun wrote:
> Yes, I had also set CLASSPATH in the profile. When executing riemann,
> how can I add classpaths on executing riemann? Does the command
> bin/riemann follow with arguments also?
>
> I can find good number of clojure plugins that riemann can use, but
> still looking for step by step of using a simple plugin in riemann.

Depends on which package you're using. For Debian, the most
straightforward way is to set EXTRA_CLASSPATH in /etc/default/riemann. See
https://github.com/aphyr/riemann/blob/master/pkg/deb/riemann#L7, or the
corresponding scripts for whichever distribution you're using.

--Kyle

[Aphyr]

On 04/05/2014 11:35 PM, Aphyr wrote:
> On 04/05/2014 11:31 PM, The Gun wrote:
>> Yes, I had also set CLASSPATH in the profile. When executing riemann,
>> how can I add classpaths on executing riemann? Does the command
>> bin/riemann follow with arguments also?

BTW, for rapid-fire questions like this, may I suggest joining #riemann
on freenode? There are quite a few people on this list, and I'd like to
keep their email volume down. :)

--Kyle

[The Gun]

I am using the RPM package on Redhat. Downloaded the latest 0.2.4-1 yesterday. It has the jar amd also the  file you mentioned with EXTRA_CLASSPATH. Uncommenting that line and putting it like this  in

riemann/etc/default/riemann-default

# Optionally add classes to the classpath for additional functionality

EXTRA_CLASSPATH=/my/esplugin/elastisch/trunk/src

# Optional JAVA_OPTS

# EXTRA_JAVA_OPTS=

I am executing 

java -jar riemann.jar

Tried

bin/riemann

Still get this

java.io.FileNotFoundException: Could not locate clojurewerkz/elastisch/native__init.class or clojurewerkz/elastisch/native.clj on classpath: 

at clojure.lang.RT.load(RT.java:443)

Will look for answers on freenode as well.

Thanks

Gunjan