Kafka client as a daemon

[Shekar Tippur]

Hello,

I am looking for a solution to run kafka client as a daemon. Appreciate if you could point me to any resource to this effect.

Thanks,

Shekar

[Joe Stein]

This is a good one that does that https://github.com/tagged/bruce also https://github.com/stealthly/dropwizard-kafka-http for producing. For consuming if you are thinking you need a daemon it would better to understand your use case more those are almost always client libraries unless you use Storm and/or Spark.

/*******************************************

 Joe Stein

 Founder, Principal Consultant

 Big Data Open Source Security LLC

 http://www.stealth.ly

 Twitter: @allthingshadoop

********************************************/

--
You received this message because you are subscribed to the Google Groups "kafka-clients" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kafka-client...@googlegroups.com.
To post to this group, send email to kafka-...@googlegroups.com.
Visit this group at http://groups.google.com/group/kafka-clients.
To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/c8da39a9-6133-4295-9d5b-21d6b044cd31%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joe Stein]

Or Samza http://samza.incubator.apache.org/ :)

[Shekar Tippur]

Joe,

Thanks for the swift response. I was looking for a non-java solution (I should have been more clear in my earlier post)

I looked at https://github.com/tagged/bruce - Looks to be a good solution for producers (similar to fluentd).

- Shekar

[Joe Stein]

We plan on adding a http end point into
https://github.com/stealthly/go_kafka_client later next week unless a pull request for that comes in first.

/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
********************************************/

--
You received this message because you are subscribed to the Google Groups "kafka-clients" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kafka-client...@googlegroups.com.
To post to this group, send email to kafka-...@googlegroups.com.
Visit this group at http://groups.google.com/group/kafka-clients.

To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/55b64007-c287-4ce0-a5de-b75e0f682ed0%40googlegroups.com.

[Andrew Otto]

Are you looking for a producer or a consumer?

If a producer, check out:  varnishkafka: https://github.com/wikimedia/varnishkafka

Simple to: set up varnish as a web server and then varnishkafka to send specific http requests formatted however you like to kafka.

If a consumer, check out: kafkatee: https://github.com/wikimedia/analytics-kafkatee

  Configurable multi topic in -> multi file or piped process output daemon.  

--
You received this message because you are subscribed to the Google Groups "kafka-clients" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kafka-client...@googlegroups.com.
To post to this group, send email to kafka-...@googlegroups.com.
Visit this group at http://groups.google.com/group/kafka-clients.

To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/55b64007-c287-4ce0-a5de-b75e0f682ed0%40googlegroups.com.

[Joe Stein]

You could also just pipe to and from kafka with https://github.com/edenhill/kafkacat that works cleanly too.

To view this discussion on the web visit https://groups.google.com/d/msgid/kafka-clients/7975BF33-7F80-4D0D-858D-F7D2B7C45791%40wikimedia.org.

[Shekar Tippur]

Thanks a lot for this information. Appreciate it.

- Shekar

[iman teimoornegad]

you can run it as a crontab in any linux dist.