R client for Kafka, KStream, and KTable

[Bill Evans]

I'm working on a project that is already benefiting from Kafka in general. It would really benefit from some of the abstractions of KStream and potentially KTable, but for one hold-up: R. In other forums it has been recommended that I write what I need in a more compatible language; I'd like to avoid that discussion for the moment, based both on (1) the size of the current code base, and (2) my belief that the language chosen for a particular project should be based on many factors, of which speed-of-execution is just one.

I'm effectively creating a streaming topology through the use of multiple topics and nodes, where most nodes poll one topic, some poll multiple topics; some push to one topic, some push to multiple topics. There is definitely room for me to migrate from "R" to another language in some of the nodes, but since at least some of them are currently much better (for me) in R, the dilemma remains.

I'm currently using rkafka (https://cran.r-project.org/web/packages/rkafka/index.html) for accessing the kafka server. For many reasons, this driver is insufficient and needs to be updated. I might be able to put together another R package that is less problematic and provides at a minimum Consumer and Producer methods. The more I read about kafka streaming, however, I'm led to the conclusion that my project and R/Kafka development in general would seriously benefit from a client that speaks a little more than the basic interfaces.

(1) Is there a better method for accessing Kafka from R? I have considered writing a new one, if needed, with librdkafka (https://github.com/edenhill/librdkafka) as well as rJava and the native java clients.

(2) Does somebody (inside or outside of Confluent) have an existing client that speaks at least the basics of KStream and KTable? (Recognize that it will likely be single-threaded, therefore potentially glossing over some of the functionality.) (Beta is acceptable, and I'm to do a lion's share of coding on the R side.)

Thanks!

[Guozhang Wang]

Hello Bill,

Re 1), I think it is a good idea to consider building a new R client based on librdkafka. I am current not aware of any R clients that are in newer (0.8+) versions.

Re 2), are you interested in R client only, or any non-Java client?

Guozhang

[Bill Evans]

Though I readily admit that the Java route might be simpler and faster, I agree that a C-based solution has a bunch of appeal.

My interest is in efficiency and speed and correctness for an R client. Occasionally, rJava can be problematic, but perhaps most commonly when (a) multiple rJava-dependent packages are loaded simultaneously (with conflicting parameter requirements), and/or (b) users don't read the docs.

Do you by chance know if librdkafka directly supports KStream and KTable?

Thanks!

[Michael Noll]

Bill,

thanks for elaborating your use case(s).

> Do you by chance know if librdkafka directly supports KStream and KTable?

Librdkafka does currently not support any Kafka Streams related features (such as KStream or KTable).  The focus of librdkafka is to be a first-class C/C++ implementation of Kafka's "normal" producer and consumer clients.

A polyglot Kafka Streams is definitely on our radar at Confluent.  The reasons for that are similar to the reasons for us supporting the normal Kafka producer/consumer clients in languages other than Java, such as C/C++ via librdkafka (Magnus Edenhill, the author of librdkafka, is in our team), or Python via confluent-kafka-python [1].  That said, we don't have an ETA yet for non-Java implementations of Kafka Streams; it will definitely not be in this year.

Best,

Michael

[1] https://github.com/confluentinc/confluent-kafka-python

Thanks!

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/2930964c-7dfa-4154-98ee-fbcca2b2bdfb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Michael G. Noll | Product Manager | Confluent | +1 650.453.5860
Download Apache Kafka and Confluent Platform: www.confluent.io/download

[Bill Evans]

That's what I expected and feared, since it is relatively new. KStream and KTable aside, do you anticipate a native R client with vanilla producer/consumer functionality any time this year?

Thanks!

[Michael Noll]

Bill,

a native R client is not on our radar for this year.

-Michael

On Tue, Aug 2, 2016 at 10:34 PM, Bill Evans <8pawe...@gmail.com> wrote:

That's what I expected and feared, since it is relatively new. KStream and KTable aside, do you anticipate a native R client with vanilla producer/consumer functionality any time this year?

Thanks!

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/a5618dfa-1048-40f6-ba5e-ca52eef1b09f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dmitriy Selivanov]

Hi there. Does anyone have working solution for R<->kafka interface?

среда, 3 августа 2016 г., 13:36:22 UTC+4 пользователь Michael Noll написал:

Bill,

a native R client is not on our radar for this year.

-Michael

On Tue, Aug 2, 2016 at 10:34 PM, Bill Evans <8pawe...@gmail.com> wrote:

That's what I expected and feared, since it is relatively new. KStream and KTable aside, do you anticipate a native R client with vanilla producer/consumer functionality any time this year?

Thanks!

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/a5618dfa-1048-40f6-ba5e-ca52eef1b09f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dmitriy Selivanov]

Simple consumer/producer interface with rscala - https://github.com/dselivanov/rscalaKafka.
For my needs it is enough for the moment (and I created it mainly because it was very simple). But of course would be great to have native client.

воскресенье, 5 марта 2017 г., 17:09:34 UTC+4 пользователь Dmitriy Selivanov написал:

[Michael Noll]

Thanks for sharing this, Dmitriy!

-Michael

On Wed, Mar 15, 2017 at 2:54 PM, Dmitriy Selivanov <selivano...@gmail.com> wrote:

Simple consumer/producer interface with rscala - https://github.com/dselivanov/rscalaKafka.
For my needs it is enough for the moment (and I created it mainly because it was very simple). But of course would be great to have native client.

воскресенье, 5 марта 2017 г., 17:09:34 UTC+4 пользователь Dmitriy Selivanov написал:

Hi there. Does anyone have working solution for R<->kafka interface?

среда, 3 августа 2016 г., 13:36:22 UTC+4 пользователь Michael Noll написал:

Bill,

a native R client is not on our radar for this year.

-Michael

On Tue, Aug 2, 2016 at 10:34 PM, Bill Evans <8pawe...@gmail.com> wrote:

That's what I expected and feared, since it is relatively new. KStream and KTable aside, do you anticipate a native R client with vanilla producer/consumer functionality any time this year?

Thanks!

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/a5618dfa-1048-40f6-ba5e-ca52eef1b09f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Michael G. Noll | Product Manager | Confluent | +1 650.453.5860
Download Apache Kafka and Confluent Platform: www.confluent.io/download

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/07888cd8-3c51-4567-a778-3fb4fd375bfe%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Michael G. Noll

Product Manager | Confluent

+1 650 453 5860 | @miguno

Follow us: Twitter | Blog