Combining Confluent Kafka Connect with Apache Flink vs Spark?

1,054 views
Skip to first unread message

Baden

unread,
Dec 29, 2016, 2:25:03 PM12/29/16
to Confluent Platform
I am new to the enterprise big data landscape and just getting to know the big players and confused with some of the names - https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/confluent-platform/Q0mE3eHeVk8/Sp2fji4eAwAJ - I meant Flink, not Tika.

In reading about Flink, I would like to use this in our data pipeline, rather than Spark.  Now, combining that with Confluent Kafka Stream and Connect - I see this article https://www.confluent.io/blog/apache-flink-apache-kafka-streams-comparison-guideline-users/ - but not a lot on combining them yet.  Is this a good way to go or should we still be looking at Spark vs Flink?



Ewen Cheslack-Postava

unread,
Dec 29, 2016, 2:41:26 PM12/29/16
to Confluent Platform
Flink will work just fine -- stream processing frameworks have their tradeoffs (e.g. see https://www.confluent.io/blog/apache-flink-apache-kafka-streams-comparison-guideline-users/), but Kafka Connect and stream processing frameworks complement each other. You can think of Kafka Connect as providing import/export and any stream processing layer as providing the transformation/processing engine.

-Ewen

On Thu, Dec 29, 2016 at 2:25 PM, Baden <bade...@gmail.com> wrote:
I am new to the enterprise big data landscape and just getting to know the big players and confused with some of the names - https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/confluent-platform/Q0mE3eHeVk8/Sp2fji4eAwAJ - I meant Flink, not Tika.

In reading about Flink, I would like to use this in our data pipeline, rather than Spark.  Now, combining that with Confluent Kafka Stream and Connect - I see this article https://www.confluent.io/blog/apache-flink-apache-kafka-streams-comparison-guideline-users/ - but not a lot on combining them yet.  Is this a good way to go or should we still be looking at Spark vs Flink?



--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/8bb0701e-7165-40aa-9084-a532b5838fbe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Matthias J. Sax

unread,
Jan 1, 2017, 11:04:20 AM1/1/17
to confluent...@googlegroups.com
If you decide to use Flink, could you let us know why? Kafka Streams
offers a lot of functionality and it would be helpful to know why you
decided not to use it in favor of Flink.


-Matthias
> send an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to
> confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/8bb0701e-7165-40aa-9084-a532b5838fbe%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/8bb0701e-7165-40aa-9084-a532b5838fbe%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
> <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
signature.asc

Baden

unread,
Jan 2, 2017, 11:57:52 AM1/2/17
to Confluent Platform
Good day,

It appears that I needed to do more research on Kafka Streams and these links really drove the message home:


One my my newly-found attractions to KStreams over Flink is the ability to embed the library in to any Java application managed by existing Kafka brokers not as a job in a Flink cluster.  This also simplifies our architecture in not needing an additional Flink layer.



>     To post to this group, send email to
>     confluent...@googlegroups.com
>     <mailto:confluent...@googlegroups.com>.
>     To view this discussion on the web visit
>     https://groups.google.com/d/msgid/confluent-platform/8bb0701e-7165-40aa-9084-a532b5838fbe%40googlegroups.com
>     <https://groups.google.com/d/msgid/confluent-platform/8bb0701e-7165-40aa-9084-a532b5838fbe%40googlegroups.com?utm_medium=email&utm_source=footer>.
>     For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

Matthias J. Sax

unread,
Jan 2, 2017, 1:51:28 PM1/2/17
to confluent...@googlegroups.com
Nice to hear :)

Just one remark:

If you write an application with Kafka Streams, the application can be
deployed anywhere (no need to put it on the same servers a the brokers
-- you could do this, but it is actually not really recommended.)

Just want to mention this, because of your comment about "any Java
application managed by existing Kafka brokers" -- not sure what you mean
by "managed" -- broker are involved via consumer group protocol of course.

-Matthias

On 1/2/17 8:57 AM, Baden wrote:
> Good day,
>
> It appears that I needed to do more research on Kafka Streams and these
> links really drove the message home:
>
> https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/
> https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/
>
> One my my newly-found attractions to KStreams over Flink is the ability
> to embed the library in to any Java application managed by existing
> Kafka brokers not as a job in a Flink cluster. This also simplifies our
> architecture in not needing an additional Flink layer.
>
>
>
>
>
> On Sunday, January 1, 2017 at 11:04:20 AM UTC-5, Matthias J. Sax wrote:
>
> If you decide to use Flink, could you let us know why? Kafka Streams
> offers a lot of functionality and it would be helpful to know why you
> decided not to use it in favor of Flink.
>
>
> -Matthias
>
>
> On 12/29/16 7:41 PM, Ewen Cheslack-Postava wrote:
> > Flink will work just fine -- stream processing frameworks have their
> > tradeoffs (e.g.
> > see
> https://www.confluent.io/blog/apache-flink-apache-kafka-streams-comparison-guideline-users/
> <https://www.confluent.io/blog/apache-flink-apache-kafka-streams-comparison-guideline-users/>),
>
> > but Kafka Connect and stream processing frameworks complement each
> > other. You can think of Kafka Connect as providing import/export
> and any
> > stream processing layer as providing the transformation/processing
> engine.
> >
> > -Ewen
> >
> > On Thu, Dec 29, 2016 at 2:25 PM, Baden <bade...@gmail.com
> <javascript:>
> confluent-platf...@googlegroups.com <javascript:>
> > <mailto:confluent-platf...@googlegroups.com
> <javascript:>>.
> > To post to this group, send email to
> > confluent...@googlegroups.com <javascript:>
> > <mailto:confluent...@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/8bb0701e-7165-40aa-9084-a532b5838fbe%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/8bb0701e-7165-40aa-9084-a532b5838fbe%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/8bb0701e-7165-40aa-9084-a532b5838fbe%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/8bb0701e-7165-40aa-9084-a532b5838fbe%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Confluent Platform" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to confluent-platf...@googlegroups.com
> <javascript:>
> > <mailto:confluent-platf...@googlegroups.com
> <javascript:>>.
> > To post to this group, send email to confluent...@googlegroups.com
> <javascript:>
> > <mailto:confluent...@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
> <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer>.
signature.asc

Baden

unread,
Jan 2, 2017, 2:00:14 PM1/2/17
to Confluent Platform
Since KStreams offers a RESTful API, could the KStreams application be written in a different framework, such as .Net?  Or is it suggested to write everything in Java using the libraries rather than RESTful interfaces?

I haven't found hard numbers that compare KStreams to Flink's message bandwidth (some say 15 million / second, some say tens of millions per second - for Flink).  Also want to confirm that KStreams is not micro batch processing, as near-realtime would be required for clickstream type data.
>     <javascript:>>.
>     >     To post to this group, send email to
>     >     confluent...@googlegroups.com <javascript:>
>     >     <mailto:confluent...@googlegroups.com <javascript:>>.
>     >     To view this discussion on the web visit
>     >    
>     https://groups.google.com/d/msgid/confluent-platform/8bb0701e-7165-40aa-9084-a532b5838fbe%40googlegroups.com
>     <https://groups.google.com/d/msgid/confluent-platform/8bb0701e-7165-40aa-9084-a532b5838fbe%40googlegroups.com>
>
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/8bb0701e-7165-40aa-9084-a532b5838fbe%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/8bb0701e-7165-40aa-9084-a532b5838fbe%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>     >     For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>
>     >     <https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>>.
>     >
>     >
>     > --
>     > You received this message because you are subscribed to the Google
>     > Groups "Confluent Platform" group.
>     > To unsubscribe from this group and stop receiving emails from it,
>     send
>     > an email to confluent-platform+unsub...@googlegroups.com
>     <javascript:>
>     > <mailto:confluent-platform+unsub...@googlegroups.com
>     <javascript:>>.
>     > To post to this group, send email to confluent...@googlegroups.com
>     <javascript:>
>     > <mailto:confluent...@googlegroups.com <javascript:>>.
>     > To view this discussion on the web visit
>     >
>     https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>
>
>     >
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>>.
>
>     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

Matthias J. Sax

unread,
Jan 2, 2017, 2:23:21 PM1/2/17
to confluent...@googlegroups.com
Kafka Streams does not offer a REST API... Not sure what you are
referring to?


About benchmarks:

DataArtisans published some benchmarks on their blog:
http://data-artisans.com/extending-the-yahoo-streaming-benchmark/

There is no official benchmark for Kafka Streams yet, but it used
KafkaConsumer and KafkaProducer internally and there are benchmarks for
those. Even if Kafka Streams put some overhead compared to plain
Consumer/Producer the number should give you some comparison.

https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines


And yes, Kafka Streams offers real record-by-record semantics -- it's
NOT micro batching and latency is in the range of milliseconds and thus
comparable what Flink or Storm offers.


-Matthias
> <https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/>
>
> >
> https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/
> > confluent-platf...@googlegroups.com <javascript:>
> <javascript:>
> > > <mailto:confluent-platf...@googlegroups.com
> <javascript:>
> > > an email to confluent-platf...@googlegroups.com
> <javascript:>
> > <javascript:>
> > > <mailto:confluent-platf...@googlegroups.com
> <javascript:>
> > <javascript:>>.
> > > To post to this group, send email to
> confluent...@googlegroups.com
> > <javascript:>
> > > <mailto:confluent...@googlegroups.com <javascript:>>.
> > > To view this discussion on the web visit
> > >
> >
> https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
> <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
> <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>>
>
> >
> > >
> >
> <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>>>.
>
> >
> > > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Confluent Platform" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to confluent-platf...@googlegroups.com
> <javascript:>
> > <mailto:confluent-platf...@googlegroups.com
> <javascript:>>.
> > To post to this group, send email to confluent...@googlegroups.com
> <javascript:>
> > <mailto:confluent...@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/e597a2ab-5415-423a-9f2c-1b76591a21e3%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/e597a2ab-5415-423a-9f2c-1b76591a21e3%40googlegroups.com?utm_medium=email&utm_source=footer>.
signature.asc

Baden

unread,
Jan 2, 2017, 4:05:54 PM1/2/17
to Confluent Platform
It may be possible that I confused the Confluent Kafka Connect REST API for having access to KStreams.




I take it that this has to access to KStreams?  Therefore, the only option is to use the Java libraries, correct?
>     >     confluent-platform+unsub...@googlegroups.com <javascript:>
>     <javascript:>
>     >     >     <mailto:confluent-platform+unsub...@googlegroups.com
>     >     > an email to confluent-platform+unsub...@googlegroups.com
>     <javascript:>
>     >     <javascript:>
>     >     > <mailto:confluent-platform+unsub...@googlegroups.com
>     <javascript:>
>     >     <javascript:>>.
>     >     > To post to this group, send email to
>     confluent...@googlegroups.com
>     >     <javascript:>
>     >     > <mailto:confluent...@googlegroups.com <javascript:>>.
>     >     > To view this discussion on the web visit
>     >     >
>     >    
>     https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>
>
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>>
>
>     >
>     >     >
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>>>.
>
>     >
>     >     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>
>     >     <https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>>.
>     >
>     > --
>     > You received this message because you are subscribed to the Google
>     > Groups "Confluent Platform" group.
>     > To unsubscribe from this group and stop receiving emails from it,
>     send
>     > an email to confluent-platform+unsub...@googlegroups.com
>     <javascript:>
>     > <mailto:confluent-platform+unsub...@googlegroups.com
>     <javascript:>>.
>     > To post to this group, send email to confluent...@googlegroups.com
>     <javascript:>
>     > <mailto:confluent...@googlegroups.com <javascript:>>.
>     > To view this discussion on the web visit
>     >
>     https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com>
>
>     >
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

Ewen Cheslack-Postava

unread,
Jan 2, 2017, 4:29:23 PM1/2/17
to Confluent Platform
Confluent's REST proxy provides access to some metadata and low-level client APIs (i.e. producer and consumer). Streams is not covered since it is a separate Java API that is expected to be compiled and deployed as a separate app.

There is support for Interactive Queries (http://docs.confluent.io/3.1.1/streams/developer-guide.html#interactive-queries) which allow you to expose the state of your streams app via an HTTP API. The feature doesn't provide an HTTP API directly, you build that on top of the API that gives you access to the current state of the app. So to use this feature in streams, you'd still implement a Java-based Streams application, but you could then expose its state to non-Java apps if you want to.

-Ewen

>     >     confluent-platform+unsubscribe@googlegroups.com <javascript:>
>     <javascript:>
>     >     >     <mailto:confluent-platform+unsubsc...@googlegroups.com
>     >     > an email to confluent-platform+unsubscribe@googlegroups.com
>     <javascript:>
>     >     <javascript:>
>     >     > <mailto:confluent-platform+unsubsc...@googlegroups.com
>     <javascript:>
>     >     <javascript:>>.
>     >     > To post to this group, send email to
>     confluent...@googlegroups.com
>     >     <javascript:>
>     >     > <mailto:confluent...@googlegroups.com <javascript:>>.
>     >     > To view this discussion on the web visit
>     >     >
>     >    
>     https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>
>
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>>
>
>     >
>     >     >
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>>>.
>
>     >
>     >     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>
>     >     <https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>>.
>     >
>     > --
>     > You received this message because you are subscribed to the Google
>     > Groups "Confluent Platform" group.
>     > To unsubscribe from this group and stop receiving emails from it,
>     send
>     > an email to confluent-platform+unsubscribe@googlegroups.com
>     <javascript:>
>     > <mailto:confluent-platform+unsubsc...@googlegroups.com
>     <javascript:>>.
>     > To post to this group, send email to confluent...@googlegroups.com
>     <javascript:>
>     > <mailto:confluent...@googlegroups.com <javascript:>>.
>     > To view this discussion on the web visit
>     >
>     https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com>
>
>     >
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/59e9cf14-af0d-4288-b008-985dcd03b2a7%40googlegroups.com.

Baden

unread,
Jan 2, 2017, 5:01:21 PM1/2/17
to Confluent Platform
As part of our data ingestion (other than streaming), meta data from files, RSS sources, etc. will be included via the ingest attachment processor plugin (Tika under the hood) for ElasticSearch.  Can I do this all with Streams - I was considering using this as a microservices broker rather than implementing a full ESB.  So a consumer would make the request out to the RSS feeds or S/FTP location for files and processing would occur after the stream is received.  This doesn't have to be real-time, so would connect be better for this?

I wouldn't want any direct access from the client (users setting up their projects) and the back-office layers (Streams, etc.)  Since a new stream project would require a new KStream I could expose a KStream API that the client side can communicate with in order to create new streams - yes or no?  This stream data would be based on whatever configurations are done by the user - all dynamic.  So at compile time, no streams will be known and added through application life-cycle.

I think I'm getting closer to understanding this and thank you for all the informative suggestions and direction.

>     >     confluent-platform+unsub...@googlegroups.com <javascript:>
>     <javascript:>
>     >     >     <mailto:confluent-platform+unsub...@googlegroups.com
>     >     > an email to confluent-platform+unsub...@googlegroups.com
>     <javascript:>
>     >     <javascript:>
>     >     > <mailto:confluent-platform+unsub...@googlegroups.com
>     <javascript:>
>     >     <javascript:>>.
>     >     > To post to this group, send email to
>     confluent...@googlegroups.com
>     >     <javascript:>
>     >     > <mailto:confluent...@googlegroups.com <javascript:>>.
>     >     > To view this discussion on the web visit
>     >     >
>     >    
>     https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>
>
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>>
>
>     >
>     >     >
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>>>.
>
>     >
>     >     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>
>     >     <https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>>.
>     >
>     > --
>     > You received this message because you are subscribed to the Google
>     > Groups "Confluent Platform" group.
>     > To unsubscribe from this group and stop receiving emails from it,
>     send
>     > an email to confluent-platform+unsub...@googlegroups.com
>     <javascript:>
>     > <mailto:confluent-platform+unsub...@googlegroups.com
>     <javascript:>>.
>     > To post to this group, send email to confluent...@googlegroups.com
>     <javascript:>
>     > <mailto:confluent...@googlegroups.com <javascript:>>.
>     > To view this discussion on the web visit
>     >
>     https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com>
>
>     >
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

Ewen Cheslack-Postava

unread,
Jan 3, 2017, 3:56:19 PM1/3/17
to Confluent Platform
On Mon, Jan 2, 2017 at 2:01 PM, Baden <bade...@gmail.com> wrote:
As part of our data ingestion (other than streaming), meta data from files, RSS sources, etc. will be included via the ingest attachment processor plugin (Tika under the hood) for ElasticSearch.  Can I do this all with Streams - I was considering using this as a microservices broker rather than implementing a full ESB.  So a consumer would make the request out to the RSS feeds or S/FTP location for files and processing would occur after the stream is received.  This doesn't have to be real-time, so would connect be better for this?

Streams processes data that is already in Kafka, and outputs data to Kafka (or makes it queryable via interactive queries). You could implement this with the low level clients (i.e. make the request for the data, then process/produce the data into Kafka). Or you could use Connect if you just want to load the raw data into Kafka.
 

I wouldn't want any direct access from the client (users setting up their projects) and the back-office layers (Streams, etc.)  Since a new stream project would require a new KStream I could expose a KStream API that the client side can communicate with in order to create new streams - yes or no?  This stream data would be based on whatever configurations are done by the user - all dynamic.  So at compile time, no streams will be known and added through application life-cycle.

Note that Kafka Streams is a stream processing library. I think what you're getting at is that you'd like users to have an API they can hit to create new topics? For that you would need to use the admin tools included with Kafka (or the new AdminClient Java API when that becomes available).
 

I think I'm getting closer to understanding this and thank you for all the informative suggestions and direction.

-Ewen
 

>     >     confluent-platform+unsubscribe@googlegroups.com <javascript:>
>     <javascript:>
>     >     >     <mailto:confluent-platform+unsubsc...@googlegroups.com
>     >     > an email to confluent-platform+unsubscribe@googlegroups.com
>     <javascript:>
>     >     <javascript:>
>     >     > <mailto:confluent-platform+unsubsc...@googlegroups.com
>     <javascript:>
>     >     <javascript:>>.
>     >     > To post to this group, send email to
>     confluent...@googlegroups.com
>     >     <javascript:>
>     >     > <mailto:confluent...@googlegroups.com <javascript:>>.
>     >     > To view this discussion on the web visit
>     >     >
>     >    
>     https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>
>
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>>
>
>     >
>     >     >
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>>>.
>
>     >
>     >     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>
>     >     <https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>>.
>     >
>     > --
>     > You received this message because you are subscribed to the Google
>     > Groups "Confluent Platform" group.
>     > To unsubscribe from this group and stop receiving emails from it,
>     send
>     > an email to confluent-platform+unsubscribe@googlegroups.com
>     <javascript:>
>     > <mailto:confluent-platform+unsubsc...@googlegroups.com
>     <javascript:>>.
>     > To post to this group, send email to confluent...@googlegroups.com
>     <javascript:>
>     > <mailto:confluent...@googlegroups.com <javascript:>>.
>     > To view this discussion on the web visit
>     >
>     https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com>
>
>     >
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

Baden

unread,
Jan 3, 2017, 9:13:17 PM1/3/17
to Confluent Platform
Thank you for the clarification on Streams.  Since we will want to connect to IoT producers it sounds like we would use Connect for that.  Now what about real-time in-flight data we want to process.  I thought this is where Streams comes in, but I think I'm still missing something on that.  That's why I thought Flink was a candidate for consuming this real-time data - it would get it into Kafka, then use Streams to ultimately get it into ElasticSearch?

Maybe an overall flow would help:

IoT devices, real-time data --- Flink --- Kafka --- ElasticSearch
RSS Feeds, database query outputs --- Kafka Connect --- ElasticSearch

Or Kafka sits right in between any external data sources and the back-office processes - taking the place of an ESB - is that correct?



As for dynamically creating topics, I found a few code samples online and this library that exposes a Swagger endpoint - https://github.com/trustedanalytics/kafka-admin-api



>     >     confluent-platform+unsub...@googlegroups.com <javascript:>
>     <javascript:>
>     >     >     <mailto:confluent-platform+unsub...@googlegroups.com
>     >     > an email to confluent-platform+unsub...@googlegroups.com
>     <javascript:>
>     >     <javascript:>
>     >     > <mailto:confluent-platform+unsub...@googlegroups.com
>     <javascript:>
>     >     <javascript:>>.
>     >     > To post to this group, send email to
>     confluent...@googlegroups.com
>     >     <javascript:>
>     >     > <mailto:confluent...@googlegroups.com <javascript:>>.
>     >     > To view this discussion on the web visit
>     >     >
>     >    
>     https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>
>
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>>
>
>     >
>     >     >
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>>>.
>
>     >
>     >     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>
>     >     <https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>>.
>     >
>     > --
>     > You received this message because you are subscribed to the Google
>     > Groups "Confluent Platform" group.
>     > To unsubscribe from this group and stop receiving emails from it,
>     send
>     > an email to confluent-platform+unsub...@googlegroups.com
>     <javascript:>
>     > <mailto:confluent-platform+unsub...@googlegroups.com
>     <javascript:>>.
>     > To post to this group, send email to confluent...@googlegroups.com
>     <javascript:>
>     > <mailto:confluent...@googlegroups.com <javascript:>>.
>     > To view this discussion on the web visit
>     >
>     https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com>
>
>     >
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

Ewen Cheslack-Postava

unread,
Jan 4, 2017, 1:26:29 AM1/4/17
to Confluent Platform
On Tue, Jan 3, 2017 at 6:13 PM, Baden <bade...@gmail.com> wrote:
Thank you for the clarification on Streams.  Since we will want to connect to IoT producers it sounds like we would use Connect for that.  Now what about real-time in-flight data we want to process.  I thought this is where Streams comes in, but I think I'm still missing something on that.  That's why I thought Flink was a candidate for consuming this real-time data - it would get it into Kafka, then use Streams to ultimately get it into ElasticSearch?

Maybe an overall flow would help:

IoT devices, real-time data --- Flink --- Kafka --- ElasticSearch
RSS Feeds, database query outputs --- Kafka Connect --- ElasticSearch

Or Kafka sits right in between any external data sources and the back-office processes - taking the place of an ESB - is that correct?

IoT devices might either produce data directly to Kafka (depending on where they are located) or via REST proxy. You don't really need Flink (or any other stream processing framework/library) unless you have some transformation to perform. And even then, you'd usually want to get the raw data into Kafka *first*, and then deal with any transformations/enrichment/etc.

So yes, you'll generally want to do raw import/ingest into Kafka, then deal with any additional processes to enrich/modify that data.

-Ewen
 

>     >     confluent-platform+unsubscribe@googlegroups.com <javascript:>
>     <javascript:>
>     >     >     <mailto:confluent-platform+unsubsc...@googlegroups.com
>     >     > an email to confluent-platform+unsubscribe@googlegroups.com
>     <javascript:>
>     >     <javascript:>
>     >     > <mailto:confluent-platform+unsubsc...@googlegroups.com
>     <javascript:>
>     >     <javascript:>>.
>     >     > To post to this group, send email to
>     confluent...@googlegroups.com
>     >     <javascript:>
>     >     > <mailto:confluent...@googlegroups.com <javascript:>>.
>     >     > To view this discussion on the web visit
>     >     >
>     >    
>     https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>
>
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com>>
>
>     >
>     >     >
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>
>     >    
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/CAE1jLMPg9Bsaq4hGD8mf2TX6fsJUfNX3EJSzPBCd35WGvtB2Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>>>.
>
>     >
>     >     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>
>     >     <https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>>.
>     >
>     > --
>     > You received this message because you are subscribed to the Google
>     > Groups "Confluent Platform" group.
>     > To unsubscribe from this group and stop receiving emails from it,
>     send
>     > an email to confluent-platform+unsubscribe@googlegroups.com
>     <javascript:>
>     > <mailto:confluent-platform+unsubsc...@googlegroups.com
>     <javascript:>>.
>     > To post to this group, send email to confluent...@googlegroups.com
>     <javascript:>
>     > <mailto:confluent...@googlegroups.com <javascript:>>.
>     > To view this discussion on the web visit
>     >
>     https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com>
>
>     >
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/cd9310e8-27cc-4508-beb4-5461c6ba42fc%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsubscribe@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

Baden

unread,
Jan 4, 2017, 11:26:12 AM1/4/17
to Confluent Platform


On Tuesday, January 3, 2017 at 3:56:19 PM UTC-5, Ewen Cheslack-Postava wrote:


Note that Kafka Streams is a stream processing library. I think what you're getting at is that you'd like users to have an API they can hit to create new topics? For that you would need to use the admin tools included with Kafka (or the new AdminClient Java API when that becomes available).


That is really the critical point here - is to be able to allow clients to dynamically edit, delete and create new topics (abstracted by "Project" naming convention, or something similar on the front-end).

I found the C# client for Kafka - https://github.com/Microsoft/CSharpClient-for-Kafka - but no indication about creating new topcs.


You have the option of either adding topics manually or having them be created automatically when data is first published to a non-existent topic. If topics are auto-created then you may want to tune the default topic configurations used for auto-created topics. Topics are added and modified using the topic tool...

  > bin/kafka-topics.sh --zookeeper zk_host:port/chroot --create --topic my_topic_name
        --partitions 20 --replication-factor 3 --config x=y

I didn't see any library support to dynamically create topics and don't see it in http://docs.confluent.io/3.0.0/kafka-rest/docs/api.html?highlight=topics#topics.

Since we want to dynamically add topics we would also need to provide an interface to users to define the schema, which would then be added to the Confluent Schema Registry, correct?

I believe I found the answer to that, using the REST proxy API: http://docs.confluent.io/3.0.0/app-development.html?highlight=topics#rest-proxy - we would store that metadata in ElasticSearch for the topic/project, correct?

Is the over-all Kafka use case for a pre-determined list of topics?

I'm fine with posting to a non-existing topic as Kafka will automatically create a new one.  However, no editing/deletion support?

Since topics will be dynamic, some of them, for example fetching RSS data, will require workers to pull the data and possibly perform some ETL.  I see from http://docs.confluent.io/3.1.1/connect/userguide.html#connect-configuring-workers that workers have to be configured through property files.  Again, this is problematic if topics are dynamic.  I assume using a Kafka worker for pulling data is THE way to go, rather than writing a cron job or Windows service to get data from a RSS feed / producer.  In v1,0 I have accomplished this with a Windows service, where RSS fee URLs are read by the service and the job performed on a scheduled basis.  I want to be able to accomplish this all through Kafka and be multi-threaded and store the data in ES.  The same will hold true for things like SQL Server connectors / producers, so I need a way to dynamically add producers, consumers, new topics, etc.  Everything required in dynamically creating a data pipeline - from producer, sink (ElasticSearch), to consumer.

If there is no dynamic worker support - would we have to write config files (through Java) and perform a worker restart or something on a daily basis?  This doesn't seem like a good option.  Basically what is the alternative to the command-line tools?








Michael Noll

unread,
Jan 4, 2017, 1:41:41 PM1/4/17
to confluent...@googlegroups.com
Baden,

you can programmatically create Kafka topics.

See e.g. my answer at http://stackoverflow.com/a/23360100/1743580 for how to do so (and the note at http://stackoverflow.com/a/38442666/1743580 when using Kafka 0.10+).  We also have example code that demonstrate under https://github.com/confluentinc/examples, see e.g. https://github.com/confluentinc/examples/blob/3.1.x/kafka-streams/src/test/java/io/confluent/examples/streams/kafka/KafkaEmbedded.java#L118-L120.  Maybe this is what you have been looking for?

Best,
Michael





--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages