How do I use EmbeddedSingleNodeKafkaCluster?

[Dean Morin]

There is a class, "EmbeddedSingleNodeKafkaCluster" that looks like it would be really useful for tests. How can I include this in my project?

It's mentioned here:

http://docs.confluent.io/3.1.1/streams/developer-guide.html#application-examples-for-kafka-streams-provided-by-confluent

And referred to here:

https://github.com/confluentinc/examples/blob/master/kafka-streams/src/test/java/io/confluent/examples/streams/MixAndMatchLambdaIntegrationTest.java

These are my dependencies:

  <dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-streams</artifactId>
    <version>0.10.1.1-cp1</version>
  </dependency>
  <dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>0.10.1.1-cp1</version>
  </dependency>

[Matthias J. Sax]

We do not publish a test maven artifact atm. You will need to c&p for
now to use it. :(


-Matthias

> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.
> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/c1548e6a-70c7-446c-bd06-a8d37deb67dd%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/c1548e6a-70c7-446c-bd06-a8d37deb67dd%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Dean Morin]

I went with the copy & paste method which Matthias mentioned. It was a little tricky, so I’ll leave my findings here in case it’s useful to someone else.

There were 3 files I had to add to my project which can be found in these two packages:

• https://github.com/confluentinc/examples/tree/3.1.x/kafka-streams/src/test/java/io/confluent/examples/streams/kafka
• https://github.com/confluentinc/examples/tree/3.1.x/kafka-streams/src/test/java/io/confluent/examples/streams/zookeeper
  

Then I found the pom.xml here:

• https://github.com/confluentinc/examples/blob/3.1.x/kafka-streams/pom.xml
  

I added dependencies one-by-one until I had just the ones I need. Originally I tried using code from the master branch, but that branch has -SNAPSHOT dependencies which are not publicly available. Switching to the latest release branch "3.1.x" worked.

These are the dependencies I needed:

  :profiles {:dev {:dependencies [[io.confluent/kafka-schema-registry "3.1.2"]
                                  [io.confluent/kafka-schema-registry "3.1.2" :classifier "tests"]
                                  [junit/junit "4.12"]
                                  [org.apache.curator/curator-test "2.9.0"]
                                  [org.apache.kafka/kafka_2.11 "0.10.1.1-cp1"]
                                  [org.apache.kafka/kafka_2.11 "0.10.1.1-cp1" :classifier "test"]
                                  [org.apache.kafka/kafka-clients "0.10.1.1-cp1" :classifier "test"]]

I’m using lein instead of maven, but these dependencies are exactly the same, just with slightly different syntax.

​

[Shannon Ma]

This and some util classes in the example are useful, any plan to add them to the confluent package?

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.

[Matthias J. Sax]

We got this request multiple times already and yes, there are plans to
publish those classes. But there is no timeline yet...

-Matthias

> > an email to confluent-platf...@googlegroups.com
> <javascript:>
> > <mailto:confluent-platf...@googlegroups.com
> <javascript:>>.

> > To post to this group, send email to confluent...@googlegroups.com

> <javascript:>
> > <mailto:confluent...@googlegroups.com <javascript:>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/confluent-platform/c1548e6a-70c7-446c-bd06-a8d37deb67dd%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/c1548e6a-70c7-446c-bd06-a8d37deb67dd%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/confluent-platform/c1548e6a-70c7-446c-bd06-a8d37deb67dd%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/confluent-platform/c1548e6a-70c7-446c-bd06-a8d37deb67dd%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platf...@googlegroups.com
> <mailto:confluent-platf...@googlegroups.com>.

> To post to this group, send email to confluent...@googlegroups.com
> <mailto:confluent...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/confluent-platform/3ac8f454-dadc-4166-9d01-92852f5ea87d%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/3ac8f454-dadc-4166-9d01-92852f5ea87d%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Ryan Worsley]

I'd like to +1 this - would speed up development a lot.

>     > an email to confluent-platform+unsub...@googlegroups.com
>     <javascript:>
>     > <mailto:confluent-platform+unsub...@googlegroups.com

>     <javascript:>>.
>     > To post to this group, send email to confluent...@googlegroups.com
>     <javascript:>
>     > <mailto:confluent...@googlegroups.com <javascript:>>.
>     > To view this discussion on the web visit
>     >
>     https://groups.google.com/d/msgid/confluent-platform/c1548e6a-70c7-446c-bd06-a8d37deb67dd%40googlegroups.com
>     <https://groups.google.com/d/msgid/confluent-platform/c1548e6a-70c7-446c-bd06-a8d37deb67dd%40googlegroups.com>
>
>     >
>     <https://groups.google.com/d/msgid/confluent-platform/c1548e6a-70c7-446c-bd06-a8d37deb67dd%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/confluent-platform/c1548e6a-70c7-446c-bd06-a8d37deb67dd%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>     > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to confluent-platform+unsub...@googlegroups.com
> <mailto:confluent-platform+unsub...@googlegroups.com>.

[Dave Thomas]

+1 from me too.  I see this is in the new kafka-streams-examples in the new repo repo on github : confluentinc/kafka-streams-examples.git - would be quite handy to have it as a testing-scope jar.

[ryan....@zefr.com]

Another +1 from us. 

It's been over a year and making a test artifact at this point should not be that big of a deal

[Sacha Barber]

I'd also like to add +1 to this request, it would really help trying stuff out, ok there is this

- https://github.com/manub/scalatest-embedded-kafka

- https://github.com/jpzk/mockedstreams