Unit testing KS with Avro

[Branden Makana]

Hello,

I'm trying to write unit tests for my kafka-streams application that uses Avro. I've been successful with unit testing using the MockedStreams library and JSON, but now I'm moving to using Avro and having trouble. 

I know about MockSchemaRegistryClient, and from this thread: https://groups.google.com/forum/#!msg/confluent-platform/Qwp9xhtzDaI/LgtDkd0aAgAJ it seems like it should be easy to plug in, but I'm running 3.3.0 and I notice the SpecificAvroSerde now has the constructor that takes a SchemaRegistryClient as package private, so the linked example no longer is doable. 

How can I unit test kafka streams, with avro serialization, but with a mocked schema registry? 

[Michael Noll]

To confirm: You want to do unit testing, not integration testing, right?

Because we do have working end-to-end examples of how to do integration testing with Kafka's Streams API, Confluent Schema Registry, and Avro.

e.g. for CP 3.3.0:

https://github.com/confluentinc/kafka-streams-examples/blob/3.3.0-post/src/test/java/io/confluent/examples/streams/SpecificAvroIntegrationTest.java

https://github.com/confluentinc/kafka-streams-examples/blob/3.3.0-post/src/test/java/io/confluent/examples/streams/GenericAvroIntegrationTest.java

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.
To post to this group, send email to confluent-platform@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/6c49c167-911d-42ac-a38d-3cbb212a8995%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Branden Makana]

Yes, unit testing not integration.

You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/ZsIaS5x516U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAA4zQx%2BPK2NK7Fao55Qy9jtK91_cTaZP%3DqG0BjTt_aQk5cfyLA%40mail.gmail.com.

[Branden Makana]

I'm kind of getting the impression that no one unit tests with the Avro serializers, at least not lately, given the lack of anything in ze Googles. 

I can always mock out, replace, or otherwise ignore the avro serializers for a unit test (after all, I'm not unit testing them) but it's pretty annoying to have to do that only when the serialization is Avro (versus say, JSON). 

[Peter Figliozzi]

Did you ever find a good solution for unit testing streams topologies with MockedStreams and Avro?  I've been pointing it at my dev Schema Registry. 

[Peter Figliozzi]

Aside: found that when using MockedStreams, I have to make and pass my own Properties to get it to work with Avro:

val props = new Properties

    props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,

      Serdes.String().getClass.getName)

    props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,

      "io.confluent.kafka.streams.serdes.avro.GenericAvroSerde")

    props.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG,

      "http://watever:8081"

Then, pass this when building your MockedStreams:

val mstream = MockedStreams()

      .withTopology(getTopology)

      .config(props)

      ...

This doesn't answer OP's question (I have a real SchemaRegistry running), but I wasted a few hours before figuring the Properties riddle out... hopefully this helps someone.

[Branden Makana]

Hi Peter,

No, I was not able to test without a schema registry; I instead just did what you did, and used a local development copy.