AVRO in Kafka Streams and the WikiFeed example

[Roel R]

Dear group,

I want to use Kafka Streams with AVRO data. I thought that it was a good idea to borrow the SpecificAvroDeserializer which is used by WikipediaFeedAvroExample.java, but I couldn't get my code working with it. After a day of digging, I noticed that I couldn't even get the example working.

In both my code as the example, deserialize(...) throws because "KafkaAvroDeserializer inner" is null, it is never set.

I guess I can fix this, but at the moment I'm not sure if this is a bug or that I'm simply doing something wrong and that there is nothing to fix.

As far as I understood, KafkaAvroDeserializer needs to know where the schema registry lives.

I found out that the Deserializer is constructed by getConfiguredInstance, which does not call configure() because the Deserializer is not a Configurable.

Would fixing that be the right direction?

Thanks,

Regards,

Roel

PS. Confluent, I think you did a great job with Kafka Streams, and the documentation is also high quality, well done!

[Roel R]

I still don't know if I'm working around a problem I created myself, or that there is some bug in the example, but this might help other, what I do is:

      settings.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8081")

      settings.put(KafkaAvroDeserializerConfig.SPECIFIC_AVRO_READER_CONFIG, "true")

And then in the SpecificAvroDeserializer, implement Configurable with:

    @Override

    public void configure(Map<String, ?> configs) {

        inner = new KafkaAvroDeserializer();

        inner.configure(configs, false); //TODO

    }

I don't know what the implications of setting isKey to false are, but with these changes the streams work as advertised.

[Guozhang Wang]

Hello Roel,

Thanks for reporting this, I will have a quick look into the examples source code and get back to you asap.

Guozhang

[Roel R]

Thanks Guozhang, 

If you need more information please ask me.

Regards,

Roel

[Michael Noll]

Hey everyone,

thanks for your patience.  We have sorted out the Avro-related issues in the meantime and also updated the confluentinc/examples repository along the way.

See my message at https://groups.google.com/d/msg/confluent-platform/JLZWG1Ybpjc/YJiUW-p1BQAJ for details.

Best,

Michael

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/08ddb0b1-68c2-4d19-b494-797693f7b5a2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Roel R]

Hi Michael,

Thanks for the information and solution!

Regards,

Roel