error while consuming topic created from kaka-connect-jdbc source with specific.avro.reader = true

3,500 views
Skip to first unread message

Andrey Plaksin

unread,
Aug 2, 2016, 6:11:17 AM8/2/16
to Confluent Platform
Hi 

we are trying to consume a topic with avro consumer 

the topic is created with connect-standalone from confluent platform 3.0 using the kafka-connect-jdbc 

we created the specific class from the schema and trying to consume with specific.avro.reader = true

getting the following error 

org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id 201
Caused by: org.apache.kafka.common.errors.SerializationException: Could not find class accounts specified in writer's schema whilst finding reader's schema for a SpecificRecord.

when setting the specific reader to false we can get the messages but not to specific class 

any help will be appreciated 

Andrey Plaksin

unread,
Aug 2, 2016, 6:42:55 AM8/2/16
to Confluent Platform
it seems that there is an issue with the namespace value 

in my avsc i set up a namespace so that the class will be created in a specific package but the schema that was generated by the kafka connect has no namespace 

Gwen Shapira

unread,
Aug 2, 2016, 1:30:51 PM8/2/16
to confluent...@googlegroups.com
Thats a good point.

Currently we are sharing the converter between all connectors running
in a Connect cluster, so while we could make namespace configurable -
it would give the same namespace to everything Connect writes to Kafka
- which is probably not what you want...

Gwen
> --
> You received this message because you are subscribed to the Google Groups
> "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to confluent-platf...@googlegroups.com.
> To post to this group, send email to confluent...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/874bed6b-7a64-470c-8a90-0159d315e8e6%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

Andrey Plaksin

unread,
Aug 2, 2016, 2:06:11 PM8/2/16
to Confluent Platform
No that is not what i want but what is the resolution for this issue ? 

The only way i was able to consume specific avro message created by kafka connect jdbc is when i defined the class in the default package and moving the consumer class there too. 

is this by design ?

Andrey Plaksin

unread,
Aug 2, 2016, 2:11:42 PM8/2/16
to Confluent Platform
By the way charing the namespace would be a better solution than the current one 

i could live with using the same namespace for avro generated objects across organization better than with having to write all the consumers in the default package since you cannot import classes from there to other packages  


On Tuesday, August 2, 2016 at 8:30:51 PM UTC+3, Gwen Shapira wrote:

Gwen Shapira

unread,
Aug 2, 2016, 2:26:02 PM8/2/16
to confluent...@googlegroups.com
It isn't by design. It is a limitation we did not think about when designing.

I'll let Ewen discuss solutions since he has better grasp of what's
possible with the current design.

On Tue, Aug 2, 2016 at 11:06 AM, Andrey Plaksin
>> > email to confluent-platf...@googlegroups.com.
>> > To post to this group, send email to confluent...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/confluent-platform/874bed6b-7a64-470c-8a90-0159d315e8e6%40googlegroups.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to confluent-platf...@googlegroups.com.
> To post to this group, send email to confluent...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/bfc2515a-9cf9-4e87-b009-08f1ab4d35a4%40googlegroups.com.

Gwen Shapira

unread,
Aug 2, 2016, 2:26:39 PM8/2/16
to confluent...@googlegroups.com
Maybe open an issue on the schema-registry (when the Avro converters
live)? This way we can track this improvement.

On Tue, Aug 2, 2016 at 11:11 AM, Andrey Plaksin
>> > email to confluent-platf...@googlegroups.com.
>> > To post to this group, send email to confluent...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/confluent-platform/874bed6b-7a64-470c-8a90-0159d315e8e6%40googlegroups.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to confluent-platf...@googlegroups.com.
> To post to this group, send email to confluent...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/confluent-platform/16cec0bd-85f4-469a-b2e0-23fc1099d025%40googlegroups.com.

Andrey Plaksin

unread,
Aug 2, 2016, 2:31:45 PM8/2/16
to confluent...@googlegroups.com

Thank you

I appreciate it

What is the best course of action for now in your opinion ? Since we have to proceed with the development of the solution I was thinking to go ahead with the default package and refactor it later when better options are available.


You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/GQDKji8gUR4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAA9QSJKsk_QfN-1Kn8Vd5vPSWOz1TGMQmW-GJw9p3ib8dGzdzQ%40mail.gmail.com.

Gwen Shapira

unread,
Aug 2, 2016, 2:35:24 PM8/2/16
to confluent...@googlegroups.com
Yeah, it kind of sucks, but I can't think of anything better :(

On Tue, Aug 2, 2016 at 11:31 AM, Andrey Plaksin
> https://groups.google.com/d/msgid/confluent-platform/CAAjyUROOQd_HRhXkHM9w28Fj_xff04JbQqNYNFzoijEs5J%2Bj%2BA%40mail.gmail.com.

Sagar Rao

unread,
Aug 8, 2016, 3:48:36 AM8/8/16
to Confluent Platform
Hi,

I am currently facing the same issue when trying to read SpecificRecord from Kafka topics and even I had set the namespace to a value. Reason is I downloaded the .avsc from SchemaRegistry(using a locally built schema-registry-maven-plugin) and when I tried to generate the class file using maven-avro-plugin, it complained for the lack of absence of package(which corresponds to namespace from the avsc file).

After adding a namespace I get the exact same error as in the OP and I get it even when specific.avro.reader is set to false.

So, just wanted to know if there's any resolution/quick-fix for this?

Sagar.
>> >> > To post to this group, send email to confluent...@googlegroups.com.
>> >> > To view this discussion on the web visit
>> >> >
>> >> >
>> >> > https://groups.google.com/d/msgid/confluent-platform/874bed6b-7a64-470c-8a90-0159d315e8e6%40googlegroups.com.
>> >> >
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Confluent Platform" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > To post to this group, send email to
>> > confluent...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/confluent-platform/16cec0bd-85f4-469a-b2e0-23fc1099d025%40googlegroups.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Confluent Platform" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/confluent-platform/GQDKji8gUR4/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> To post to this group, send email to confluent...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/confluent-platform/CAA9QSJKsk_QfN-1Kn8Vd5vPSWOz1TGMQmW-GJw9p3ib8dGzdzQ%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send an

Andrey Plaksin

unread,
Aug 8, 2016, 5:06:41 AM8/8/16
to Confluent Platform

As I understand right now the only solution is to place the generated classes in the default package , this means that the code using them has to be there too

There is no other resolution


>> >> > To post to this group, send email to confluent...@googlegroups.com.
>> >> > To view this discussion on the web visit
>> >> >
>> >> >
>> >> > https://groups.google.com/d/msgid/confluent-platform/874bed6b-7a64-470c-8a90-0159d315e8e6%40googlegroups.com.
>> >> >
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Confluent Platform" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > To post to this group, send email to
>> > confluent...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/confluent-platform/16cec0bd-85f4-469a-b2e0-23fc1099d025%40googlegroups.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Confluent Platform" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/confluent-platform/GQDKji8gUR4/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> To post to this group, send email to confluent...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/confluent-platform/CAA9QSJKsk_QfN-1Kn8Vd5vPSWOz1TGMQmW-GJw9p3ib8dGzdzQ%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To post to this group, send email to confluent...@googlegroups.com.
> To view this discussion on the web visit

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/GQDKji8gUR4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/08bd6b20-288b-417a-82f8-451a810dabf7%40googlegroups.com.

Franziska Kühn

unread,
May 12, 2020, 1:35:32 PM5/12/20
to Confluent Platform
Hi, I've written a kafka connector with a dedicate avro-scheme in scala and a microservice with kafka-streams using scala dsl and I'm facing the same issue. Putting avro-schemes in one directory with consumer class under default src-package and given the package name with class name for kafka data scheme solved this issue. but it's really ugly, why is there still no namespace possibility by creating a struct scheme , shouldn't be a problem?
>> >> > To post to this group, send email to confluent...@googlegroups.com.
>> >> > To view this discussion on the web visit
>> >> >
>> >> >
>> >> > https://groups.google.com/d/msgid/confluent-platform/874bed6b-7a64-470c-8a90-0159d315e8e6%40googlegroups.com.
>> >> >
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Confluent Platform" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > To post to this group, send email to
>> > confluent...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/confluent-platform/16cec0bd-85f4-469a-b2e0-23fc1099d025%40googlegroups.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Confluent Platform" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/confluent-platform/GQDKji8gUR4/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> To post to this group, send email to confluent...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/confluent-platform/CAA9QSJKsk_QfN-1Kn8Vd5vPSWOz1TGMQmW-GJw9p3ib8dGzdzQ%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To post to this group, send email to confluent...@googlegroups.com.
> To view this discussion on the web visit

--
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/GQDKji8gUR4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platform+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages