kafka-avro-console-consumer requires a key schema?

5,150 views
Skip to first unread message

Kevin Hutson

unread,
Apr 13, 2016, 7:25:15 PM4/13/16
to Confluent Platform
We are new to the schema registry, so I hope this isn't a dumb question.

Our Kafka producer is using io.confluent.kafka.serializers.KafkaAvroEncoder as the value serializer (value.serializer). But, we aren't using it for the key. This means we end up with a schema for the message (subject=MYTOPIC-value) but not for the key. It's just plain text. 

From the Java API, I can consume messages for this topic and see both the message and the key. It talks to the schema registry fine.
From kafka-avro-console-consumer, I can see the messages if I don't ask it to print the key.
But, If I try to consume with the kafka-avro-console-consumer tool, it throws an exception if I try to print the key (--property print.key=true). 

./kafka-avro-console-consumer --zookeeper local.docker:2181 --topic MYTOPIC --from-beginning --property schema.registry.url=http://local.docker:8081 --property print.key=true
Processed a total of 1 messages
[2016-04-13 18:17:46,989] ERROR Unknown error when running consumer:  (kafka.tools.ConsoleConsumer$:103)
org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!

Is this because the kafka-avro-console-consumer tool expects both the key and value to have schemas?
If so, will POSTing the key as a simple String be sufficient. At this time, our key is just a String.
That would mean we would need to set key.serializer to io.confluent.kafka.serializers.KafkaAvroEncoder?


thank you!
Kevin Hutson

Liquan Pei

unread,
Apr 13, 2016, 7:30:46 PM4/13/16
to confluent...@googlegroups.com
Hi Kevin,

You need to provide the value-schema as a command line parameter in kafka-avro-console-consumer. The quickstart of the confluent documentation constains some example (http://docs.confluent.io/2.0.1/quickstart.html). There are more information here http://docs.confluent.io/2.0.1/schema-registry/docs/serializer-formatter.html on serializers and formatters. 

Thanks,
Liquan

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/ff546bff-79ee-43f1-952d-9d6e89f031e7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Liquan Pei | Software Engineer | Confluent | +1 413.230.6855
Download Apache Kafka and Confluent Platform: www.confluent.io/download

Kevin Hutson

unread,
Apr 13, 2016, 8:10:40 PM4/13/16
to Confluent Platform
Hi Liquan ,

Thank you for the quick reply. But wait. I'm confused by your answer. I already have my value schema in the schema registry. And the message is serialized with the magic byte. In fact, I can deserialize messages from this topic in Java and see the key (which is just a String).

So, we aren't currently using a key schema. Will kafka-avro-console-consumer not print the message key unless I provide the full schema via the command line? That doesn't sound right. 
I've even provided the url to the schema registry. And, this prints the message (not the key) just fine if I remove '--property print.key=true'.

Let me try to simplify my question:
-> Can I use kafka-avro-console-consumer CLI tool to view the message keys of a topic IF I've only registered the value-schema and not a key-schema for this topic? To me, it seems like this tool expects both a key and value schema to be registered for a topic?

best,
Kevin


On Wednesday, April 13, 2016 at 6:30:46 PM UTC-5, Liquan Pei wrote:
Hi Kevin,

You need to provide the value-schema as a command line parameter in kafka-avro-console-consumer. The quickstart of the confluent documentation constains some example (http://docs.confluent.io/2.0.1/quickstart.html). There are more information here http://docs.confluent.io/2.0.1/schema-registry/docs/serializer-formatter.html on serializers and formatters. 

Thanks,
Liquan
On Wed, Apr 13, 2016 at 4:25 PM, Kevin Hutson <mrj...@gmail.com> wrote:
We are new to the schema registry, so I hope this isn't a dumb question.

Our Kafka producer is using io.confluent.kafka.serializers.KafkaAvroEncoder as the value serializer (value.serializer). But, we aren't using it for the key. This means we end up with a schema for the message (subject=MYTOPIC-value) but not for the key. It's just plain text. 

From the Java API, I can consume messages for this topic and see both the message and the key. It talks to the schema registry fine.
From kafka-avro-console-consumer, I can see the messages if I don't ask it to print the key.
But, If I try to consume with the kafka-avro-console-consumer tool, it throws an exception if I try to print the key (--property print.key=true). 

./kafka-avro-console-consumer --zookeeper local.docker:2181 --topic MYTOPIC --from-beginning --property schema.registry.url=http://local.docker:8081 --property print.key=true
Processed a total of 1 messages
[2016-04-13 18:17:46,989] ERROR Unknown error when running consumer:  (kafka.tools.ConsoleConsumer$:103)
org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!

Is this because the kafka-avro-console-consumer tool expects both the key and value to have schemas?
If so, will POSTing the key as a simple String be sufficient. At this time, our key is just a String.
That would mean we would need to set key.serializer to io.confluent.kafka.serializers.KafkaAvroEncoder?


thank you!
Kevin Hutson

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

Liquan Pei

unread,
Apr 13, 2016, 8:26:06 PM4/13/16
to confluent...@googlegroups.com
Hi Kevin,

Sorry for the incorrect answer to your question. I think you are right and you don't need to provide the value-schema as the command line parameter. 

The kafka-avro-consumer calls AvroMessageFormatter to print out the deserialized Avro records in the console. AvroMessageFormatter is a subclass of AbstractKafkaAvroDeserializer which expects the record to have serialized Avro format generated by KafkaAvroEncoder or KafkaAvroSerializer. This means that the kafka-avro-consumer is not able to display the key if the key is not serialized using the Avro serializers. 

BTW, it seems that you are using the old producer. Which version of Kafka you are currently using?

Thanks,
Liquan

Hi Liquan ,

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.



--
Liquan Pei | Software Engineer | Confluent | +1 413.230.6855
Download Apache Kafka and Confluent Platform: www.confluent.io/download

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/bdfdcffd-edae-4a46-a6bc-85ac69888e09%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Kevin Hutson

unread,
Apr 13, 2016, 8:40:48 PM4/13/16
to Confluent Platform
Hi Liquan,

Ah. That makes sense and answers my question. We'll probably try to serialize our key then since sanity testing with the CLI is a must. :-)

re: our version:
Well, our cluster is currently on confluent-2.0.1-2.11.7 which I believe corresponds to kafka 0.9.0.1.
Yes. Unfortunately, our Producer (well, this particular one) is older, on 0.8.2. 
We are working with Oracle for this particular producer and they lag behind a bit on the Producer version.

Thanks again!
Kevin
Hi Liquan ,

To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.



--
Liquan Pei | Software Engineer | Confluent | +1 413.230.6855
Download Apache Kafka and Confluent Platform: www.confluent.io/download

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages