Problem deserializing Avro messages using REST Proxy

[Giulio Vito de Musso]

Hello you all,

I wrote some data on a Kafka topic using Avro. In particulat I used the KafkaAvroSerializer both for the key and the value and I wrote some GenericData.Record data.

Properties kafkaProps = new Properties();

kafkaProps.put("bootstrap.servers", "kafka-server-url");
kafkaProps.put("schema.registry.url", "schema-registry-url");
kafkaProps.put("key.serializer", KafkaAvroSerializer.class.getCanonicalName());
kafkaProps.put("value.serializer", KafkaAvroSerializer.class.getCanonicalName());


KafkaProducer<GenericRecord, GenericRecord> producer = new KafkaProducer<>(kafkaProps);


GenericRecord key = new GenericData.Record(keySchema);

key.put("idField1, "1234");

key.put("idField2", "5678");

GenericRecord value = new GenericData.Record(valueSchema);

key.put("valueField1", "abcd");

key.put("valueField2", "efgh");

ProducerRecord<GenericRecord, GenericRecord> record = new ProducerRecord<>("theTopic", key, value);


producer.send(record, (RecordMetadata metadata, Exception exception) -> {
 if (exception != null) {
 LOG.error(exception.getMessage());
 }
});

The message is sent in fact I can read the GenericData.Record objects using a java consumer (which uses the kafka-clients and kafka-avro-serializer libraries).

Now I need to implement a javascript application which reads the topic, so I should read the data using the REST Proxy, but in a test with cURL I'm getting a Avro deserialize error.

curl -X POST -H "Content-Type: application/vnd.kafka.v1+json" --data '{"name": "my_consumer_instance", "format": "avro", "auto.offset.reset": "smallest"}' http://localhost:8082/consumers/my_avro_consumer

curl -X GET -H "Accept: application/vnd.kafka.avro.v1+json" http://localhost:8082/consumers/my_avro_consumer/instances/my_consumer_instance/topics/theTopic

This is the response I get from the GET request

{"error_code":50002,"message":"Kafka error: Error deserializing Avro message for id 1"}

It seems that Avro cannot deserialize the encoded message even though I used Avro for the serialization.

What do you think the problem could be? 

Thank you

Regards

[Giulio Vito de Musso]

Ok solved, it was a silly configuration issue of the Kafka REST proxy. 

I misconfigured the schema.registry.url property.

Sorry for having added a non useful post.

[kingch...@gmail.com]

hi  @gvdm  I have the same question. how did you fixed it .

thank you !!

在 2017年4月28日星期五 UTC+8下午9:21:37，gvdm写道：

[gvdm]

Hi @kingchai0184

I don't remember but as told in my last message I misconfigured the schema.registry.url property in the REST proxy configurations. 

Try checking that the URL format is right (http://SR_host:SR_port) and that the host and port are correct.