Schema Identifier Not Present in the Message

[Prateek Gupta]

As per the documentation, Avro Serialization, the Confluent Schema Registry tracks all of the Avro schemas used in Kafka topics, and where the Avro Converter sends the generated Avro schemas. Since the Avro schemas are stored in this registry, each message need only include a tiny schema identifier.

PFB the json decoded message value for reference.

{

"before": null,

"after": {

"localhost.demo.Demo.Value": {

"id": {

                               "int": 1,

                        }

"Name": {

"string": "John"

},

"Age": {

"int": 24

},

"Department": {

"string": "Tech"

}

}

},

"source": {

"name": "localhost",

"server_id": 0,

"ts_sec": 0,

"gtid": null,

"file": "bin.000012",

"pos": 154,

"row": 0,

"snapshot": {

"boolean": true

},

"thread": null,

"db": {

"string": "demo"

},

"table": {

"string": "Demo"

}

},

"op": "c",

"ts_ms": {

"long": 1510237013611

}

}

[Jiri Pechanec]

Hi,

could you please try our Avro tutorial - https://github.com/debezium/debezium-examples/tree/master/tutorial#using-mysql-and-the-avro-message-format - to verify wheter it works for you or not?

Thanks a lot

J.

[Prateek Gupta]

Thanks for the response, J.

But, we are not using the docker environment.

PFA the config file we are using for schema registry.

[Gunnar Morling]

Hey Prateek,

I'm not exactly sure what your question is.

Is it that you don't see the schema id in that JSON representation probably created by the console consumer? If so, that's to be expected, as that output will only represent the actual message data. I'd assume if you take a look at the binary representation, you'll find the reference.

But perhaps that was not your question actually :)

Cheers,

--Gunnar

[Prateek Gupta]

Thanks for the response, Gunnar.

And yes, the question is directed towards the omission of 'global schema id' from the message. :)

The JSON referred is actually an Avro message that has been decoded using Avro tools.

We are trying to map a message to the corresponding schema, in case of schema evolution. But, if  response is as expected, can you please guide us through on how to go about the mapping issue?

[Gunnar Morling]

I recommend you take a look at the Avro deserializer implementation at https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/AbstractKafkaAvroDeserializer.java

You'll find that the key and value of the Kafka messages are structured like this:

* byte 0

* four bytes for an int representing the schema id (that's what you're after)

* remaining bytes representing the actual serialized Avro data, adhering to that schema

Hth,

--Gunnar