multi tables CDC events to single topic

[Hong Cheng]

hi all,

By using Reroute, i am able to route different tables ( with difference schema) to one topic. Is it possible i am using Avro serializers ? 

How can i read message from kafka stream since every event can be serialize for different type of objects ? 

Anyway I can convert debezium CDC event to customer schema before sent to kafka for such use case ?

Thanks

hong

[Gunnar Morling]

ho...@interactivelabs.co schrieb am Freitag, 5. November 2021 um 06:44:22 UTC+1:

hi all,

By using Reroute, i am able to route different tables ( with difference schema) to one topic. Is it possible i am using Avro serializers ? 

Yes, it is. When using Avro, each Kafka record contains an id for its schema, which can be resolved against a schema registry such as Apicurio (https://www.apicur.io/registry/). All records with the same schema will have the same id.

 

How can i read message from kafka stream since every event can be serialize for different type of objects ? 

Deserializers (or Kafka Connect converters) will fetch the right Avro schema for the incoming record from the registry and apply that one.  You can find a complete example here: https://github.com/debezium/debezium-examples/tree/main/tutorial#avro-format-using-apicurio-avro-converter.

Anyway I can convert debezium CDC event to customer schema before sent to kafka for such use case ?

I don't quite get that part. You can modify an event's schema by applying an SMT before sending. If no schema with that "shape" already exists within the registry, the converter/serializer will register that new schema.

 

Thanks

hong

Hth,

--Gunnar