Debezium Mysql Connector - How to provide the custom name for Schema Change topic

[rohit singh]

Hi Team,

I am trying to capture schema change using debezium mysql connector. I want to provide the custom name for the topic used to publish the schema change.

So that we can use that topic and consume the data further.

I didtn find any solution , please help me to resolve this.

Note: We have kafka and kafka connect debezium mysql connector running on different servers and we cannot create auto topic creation option.

Thanks in advance.

[Chris Cranford]

Hi -

There are two schema history topics that can be created by any relational connector (except PostgreSQL).  One is configured using the `schema.history.internal.kafka.topic` [1] configuration.  This is the internal schema history topic used and consumed by the connector and it should not be used by other sources as the format of this topic is not guaranteed to remain compatible across releases. 

The second is one that is created when you set `include.schema.changes` [2] as `true`, and it will have the same name as the value supplied in `topic.prefix` [3] by default.  If this is insufficient for you needs, you can always use the ByLogicalTableRouter [4] or any other custom SMT to rename the topic to a more desired state during an in-flight transformation step to avoid affecting the prefixes applied to your captured table topics.

Let us know if you have other questions.
Chris

[1]: https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-database-history-kafka-topic
[2]: https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-include-schema-changes
[3]: https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-topic-prefix
[4]: https://debezium.io/documentation/reference/stable/transformations/topic-routing.html

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/49d63b0e-0867-473d-9f84-6aeaf38c97acn%40googlegroups.com.

[rohit singh]

Thanks for your reply,

Yes i am talking about the second one , where i added the property 'include.schema.changes' to true. Is there any way instead of  'topic.prefix' , i can give my topic name and it will send message to custom topic name provided.

Because on our server i am not able to create the topic with same 'topic.prefix' value.

If we use SMT , do we still need topic created with  'topic.prefix' . so it will route the message directly to the custom topic provided. can you please share any example .

Thanks in advance.

[Chris Cranford]

Hi -

Yes, when you use a single message transformation, Debezium will still generate the events to the `topic.prefix` topic name.  The single message transformation runs just before the event gets sent to Kafka, so this is a perfect time to do any in-flight adjustments of events, such as changing the topic, adding fields, removing fields, etc. 

The ByLogicalTableRouter [1] can be quite useful here for this situation:

transforms=reroute-schema-changes
transforms.reroute-schema-changes.type=io.debezium.transforms.ByLogicalTableRouter
transforms.reroute-schema-changes.topic.regex=
transforms.reroute-schema-changes.topic.replacement=

In the above configuration, you set the "transforms.reroute-schema-changes.topic.regex" to a regular expression that matches the value of your "topic.prefix" in your configuration.  Lastly, you set a value for the "transforms.reroute-schema-changes.topic.replacement" which will be the new topic that you want the messages sent to.

In this example, any events sent to a topic that matches the "topic.regex" will be re-routed (topic changed) to be the value in "topic.replacement".

Hope that helps.
Chris

[1]: https://debezium.io/documentation/reference/2.7/transformations/topic-routing.html

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/d5264c5e-4cb4-4ccc-ae35-f9a6dd698aedn%40googlegroups.com.

[rohit singh]

Thank you Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/ebff179f-56bd-4359-8297-c4b84072784c%40gmail.com.