Heartbeat topic names

[Nic Snoek]

Hi All,

I am running a number of debezium connectors that are using the heartbeat feature, and I need to create heartbeat topics for each of them. Ideally they would just all publish to the same topic, but event though the heartbeat.topics.prefix is configurable, the suffix equals the server.name/topic.prefix which must be unique. 

Is there a reason that they all need a different heartbeat topic? If not, is there a way to configure the connectors to do so?

Nic

[Nic Snoek]

I meant to say, "If not, is there a way to configure the connectors to use the same heartbeat topic?"

[Chris Cranford]

Hi Nic -

There are two bits about heartbeats we need to consider.

The first is whether the connector is configured using "heartbeat.action.query".  If so, you can share this table among connectors; however, doing so means that without proper configuration you compound the frequency that heartbeats are emitted from each connector.  Imagine a situation where you have 2 connectors, both sharing the same table with a heartbeat interval of 120 seconds.  If they're directly opposite of one another, meaning connector A emits at t0 and connector B emits at t0+60, you end up with both connectors emitting heartbeats exactly every 60 seconds because any insert or update into that heartbeat table is seen by all connectors.  Now you can avoid this by using a 2 column heartbeat table where one of the columns is a static, unique value to that connector and then leveraging a SMT to filter out heartbeat events just before going to Kafka that aren't for that specific connector based on that static column value.  While this works, it's generally just simpler to deploy a connector with its own heartbeat action table assignment for a more straightforward configuration.

In much the same way, you can also route all heartbeat events for multiple connectors to exactly the same topic, as long as the structure of the heartbeat events is identical.  Please see our ByLogicalTableRouter [1] transform as it's sole purpose is to handle situations where you may want to combine events from multiple topics into a single destination topic.

Thanks,
Chris

[1] https://debezium.io/documentation/reference/stable/transformations/topic-routing.html

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/9b9ea9e6-0d05-46af-8564-bd672e5da43dn%40googlegroups.com.

[Nic Snoek]

Hi Chris, thanks so much for your reply, that is a great idea. Can you please explain why the structure of the heartbeat events must be the same? I can see that in the general use case for ByLogicalTableRouter the structure should be the same, but it is my understanding that in the case of heartbeats, it is the action of writing these events that is important, and that no one is actually interested in reading them.

[Chris Cranford]

Hi Nic -

While you can store different event structures in the same topic in Kafka, if you're using schema registry this can be problematic if you have differing schemas for the same table, particularly depending on the compatibility setting you've applied.  This is less of a concern when not using Avro and Schema Registry.

Thanks,
Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/dd8d242b-f897-4171-9ab0-da72fa6b0d69n%40googlegroups.com.

[Nic Snoek]

Understood, thank you for your help