Heartbeat Implementation Details

[Josh Ribera]

Hello,

I was hoping to get some clarification on how the heartbeat works. After looking through the code I see that the heartbeat is emitted once a data change event occurs. What we are hoping to do is have the connector emit to the heartbeat topic on the interval defined. I found a few posts regarding this on gitter but they're from a few years ago so I'm not sure if the recommendations are still valid (create a dummy table, create some change event every X seconds/minutes/hours). I mostly wanted to refresh this topic and see if there are any new recommendations.

Thanks!

-- Josh

[Chris Cranford]

Hi Josh -

There are two heartbeat concepts in Debezium and it depends entirely on which problem you're trying to solve.

In PostgreSQL, we have the "heartbeat.action.query" which is meant as a way to keep events periodically coming into Debezium from PostgreSQL. This process continues to advance the replication slot's LSNs forward so that the database transaction log doesn't grow out of proportion.  This is needed particularly in cases where the table being captured changes infrequently compared relative to other tables in the database. 

The other concept revolves around the "heartbeat.interval.ms" setting which controls how frequent does the connector emit a simple event to Kafka.  The goal of the heartbeat event is to prevent the connector's offset data from becoming stale in Kafka.  This is particularly useful for a connector that receives periodic events from the source, requires updating its offsets, but the events read don't cause an event to be emitted to Kafka to sync offsets.  By emitting a heartbeat event on a semi-regular interval when necessary, this allows keeping Kafka's offset data aligned with the connector so that in the event that the connector is restarted, it starts where it left off rather than some point in time in the past due to old offset data.

Now, it is important to note that how "heartbeat.interval.ms" is managed varies by connector depending on the integration with the source system's transaction log.

In MongoDB as an example, we use a non-blocking loop to check whether any new events have been written to its log.  If no events are written, we do nothing but do brief pause and then re-poll.  If an event is read and the event is processed by the connector, we again don't emit any heartbeat events.  We only emit heartbeats if we read an event but its not an event the connector cares about.  This allows us to replicate the offsets oplog position to Kafka even though the event wasn't anything of interest.

In PostgreSQL we also use a non-blocking IO call to check for changes.  But unlike above, if no event is returned and we have at least processed 1 event since the connector started, a Heartbeat will always be emitted as long as the Heartbeat's interval timer has expired.

If you look at other connectors, you'll see again the interaction with emitting heartbeats varies depending on the use case.

HTH,
CC

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/64e55148-8294-4dc9-902c-bd275f51dde6n%40googlegroups.com.

[Prabhatika Vij]

Hey Josh, Chris,

Thank you so much for this conversation, this has helped me gain a better understanding of how the heartbeat mechanism is implemented for MongoDB and PostgreSQL connectors.

I want some help understanding how heartbeat works in case of a MySQL connector and why it is required for MySQL. 

Both MongoDB and PostgreSQL, as far as I understand, notify the client when there is a write operation on the subscribed collections/tables. There might be a case where the there might be changes happening frequently to other databases and collections/tables in the cluster but not to the subscribed collections/tables. In that case, the connector's recorded resume token/LSN might be lost. To avoid this, the heartbeat mechanism can be enabled for both types of connectors.

As far as I understand, a MySQL connector works by reading the binlog files from the database but the database doesn't notify the client of the changes, the connector is responsible for reading the binlog files, filtering out the records pertaining to the subscribed tables and managing till what binlog file and position it has read. In such a case, even if there is no change in the subscribed table, the connector will read the binlog files, realize that there are no changes and record till what binlog file and position it has read. Is this understanding correct? If yes, then why do we need heartbeats for a MySQL connector? If no, could you please help me understand how a MySQL connector works?

Thanks a ton,

Prabhatika 

[Gunnar Morling]

Hey Prabhatika,

You are right that heartbeats aren't needed for MySQL as far as the database itself is concerned.

The issue is that a source connector can only commit its offsets to Kafka if it actually emits records. So if there is a situation where all source tables are filtered out and the connector thus never emits any record, no source offsets can be committed to Kafka. Then, when restarting the connector e.g. after a reconfiguration or an upgrade, it would have to reprocess large sections of the binlog which isn't desirable. Emitting heartbeat records to Kafka mitigates this, allowing the connector to commit the latest source offsets from the database it has received, also when not emitting any of these changes actually to Kafka.

Hth,

--Gunnar

To view this discussion visit https://groups.google.com/d/msgid/debezium/b21c5da1-26fe-44ed-8604-3f000c84e2b7n%40googlegroups.com.

[Prabhatika Vij]

Understood, it does help. Thank you for your reply Gunnar!

Prabhatika 

You received this message because you are subscribed to a topic in the Google Groups "debezium" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/debezium/JFXqYFvFIns/unsubscribe.
To unsubscribe from this group and all its topics, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/CADGJaX_xSj8nQ1q7XaOmAd6xPYizS8FcCM9h7QhLp83eJmkcug%40mail.gmail.com.