Question Regarding debezium monitor
26 views
Skip to first unread message
Shalvin K
Mar 27, 2026, 8:33:26 AM (5 days ago) Mar 27
to debezium
Hi Debezium Community,
We're running a Debezium embedded engine for MySQL with a custom ChangeConsumer implementation. I'd love to get some thoughts on our data loss prevention and monitoring strategy.
Background
Previously we were on Debezium 1.8 without a ChangeConsumer. In that setup, our internal Kafka producer and Debezium's offset commit producer were independent, which we believe led to occasional data loss — Debezium would commit the offset while the corresponding Kafka batch was silently dropped.
What we've done
We've since upgraded Debezium and implemented a ChangeConsumer, making Kafka message delivery and offset commits atomic. This resolves the original race condition, but we want to go further.
What we need
Given that data quality is now a P0 concern for us, we want to ensure:
1. Zero data loss from the Debezium pipeline, and
2. Active alerting if any loss does occur, so we can detect and remediate it quickly.
Our current thinking
We're considering running a binlog client as a sidecar that independently reads the MySQL binlog, filters for the tables we care about, and maintains its own event count. We'd then compare this count against what Debezium has actually pushed to Kafka and alert on any mismatch.
Before we build this out, we wanted to ask:
- Is there a recommended or community-standard approach for Debezium data loss detection?
- Are there any built-in metrics or hooks (e.g., JMX, event counters) we should be leveraging?
- Has anyone implemented a similar sidecar/shadow-reader pattern and found pitfalls we should watch out for?
Any guidance or pointers to prior discussions would be greatly appreciated. Thanks!
We're running a Debezium embedded engine for MySQL with a custom ChangeConsumer implementation. I'd love to get some thoughts on our data loss prevention and monitoring strategy.
Background
Previously we were on Debezium 1.8 without a ChangeConsumer. In that setup, our internal Kafka producer and Debezium's offset commit producer were independent, which we believe led to occasional data loss — Debezium would commit the offset while the corresponding Kafka batch was silently dropped.
What we've done
We've since upgraded Debezium and implemented a ChangeConsumer, making Kafka message delivery and offset commits atomic. This resolves the original race condition, but we want to go further.
What we need
Given that data quality is now a P0 concern for us, we want to ensure:
1. Zero data loss from the Debezium pipeline, and
2. Active alerting if any loss does occur, so we can detect and remediate it quickly.
Our current thinking
We're considering running a binlog client as a sidecar that independently reads the MySQL binlog, filters for the tables we care about, and maintains its own event count. We'd then compare this count against what Debezium has actually pushed to Kafka and alert on any mismatch.
Before we build this out, we wanted to ask:
- Is there a recommended or community-standard approach for Debezium data loss detection?
- Are there any built-in metrics or hooks (e.g., JMX, event counters) we should be leveraging?
- Has anyone implemented a similar sidecar/shadow-reader pattern and found pitfalls we should watch out for?
Any guidance or pointers to prior discussions would be greatly appreciated. Thanks!
Chris Cranford
Mar 31, 2026, 3:10:54 AM (yesterday) Mar 31
to debe...@googlegroups.com
Hi -
There's no one size fits all answer.
The most common way is to take an extract from the MySQL source and compare that using some unique-identity means with your target. If your target is your Kafka topics, you'd need to guarantee that you have infinite retention so that you have at least the last logical event for a given primary key in the topic. But this is rarely the case, so your target will most likely be whatever your destination system is for your change consumer.
As for metrics, yes the connector does maintain counts for number of events seen, broken down by create, update, and deletes on newer versions. However, metrics don't persist across connector restarts, so these are not going to be as useful if you need to look across a long time period.
As for a sidecar, no one has specifically mentioned that across all our support channels, but it's possible someone may have in some capacity. As for pitfalls, the only thing I would watch for are changes to primary keys. In transaction logs, this is seen as a singular logical event, but Debezium emits this operation as two logical events (1 delete for the old primary key, and 1 create for the new primary key). So when you're looking at metrics or doing comparisons at the binlog level, just be aware of certain nuances like this.
Hope that helps.
-cc
There's no one size fits all answer.
The most common way is to take an extract from the MySQL source and compare that using some unique-identity means with your target. If your target is your Kafka topics, you'd need to guarantee that you have infinite retention so that you have at least the last logical event for a given primary key in the topic. But this is rarely the case, so your target will most likely be whatever your destination system is for your change consumer.
As for metrics, yes the connector does maintain counts for number of events seen, broken down by create, update, and deletes on newer versions. However, metrics don't persist across connector restarts, so these are not going to be as useful if you need to look across a long time period.
As for a sidecar, no one has specifically mentioned that across all our support channels, but it's possible someone may have in some capacity. As for pitfalls, the only thing I would watch for are changes to primary keys. In transaction logs, this is seen as a singular logical event, but Debezium emits this operation as two logical events (1 delete for the old primary key, and 1 create for the new primary key). So when you're looking at metrics or doing comparisons at the binlog level, just be aware of certain nuances like this.
Hope that helps.
-cc
Any guidance or pointers to prior discussions would be greatly appreciated. Thanks! --
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/db7e6e19-2c08-427f-acfc-a6ba3b263309n%40googlegroups.com.
Shalvin K
7:55 AM (6 hours ago) 7:55 AM
to debezium
Hi Chris,
Thanks for the response and for the helpful insights. This gives us a good direction as we evaluate possible approaches for monitoring and validation.
Appreciate the guidance!
Reply all
Reply to author
Forward
0 new messages