Monitor Incremental Snapshotting

[Drew von Zweck]

We are trying to move away from pausing long-running connectors and kicking off a full new snapshot-only connector by switching to Incremental Snapshotting. A concern we have with this is the ability to have insight into debeziums incremental snapshotting process. Specifically if there are any metrics or methods to track...

1. The connector consumed the signal from the signal topic

2. The incremental snapshot for each table has begun and/or completed

Right now we know the snapshot failed/finished by tracking the snapshot-only connector's status using the metric SnapshotCompleted 

(https://debezium.io/documentation/reference/stable/connectors/mysql.html#connectors-snaps-metric-snapshotcompleted_mysql), and are hesitant to give up this level of insight and monitoring..

Thanks.

[Chris Cranford]

Hi Drew -

There isn't anything concrete to provide either of those two directly; however these could possibly be deduced indirectly.  The RemainingTableCount and TotalTableCount do not increase except when an Incremental Snapshot signal is received.  So you could use the increase to detect a new signal was seen.  In terms of the latter, that would gets to be a bit more tricky.  I do think we could consider adding a String-based property to the metrics, similar to the ChunkFrom/ChunkTo but this would hold the current table being snapshot.  Could you raise a Jira for this?

Thanks,
Chris

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/1aadc97b-fd14-493c-b801-3373b576a678n%40googlegroups.com.

[Drew von Zweck]

We have created a Jira Ticket here. 

Thanks for your help, and please reach out with any questions or concerns.