Supported Signal Channels for Incremental Snapshot for MySQL and MongoDB
15 views
Skip to first unread message
Divya Dass
Aug 6, 2025, 9:38:22 PMAug 6
to debezium
Hello,
We have a setup where the Debezium server is running in K8s and pushing data Pub/Sub, from where it goes to BQ via bq subscriptions.
We are facing issues with a snapshot of a large tables (>50 GB,) and we are looking for an incremental snapshot.
The problem is we are not allowed to have write permission on the source database. The official documentation says that we can either use source or kafka topic to trigger an incremental snapshot.
My Question is:
Can we use "file" as the signal mechanism to trigger the incremental snapshot? Since it's part of signal.enabled.channels property. I am thinking of using external PVC for the pod for this.
Regards,
Divya
Chris Cranford
Aug 6, 2025, 9:58:18 PMAug 6
to debe...@googlegroups.com
Hi -
Yes, you can use the "file" signal to trigger the incremental snapshot, but that won't be sufficient as write access is required for Incremental Snapshots unless the source database supports what is called "read-only" incremental snapshots.
By default, incremental snapshots write window markers to the signal table. These markers are then captured in the CDC processing loop to identify where the incremental snapshot chunk read should be performed and what data set read from the transaction logs should undergo a process called de-duplication so that you don't get a snapshot and a change event for the same row.
For MySQL, MariaDB, and PostgreSQL, there is a specialization of incremental snapshots called read-only, where these window markers are managed in a different way that does not require write permissions on the database. For MySQL and MariaDB, this requires that GTID mode be enabled.
So in your case, while you can use the "file" channel to trigger the incremental snapshot, if MySQL does not have GTID mode enabled, then write access to the database is mandatory. For MongoDB there is no read-only specialization, so write access is required.
So I'm afraid without write permissions, MongoDB won't be able to use Incremental Snapshots and potentially MySQL if you don't have GTIDs enabled.
-cc
Yes, you can use the "file" signal to trigger the incremental snapshot, but that won't be sufficient as write access is required for Incremental Snapshots unless the source database supports what is called "read-only" incremental snapshots.
By default, incremental snapshots write window markers to the signal table. These markers are then captured in the CDC processing loop to identify where the incremental snapshot chunk read should be performed and what data set read from the transaction logs should undergo a process called de-duplication so that you don't get a snapshot and a change event for the same row.
For MySQL, MariaDB, and PostgreSQL, there is a specialization of incremental snapshots called read-only, where these window markers are managed in a different way that does not require write permissions on the database. For MySQL and MariaDB, this requires that GTID mode be enabled.
So in your case, while you can use the "file" channel to trigger the incremental snapshot, if MySQL does not have GTID mode enabled, then write access to the database is mandatory. For MongoDB there is no read-only specialization, so write access is required.
So I'm afraid without write permissions, MongoDB won't be able to use Incremental Snapshots and potentially MySQL if you don't have GTIDs enabled.
-cc
--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/46246652-6e07-4922-8116-727434662a58n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages