Incremental Snapshot and SCHEMA_ONLY snapshot mode.

[vamsee arcot]

Hello Team, has anyone tried setting up incremental snapshot.

can someone share the steps please..

Also does incremental snapshot work with snapshot.mode set to "SCHEMA_ONLY".?

Please let me know.

Thanks and regards

Vamsee

[Chris Cranford]

Hi Vamsee -

The Incremental Snapshot feature is available only to specific connectors at this point, namely MySQL, PostgreSQL, SQL Server, and DB2.  We are looking to extend this feature to MongoDB and Oracle during the 1.7 release stream.

The first steps is to setup the signaling feature needed to trigger incremental snapshots.  You can find how to do this below:
https://debezium.io/documentation/reference/1.6/configuration/signalling.html

Once the signal table is defined and the connector configuration updated to reflect that, then you're all set. 

To initiate the Incremental Snapshot, insert a record into this table using this criteria:

ID   - A unique identifier for the row
TYPE - Should be set to "execute-snapshot"
DATA - Should be a JSON document that describes the collections to snapshot.

An example JSON document to incrementally snapshot two tables would look like:

{
    "data-collections": [
        "DEBEZIUM.TABLEA",
        "DEBEZIUM.TABLEB"
    ]
}

Lastly, the Incremental Snapshot feature does not care what mode you've set for "snapshot.mode" nor does it use any of the "snapshot.select.override.*" configurations.  The feature itself is based on an algorithm that relies on some deterministic factors to be able to properly merge the state in parallel with streaming and so those other traditional snapshot settings have no impact on the incremental snapshot behavior at all.

HTH,
CC

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/CAD1L9d3COEzi909FF1fDTL-em4EAoXcbvF26Hg_8%2Bc4b0PKXJQ%40mail.gmail.com.

[Gunnar Morling]

> Lastly, the Incremental Snapshot feature does not care what mode you've set for "snapshot.mode" nor does it use any of the "snapshot.select.override.*" configurations.  The feature itself is based on an algorithm that relies on some deterministic factors to be able to properly merge the state in parallel with streaming and so those other traditional snapshot settings have no impact on the incremental snapshot behavior at all.

Thanks a lot for this great reply!

To expand on this particular point: as you say, "snapshot.mode" doesn't apply to incremental snapshots; in particular, "schema_only" wouldn't make much sense. If a new schema snapshot is needed, just use the traditional snapshotting with "schema_only".

Now "snapshot.select.override.*"  is something which would make sense to support for incremental snapshotting, too. It just came up the other day, and it's tracked via https://issues.redhat.com/browse/DBZ-3704. This will require some thinking, as the incremental snapshot SELECT statements are built dynamically. So we may consider to have a way for amending those statements with user-provided WHERE clauses, so to narrow down the set of snapshotted rows.

--Gunnar