Starting Debezium from an offset

[Saurabh Godbole]

Hi,

I have a use-case where I need to start debezium from an offset. I have recorded an offset from a binlog file earlier. I am using a customOffsetStore based on the MemoryOffsetBackingStore and FileDatabaseHistory for dbhistory. 

Here is what I did to test resuming from the pre-recorded offset:

1. I deleted all the dbhistory files that had been created locally.
2. I started Debezium in snapshot_mode=schema_only. I verified that the dbhistory file had been generated
3. Then I restarted the connector, loading the offset values that I had previously recorded.
4. Debezium gave the following message:
   "MySqlSnapshotChangeEventSource.getSnapshottingTask: A previous offset indicating a completed snapshot has been found. Neither schema nor data will be snapshotted."
   Followed by the error after a few lines:
   "MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired: Encountered change event {<event>} at offset {<supplied_offset>} for table {xyz} whose schema isn't known to this connector. One possible cause is an incomplete database history topic. Take a new snapshot in this case"
   

Can anyone help me regarding this error? 

[Gunnar Morling]

Hum, that's interesting. Does your DB history file contain an entry for the table in question?

[Saurabh Godbole]

Yes, the db history file does contain 2 entries with the table name in question:

One is a drop table entry, and the other is a create table entry

[jiri.p...@gmail.com]

Hi,

please enable DEBUG logging. It should show you which DDLs and when are they applied when read from history.

I am suspecting that the issue might be causes by mismatch between offsets stored in database history and the restart offsets.

J.