Re: Starting Debezium from an offset

[Saurabh Godbole]

Hi,

This is in continuation to https://groups.google.com/g/debezium/c/un_Z31pnLlw. My messages there were getting auto-deleted.

Thanks Jiri and Gunnar for your suggestions. 

I looked at the logs and saw that the Dbhistory entries were getting skipped.
 io.debezium.relational.history.FileDatabaseHistory: Skipping: CREATE TABLE `sbtest` (...

I realized after looking at the source code that this was because the file/pos stamped on the Dbhistory entries was later than the offset I was providing Debezium. 

Is there a way to start Debezium from an earlier point in time ie. an older offset value?

Thanks,

Saurabh Godbole 

[Gunnar Morling]

Sorry again for the mess caused by the moderation hick-ups. Not sure whether it's still relevant, so just in case: the connector will continue from what ever offset it reads back from the offset store. So if you set that to an earlier offset, it will continue from there.

--Gunnar

[Saurabh Godbole]

No worries!

I am using a customOffsetStore based on the MemoryOffsetBackingStore. I am setting the offset to an older value, but it is not allowing me to start streaming from that point as the binlog file and pos stamped on the DBHistory file entries are later than the offset I provided.

Summarizing the steps from the earlier conversation :

1. I deleted all the dbhistory files that had been created locally.
2. I started Debezium in snapshot_mode=schema_only. I verified that the dbhistory file had been generated
3. Then I restarted the connector, loading the offset values that I had previously recorded.
4. Debezium gave the following message:
   "MySqlSnapshotChangeEventSource.getSnapshottingTask: A previous offset indicating a completed snapshot has been found. Neither schema nor data will be snapshotted."
   Followed by the error after a few lines:
   "MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired: Encountered change event {<event>} at offset {<supplied_offset>} for table {xyz} whose schema isn't known to this connector. One possible cause is an incomplete database history topic. Take a new snapshot in this case"