Long-running MySQL incremental snapshot restarting or failing to resume at last_processed_key?

[Stephen Tyndall]

I'm performing incremental snapshots of some very large (~20 billion row) MySQL tables. I'm running the kafka-connect workers on kubernetes, and the workers get restarted occasionally as we consolidate workloads on different nodes as total kubernetes cluster load changes. This has not been an issue for normal operation.

During these long snapshots, I've seen the snapshot start over from the beginning after about 5 days (or in row count, ~2 billion rows). 

The notification service gives me the following log lines:
Final normal read event:

[Notification Service] {id='d32f8457-e65b-42f5-8871-a88e70e85ed5', aggregateType='Incremental Snapshot', type='IN_PROGRESS', additionalData={current_collection_in_progress=myschema.mytable, connector_name=my-connector, maximum_key=16525195604, last_processed_key=2362771751, data_collections=myschema.mytable}, timestamp=1764192175391}

Apparent restart:
[Notification Service] {id='2ebb1e87-b61d-4dc8-a0d9-c494700d820e', aggregateType='Incremental Snapshot', type='IN_PROGRESS', additionalData={current_collection_in_progress=myschema.mytable, connector_name=my-connector, maximum_key=16525195604, data_collections=myschema.mytable, last_processed_key=382}, timestamp=1764192203298}

I'm running debezium 3.0. Is there a reason that an incremental snapshot might start from the beginning? Where does the connector store the current last_processed_key for resumption?

Thanks for your help.

[Mario Fiore Vitale]

Hi, 

could you please enable trace logs for `io.debezium.pipeline.source.snapshot.incremental` package and send the full logs (before and after the restart). 

Thanks, 
Mario.