schema_only_recovery does not work after purging the db history topic

[Faizan Hadfa]

I had a mysql connector running. After some time the db history topic grew really large and had lot of messages. Due to this history recovery used to fail. 

So I decided to purge the db history topic and tried to start the connector in schema_only_recovery mode. 

But my connector kept failing saying that binlog position recorded is not present, which was true; the last recorded binlog position was not present due to mysql binlog cleanup. I expected it to repopulate the db history topic and then seek for mysql binlog position, which went missing. However it failed and prompted me to start connector using snapshot mode "when_needed". 

When i was trying to do that it stated 

"org.apache.kafka.connect.errors.ConnectException: The db history topic is missing. You may attempt to recover it by reconfiguring the connector to SCHEMA_ONLY_RECOVERY"

I was left with no option but start a connector with new name.

My question is-

1. Why did schema_only_recovery not populate db history topic before failing with missing binlog position error?

2. I also tried using schema_only recovery mode, which captures the current schema and start reading from current binlog position. This also failed saying that 'The db history topic is missing.'. Why did not it create a new history using the current schema.

Regards,

Faizan

[Jiri Pechanec]

Hi,

this is usually a sign of having otdated binlog coordinates stored in the offsets. If it is the case you either need to rename the connector (hence clearing offsets) or manipulate the offsets manually using https://debezium.io/docs/faq/#how_to_change_the_offsets_of_the_source_database procedure. There is request in Kafka community for a tool to manupulate the offsets but nothing has been developed yet.

J.