Database history couldn't be recovered Issue

[Ankit Khandelwal]

Hi Team

We are using debezium 2.4.0 as a connector with kafka connect and we are live with 100+ dbservers with 1000+ schemas and 400+ tables from each schema. We are facing this database history couldn't be recovered issue in some connectors. To be precise, we see the error : org.apache.kafka.connect.errors.ConnectException: java.lang.IllegalStateException:
The database history couldn't be recovered.
Consider to increase the value for database.history.kafka.recovery.poll.interval.ms

As per the suggestion, for some of the cases it gets fixed by increasing this parameter, but in other cases it doesn't get fixed. What is the exact reason for which it fails to recover the history data from the topics. We have the history topic for each connector with infinite retention period.

Thanks 

Ankit Khandelwal

NOTICE: The information contained in this transmission and any attachments is confidential and is for the use of the intended recipient(s) only. If you have received in error, please notify the sender and delete it. If you are not the intended recipient, you must not use, disclose or distribute this e-mail without the author's prior permission.

[Chris Cranford]

Hi Ankit -

If you have not set "schema.history.internal.store.only.captured.tables.ddl" as "true" in your connector deployments, then each connector's schema history tracks schema changes for all tables in that database instance. This can add a lot of activity over time in the history topic, particularly if you are capturing fewer tables than those not captured and you have a high volume of DDL changes applied to your database. You may want to consider changing this setting if that's the case for your environment to reduce the volume of history records tracked in the history topic.

Hope that helps.
-cc

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/0bab4dd3-3d7a-4a4a-918c-b1057aa5de15n%40googlegroups.com.