Error "The database schema history couldn't be recovered" in debezium source conector sql server
294 views
Skip to first unread message
Reginaldo Silva
Mar 7, 2025, 8:31:25 AM3/7/25
to debezium
Hello everyone, I'm constantly getting this error in the Debezium source connector for SQL Server version 3.0.
I've already done some tests by increasing the value of "schema.history.internal.kafka.recovery.poll.interval.ms" to 1000, 5000 and 8000. None of these worked, the connector always fails again for this reason. What can I do to resolve this situation? Thanks
log error :
java.lang.IllegalStateException: The database schema history couldn't be recovered. Consider to increase the value for schema.history.internal.kafka.recovery.poll.interval.ms at io.debezium.storage.kafka.history.KafkaSchemaHistory.recoverRecords(KafkaSchemaHistory.java:312) at io.debezium.relational.history.AbstractSchemaHistory.recover(AbstractSchemaHistory.java:100) at io.debezium.relational.history.SchemaHistory.recover(SchemaHistory.java:192) at io.debezium.relational.HistorizedRelationalDatabaseSchema.recover(HistorizedRelationalDatabaseSchema.java:72) at io.debezium.schema.HistorizedDatabaseSchema.recover(HistorizedDatabaseSchema.java:40) at io.debezium.connector.common.BaseSourceTask.validateAndLoadSchemaHistory(BaseSourceTask.java:151) at io.debezium.connector.sqlserver.SqlServerConnectorTask.start(SqlServerConnectorTask.java:109) at io.debezium.connector.common.BaseSourceTask.startIfNeededAndPossible(BaseSourceTask.java:401) at io.debezium.connector.common.BaseSourceTask.poll(BaseSourceTask.java:311) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.poll(AbstractWorkerSourceTask.java:465) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:353) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:225) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:280) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:78) at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:237) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583)
I've already done some tests by increasing the value of "schema.history.internal.kafka.recovery.poll.interval.ms" to 1000, 5000 and 8000. None of these worked, the connector always fails again for this reason. What can I do to resolve this situation? Thanks
log error :
java.lang.IllegalStateException: The database schema history couldn't be recovered. Consider to increase the value for schema.history.internal.kafka.recovery.poll.interval.ms at io.debezium.storage.kafka.history.KafkaSchemaHistory.recoverRecords(KafkaSchemaHistory.java:312) at io.debezium.relational.history.AbstractSchemaHistory.recover(AbstractSchemaHistory.java:100) at io.debezium.relational.history.SchemaHistory.recover(SchemaHistory.java:192) at io.debezium.relational.HistorizedRelationalDatabaseSchema.recover(HistorizedRelationalDatabaseSchema.java:72) at io.debezium.schema.HistorizedDatabaseSchema.recover(HistorizedDatabaseSchema.java:40) at io.debezium.connector.common.BaseSourceTask.validateAndLoadSchemaHistory(BaseSourceTask.java:151) at io.debezium.connector.sqlserver.SqlServerConnectorTask.start(SqlServerConnectorTask.java:109) at io.debezium.connector.common.BaseSourceTask.startIfNeededAndPossible(BaseSourceTask.java:401) at io.debezium.connector.common.BaseSourceTask.poll(BaseSourceTask.java:311) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.poll(AbstractWorkerSourceTask.java:465) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:353) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:225) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:280) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:78) at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:237) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583)
jiri.p...@gmail.com
Mar 7, 2025, 8:36:10 AM3/7/25
to debezium
Hi,
please try to set it to 30/60 seconds.
Jiri
Reginaldo Silva
Mar 7, 2025, 9:09:15 AM3/7/25
to debezium
Thanks Jiri, I'll do this test.
Reginaldo Silva
Mar 17, 2025, 9:16:55 AM3/17/25
to debezium
Good morning Jiri, I made both suggestions, 60000 solved it for some but it still happens. Is there any other suggestion I can make for these cases? Thank you very much!
jiri.p...@gmail.com
Mar 20, 2025, 6:45:34 AM3/20/25
to debezium
I think you can increase it even more. It seems that the combination of the stored schema and your Kafka broker leads sometimes to large wait times.
You can try to monitor Kafka broker traffic/load and check if there is a relation.
Jiri
Reply all
Reply to author
Forward
0 new messages