The db history topic or its content is fully or partially missing. Please check database history topic configuration and re-execute the snapshot

[Logu Venkatachalam]

I see this thread in the Confluence developer forum.

https://forum.confluent.io/t/debezium-the-db-history-topic-or-its-content-is-fully-or-partially-missing-please-check-database-history-topic-configuration-and-re-execute-the-snapshot/4614

We see this exception often in our logs. After this point the Connector stopped and no events flow after that.

As the above URL suggests recreating the connector with different name seem to work, but that is not a solution for us.

Any suggestions how we can fix this?

Logu

[Logu Venkatachalam]

Today we upgraded to debezium 1.9.2 and the exception

io.debezium.DebeziumException: The db history topic or its content is fully or partially missing. Please check database history topic configuration and re-execute the snapshot.

started appearing immediately.

Any suggestion how we can fix this?

[CCHsu]

Hi, Logu,

As far as I know, the only way I can find is to use a different task name when trying DBZ for SQL Server, 

and I also find it not good at the first glance.

The current steps I find acceptable is outlined as below.

1) `curl -X DELETE /connectors/{name}` the current task 

2) Rename the task (`name` property)

3) Additionally set `"snapshot.mode": "schema_only"` property

4) Load the "new" task with `curl -X POST /connectors`

5) Check the kafka connect log to confirm the issue solved

6) Remove `"snapshot.mode": "schema_only"` property and reload the "new" task with `curl -X PUT /connectors/{new_name}`

Disclaimer: I am also on the way trying/testing DBZ.  😅

log...@gmail.com 在 2022年6月21日 星期二清晨7:42:03 [UTC+8] 的信中寫道：

[Chris Cranford]

Hi Logu -

All Debezium relational connectors except PostgreSQL require a database history topic in order to function.  This topic must be configured with the correct retention, partition, and clean-up policies based on the documentation [1].  I would first start by checking the configuration of the history topic with what we outline in the documentation and adjust those settings if they're not correct.  This is more often than not the reason for that error message on restart.  The less common scenario is someone inadvertently removed the history topic from Kafka and if that happened, you may be able to repair it by using "snapshot.mode=schema_only_recovery" if the connector involved supports that mode and no schema changes have occurred since the last execution of the connector.  If you can't guarantee that point about schema changes, then unfortunately you'll need to redeploy the connector.

A redeploy of the connector does not necessarily require an initial snapshot.  If you are on the latest versions of the connector, you should be able to use a schema-only snapshot mode (if the connector supports it) to begin streaming changes from the current time and then trigger an incremental snapshot using a signal (requires configuring signals) and that would capture the changes missed from the last successful execution and the time you redeployed.

Hope that helps,
CC

[1]: https://debezium.io/documentation/reference/stable/install.html#_configuring_debezium_topics

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/3f74bfa0-784d-4b0f-b547-fe122ad8696dn%40googlegroups.com.

[Logu Venkatachalam]

Thanks Chris. I am going to check this. Checked the document you gave. It is not clear how to set this up from the doc.

Do you have a sample configuration for SQLServer? That will help a lot.

[Chris Cranford]

Hi Logu -

The history topic configuration is universal and isn't specific to a connector (outside of the fact that PG doesn't use it naturally).  What you want to confirm at the topic side is the following:

    - Infinite retention policy
    - No compaction
    - Single partition

If that isn't the case, this will impact the validity of the history topic and lead to problems.

Hope that helps
CC

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/30af4bf1-0f72-49c3-a712-bd3ad596b908n%40googlegroups.com.

[Logu Venkatachalam]

Thanks Chris for the reply. I am new to debezium and I really do not understand what you are saying. I need to read about debezium. Let me read more and see if can follow these steps.

[Logu Venkatachalam]

Chris, we are using Azure EventHub. Just came to know that EventHub does not support Infinite retention policy. It only supports max 60 days. Is there a workaround for this combination?

[Chris Cranford]

Logu, you could configure the database history topic configuration to either a Kafka-based topic or you could point to a file on the filesystem if you have access to a persisted volume.  See the debezium.source.database.history configuration options found here: https://debezium.io/documentation/reference/stable/operations/debezium-server.html#debezium-source-database-history-class

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/53d44b92-9d2e-4b8d-83be-48adcce9cbc8n%40googlegroups.com.

[Logu Venkatachalam]

Thanks Chris. We applied these configs and they were rejected as "Unrecognized config variables". Later found in the document you sent these are for Debezium Server.

This is our setting

SQL Server -> Debezium SQL Server connector (we are just using the jar files) -> Kafka Connect -> Event Hub.

Looking for instructions to setup database history topic for this environment. I read through the document online and I am unable to figure this out. I would appreciate any and every help on this.

[Chris Cranford]

Hi Logu -

I apologize for the confusion, you mentioned EventHubs not knowing about retention policies so I just assumed you were using Debezium Server + EventHub sink. 

But I think based on what you have provided, the same philosophy holds true.  As I understand it, EventHubs has a pretty strict retention policy and I believe the default is something like 24 hours, which won't work for database history topics at all.  So what you will need to do is explicitly configure the database history to be stored in a local file or in a separate Kafka broker from EventHubs.

The first option you need to adjust is this:

    database.history

This uses the "io.debezium.relational.history.KafkaDatabaseHistory" implementation by default.  You have two options here.  The first is to change this to use the file-based implementation, io.debezium.relational.history.FileDatabaseHistory class name.  This will require you have a shared volume available across the Kafka Connect cluster so that the connector task can access the file.  If you'd rather keep the history topic in a remote topic, you could also consider standing up a separate Kafka broker where you store just the history topic with infinite retention since EventHubs does not offer that.  For this you would need to adjust the broker host/ip and credentials using the database.history.* settings pass those through to the underlying Kafka Connect APIs like you would when configuring credentials and SSL communications with the broker.  See the pass-thru section at https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#debezium-sqlserver-connector-database-history-configuration-properties

Hope that helps and let us know if you have any other questions.
CC

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/593902c9-f87f-4171-ae34-2d4ea497f213n%40googlegroups.com.

[Logu Venkatachalam]

Wow! Thank you so much, Chris! I am going to try this today.