Debezium Oracle Connector 3.1.x sends null key for Schema History messages

[Renan Rocha]

Hello Debezium Team and Community,

I have found a situation in the Debezium Oracle connector across the 3.1.x release series. After extensive troubleshooting on a stable Kafka cluster (3 nodes), the connector consistently fails when using the cleanup.policy=compact for the schema history topic.

Here is a summary of the findings:

Environment:

• Debezium Connector Version: Tested with 3.1.0.Final, 3.1.2.Final, and 3.1.3.Final. The error occurs in all of them.
• Apache Kafka Version: 3.9.1
• Database: Oracle (using Logminer adapter)
• Schema History Topic Configuration: Manually created with cleanup.policy=compact, partitions=1, replication-factor=3.

Observed Behavior: The connector fails during the initial schema snapshot. The Kafka Connect log shows a DebeziumException caused by org.apache.kafka.common.InvalidRecordException: Compacted topic cannot accept message without key...

Our investigation confirmed that the connector is attempting to produce messages to the schema history topic with a null key. We also observed that the connector was erratically writing to two different history topics (topic.prefix and schema.history.internal.kafka.topic) with two different message formats (one with a valid key, and one with a null key).

Expected Behavior: The connector should generate a valid, non-null key for all messages produced to the schema history topic, allowing it to start successfully with a topic configured with cleanup.policy=compact.

Workaround Found: The functional workaround was to use the schema history topic's configuration to cleanup.policy=delete and retention.ms=-1. This allows the Kafka broker to accept the null-key messages, but it forces the connector to use a less efficient, log-based history mechanism.

I hope this information is helpful, maybe for debugging the issue.

Thank you for your excellent work on the Debezium project.

Best regards,

Renan

[Renan Rocha]

[Gunnar Morling]

Hey,

The schema history topic must not be compacted, all the records are needed in order to recover the history after a restart.

--Gunnar

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/9d007db8-df45-48c9-b5bf-d5497ca89e1cn%40googlegroups.com.

[Renan Rocha]

Hi Gunnar, 

Thank you for the definitive clarification. This is very helpful. 

Best regards,  
Renan