Hello everyone,
I’m currently facing a blocking issue with using Debezium to replicate near real-time changes from a SQL Server database to an Oracle database. Here is my architecture :
SQL Server database with CDC enabled.
An Azure Container App running a Debezium image with its Debezium Connect connector.
An Azure Event Hubs Namespace containing several Azure Event Hubs acting as Kafka topics.
An on-premises VM running the same Debezium image with a Debezium Sink connector.
Oracle database.
At the moment, the data flow is working correctly from the SQL Server database to Azure Event Hubs. For each change made in the SQL Server database, I can see the logs appearing in the corresponding Event Hubs.
The issue arises with the connector running on my VM (the Debezium Sink connector). It successfully connects to the Azure Event Hubs Namespace but does not "react" when a new log arrives in an Event Hub. I also tested simulating a log sent directly to an Azure Event Hub, and in that case, the Debezium Sink connector returns an error like:
Configured primary key mode 'record_key' cannot have null schema
I suspect an issue with the format of the JSON logs in Azure Event Hubs, but I haven’t found anything online or through AI tools that helps me resolve the problem. Below I’m including additional information about the structure of my test project :
--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/ab9bdd7f-ad17-4857-9cb7-8aad611d65e9n%40googlegroups.com.
SQL Server database with CDC enabled.
An Azure Container App running a Debezium image with its Debezium Connect connector.
An Azure Event Hubs Namespace containing several Azure Event Hubs acting as Kafka topics.
An on-premises VM running the same Debezium image with a Debezium Sink connector.
Oracle database.
At the moment, the data flow is working correctly from the SQL Server database to Azure Event Hubs. For each change made in the SQL Server database, I can see the logs appearing in the corresponding Event Hubs.
TRACE || WorkerSinkTask{id=debezium-sink-connector-0} Applying transformations to record in topic 'sqlserver.ebx_dev.dbo.test' partition 0 at offset 5 and timestamp 1753274414865 with key Struct{ID=1} and value Struct{before=Struct{ID=1,NAME=Hey},after=Struct{ID=1,NAME=Ho},source=Struct{version=3.1.0.Final,connector=sqlserver,name=sqlserver,ts_ms=1753274409747,snapshot=false,db=EBX_DEV,ts_us=1753274409747000,ts_ns=1753274409747000000,schema=dbo,table=TEST,change_lsn=00000247:00063cb8:0002,commit_lsn=00000247:00063cb8:0003,event_serial_no=2},op=u,ts_ms=1753274414369,ts_us=1753274414369758,ts_ns=1753274414369758860} [org.apache.kafka.connect.runtime.WorkerSinkTask]
Bind field 'NAME' at position 2 with type io.debezium.connector.jdbc.type.connect.ConnectStringType: Ho [io.debezium.connector.jdbc.dialect.GeneralDatabaseDialect]
To view this discussion visit https://groups.google.com/d/msgid/debezium/e4e25dec-c300-4f18-bde1-55e1c41d820en%40googlegroups.com.
Hi,
Here are all the logs from the JDBC Sink connector with TRACE logging enabled.
To view this discussion visit https://groups.google.com/d/msgid/debezium/55a886c0-ad04-481a-9577-a283b9910c82n%40googlegroups.com.
Hi Chris,
I realized that I had sent the wrong log file in my previous message, which is why I deleted it. I'm sorry about that — I'll get back to you as soon as I can when I have the correct file.
Thanks for your quick response, by the way !
--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/dfe410b9-0f01-472a-a1e6-26e1b1081121n%40googlegroups.com.
I ended up finding the root cause of the issue by chance, and it was something very simple : the Oracle database credentials provided to the connector were incorrect. Because of the project’s complex architecture, I didn’t realize that the Oracle credentials weren’t working properly. The data pipeline started working once I replaced the Oracle credentials in the Debezium Sink connector.
What’s unfortunate is that the Debezium Sink connector never alerted us that it was having trouble with the credentials.
I want to thank you very much for helping us and pointing us in the right direction, especially by suggesting we enable TRACE logging.