Debezium server custom converter and sql server default value issue

[Liudas Sodonis]

Hello fellow engineers,

I made a custom debezium converter that converts all values to strings.

https://gist.github.com/lfx/854cb6ffd9f566fedf0aa0a373648ec8

It works well however in some odd conditions it crashes (dump below).

The culprit seems to be line  this https://github.com/debezium/debezium/blob/2.3/debezium-core/src/main/java/io/debezium/relational/TableSchemaBuilder.java#L404

--

403 | fieldBuilder
404 | .defaultValue(customConverterRegistry.getValueConverter(table.id(), column)
405 | .orElse(ValueConverter.passthrough()).convert(defaultValue));

--

The column DDL: EmailPromotion int DEFAULT 0 NOT NULL,

So I wonder if I doing something wrong with my converter?

Or there is a bug somewhere in Debezium it self? 

If I remove the "default 0" and make it empty, the debezium works like a charm.

Debeizum server version 2.3.4-Final. 

Full table definition:

CREATE TABLE aw2019.Person.Person (

BusinessEntityID int NOT NULL,

PersonType nchar(2) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,

NameStyle NameStyle DEFAULT 0 NOT NULL,

Title nvarchar(8) COLLATE SQL_Latin1_General_CP1_CI_AS DEFAULT '' NULL,

FirstName Name COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,

MiddleName Name COLLATE SQL_Latin1_General_CP1_CI_AS NULL,

LastName Name COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,

Suffix nvarchar(10) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,

EmailPromotion int DEFAULT 0 NOT NULL,

AdditionalContactInfo xml DEFAULT '' NULL,

Demographics xml DEFAULT '' NULL,

rowguid uniqueidentifier DEFAULT newid() NOT NULL,

ModifiedDate datetime DEFAULT getdate() NOT NULL,

CONSTRAINT PK_Person_BusinessEntityID PRIMARY KEY (BusinessEntityID)

);

CREATE UNIQUE NONCLUSTERED INDEX AK_Person_rowguid ON aw2019.Person.Person (rowguid);

CREATE NONCLUSTERED INDEX IX_Person_LastName_FirstName_MiddleName ON aw2019.Person.Person (LastName, FirstName, MiddleName);

CREATE INDEX PXML_Person_AddContact ON aw2019.Person.Person (AdditionalContactInfo);

CREATE INDEX PXML_Person_Demographics ON aw2019.Person.Person (Demographics);

CREATE INDEX XMLPATH_Person_Demographics ON aw2019.Person.Person (Demographics);

CREATE INDEX XMLPROPERTY_Person_Demographics ON aw2019.Person.Person (Demographics);

CREATE INDEX XMLVALUE_Person_Demographics ON aw2019.Person.Person (Demographics);

ALTER TABLE aw2019.Person.Person WITH NOCHECK ADD CONSTRAINT CK_Person_EmailPromotion CHECK ([EmailPromotion]>=(0) AND [EmailPromotion]<=(2));

ALTER TABLE aw2019.Person.Person WITH NOCHECK ADD CONSTRAINT CK_Person_PersonType CHECK ([PersonType] IS NULL OR (upper([PersonType])='GC' OR upper([PersonType])='SP' OR upper([PersonType])='EM' OR upper([PersonType])='IN' OR upper([PersonType])='VC' OR upper([PersonType])='SC'));

Dump:

2023-10-13 13:56:31,421 ERROR [io.deb.ser.ConnectorLifecycle] (pool-7-thread-1) Connector completed: success = 'false', message = 'Error while trying to run connector class 'io.debezium.connector.sqlserver.SqlServerConnector'', error = 'org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.': org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:67)
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:125)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1623)
Caused by: io.debezium.DebeziumException: io.debezium.DebeziumException: io.debezium.DebeziumException: Failed to set field default value for 'aw2019.Person.Person.EmailPromotion' of type int, the default value is 0 of type class java.lang.Integer
at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:85)
at io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:186)
at io.debezium.connector.sqlserver.SqlServerChangeEventSourceCoordinator.executeChangeEventSources(SqlServerChangeEventSourceCoordinator.java:84)
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:118)
... 5 more
Caused by: io.debezium.DebeziumException: io.debezium.DebeziumException: Failed to set field default value for 'aw2019.Person.Person.EmailPromotion' of type int, the default value is 0 of type class java.lang.Integer
at io.debezium.relational.RelationalSnapshotChangeEventSource.lambda$createSchemaChangeEventsForTables$2(RelationalSnapshotChangeEventSource.java:374)
at io.debezium.pipeline.EventDispatcher.dispatchSchemaChangeEvent(EventDispatcher.java:396)
at io.debezium.relational.RelationalSnapshotChangeEventSource.createSchemaChangeEventsForTables(RelationalSnapshotChangeEventSource.java:369)
at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:145)
at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:76)
... 8 more
Caused by: io.debezium.DebeziumException: Failed to set field default value for 'aw2019.Person.Person.EmailPromotion' of type int, the default value is 0 of type class java.lang.Integer
at io.debezium.relational.TableSchemaBuilder.addField(TableSchemaBuilder.java:410)
at io.debezium.relational.TableSchemaBuilder.lambda$create$2(TableSchemaBuilder.java:148)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at io.debezium.relational.TableSchemaBuilder.create(TableSchemaBuilder.java:146)
at io.debezium.relational.RelationalDatabaseSchema.buildAndRegisterSchema(RelationalDatabaseSchema.java:122)
at io.debezium.connector.sqlserver.SqlServerDatabaseSchema.applySchemaChange(SqlServerDatabaseSchema.java:53)
at io.debezium.pipeline.EventDispatcher$SchemaChangeEventReceiver.schemaChangeEvent(EventDispatcher.java:651)
at io.debezium.relational.RelationalSnapshotChangeEventSource.lambda$createSchemaChangeEventsForTables$2(RelationalSnapshotChangeEventSource.java:371)
... 12 more
Caused by: org.apache.kafka.connect.errors.SchemaBuilderException: Invalid SchemaBuilder call: default has already been set.
at org.apache.kafka.connect.data.SchemaBuilder.checkCanSet(SchemaBuilder.java:437)
at org.apache.kafka.connect.data.SchemaBuilder.defaultValue(SchemaBuilder.java:126)
at io.debezium.relational.TableSchemaBuilder.addField(TableSchemaBuilder.java:404)
... 27 more

2023-10-13 13:56:31,423 DEBUG [io.qua.run.Application] (main) Stopping application

[jiri.p...@gmail.com]

Hi,

this looks like a bug. Could you please create a Jira issue for it? It seems the only missing info for proper report is connector configuration.

Thanks a lot!

J.

[Liudas Sodonis]

Thanks J,

Created https://issues.redhat.com/browse/DBZ-7045