JDBC Sink: Failed to resolve column type for schema: ARRAY (null)
xiong chen
Hello Team,I am using the Debezium sink connector to process Debezium-format topic data from a PostgreSQL database. I found that the active_weekdays value coming as an array is null; it seems it cannot recognize it and throws an error. How should I handle this issue? thank you.
Caused by: org.apache.kafka.connect.errors.ConnectException: Failed to resolve column type for schema: ARRAY (null)
{
"type":"array",
"items":{
"type":"int32",
"optional":true
},
"optional":true,
"field":"active_weekdays"
}
after:{
"active_weekdays":null
}
full topic data：{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"string","optional":false,"field":"earning_bucket_id"},{"type":"string","optional":false,"field":"rule_name"},{"type":"string","optional":false,"field":"rating_type"},{"type":"string","optional":false,"field":"earning_qualifying_rule_id"},{"type":"string","optional":false,"field":"earn_method"},{"type":"string","optional":false,"field":"to_earn_one_unit"},{"type":"string","optional":false,"field":"period_type"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"period_from"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"period_to"},{"type":"boolean","optional":false,"default":false,"field":"is_active_weekdays"},{"type":"array","items":{"type":"int32","optional":true},"optional":true,"field":"active_weekdays"},{"type":"boolean","optional":false,"default":false,"field":"is_active_times"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTime","version":1,"field":"active_times_from"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTime","version":1,"field":"active_times_to"},{"type":"boolean","optional":false,"default":false,"field":"is_deleted"},{"type":"string","optional":false,"field":"created_by"},{"type":"int64","optional":false,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"created_date"},{"type":"string","optional":false,"field":"last_modified_by"},{"type":"int64","optional":false,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"last_modified_date"},{"type":"string","optional":false,"default":"","field":"earning_qualifying_rule_main_id"}],"optional":true,"name":"CDC_ERN.earning.earning_rule.Value","field":"before"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"string","optional":false,"field":"earning_bucket_id"},{"type":"string","optional":false,"field":"rule_name"},{"type":"string","optional":false,"field":"rating_type"},{"type":"string","optional":false,"field":"earning_qualifying_rule_id"},{"type":"string","optional":false,"field":"earn_method"},{"type":"string","optional":false,"field":"to_earn_one_unit"},{"type":"string","optional":false,"field":"period_type"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"period_from"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"period_to"},{"type":"boolean","optional":false,"default":false,"field":"is_active_weekdays"},{"type":"array","items":{"type":"int32","optional":true},"optional":true,"field":"active_weekdays"},{"type":"boolean","optional":false,"default":false,"field":"is_active_times"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTime","version":1,"field":"active_times_from"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTime","version":1,"field":"active_times_to"},{"type":"boolean","optional":false,"default":false,"field":"is_deleted"},{"type":"string","optional":false,"field":"created_by"},{"type":"int64","optional":false,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"created_date"},{"type":"string","optional":false,"field":"last_modified_by"},{"type":"int64","optional":false,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"last_modified_date"},{"type":"string","optional":false,"default":"","field":"earning_qualifying_rule_main_id"}],"optional":true,"name":"CDC_ERN.earning.earning_rule.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false,incremental"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"sequence"},{"type":"int64","optional":true,"field":"ts_us"},{"type":"int64","optional":true,"field":"ts_ns"},{"type":"string","optional":false,"field":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"int64","optional":true,"field":"txId"},{"type":"int64","optional":true,"field":"lsn"},{"type":"int64","optional":true,"field":"xmin"}],"optional":false,"name":"io.debezium.connector.postgresql.Source","field":"source"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"name":"event.block","version":1,"field":"transaction"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"int64","optional":true,"field":"ts_us"},{"type":"int64","optional":true,"field":"ts_ns"}],"optional":false,"name":"CDC_ERN.earning.earning_rule.Envelope","version":2},"payload":{"before":null,"after":{"id":"D5E0CE371AD14EE2BDFA13A79BD1C133","earning_bucket_id":"4862960A685C4D02BB879283085A40EF","rule_name":"NST_OneBucket_MassHybridTable_DiamondBlack","rating_type":"NonSmartTable","earning_qualifying_rule_id":"53B90B5406A74901A72CF564DFCB7C89","earn_method":"TheoWin","to_earn_one_unit":"25.00000000","period_type":"Perpetual","period_from":null,"period_to":null,"is_active_weekdays":false,"active_weekdays":null,"is_active_times":false,"active_times_from":null,"active_times_to":null,"is_deleted":false,"created_by":"qa.tracy.leong","created_date":1762916845798519,"last_modified_by":"qa.tracy.leong","last_modified_date":1762916845798519,"earning_qualifying_rule_main_id":"8E7D9434AA863F25C6E260C2B7034CCC"},"source":{"version":"2.7.3.Final","connector":"postgresql","name":"CDC_ERN","ts_ms":1762916846049,"snapshot":"false","db":"gegcrm","sequence":"[\"52848522888\",\"52848593584\"]","ts_us":1762916846049271,"ts_ns":1762916846049271000,"schema":"earning","table":"earning_rule","txId":15879261,"lsn":52848593584,"xmin":null},"transaction":null,"op":"c","ts_ms":1762916829942,"ts_us":1762916829942078,"ts_ns":1762916829942078000}}
xiong chen
xiong chen: Hello Team,I am using the Debezium sink connector to process Debezium-format topic data from a PostgreSQL database. I found that the active_weekdays value coming as an array is null; it seems it cannot recognize it and throws an error. How should I handle this issue? thank you.
Caused by: org.apache.kafka.connect.errors.ConnectException: Failed to resolve column type for schema: ARRAY (null)
{
"type":"array",
"items":{
"type":"int32",
"optional":true
},
"optional":true,
"field":"active_weekdays"
}
after:{
"active_weekdays":null
}
Connector Config:{
"name": "DMT_ERN_Sink_Connector",
"config": {
"connector.class": "io.debezium.connector.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:sqlserver://XXXX:1433;databaseName=XXX;encrypt=false",
"connection.username": "XXX",
"connection.password": "XXX",
"topics": "CDC_ERN.earning.earning_rule",
"table.name.format": "${source.schema}.${source.table}",
"primary.key.mode": "record_key",
"insert.mode": "upsert",
"delete.enabled": "true",
"name": "DMT_ERN_Sink_Connector",
"consumer.override.group.id": "CDC_DMT_ERN_TO_MSSQL",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable": "true",
"value.converter.schemas.enable": "true"
}
}
Full Topic Data: {"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"string","optional":false,"field":"earning_bucket_id"},{"type":"string","optional":false,"field":"rule_name"},{"type":"string","optional":false,"field":"rating_type"},{"type":"string","optional":false,"field":"earning_qualifying_rule_id"},{"type":"string","optional":false,"field":"earn_method"},{"type":"string","optional":false,"field":"to_earn_one_unit"},{"type":"string","optional":false,"field":"period_type"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"period_from"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"period_to"},{"type":"boolean","optional":false,"default":false,"field":"is_active_weekdays"},{"type":"array","items":{"type":"int32","optional":true},"optional":true,"field":"active_weekdays"},{"type":"boolean","optional":false,"default":false,"field":"is_active_times"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTime","version":1,"field":"active_times_from"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTime","version":1,"field":"active_times_to"},{"type":"boolean","optional":false,"default":false,"field":"is_deleted"},{"type":"string","optional":false,"field":"created_by"},{"type":"int64","optional":false,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"created_date"},{"type":"string","optional":false,"field":"last_modified_by"},{"type":"int64","optional":false,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"last_modified_date"},{"type":"string","optional":false,"default":"","field":"earning_qualifying_rule_main_id"}],"optional":true,"name":"CDC_ERN.earning.earning_rule.Value","field":"before"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"string","optional":false,"field":"earning_bucket_id"},{"type":"string","optional":false,"field":"rule_name"},{"type":"string","optional":false,"field":"rating_type"},{"type":"string","optional":false,"field":"earning_qualifying_rule_id"},{"type":"string","optional":false,"field":"earn_method"},{"type":"string","optional":false,"field":"to_earn_one_unit"},{"type":"string","optional":false,"field":"period_type"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"period_from"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"period_to"},{"type":"boolean","optional":false,"default":false,"field":"is_active_weekdays"},{"type":"array","items":{"type":"int32","optional":true},"optional":true,"field":"active_weekdays"},{"type":"boolean","optional":false,"default":false,"field":"is_active_times"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTime","version":1,"field":"active_times_from"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTime","version":1,"field":"active_times_to"},{"type":"boolean","optional":false,"default":false,"field":"is_deleted"},{"type":"string","optional":false,"field":"created_by"},{"type":"int64","optional":false,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"created_date"},{"type":"string","optional":false,"field":"last_modified_by"},{"type":"int64","optional":false,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"last_modified_date"},{"type":"string","optional":false,"default":"","field":"earning_qualifying_rule_main_id"}],"optional":true,"name":"CDC_ERN.earning.earning_rule.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false,incremental"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"sequence"},{"type":"int64","optional":true,"field":"ts_us"},{"type":"int64","optional":true,"field":"ts_ns"},{"type":"string","optional":false,"field":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"int64","optional":true,"field":"txId"},{"type":"int64","optional":true,"field":"lsn"},{"type":"int64","optional":true,"field":"xmin"}],"optional":false,"name":"io.debezium.connector.postgresql.Source","field":"source"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"name":"event.block","version":1,"field":"transaction"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"int64","optional":true,"field":"ts_us"},{"type":"int64","optional":true,"field":"ts_ns"}],"optional":false,"name":"CDC_ERN.earning.earning_rule.Envelope","version":2},"payload":{"before":null,"after":{"id":"D5E0CE371AD14EE2BDFA13A79BD1C133","earning_bucket_id":"4862960A685C4D02BB879283085A40EF","rule_name":"NST_OneBucket_MassHybridTable_DiamondBlack","rating_type":"NonSmartTable","earning_qualifying_rule_id":"53B90B5406A74901A72CF564DFCB7C89","earn_method":"TheoWin","to_earn_one_unit":"25.00000000","period_type":"Perpetual","period_from":null,"period_to":null,"is_active_weekdays":false,"active_weekdays":null,"is_active_times":false,"active_times_from":null,"active_times_to":null,"is_deleted":false,"created_by":"qa.tracy.leong","created_date":1762916845798519,"last_modified_by":"qa.tracy.leong","last_modified_date":1762916845798519,"earning_qualifying_rule_main_id":"8E7D9434AA863F25C6E260C2B7034CCC"},"source":{"version":"2.7.3.Final","connector":"postgresql","name":"CDC_ERN","ts_ms":1762916846049,"snapshot":"false","db":"gegcrm","sequence":"[\"52848522888\",\"52848593584\"]","ts_us":1762916846049271,"ts_ns":1762916846049271000,"schema":"earning","table":"earning_rule","txId":15879261,"lsn":52848593584,"xmin":null},"transaction":null,"op":"c","ts_ms":1762916829942,"ts_us":1762916829942078,"ts_ns":1762916829942078000}}
Chris Cranford
As mentioned on Zulip [1], when you begin to cross vendor boundaries and use non-standard ANSI92 data types, you introduce risk that there isn't a direct mapping from source to sink. Since SQL Server does not have native array type support like PostgreSQL, there's no direct mapping. You could introduce a `CustomConverter` or a `Transformation` to your pipeline so that the data from PostgreSQL in array-type columns is serialized in another format, such as JSON and written as JSON at the target, as an option.
Otherwise, there would need to be some discussion and code change if there is a reasonable and deterministic way to handle such corner cases.
Thanks,
-cc
[1]: #community-jdbc > Sink: Failed to resolve column type for schema: ARRAY (null)
--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/afa33896-6196-43ce-8500-79d3501296fbn%40googlegroups.com.
cx1078...@gmail.com
Could you give me some examples for me to try?
Beat Regards
| From | Chris Cranford<cran...@gmail.com> |
| Date | 11/12/2025 20:14 |
| To | debe...@googlegroups.com |
| Subject | Re: [debezium] JDBC Sink: Failed to resolve column type for schema: ARRAY (null) |