Outbox - expand json payload throws error

[Sebastian Wallace]

We're using Debezium 1.8 with the outbox transform. We're happy with the results so far, but we just have one issue.

When we apply:

"transforms.outbox.table.expand.json.payload": "true"

We get an error from the connector:

"org.apache.kafka.connect.errors.DataException: Invalid Java object for schema type STRING: class org.apache.kafka.connect.data.Struct for field: \"payload\"\n\tat org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:245)\n\tat org.apache.kafka.connect.data.Struct.put(Struct.java:216)\n\tat org.apache.kafka.connect.data.Struct.put(Struct.java:203)\n\tat io.debezium.transforms.outbox.EventRouterDelegate.apply(EventRouterDelegate.java:176)\n\tat io.debezium.transforms.outbox.EventRouter.apply(EventRouter.java:25)"

Here is the outbox config:

transforms": "outbox",
"transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
"transforms.outbox.route.topic.replacement": "topics.${routedByValue}",
"transforms.outbox.route.by.field": "streamType",
"route.by.field": "streamType",
"transforms.outbox.table.field.id": "eventId",
"transforms.outbox.table.field.key": "aggregateId",
"transforms.outbox.table.field.event.key": "aggregateId",
"transforms.outbox.table.field.type": "eventType",
"transforms.outbox.table.field.event.payload": "payload",
"transforms.outbox.table.field.event.payload.id": "aggregateId",
"transforms.outbox.table.expand.json.payload": "true",
"transforms.outbox.table.fields.additional.placement": "aggregateId:envelope:aggregateId,eventType:envelope:eventType,metadata:envelope:metadata,changesetId:envelope:changesetId,streamType:envelope:streamType,eventId:envelope:eventId",
"tombstones.on.delete": "false",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"key.converter.schemas.enable": "false",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
"include.schema.changes": "false"

Our issue is very similar to this:
https://stackoverflow.com/questions/70323167/debezium-outbox-pattern-property-transforms-outbox-table-expand-json-payload-not

Except I get the same error if I try to use the StringConverter for the value.converter.

Any help to resolve this will be very much appreciated.

[Gunnar Morling]

Could you double-check whether you indeed are using the StringConverter for the message values? In the configuration above you use the StringConverter for message keys, but the JSON one for message values.

Thanks,

--Gunnar