Issue with ${routedByValue} var with debezium operator on Kubernetes

[Isabel Fernández Naranjo]

Hello,

I'm running into a challenging configuration issue with the Debezium Operator on Kubernetes and hope someone can provide some insight.

My Setup:

• Environment: Kubernetes

• Operator: Debezium Operator

• Pipeline: MySQL (Source) -> Debezium Server -> RabbitMQ (Sink)

• Debezium Server Version: 3.1

I am implementing the outbox pattern and need to use the io.debezium.transforms.outbox.EventRouter SMT. The key requirement is to set the dynamic routing property: debezium.transforms.outbox.route.topic.replacement=${routedByValue}

The EventRouter transform itself needs to process the literal string ${routedByValue} at runtime.

I cannot find a reliable way to get this literal string ${routedByValue} into the final application.properties file inside the Debezium pod. The Kubernetes and shell environment consistently tries to substitute this variable at startup, which causes the application to fail with errors like a NullPointerException.

I need to substitute environment variables like ${RABBITMQ_PASSWORD} (from a Kubernetes Secret), but I must prevent the substitution of ${routedByValue}.

What is the canonical, recommended way to set a property that contains a Debezium-internal placeholder like ${routedByValue} when using the Debezium Operator?

It seems like a fundamental use case, but I've hit a wall with shell escaping. Is there a simpler, more official method that I'm completely missing?

Any help would be greatly appreciated!

Isabel.

La información contenida en este mensaje de correo electrónico y, en su caso, en cualquier fichero anexo al mismo, tiene carácter reservado y confidencial para uso exclusivo de su/s destinatario/s, por lo que su divulgación, copia o distribución a terceros están expresamente prohibidas. En caso de haber recibido este mensaje por error se ruega comunicarlo inmediatamente al remitente y eliminarlo a continuación.

Sus datos serán tratados por Zertiban S.L.U. con la finalidad de mantener las relaciones contractuales, comerciales y profesionales que nos unen a Ud. Para obtener más información sobre cesiones, legitimidad del tratamiento, plazo de conservación de sus datos y dirección donde ejercer sus derechos de protección de datos, por favor, acceda a Política de Privacidad de Zertiban S.L.U. o diríjase a le...@zertiban.com

[Mario Fiore Vitale]

Hi Isabel, 

can you please share the full DebeziumServer configuration? Also you also attach full logs that contains the NPE?

Thanks, 
Mario.

[Isabel Fernández Naranjo]

Hi Mario,

This is the configuration:

apiVersion: debezium.io/v1alpha1

kind: DebeziumServer

metadata:

name: debezium-server-user-service

namespace: debezium-operator

labels:

kustomize.toolkit.fluxcd.io/substitute: disabled

spec:

quarkus:

config:

log.console.json: false

log.level: INFO

http.port: 8081

runtime:

environment:

from:

- secretRef:

name: debezium-secrets

templates:

pod:

metadata:

labels:

app.kubernetes.io/type: infra

sink:

type: rabbitmq

config:

connection.host: rabbitmq.rabbitmq-system.svc

connection.port: 5672

connection.username: rabbitmq-user-service-test

connection.password: ${RABBITMQ_PASSWORD}

connection.virtual.host: /

ackTimeout: 3000

delivery.mode: 2

routingKey.source: key

source:

class: io.debezium.connector.mysql.MySqlConnector

config:

database.hostname: ${MYSQL_HOST}

database.dbname: user-service-test

database.port: 3306

database.user: debezium-user-service-test

database.password: ${MYSQL_PASSWORD}

table.include.list: user-service-test.OUTBOX

database.include.list: debezium-user-service-test

database.server.id: 184060

offset.flush.interval.ms: 0

topic.prefix: outbox

schema.history.internal: io.debezium.storage.file.history.FileSchemaHistory

schema.history.internal.file.filename: /debezium/data/schistory.dat

offset.storage.file.filename: /debezium/data/offsets.dat

snapshot.mode: initial

schema.history.internal.store.only.captured.databases.ddl: true

schema.history.internal.store.only.captured.tables.ddl: false

format:

header:

config:

schemas.enable: false

type: json

key:

type: simplestring

value:

config:

schemas.enable: false

type: json

transforms:

- type: filter

config:

language: jsr223.groovy

condition: value.schema().field("op") && value.getString("op") == "c"

type: io.debezium.transforms.Filter

- type: outbox

config:

route.by.field: EXCHANGE

route.topic.regex: (?<routedByValue>.*)

route.topic.replacement: ${routedByValue}

table.field.event.id: UUID

table.field.event.key: ROUTING_KEY

table.field.event.timestamp: PRODUCED_TIMESTAMP

table.field.event.payload: PAYLOAD

table.expand.json.payload: true

table.json.payload.null.behavior: ignore

table.fields.additional.placement: EVENT_TYPE:header:__TypeId__,PRODUCED_TIMESTAMP:header:timestamp

table.fields.additional.error.on.missing: true

route.tombstone.on.empty.payload: false

table.op.invalid.behavior: error

type: io.debezium.transforms.outbox.EventRouter

And the logs are attached to this message.

Thank you,

Isabel.

[Mario Fiore Vitale]

Hi Isabel, 

I had a look at the issue and it seems that the problem is not with Kubernetes or Operator itself but with Debezium Server. In particular, the ${something} interferes with the Quarkus property expansion [1].

@Jiri Pechanec to me this seems to be a bug on the SMT side since it seems not to be compatible to be used with Debezium Server. WDYT? 
If I have not missed something I think we can open a bug. 

[1] https://quarkus.io/guides/config-reference#property-expressions

--
You received this message because you are subscribed to a topic in the Google Groups "debezium" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/debezium/KoaWgfAOLEY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/6b661df3-37b8-4468-81d5-c11dd8cfd5b5n%40googlegroups.com.

--

Mario Fiore Vitale

Senior Software Engineer

Red Hat

[Chris Cranford]

Hi Mario & Isabel -

Have we tried to escape the `$` character like as follows:

    \${routedByValue}
    $${routedByValue}

If I recall, this is one way in Quarkus to treat the `$` token so that MicroProfile does not expand.

-cc

You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/CALDUC%2Bbofyn4xA0_DmnUMeZPgyS_80pfKudcTLsksZcJ1ZFisQ%40mail.gmail.com.

[Mario Fiore Vitale]

Hi Chris, 

Yes, I have tried with  $${routedByValue} but even if Quarkus doesn't break then the value of the property is $${routedByValue} the will be not recognized by the SMT. Isn't it?

To view this discussion visit https://groups.google.com/d/msgid/debezium/c8a90116-0dc6-499d-bb31-1c7090e43c16%40gmail.com.

[Chris Cranford]

Hi Mario -

I have not tested this myself, but my understanding is that the `$$` avoids MicroProfile doing the replacement, but then Quarkus has a pass where seeing `$$` is converted to just `$`. So my understanding is the transform should get the value as it expects, `${routedByValue}`.

-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/CALDUC%2BayksNYBto5bXcNQfHBi1%2BF5BjDBKkfN02cyu%2B-gaaSOw%40mail.gmail.com.

[Mario Fiore Vitale]

Well, Isabel, can you test it with $${routedByValue} and let me know if it works?

Thanks,

Mario.

To view this discussion visit https://groups.google.com/d/msgid/debezium/09ab9728-7b61-47f0-908f-e89a3f93b4f3%40gmail.com.

[Isabel Fernández Naranjo]

Hi Mario,

I tested it, but it shows the same null pointer exception error, as if the value of the variable was empty or something.

Thank you,

Isabel.

[Mario Fiore Vitale]

Hi Isabel, 

I have tests it with Debezium Server distribution adding the following config 

debezium.api.enabled=true
debezium.format.header=json
debezium.format.key=json
debezium.format.value=json
debezium.sink.http.url=https://www.postb.in/1755786226203-2970293918624
debezium.sink.type=http
debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
debezium.source.database.dbname=postgres
debezium.source.database.hostname=localhost
debezium.source.database.password=postgres
debezium.source.database.port=5432
debezium.source.database.user=postgres
debezium.source.offset.flush.interval.ms=60000
debezium.source.offset.storage=org.apache.kafka.connect.storage.MemoryOffsetBackingStore
debezium.source.schema.history.internal=io.debezium.relational.history.MemorySchemaHistory
debezium.source.schema.include.list=inventory
debezium.source.topic.prefix=inventory
debezium.transforms.t0.condition=value.schema().field("op") && value.getString("op") == "c"
debezium.transforms.t0.language=jsr223.groovy
debezium.transforms.t0.negate=false
debezium.transforms.t0.type=io.debezium.transforms.Filter
debezium.transforms.t1.negate=false
debezium.transforms.t1.route.by.field=EXCHANGE
debezium.transforms.t1.route.tombstone.on.empty.payload=true
debezium.transforms.t1.route.topic.regex=(?<routedByValue>.*)
debezium.transforms.t1.route.topic.replacement=$${routedByValue}
debezium.transforms.t1.table.expand.json.payload=true
debezium.transforms.t1.table.field.event.id=UUID
debezium.transforms.t1.table.field.event.key=ROUTING_KEY
debezium.transforms.t1.table.field.event.payload=PAYLOAD
debezium.transforms.t1.table.field.event.timestamp=PRODUCED_TIMESTAMP
debezium.transforms.t1.table.fields.additional.error.on.missing=true

and as a result it starts with the correct configuration 

2025-08-21 16:32:38,115 INFO  [io.deb.con.com.BaseSourceTask] (pool-9-thread-1) Starting PostgresConnectorTask with configuration:
   connector.class = io.debezium.connector.postgresql.PostgresConnector
   transforms = t0,t1
   schema.include.list = inventory
   transforms.t1.table.field.event.timestamp = PRODUCED_TIMESTAMP
   record.processing.threads =

   transforms.t1.route.topic.replacement = ${routedByValue} 

Can you please check again?

Thanks, 
Mario. 

To view this discussion visit https://groups.google.com/d/msgid/debezium/ab7a36dc-b8ce-422a-a7e1-95f9e26a83a2n%40googlegroups.com.

[Isabel Fernández Naranjo]

Hi Mario,

I don't know why but now it is working :/ I remember trying setting double $$ a few days ago...

Anyway thank you all so much for your time!

Regards,

Isabel.

[Chris Cranford]

As this isn't well documented on the Quarkus side and I only found it via a GH issue search, I've added DBZ-9385 so we can consider adding something to the Debezium Server docs to clarify this point for future users. Thanks for all the work and help on this Isabel and Mario.

-cc

[1]: https://issues.redhat.com/browse/DBZ-9385

To view this discussion visit https://groups.google.com/d/msgid/debezium/966a0cfd-5c2b-42b5-bb15-89c96599b48cn%40googlegroups.com.