Debezium-Pub/Sub publish to multiple topic issue

48 views
Skip to first unread message

Laxinandan Rajkhowa

unread,
Jun 12, 2024, 7:47:56 AMJun 12
to debezium
Hi All,
We have integrated Debezium CDC and Pub/Sub but facing a issue when publishing messages from single table to multiple pub/sub topics.

Problem Statement:
We want to publish messages from outbox table to multiple pub/sub topics based on the "transforms.route.topic.expression". But this is not working. 
Whereas we are able to publish messages to common Pub/Sub topic(<<db_name.schema.table_name>>).

Can you please let us know if Debezium has already a solution for this in place.

Attached the application.properties file.

Thanks,
Laxinandan Rajkhowa
application.properties

jiri.p...@gmail.com

unread,
Jun 12, 2024, 7:49:55 AMJun 12
to debezium
Hi,

could you please share the Server logfile too?

Jiri

Laxinandan Rajkhowa

unread,
Jun 12, 2024, 8:25:28 AMJun 12
to debe...@googlegroups.com

Hi Jiri,

Please find the attached server logs and application.properties file.

 

 

Thanks,

Laxinandan Rajkhowa

 

From: debe...@googlegroups.com <debe...@googlegroups.com> On Behalf Of jiri.p...@gmail.com
Sent: Wednesday, June 12, 2024 5:20 PM
To: debezium <debe...@googlegroups.com>
Subject: [debezium] Re: Debezium-Pub/Sub publish to multiple topic issue

 

[Email from a non-Nagarro source: please exercise caution with links and attachments]

 

--
You received this message because you are subscribed to a topic in the Google Groups "debezium" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/debezium/qbHuFToXuUA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/1987e156-37ca-437b-a88d-c56ca982d84bn%40googlegroups.com.

Debezium-server-logs.txt
application.properties

jiri.p...@gmail.com

unread,
Jun 13, 2024, 3:39:37 AMJun 13
to debezium
Hi,


You need to add both scripting package and the languages itself among Debezium Server JARs as these are not embedded by default.

Jiri

Laxinandan Rajkhowa

unread,
Jun 13, 2024, 1:43:33 PMJun 13
to debe...@googlegroups.com

Hi Jiri,

I am pleased to inform that the issue is resolved after adding the debezium-scripting-2.6.2.Final.jar to Debezium Environment.

Thank You for the suggestion !! 😊

 

However, I also want to discuss on another issue. The message sent to Pub/Sub has schema details and payload. We only want the payload to be sent and not schema details. The green shaded text is the payload.

We have added the “debezium.source.transforms.wrap.type=io.debezium.transforms.ExtractNewRecordState” in application.properties and is not working.

Can you please suggest a solution for this.

 

 

Message Body:

{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"string","optional":true,"field":"aggregate_id"},{"type":"string","optional":true,"field":"aggregate_type"},

{"type":"string","optional":true,"field":"created_by"},{"type":"int64","optional":false,"name":"io.debezium.time.Timestamp","version":1,"field":"created_on"},{"type":"string","optional":true,"field":"event_type"},

{"type":"string","optional":true,"field":"payload"}],"optional":false,"name":"outbox.outbox.outbox_event.Value"},

"payload":{"id":"24242","aggregate_id":"23222","aggregate_type":"com.nagarro.oms.common.audit.AuditDomainEvent","created_by":

"System","created_on":1718236800000,"event_type":"com.nagarro.oms.common.audit.AuditDomainEvent","payload":"Nagarro"}}

application.properties

jiri.p...@gmail.com

unread,
Jun 14, 2024, 1:43:47 AMJun 14
to debezium
Hi, could you please try debezium.format.value.schemas.enable=false ?

See

Jiri

Laxinandan Rajkhowa

unread,
Jun 17, 2024, 2:15:26 AMJun 17
to debe...@googlegroups.com

Hi Jiri,

Thanks!!

We are facing another issue and need your suggestion.

 

Debezium is working fine in my local. But if I deploy debezium to another machine then it’s not working. I have also cleared schistory.dat and offsets.dat files.

 

Problem Statement:

Debezium is trying to capture events from all schemas rather than single table. Due to this the logs has exception. We have to capture events from outbox.outbox_event table only.

 

Attached the server logs and application.properties.

Debezium_Logs.txt
application.properties

jiri.p...@gmail.com

unread,
Jun 18, 2024, 12:36:33 AMJun 18
to debezium
Hi,


Jiri

Laxinandan Rajkhowa

unread,
Jun 18, 2024, 12:45:20 PMJun 18
to debe...@googlegroups.com

Hi Jiri,

Thank You!!

We are able to resolve the issues with MySQL Database and working as expected.

 

However, we have another requirement to integrate Debezium with PostgreSQL and Pub/Sub.

And facing below issues and need your suggestion:

  1. Is it mandatory to update wal_level to logical?
  2. We are not able to start the server when connecting to PostgreSQL. We have attached the server logs and application.properties for the same.
application.properties
Server_logs.txt

Chris Cranford

unread,
Jun 18, 2024, 1:00:32 PMJun 18
to debe...@googlegroups.com
Hi,

Yes, the `wal_level` must be set to logical.

Regarding the failure, the user does not have permission to create the replication slot:

    > Creation of replication slot failed
    > ERROR: could not access file \"decoderbufs\": No such file or directory

This looks like perhaps decoderbufs logical plug-in isn't installed on the PostgreSQL database.  You may need to use `pgoutput` instead?

Chris

You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/SEZPR06MB531739132F96F9781F29A9D3E8CE2%40SEZPR06MB5317.apcprd06.prod.outlook.com.

Chris Cranford

unread,
Jun 18, 2024, 1:01:27 PMJun 18
to debe...@googlegroups.com
My apologies, I hit send too early, the failure is not because of permissions but rather than decoderbuf plug-in does not appear to be installed.

Laxinandan Rajkhowa

unread,
Jun 20, 2024, 8:01:55 AMJun 20
to debe...@googlegroups.com

Hi Chris,

As per our requirement, we have to connect to AlloyDB in GCP.

 

Problem Statement:

After doing the changes, the events are not captured by Debezium.

Attaching the server logs and application.properties. We have used pgoutput in application.properties.

 

We have configured AlloyDB with below changes. Please let us know if any additional changes are required.

 

  1. Enabled alloydb.enable_pglogical and alloydb.logical_decoding flags at AlloyDb.

 

  1. CREATE EXTENSION IF NOT EXISTS pglogical;

 

       3.  ALTER ROLE postgres WITH REPLICATION LOGIN;

             CREATE ROLE replication_role WITH REPLICATION LOGIN;

             CREATE USER debezium_user WITH PASSWORD 'password';

             GRANT debezium TO postgres;

             CREATE ROLE replication_group WITH LOGIN;

             GRANT replication_group TO postgres;

             GRANT replication_group TO debezium_user;

             ALTER TABLE outbox.outbox_event OWNER TO replication_group;

 

 

Thanks,

Laxinandan Rajkhowa

Server_logs.txt
application.properties

Chris Cranford

unread,
Jun 20, 2024, 11:29:34 AMJun 20
to debe...@googlegroups.com
Could you try without setting "database.include.list"?

Thanks,
Chris

Laxinandan Rajkhowa

unread,
Jun 20, 2024, 12:12:01 PMJun 20
to debe...@googlegroups.com

Hi Chris,

Thanks!!

Issue is resolved after removing “database.include.list”

 

Another question. We have deployed debezium to VM(Ubuntu OS) and from SSH terminal ran the ./run.sh to start debezium. However, after the SSH session is closed, the debezium server also stops.

Any better way to trigger the ./run.sh script.

Chris Cranford

unread,
Jun 20, 2024, 1:10:00 PMJun 20
to debe...@googlegroups.com
Hi -

I'd checkout nohup for this ;)

Chris
Reply all
Reply to author
Forward
0 new messages