Debezium stops sending messages to Google Cloud Pub/Sub

338 views
Skip to first unread message

Ido Shamun

unread,
Aug 22, 2021, 1:25:19 AM8/22/21
to debezium
Hi!

I deployed a Debezium instance on my Kubernetes cluster.
I allocated 1 CPU, 1 GB memory, and 100 GB SSD local disk.
The Debezium instance is connected to Postgres and has Google Pub/Sub set to sink.
Usually, it works great but every now and then (few days) it just stops sending messages. To resolve the issue I delete the pod and Kubernetes recreates it.

Here are the props:
```
debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.database.hostname=%hostname%
debezium.source.database.port=5432
debezium.source.database.user=%database_user%
debezium.source.database.password=%database_pass%
debezium.source.database.dbname=%database_dbname%
debezium.source.table.include.list=public.comment,public.comment_upvote,public.source_request,public.upvote,public.post,public.user,public.post_report
debezium.source.column.exclude.list=public.post.tsv,public.post.placeholder

debezium.transforms=Reroute
debezium.transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter
debezium.transforms.Reroute.topic.regex=(.*)
debezium.transforms.Reroute.topic.replacement=%topic%

debezium.sink.type=pubsub
```

I use Debezium 1.5 and I have another instance with the same config but for a different DB that works just fine.
If you have any suggestions this would be great. 🙏

Thanks!

Gunnar Morling

unread,
Aug 23, 2021, 3:45:08 AM8/23/21
to debezium
Hey,

Really hard to tell without knowing more. Is there anything in the logs in this situation? You might have to increase the log level to gain more insight. E.g. with TRACE, you should get some idea of whether the connector still is running and fetching events or not.

--Gunnar

Ido Shamun

unread,
Aug 29, 2021, 1:04:57 AM8/29/21
to debezium
Sorry for the late reply (for some reason I wasn't notified about your message).
I couldn't see anything in the logs but let me change the logging to TRACE level and get back to you.
It might take a while as sometimes this issue takes several days to show up.

Ido Shamun

unread,
Aug 30, 2021, 9:20:27 AM8/30/21
to debezium
I tried to follow the guide here:

I set `log4j.logger.io.debezium.connector.postgresql=TRACE, stdout` in the application.properties file and also tried to set `CONNECT_LOG4J_IO_DEBEZIUM_CONNECTOR_POSTGRESQL` to the same value as env var.
But nothing seems to change the logging level. I suspect it has to do with the fact I'm running Debezium Server so it might be a bit different, am I right?

Ido Shamun

unread,
Sep 20, 2021, 5:14:54 AM9/20/21
to debezium
Trying to bump here the topic as I can't manage to turn on the TRACE logging...
Message has been deleted
Message has been deleted
Message has been deleted
Message has been deleted

Chris Cranford

unread,
Jun 2, 2022, 12:56:31 PMJun 2
to debe...@googlegroups.com
Debezium Server is based on the Quarkus framework, which requires that logging be configured in the application.properties file [1].  You should be able to enable TRACE logging for Debezium Server by adding the following entries to that file:

    quarkus.log.category."io.debezium".level=TRACE
    quarkus.log.cateogry."io.debezium".min-level=TRACE

Hope that helps,
CC

[1]: https://quarkus.io/guides/logging#runtime-configuration

On 5/29/22 20:12, 'Максим Макаров' via debezium wrote:
I have the same problem.
And also can't enable TRACE logging in Debezium Server.

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/ada869ac-4f34-47d0-bf61-7e93ea58993cn%40googlegroups.com.

Reply all
Reply to author
Forward
0 new messages