Timeout Exception at PubSub consumer

[Ajitesh kumar]

Hi All,

I have setup a mySQL connector with debezium 2.6 running on GKE. after sending certain records the server stops with a timeout exception. Any idea how i can resolve this, Attached my application.properties and log files as well.

com.google.api.core.AbstractApiFuture$InternalSettableFuture@73b48f0c[status=SUCCESS, result=[java.lang.String@2fd5cc95]]]]]
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:531)
        at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:119)
        at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:75)
        at io.debezium.server.pubsub.PubSubChangeConsumer.handleBatch(PubSubChangeConsumer.java:244)

Thanks in advance.

Thanks.

[Nathan Smit]

I see you commented out your change to   debezium.sink.pubsub.wait.message.delivery.timeout.ms Any particular reason why?  That's the limit you're hitting up against as far as I can tell from the log.  If you increased it to some super high value could you see if that resolves it (at least to isolate the problem)

In our environment we ran into similar issues after we upgraded Debezium and the newer version had this config item added.  Our current config has  debezium.sink.pubsub.wait.message.delivery.timeout.ms=120000 and it works fine, though

[Nathan Smit]

Just to add I'm not a Debezium employee, just a user.  I've personally found the pub/sub connector more difficult to use since this config was added due to hitting up against this timeout

[Ajitesh Kumar]

Thans Nathan. I tried with same but it throws same error.

_{This email and all contents are subject to the following disclaimer: https://www.pepkor.co.za/email-disclaimer}

_{IMPORTANT: This electronic message may contain information that is considered confidential. If received in error, please notify the sender immediately and then delete the original and all subsequent/related messages.}

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/3bd22512-3144-47bc-9f25-c5c373c7db2dn%40googlegroups.com.

--

Thanks & Regards
Ajitesh Kumar
09673799966

[Nathan Smit]

Could you try making it something stupidly high like 1200000000?

An issue we had as well previously was where this error was hiding other problems.  Are any of your messages going through or are they all failing?

[Ajitesh Kumar]

As soon as i increase it to random large number , its stuck at Keepalive: Trying to restore lost connection to

2024-06-20 10:13:41,124 INFO  [io.deb.con.mys.MySqlStreamingChangeEventSource] (debezium-mysqlconnector-test-change-event-source-coordinator) Waiting for keepalive thread to start
2024-06-20 10:13:41,126 INFO  [io.deb.uti.Threads] (blc-10.106.96.5:3306) Creating thread debezium-mysqlconnector-test-binlog-client
2024-06-20 10:13:41,225 INFO  [io.deb.con.mys.MySqlStreamingChangeEventSource] (debezium-mysqlconnector-test-change-event-source-coordinator) Keepalive thread is running
2024-06-20 10:13:42,233 INFO  [io.deb.con.com.BaseSourceTask] (pool-8-thread-1) 165 records sent during previous 00:00:10.712, last recorded offset of {server=test} partition is {ts_sec=1718437101, file=mysql-bin.003266, pos=104292624, gtids=5034a449-bb4a-11ec-8652-42010a800002:1-515264397, row=1, server_id=2828484328, event=2}
2024-06-20 10:13:47,793 INFO  [io.deb.con.com.BaseSourceTask] (pool-8-thread-1) 4096 records sent during previous 00:00:05.56, last recorded offset of {server=test} partition is {ts_sec=1718437101, file=mysql-bin.003267, pos=355702, gtids=5034a449-bb4a-11ec-8652-42010a800002:1-515268493, row=1, server_id=2828484328, event=2}
2024-06-20 10:14:06,597 INFO  [io.deb.con.com.BaseSourceTask] (pool-8-thread-1) 45056 records sent during previous 00:00:18.804, last recorded offset of {server=test} partition is {ts_sec=1718437104, file=mysql-bin.003267, pos=16951170, gtids=5034a449-bb4a-11ec-8652-42010a800002:1-515313549, row=1, server_id=2828484328, event=2}
2024-06-20 10:14:47,246 INFO  [io.deb.con.com.BaseSourceTask] (pool-8-thread-1) 114688 records sent during previous 00:00:40.649, last recorded offset of {server=test} partition is {ts_sec=1718437112, file=mysql-bin.003267, pos=59194505, gtids=5034a449-bb4a-11ec-8652-42010a800002:1-515428237, row=1, server_id=2828484328, event=2}
2024-06-20 10:16:06,783 INFO  [io.deb.con.com.BaseSourceTask] (pool-8-thread-1) 221184 records sent during previous 00:01:19.537, last recorded offset of {server=test} partition is {ts_sec=1718437126, file=mysql-bin.003268, pos=35354746, gtids=5034a449-bb4a-11ec-8652-42010a800002:1-515649421, row=1, server_id=2828484328, event=2}
2024-06-20 10:17:41,127 INFO  [com.git.shy.mys.bin.BinaryLogClient] (blc-keepalive-10.106.96.5:3306) Keepalive: Trying to restore lost connection to <Masked DB host>

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/352a77b1-7f2d-49f2-a037-a1d0efe1ee22n%40googlegroups.com.

[Ajitesh Kumar]

Hi All,

Can any of you please help on this ?

Thanks

[jiri.p...@gmail.com]

Hi,

this looks like a different issue. I don't see how this can be related to a failure on the source side.

Jiri