Pub/Sub batch timeout with ordering keys enabled

[pramod choudhari]

Hi,

I'm facing timeouts with Debezium Server 3.0.7 publishing to Google Pub/Sub with ordering keys enabled.
Example error I faced recently:

**Error:**

java.util.concurrent.TimeoutException: Waited 600000 milliseconds for ListFuture[status=PENDING]

at io.debezium.server.pubsub.PubSubChangeConsumer.handleBatch()

After server restart, it works fine until next occurrence. 

**What i have tried**

- My initial max.batch.size was 4096, i have brought it down to 1024 now.

- wait.message.delivery.timeout.ms and retry.total.timeout.ms I increased to 10mins 

Still I am  facing this Timeout issue sometimes.

My configurations:

**Source Config:**

{

"max.batch.size": 1024,

"max.queue.size": 4096,

"poll.interval.ms": 500

}

**Pub/Sub Sink Config:**

{

"ordering.enabled": true,

"batch.delay.threshold.ms": 100,

"batch.element.count.threshold": 100,

"batch.request.byte.threshold": 5000000,

"wait.message.delivery.timeout.ms": 600000,

"retry.total.timeout.ms": 600000,

"concurrency.threads": 4

}

**Questions:**

1. What config changes can prevent this timeout? Should I reduce

batch.element.count.threshold further or adjust retry settings?

2. I cannot disable ordering. Looking for configuration-based solutions

to handle these intermittent PENDING futures or any steps to debug this issue further.

Thanks.

[Chris Cranford]

Hi,

You've started a thread [1] on Zulip, let's continue the discussion there so that it's in one place.

Thanks,
-cc

[1]: #community-dbz-server > Debezium Server TimeoutException

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/71df02f9-0e4f-40e8-b26e-6ebe49c18cbdn%40googlegroups.com.