How to horizontally scale debezium by adding more nodes for same db source to improve performace

[Roobal Jindal]

I am using debezium connector for mssql to push cdc events to kafka. There are millions of events to be pushed to kafka per minute but its lagging and is not able to keep it upto the speed of updations.

I am running connector in kafka's distributed mode. I assume since it is distributed, we can add more node to read binlog in parallel and increase performance but there is no such information on google how to do that.

I saw similar posts but everyone asked not to use multiple connectors for different set of tables for the same database as this doesn't improve the performance as both of them still will read the same serial binlog, specially when topic prefix is same.

If not, then whats the benefit of ditributed mode?

[jiri.p...@gmail.com]

Hi,

what's lagging, reading from database or writing to Kafka broker? Could you please share chart for https://debezium.io/documentation/reference/2.6/connectors/sqlserver.html#connectors-snaps-metric-queueremainingcapacity_sqlserver over some time period?

Jiri

[Roobal Jindal]

writing to Kafka broker is lagging. Its writing 1200 records/sec. Is there a way to process tables in parallel or adding more nodes? if not, then whats the advantage of distributed mode

[jiri.p...@gmail.com]

Hi,

in that case I'd recommend to add more Kafka brokers. Also make sure you've larger number of pratition count per topic. You can ca also try to reduce acks to 1. Adding more Kafka Connect nodes is not a solution as Kafka writes should definitely be much much much faster.

Jiri