Achieving faster rate of reading binlog from AWS RDS (mysql)

[Vikas Gupta]

Hello,

We are using Debezium against AWS RDS (Aurora MySQL).

Our RDS was initially hosted on db.t3.small (2 vCPUs, 2GB RAM, 5Gbps network)

It was able to read the binlog at around ~2500 messages/second.

To achieve a higher rate, we then migrated the database to db.t3.medium (2 vCPUs, 4GB RAM, 5Gbps network)

We noticed that going from db.t3.small to db.t3.medium didn't have any effect on the rate of processed events.

What properties (like batching, polling) can we tweak to achieve a faster binlog reading?

I am guessing that upgrading to a higher vCPUs/ RAM/ network help us achieve a faster binlog reading. Out of vCPUs/RAM/network, which one would have the most effect on the read speed?

Thank you!
-VG

[jiri.p...@gmail.com]

Hi,

there is no single anser to this problem. COuld you please share the chart for https://debezium.io/documentation/reference/2.4/connectors/mysql.html#connectors-snaps-metric-queueremainingcapacity_mysql ?

This will give us a hint if the issue is on reading side or on writing side. If the issue is on reading side then definitely CPU and network are the two coponents to focus on.

J.

[Vikas Gupta]

Jiri,

We have commented out the writes to kafka in the Listener. These are the 4 metrics related to queues. Their values doesn't seem to change.

debezium.mysql:context=streaming,server=debezium-cdc-events2,type=connector-metrics.MaxQueueSizeInBytes:0
debezium.mysql:context=streaming,server=debezium-cdc-events2,type=connector-metrics.QueueTotalCapacity:8192
debezium.mysql:context=streaming,server=debezium-cdc-events2,type=connector-metrics.QueueRemainingCapacity:8192
debezium.mysql:context=streaming,server=debezium-cdc-events2,type=connector-metrics.CurrentQueueSizeInBytes:0

Which queue is this?

-VG

[jiri.p...@gmail.com]

I am looking for QueueRemainingCapacity but I'd like to see it captured for some time to see if it changes or not.

J.

[Vikas Gupta]

It looks like I had been computing the lag of the Debezium connector incorrectly. The correct way seems to be obtaining the progress of the connector through its JMX metrics and then comparing it with the output of “show binary logs”

The relevant metrics we are now using are https://debezium.io/documentation/reference/2.4/connectors/mysql.html#binlog-filename & https://debezium.io/documentation/reference/2.4/connectors/mysql.html#binlog-position

Previously, I had been comparing the logline

[io.debezium.connector.common.BaseSourceTask - 6446138 records sent during previous 00:21:12.915, last recorded offset of {server=debezium-cdc-events2} partition is {transaction_id=null, ts_sec=1699284313, file=mysql-bin-changelog.004266, pos=55915217, row=8, server_id=73360770, event=6}]

with the latest binlog file that’s available on the mysql server. That seems to be inaccurate.

Regards,

-VG

--
You received this message because you are subscribed to a topic in the Google Groups "debezium" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/debezium/TyF_QjMScf4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/085eec1b-18bc-4a9e-9283-03ad8d454cean%40googlegroups.com.

[Vikas Gupta]

Jiri, 

QueueRemainingCapacity is frequently hitting 0.

Debezium connector is gradually falling behind the latest binlog file.

-VG

On 07/11/23 10:44 am, jiri.p...@gmail.com wrote:

--

[jiri.p...@gmail.com]

Hi,

hiting the zero means that Debezium is reading as fast as it can but there is a bottleneck in Kafka Connect -> Kafka broker communication. You'd need to tune the Kafka per settings.

J.

[jiri.p...@gmail.com]

Also try to increase max.queue.size and max.bacth.size config options to see if it makes any difference

J

[Vikas Gupta]

Thanks Jiri, I will try that though I feel that it won't fix the issue as the read speed from CDC is faster than writing to kafka.

How have other folks solved the speed mismatch issue? Which property of kafka can I tweak? I am not a kafka expert. We have a 3-node MSK cluster in AWS. In our spring-boot app, we are mostly using the default kafka properties.

-VG

[jiri.p...@gmail.com]

Hi,

I'd recommend to multiply the settings in the above mentioned config options to generate larger batches. Then on the writing side tuning of acks and  min.insync.replicas have significant impact.

J.