Scalability of debezium connectors

[Shuguang Xiang]

Hi

I am currently running a MySQL connector for a single database whose peak write is 500 records/second. The debezium connector is able to catch up with such write load.

From time to time, our database user will have bulk updates for a single table that has 10 million records.  In this case, the debezium connector is not able to catch up with such a write spike. 

I am trying to set up 2 connectors consuming a single database at the same time to ramp up the throughput. I split the tables evenly among these 2 connectors.

In my connect_test database, I have 2 tables: test_table1, test_table2.  

Below are 2 connectors configurations. I intentionally set database.server.name the same to make kafka topic from the same database to have the same prefix. 

My question is:

1. Is setting up multiple connectors at the same database a proper way to scale up processing speed?

2. If it is correct to set up multiple connectors at the same time, could we have the same database.server.name for 2 connectors in the same kafka connect? (I get callstack on my local machine)

{

"name": "shuguang_connector1",

"config": {

"connector.class": "io.debezium.connector.mysql.MySqlConnector",

"tasks.max": "1",

"database.hostname": "mysql",

"database.port": "3306",

"database.user": "confluent",

"database.password": "confluent",

"database.server.id": "184051",

"database.server.name": "mysql_connect_test",

"database.whitelist": "connect_test",

"table.whitelist":"test_table1",

"database.history.kafka.bootstrap.servers": "kafka:9092",

"database.history.kafka.topic": "debz.schema-changes.mysql_connect_test1",

"snapshot.mode": "schema_only",

"snapshot.locking.mode": "none"

}

}

{

"name": "shuguang_connector2",

"config": {

"connector.class": "io.debezium.connector.mysql.MySqlConnector",

"tasks.max": "1",

"database.hostname": "mysql",

"database.port": "3306",

"database.user": "confluent",

"database.password": "confluent",

"database.server.id": "184052",

"database.server.name": "mysql_connect_test",

"database.whitelist": "connect_test",

"table.whitelist":"test_table1",

"database.history.kafka.bootstrap.servers": "kafka:9092",

"database.history.kafka.topic": "debz.schema-changes.mysql_connect_test2",

"snapshot.mode": "schema_only",

"snapshot.locking.mode": "none",

"converters": "boolean",

"boolean.type": "io.debezium.connector.mysql.converters.TinyIntOneToBooleanConverter"

}

}

[Shuguang Xiang]

Hi expert,

Could you help take a look?

Best regards,

Shuguang

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

[Gunnar Morling]

Hi Shuguang,

Sorry for the late reply, had missed that one before. In general, you won't gain much in terms of performance by setting up multiple connectors for one and the same MySQL database, as both of them still will read the same serial binlog. If each of them processes a distinct subset of tables, you may see some improvements, but I'm not sure how significant it'd be. In this case, you should definitely use distinct server names, as that name impacts the history topic and they must not conflict.

>  In this case, the debezium connector is not able to catch up with such a write spike. 

Can you elaborate a bit on that one? I assume latency goes up temporarily but the connector will catch up eventually, after the spike is over? Can you share some more detailed numbers of latency etc. Thx!

--Gunnar

[Shuguang Xiang]

Hi Gunnar,

Sorry for the late reply, Gunnar.

True, I saw conflicts while setting up 2 local connectors with the same server name. There would be no messages in the destination table.

For latency in production, it was not able to catch up. This jmx metric  avg:jmx.debezium.binlog.millisecondsbehindsource kept accumulating. And the metric became 6 hours after the bulk update started.The bulk update is one shot and there was no more bulk update during this latency period. The bulk update SQL was UPDATE TABLE_NAME SET COLUMN1= VALUE1 WHERE COLUMN1= VALUE2

Best regards,

Shuguang

--
You received this message because you are subscribed to a topic in the Google Groups "debezium" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/debezium/nJjkTKz4v3A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/a9a03d18-1fdc-42f3-89e8-17f3d595dec2n%40googlegroups.com.

[Shuguang Xiang]

Beyond that, is it possible I change the destination topic name rather than default destination topic, which usually is database server name + database name+ table name?

[Chris Cranford]

Hi Shuguang,

That most definitely is possible through a single message transform configuration. 
See https://debezium.io/documentation/reference/configuration/topic-routing.html and https://debezium.io/documentation/reference/configuration/content-based-routing.html

Thanks
Chris

You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/CAHbWEPGMGp974nM_YfDXztryz4%2Bp1d%3D24S5Dsd2WJWXtJ5tRfg%40mail.gmail.com.

[Gunnar Morling]

Hi Shuguang,

Could you log a Jira issue, describing your situation, record numbers, and statements? It seems like this could be quite an actionable scenario we could leverage for some performance testing, profiling and hopefully tuning. 1.6.0.Alpha1 (out in ~two weeks) should improve the situation, as this brings in a performance improvement in the binlog client:

    https://github.com/debezium/debezium/pull/2327

Another option for the time being is increasing producer batch sizes and size of the connector internal queue.

Hth,

--Gunnar

[Shuguang Xiang]

Hi Gunnar,

Sorry for the late reply. 

I have created a jira ticket  https://issues.redhat.com/browse/DBZ-3477. After deployment, could you give some metric like what is the maximum throughput per second for a debezium connector? I know the binlog reader speed is related to memory and network. Some metric would help.

I will take a look at the 2 options you suggest here:

1. Run 2 connectors with topic routing.

2.  increasing producer batch sizes and size of the connector internal queue. Could you tell me how to increase these 2 metrics?

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/3e5d471f-07f2-4aaa-be96-3dac9640440bn%40googlegroups.com.

[jiri.p...@gmail.com]

Hi,

please see https://debezium.io/documentation/reference/1.5/connectors/mysql.html#mysql-property-max-queue-size

J.

[Wiqi Khan]

@gunnar and @shuguan any leads on this one?