Benchmarking Debezium?

[Cheng Chen]

Hi there,

Kudos for the great work of Debezium, we are testing it out :)

I am curious that have you done any benchmakring or performance testing as to how fast Debezium can consume the binlog? what could be some of the bottlenecks and limits?

Thanks,

Cheng

[Randall Hauch]

We've done no benchmarking or performance to date, since we've been focused on functionality. In part this is because we all know that Kafka can be configured to handle very large volumes of messages with quite small latencies, and this involves tuning not only the Kafka broker configurations but also the producers (and consumers). Since Kafka Connect is largely just an implementation that uses producers with well-known and highly-performant patterns (e.g., recording the offsets periodically, etc.), this also should be able to be tuned to perform quite well. And although we don't known how many high-volume connectors could be deployed/run in a single Kafka Connect service instance, in some respects that doesn't matter too much since we can always add more service instances to the cluster or even create additional Kafka Connect clusters.

Having said that, we do not know the performance or capabilities of the MySQL connector and the library it uses to read the binlog. We also don't know the impact that a single connector has on the server or that writing row changes to the binlog has on MySQL server performance, though the latter is (always?) a prerequisite for MySQL clustering.

Best regards,

Randall

[Scott Ferguson]

its been a while since this post. has any work been done in this area since? also, is anyone using debezium in production?

[Chris Riccomini]

We are currently dark launched in production for MySQL 5.7 + GTIDs. Testing some failover scenarios, but already using it to load a few DBs.

[orr.g...@equalum.io]

Hi,

I'm using Debezium Embedded in order to use the MySql connector

and I was wondering if you have any benchmarks.

Do you have any numbers?

I'm currently experiencing aroung 5-10K rows per second which is rather slow, given the fact that I cannot run multiple connectors in parallel for the same binlog.

Do you have any help on the subject?

[Jiri Pechanec]

Hi,

we have not any performance numbers to be published yet. But as a hint you can look at numbers for postgres connector - https://gitter.im/debezium/dev?at=5a2a6de4ba39a53f1a360424

J.

[Randall Hauch]

Make sure that you tune the producer settings in the Kafka Connect workers. See the Kafka documentation for details. 

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To post to this group, send email to debe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/830804e8-7fe6-4fa8-bdb0-1f345bec8292%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.