Configuring debezium connector for Sharded MySQL setup?

nitin agarwal

unread,

Mar 30, 2020, 2:01:24 AM3/30/20

to debezium

Hi,

I want to configure debezium connect for sharded MySQL setup. What are the recommendations for this?

Do I need to configure one connector for each shard ?

Thanks,

Nitin

Gunnar Morling

unread,

Mar 30, 2020, 2:27:01 AM3/30/20

to debezium

How are you implementing sharding? Manually, or using something like Vitess? In general, yes, you'd need a connector for each shard on a different host. Would be interesting to learn more about your specific set-up.

--Gunnar

nitin agarwal

unread,

Mar 30, 2020, 4:56:15 AM3/30/20

to debezium

Manually, controlled at the application. We have 5 shard and each one has 2 slaves.

One followup question, is it possible to have the topic shared across different connectors. If each connector writes to a new topic in the Kafka then downstream pipeline would also need to know about this.

Thanks for your quick reply.

Thanks,

Nitin

Jiri Pechanec

unread,

Mar 30, 2020, 5:20:43 AM3/30/20

to debezium

Hi,

for the latter question please look at https://debezium.io/documentation/reference/1.1/configuration/topic-routing.html#_logical_tables

niting...@gmail.com

unread,

Mar 18, 2021, 9:25:01 AM3/18/21

to debezium

I have a use case to merge data from all tables in the source DB into a single topic. I am using org.apache.kafka.connect.transforms.RegexRouter for it. Is there any concern in this ?

ByLogicalTableRouter is not working because topics don't have the same schema.

Thanks,

Nitin

Gunnar Morling

unread,

Mar 19, 2021, 4:02:45 AM3/19/21

to debezium

No concerns about using RegexRouter, if it does what you want. Just take care to avoid message key collisions, which is what ByLogicalTableRouter helps with.

--Gunnar

Message has been deleted

Gunnar Morling

unread,

Mar 19, 2021, 11:37:49 AM3/19/21

to debe...@googlegroups.com

Even without compaction you might want to be able to tell apart two messages with the same key from two different tables, though. Like, Customer 123 != Purchase Order 123.

--Gunnar

Am Fr., 19. März 2021 um 16:36 Uhr schrieb niting...@gmail.com <niting...@gmail.com>:

Thanks Gunnar for your reply.

I am not using Kafka log compaction so even if there are collisions in message key it should be fine for my use case.

Thanks,
Nitin

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/bb398b8c-79b3-456f-917c-0c01fe8ecc9fn%40googlegroups.com.

Reply all

Reply to author

Forward