Number of debezium connectors needed

[Rohit Panwar]

We have a multiple tenant setup. We have around 40 aurora clusters , each having close to 50-100 tenants. For each of these tenants we are looking to capture 50 tables as part of our CDC pipeline.

To summarise calculations -
Per cluster - tenants - 100 and tables 50, so around 5000 tables per cluster is what we need to capture.
Is it recommended to go with 1 connector per tenant? or what is the recommended way to handle this scale?

[Chris Cranford]

Hi -

The Debezium's team overall messaging is to always prefer a single connector because reading from transaction logs is a serial process and there's not always much to gain by splitting that action across multiple connectors since all transaction log events must be read by each connector, so multiple connectors do increase IO.

But there are most certainly business goals, reliability concerns, and potential variances in configuration where using a connector per tenant distribution strategy makes more sense. In practice, this isn't all that uncommon and it provides you with potentially more long-term flexibility as you scale and helps to avoid situations where one tenant issue taints the availability or stream of changes for other tenants.

So it's really more of a question of what matters the most to you, long-term goals, and business need.

-cc

Is it recommended to go with 1 connector per tenant? or what is the recommended way to handle this scale? --
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/1cad3924-1720-4cb3-91c3-45c00a5c9952n%40googlegroups.com.