Connector reconciliation for new tenants

[Rohit Panwar]

Hi Community,

We have a multiple tenant setup. We have around 40 aurora clusters , each having close to 50-100 tenants. For each of these tenants we are looking to capture 50 tables as part of our CDC pipeline.

To summarise calculations -
Per cluster - tenants - 100 and tables 50, so around 5000 tables per cluster is what we need to capture.

What are the recommended number of connectors per cluster for an infra like this?

[Rohit Panwar]

Building on the previous question, when a new tenant is onboarded to our system, we want to start a new connector for this tenant, however after some time we would want to move it to an existing connector for that cluster. What is the recommended way to achieve something like this?

[Chris Cranford]

Hi 

Starting a new connector for a new tenant is a straightforward task; however, the process becomes more complex when you want to merge connectors. This isn't really something we support for two concurrently running connectors capturing changes at two independent positions where the expected structures for tables may could be different due to recent schema changes. 

Furthermore, because changes are being captured independently, you run the risk of having a gap in the data or duplicates, depending on how the two connectors interpret overlapping positions. So while you can use the `recovery` snapshot mode to try and merge these two independent connectors, there can be side effects of doing so, and you may then need to rely on incremental or blocking snapshots to deal with any data gap should that occur.

I'd recommend to avoid any issues where you may need to re-take a snapshot, using per-tenant connectors may be the better alternative.

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/e1027e3d-6357-4a34-abb1-17b3facd42dan%40googlegroups.com.