Running debezium in multiple containers

[Ritesh Negi]

Hello,

I am looking to run debezium in multiple containers and run a oracle connector inside each container to the same table as I am thinking of creating containers using same dockerfile or docker-compose.yaml.

My doubt is how the initial snapshot and any changes to the monitored tables be handled.

Will a change be picked by all the instances or a single instance?

Thanks,

Ritesh

[Chris Cranford]

Hi Ritesh -

First to answer your question, whether a change is picked up by one or more than one connector will depend entirely on your connector configuration.  If multiple connector configurations specify the table in the "table.include.list", then yes each connector will emit that change; however, if the connector configuration is mutually exclusive then each connector will only emit changes for the tables that are part of it's filter configuration, i.e. include/list lists.

That said, I would highly recommend against multiple Oracle connectors this way.  Is there a specific reason why you need this set up?

As some detail, each Oracle connector interacts with Oracle LogMiner in a 1-to-1 fashion, meaning that each connector starts its own LogMiner process.  This process consumes valuable CPU and Disk IO resources, and so you ideally do not want multiple of these happening if your database is high traffic or has lots of transactions, particularly if those transactions are long-running.  And since each connector operates independently, while the filter configuration prevents the emission of events for tables that you are not interested in for that connector, every redo entry will be read by each connector regardless, so in high traffic situations, this can be quite a lot of IO operations. 

We do have plans to investigate using the connector with a standby (logical and/or physical) in the future, see DBZ-3866 [1].  But at this time, that is not yet possible and so resource utilization should also be considered when you begin to think about how to scale out connectors like this with your database source.  We are looking at alternatives to this even beyond the standby method above, but I don't have information to share on that yet.

CC

[1]: https://issues.redhat.com/browse/DBZ-3866

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/5dee75fe-4ceb-4e5b-a6cd-48b89c651f83n%40googlegroups.com.

[Ritesh Negi]

Hi Chris,

We have our private cloud on which we wish to run debezium.

We have 2 regions and 2 zones in one of the region.

So if we use debezium docker image or custom image, we will end up with minimum of 1 connector in 2 zones atleast listening to changes on the active database as I was thinking of creating the connector inside the container once the container is created using modified dockerfile.

We have active passive database in the 2 regions with dataguard replicating data from active DB to passive DB.

Can you recommend how we can architect our solution using debezium in this infra?

[Ritesh Negi]

Hi Chris,

Any thoughts on above question?

[Chris Cranford]

Hi Ritesh.  If I understand correctly you have 2 regions where you have 2 databases in each region, a primary and a secondary/fail-over using DataGuard for replication.  If that's accurate, you would then just set up two Oracle connectors, one that points to each of the primary data sources and captures changes based on the table/schema filters for that data source.  How you decide to do this with docker containers, on-prim installations, or any other cloud offering is entirely up to you.

Hope that helps,
Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/1c7caeb6-2f7c-4628-b7c2-0d3b43fe9f83n%40googlegroups.com.