Are there any problems when running multiple debezium connectors on one server?

[최동주]

Are there any problems when running multiple debezium connectors on one server?

For example, is it a condition race, scheduling problem? (i used distributed mode)

my debezium config

one

{
"name" : "tcmg-optionset",
"config" : {
"include.schema.changes":"true",
"snapshot.mode":"when_needed",
"connector.class" : "io.debezium.connector.mysql.MySqlConnector",
"database.hostname" : "",
"database.port" : "",
"database.user":"",
"database.password":"!",
"database.server.id":"20022",
"database.server.name":"klagoDev_tcmg_optionset",
"database.history.kafka.bootstrap.servers":"localhost:9092",
"database.history.kafka.topic":"history_tcmg_optionset",
"database.whitelist":"",
"table.whitelist":".tcmg_optionset",
"database.serverTimezone":"Asia/Seoul",
"tasks.max":"1",
"transforms":"unwrap,route",
"transforms.unwrap.type":"io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.add.fields":"op,table,db",
"transforms.unwrap.delete.handling.mode":"rewrite",
"transforms.route.type":"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex":"([^.]+)\\.([^.]+)\\.([^.]+)",
"transforms.route.replacement":"tcmg_optionset",
"decimal.handling.mode":"double"
}
}

two

{
"name" : "titg-item",
"config" : {
"include.schema.changes":"true",
"snapshot.mode":"when_needed",
"connector.class" : "io.debezium.connector.mysql.MySqlConnector",
"database.hostname" : "",
"database.port" : "",
"database.user":"",
"database.password":"",
"database.server.id":"20026",
"database.server.name":"klagoDev_titg_item",
"database.history.kafka.bootstrap.servers":"localhost:9092",
"database.history.kafka.topic":"history_titg_item",
"database.whitelist":"",
"table.whitelist":"neos_1234.titg_item",
"database.serverTimezone":"Asia/Seoul",
"tasks.max":"1",
"transforms":"unwrap,route",
"transforms.unwrap.type":"io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.add.fields":"op,table,db",
"transforms.unwrap.delete.handling.mode":"rewrite",
"transforms.route.type":"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex":"([^.]+)\\.([^.]+)\\.([^.]+)",
"transforms.route.replacement":"titg_item",
"decimal.handling.mode":"double"
}
}

[Chris Cranford]

Hi,

You can safely run multiple connectors in the same Kafka Connect environment, that's not a problem.  If you're referring to running multiple connectors against the same database, while that is technically possible as well, we do generally recommend you don't do that unless there is an absolute need for it.  More often than not, a single connector can easily perform the workload of many tables rather than trying to scale outward with multiple connectors.

Chris

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/ab3c1459-a3ef-45b7-a42f-a3be54396bf1n%40googlegroups.com.

[Gunnar Morling]

What Chris says. It's not quite clear what you mean by "server" here; in any case you definitely must use distinct values for database.server.id and database.server.name, respectively.

--Gunnar

[Arturo Opsetmoen A.]

Hei,

Sorry for opening an old thread but I didn't find any info on this question I have...

Is it possible to set two independent Debezium instances towards the same Oracle database? We need to test some code and don't have a dev/test version of the database we use. We don't want to make the prod system unstable and wonder if connecting two Debezium instances to the Oracle database could result in unstabilities...

Thanks a lot in advance!

--
Arturo

[jiri.p...@gmail.com]

Hi,

yes, it is technically possible but multiple log mining sessions could introduce non-negligible load on the database so we don't usually recommend it.

OTOH if it is something time limitied and database has a spare capacity then it might be a viable option.

Jiri

[Chris Cranford]

To add to what Jiri mentioned, if you want to run multiple connectors against the same Oracle database, you must use `log.mining.strategy` set to either `online_catalog` or `hybrid`.

-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/aed20b74-160c-4f2d-9de4-ad94194ed8d4n%40googlegroups.com.