Are there any problems when running multiple debezium connectors on one server?

[최동주]

Are there any problems when running multiple debezium connectors on one server?

For example, is it a condition race, scheduling problem? (i used distributed mode)

my debezium config

one

{
"name" : "tcmg-optionset",
"config" : {
"include.schema.changes":"true",
"snapshot.mode":"when_needed",
"connector.class" : "io.debezium.connector.mysql.MySqlConnector",
"database.hostname" : "",
"database.port" : "",
"database.user":"",
"database.password":"!",
"database.server.id":"20022",
"database.server.name":"klagoDev_tcmg_optionset",
"database.history.kafka.bootstrap.servers":"localhost:9092",
"database.history.kafka.topic":"history_tcmg_optionset",
"database.whitelist":"",
"table.whitelist":".tcmg_optionset",
"database.serverTimezone":"Asia/Seoul",
"tasks.max":"1",
"transforms":"unwrap,route",
"transforms.unwrap.type":"io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.add.fields":"op,table,db",
"transforms.unwrap.delete.handling.mode":"rewrite",
"transforms.route.type":"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex":"([^.]+)\\.([^.]+)\\.([^.]+)",
"transforms.route.replacement":"tcmg_optionset",
"decimal.handling.mode":"double"
}
}

two

{
"name" : "titg-item",
"config" : {
"include.schema.changes":"true",
"snapshot.mode":"when_needed",
"connector.class" : "io.debezium.connector.mysql.MySqlConnector",
"database.hostname" : "",
"database.port" : "",
"database.user":"",
"database.password":"",
"database.server.id":"20026",
"database.server.name":"klagoDev_titg_item",
"database.history.kafka.bootstrap.servers":"localhost:9092",
"database.history.kafka.topic":"history_titg_item",
"database.whitelist":"",
"table.whitelist":"neos_1234.titg_item",
"database.serverTimezone":"Asia/Seoul",
"tasks.max":"1",
"transforms":"unwrap,route",
"transforms.unwrap.type":"io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.add.fields":"op,table,db",
"transforms.unwrap.delete.handling.mode":"rewrite",
"transforms.route.type":"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex":"([^.]+)\\.([^.]+)\\.([^.]+)",
"transforms.route.replacement":"titg_item",
"decimal.handling.mode":"double"
}
}

[Chris Cranford]

Hi,

You can safely run multiple connectors in the same Kafka Connect environment, that's not a problem.  If you're referring to running multiple connectors against the same database, while that is technically possible as well, we do generally recommend you don't do that unless there is an absolute need for it.  More often than not, a single connector can easily perform the workload of many tables rather than trying to scale outward with multiple connectors.

Chris

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/ab3c1459-a3ef-45b7-a42f-a3be54396bf1n%40googlegroups.com.

[Gunnar Morling]

What Chris says. It's not quite clear what you mean by "server" here; in any case you definitely must use distinct values for database.server.id and database.server.name, respectively.

--Gunnar