Not able to start debezium

[srikant dalai]

Hi All,

I am not able to start the connector. Please find the debezium-connector details

name=coldstart_dbzm_connector3
connector.class=io.debezium.connector.mysql.MySqlConnector
database.hostname=localhost
database.port=3306
database.user=cccc
database.password=cccc
database.server.id=2
database.server.name=dbz_prod1
database.include.list=
table.include.list=
database.history.kafka.topic=coldstart_dbhistory
bigint.unsigned.handling.mode=precise
binary.handling.mode=base64
topic.creation.enable = true
topic.creation.default.partitions=20
topic.creation.default.replication.factor=2
max.queue.size = 7000
max.batch.size = 500
snapshot.mode = initial

I am getting this error in logs:-

Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic coldstart_dbhistory not present in metadata after 10000 ms.
    at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1316)
    at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:985)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:885)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:773)
    at io.debezium.relational.history.KafkaDatabaseHistory.storeRecord(KafkaDatabaseHistory.java:275)
    ... 17 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Topic coldstart_dbhistory not present in metadata after 10000 ms.

Any steps how to start this connector.

I have tried changing the connector name but it had failed. 

[Chris Cranford]

Hi Srikant -

This error typically means that the Kafka broker has either stopped or is no longer available due to some network problem.  What version of Debezium are you using?  I notice you haven't specified the "database.history.kafka.bootstrap.servers" property in your connector configuration and this seems dubious to me.

Chris

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/93404071-2f70-46b7-9def-e6628f455197n%40googlegroups.com.

[srikant dalai]

Hi Chris,

Parameter is there, missed to paste it here. I am using  debezium-connector-mysql-1.9.5.Final

database.history.kafka.bootstrap.servers=10.X.X.X:9092

[Chris Cranford]

Hi Srikant, can you check to see if you have any connectivity issues between the broker and the Kafka Connect environment.

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/a29fb2d1-e24d-4a10-9563-87584c988a6fn%40googlegroups.com.

[srikant dalai]

Hi Chris,

I am able to telnet to all the brokers. Guide me if anything else is needed.

[srikant dalai]

Hi Chris,

It worked with debezium 1.8 without any issue.i am using MySQL 5.7.38.

Please help me find out the reason.

[Nilesh Sinha]

I am also facing the same issue. Auto history topic creation not working on version 1.9.5, working on 1.8.1 though.

[Chris Cranford]

Hi Nilesh -

Can you check to see whether this is specific to 1.9.5 by checking whether it works on earlier versions of 1.9.x?

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/63dcbf44-67ec-4a4c-9292-8171a7a01045n%40googlegroups.com.

[Nilesh Sinha]

I checked for 1.9.0-final and 1.9.4-final, its not working.

[Chris Cranford]

Hi Nilesh -

Can you please share your connector configuration and the failure message you get?

Thanks,
Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/0a6b174c-7867-431c-a83f-b621faadceden%40googlegroups.com.

[Nilesh Sinha]

Hi Chris,

Config file:-

connector.class=io.debezium.connector.mysql.MySqlConnector
snapshot.locking.mode=none
database.history.kafka.topic=curefit_cultapp_schema_hist2
transforms=route
max.queue.size = 81290
include.schema.changes=true
max.batch.size = 20482
transforms.route.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.route.regex=([^.]+)\.([^.]+)\.([^.]+)
database.history.kafka.recovery.poll.interval.ms=60000
database.history.store.only.captured.tables.ddl=true
database.whitelist=xxxx
transforms.route.replacement=curefit_cultapp
database.allowPublicKeyRetrieval=true
database.user=curefit
database.server.id=11111111111
database.history.kafka.bootstrap.servers=xxx:9092,xxx:9092
database.server.name=curefit-cultapp-new111111
database.port=3306
task=1
database.hostname=xxxx
database.password=xxxx
database.include.list=xxxx
snapshot.mode=when_needed

Error Message:- 

[2022-07-24 20:56:43,700] ERROR [cultapp-stage2|task-0] WorkerSourceTask{id=cultapp-stage2-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:195)
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
    at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:116)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: io.debezium.DebeziumException: io.debezium.relational.history.DatabaseHistoryException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic curefit_cultapp_schema_hist2 not present in metadata after 10000 ms.
    at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:85)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:155)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:137)
    at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109)
    ... 5 more
Caused by: io.debezium.relational.history.DatabaseHistoryException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic curefit_cultapp_schema_hist2 not present in metadata after 10000 ms.
    at io.debezium.relational.history.KafkaDatabaseHistory.storeRecord(KafkaDatabaseHistory.java:290)
    at io.debezium.relational.history.AbstractDatabaseHistory.record(AbstractDatabaseHistory.java:96)
    at io.debezium.relational.HistorizedRelationalDatabaseSchema.record(HistorizedRelationalDatabaseSchema.java:101)
    at io.debezium.connector.mysql.MySqlDatabaseSchema.applySchemaChange(MySqlDatabaseSchema.java:197)
    at io.debezium.pipeline.EventDispatcher$SchemaChangeEventReceiver.schemaChangeEvent(EventDispatcher.java:539)
    at io.debezium.connector.mysql.MySqlSnapshotChangeEventSource.lambda$createSchemaChangeEventsForTables$7(MySqlSnapshotChangeEventSource.java:548)
    at io.debezium.pipeline.EventDispatcher.dispatchSchemaChangeEvent(EventDispatcher.java:302)
    at io.debezium.connector.mysql.MySqlSnapshotChangeEventSource.createSchemaChangeEventsForTables(MySqlSnapshotChangeEventSource.java:548)
    at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:119)
    at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:76)
    ... 8 more
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic curefit_cultapp_schema_hist2 not present in metadata after 10000 ms.
    at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1329)
    at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:998)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:889)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:775)

    at io.debezium.relational.history.KafkaDatabaseHistory.storeRecord(KafkaDatabaseHistory.java:275)
    ... 17 more

Caused by: org.apache.kafka.common.errors.TimeoutException: Topic curefit_cultapp_schema_hist2 not present in metadata after 10000 ms.

[jiri.p...@gmail.com]

Hi,

are you sure your Kafka broker does not require authentication or secured connection?

J.

[Nilesh Sinha]

Yes , I am sure.

Also same setup works with just downgrading to 1.8.1.

[Chris Cranford]

Hi Nilesh -

Out of curiosity, what is the version of your Kafka broker environment?

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/2e3e0008-b607-4c0f-b9c2-0820c48ee8c5n%40googlegroups.com.

[Nilesh Sinha]

I am on Kafka version - 2.2.1. Using AWS MSK.

[Chris Cranford]

Hi Nilesh -

So I used our tutorial [1] as the basis for deploying a MySQL connector but in the tutorial I made the following adjustments to docker-compose-mysql to align versions as discussed here:

    - debezium/zookeeper:0.9.5.Final (this uses Zookeeper 3.4.13 which shipped with AK 2.2.1)
    - debezium/kafka:0.9.5.Final (this uses AK 2.2.0, a version older than described)
    - debezium/connect:1.8 and debezium/connect:1.9

In both scenarios of deploying the MySQL connector, the connector was successfully deployed and the history topics were created.

Could you please enable TRACE logging on Kafka Connect and the connector and please deploy the connector once more and attach the logs from the start of the deployment up to the point of the failure, using Debezium 1.9.5.Final.  I'd like for us to see whether there is something we can identify in the logs that could explain the exception.  You can send the logs directly to my email address rather than attaching the logs to the mailing list if you'd prefer.

Thanks,
Chris

[1]: https://github.com/debezium/debezium-examples/tree/main/tutorial

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/2d1aa5b1-ce41-4d85-82e2-366e86ecd96fn%40googlegroups.com.

[Nilesh Sinha]

Sent on mail.

[Chris Cranford]

So Nilesh -

So looking at the configuration and the logs, I noticed quite a number of the following warnings:

    Unable to register the MBean

This indicates you either multiple connectors being deployed with the same "database.server.name", which is not allowed as this property must be unique across all connectors on the cluster, or perhaps you have a prior instance that hadn't yet stopped before a new instance was started. 

You also have a considerable number of warnings in the logs that read:

    Error while fetching metadata with correlation id

Now normally this is simply an indicator that the topic doesn't exist and its being created; however normally this process shouldn't take several minutes to complete, so this indicates that either something is happening on the broker side most likely.  Perhaps the broker is overloaded?  I did notice that at "2022-07-25 01:35:34,817" it appears the connect cluster eventually re-established connectivity with the broker, so perhaps there is some connectivity issues here as well?  Later on at "2022-07-25 01:41:06,037", it appears you lost connectivity to the broker yet again.

In any case, are you in a position to try 2.0.0.Beta1 by chance?  We recently added a small check when creating the history topic to verify the asynchronous call's result.  In the versions you've referenced, we haven't been checking that status so perhaps the topic isn't actually being created at all and that's the source of the issue.  Even testing 2.0.0.Beta1 as a temporary connector with similar configuration might help shed some light.

Thanks
Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/5287be9d-2c18-49b6-a77c-6b982663070bn%40googlegroups.com.

[Nilesh Sinha]

Hi Chris,

2.0.0.Beta1 throwing below error.

org.apache.kafka.common.config.ConfigException: Invalid value io.debezium.connector.mysql.MySqlConnectorTask for configuration task.class: Class io.debezium.connector.mysql.MySqlConnectorTask could not be found.
at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:744)
at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:490)
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:483)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:113)
at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:146)
at org.apache.kafka.connect.runtime.TaskConfig.<init>(TaskConfig.java:51)
at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:511)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1421)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$getTaskStartingCallable$22(DistributedHerder.java:1434)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Please find the screenshot of files as well.

You received this message because you are subscribed to a topic in the Google Groups "debezium" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/debezium/2jbO26yLy2w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/1cca90c3-d7bf-32bf-8eb6-8a8bde8a51a6%40gmail.com.

--

Thanks and Regards,

Nilesh Sinha

[Chris Cranford]

Hi Nilesh -

Is this using a debezium/connect image or is this after you manually installed the artifacts into a local deployment of Connect?  If the later, this appears like you may not have installed the plugin correctly.

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/CAK02z2O7L8wdc02rD4srNUx-ZVPFFF3Y1D2SM-RmWvtt3Qo7AQ%40mail.gmail.com.