Restarting the connector with "snapshot.mode": "SCHEMA_ONLY_RECOVERY" failed

[Ritesh Singh]

Hi ALL,

My connector's DB history got too big to read. Even after setting

"database.history.kafka.recovery.attempts": "20",
"database.history.kafka.recovery.poll.interval.ms": "1000",

It was not able to read the DBHistory on the start.

Then I deleted the history topic, and restarted the connector with "snapshot.mode": "SCHEMA_ONLY_RECOVERY", it failed because of an error:

{"name":"dptestsite2-connector","connector":{"state":"RUNNING","worker_id":"connect:8083"},"tasks":[{"id":0,"state":"FAILED","worker_id":"connect:8083","trace":"org.apache.kafka.connect.errors.ConnectException: Unexpected error while connecting to MySQL and looking at GTID mode: \n\tat io.debezium.connector.mysql.MySqlJdbcContext.isGtidModeEnabled(MySqlJdbcContext.java:175)\n\tat io.debezium.connector.mysql.BinlogReader.doStart(BinlogReader.java:312)\n\tat io.debezium.connector.mysql.AbstractReader.start(AbstractReader.java:116)\n\tat io.debezium.connector.mysql.ChainedReader.startNextReader(ChainedReader.java:203)\n\tat io.debezium.connector.mysql.ChainedReader.readerCompletedPolling(ChainedReader.java:157)\n\tat io.debezium.connector.mysql.AbstractReader.cleanupResources(AbstractReader.java:308)\n\tat io.debezium.connector.mysql.AbstractReader.poll(AbstractReader.java:288)\n\tat io.debezium.connector.mysql.ChainedReader.poll(ChainedReader.java:145)\n\tat io.debezium.connector.mysql.MySqlConnectorTask.poll(MySqlConnectorTask.java:416)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:245)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:221)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.sql.SQLNonTransientConnectionException: Too many connections\n\tat com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)\n\tat com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)\n\tat com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)\n\tat com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:835)\n\tat com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:455)\n\tat com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:240)\n\tat com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:207)\n\tat io.debezium.jdbc.JdbcConnection.lambda$patternBasedFactory$1(JdbcConnection.java:179)\n\tat io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:756)\n\tat io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:751)\n\tat io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:437)\n\tat io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:385)\n\tat io.debezium.connector.mysql.MySqlJdbcContext.isGtidModeEnabled(MySqlJdbcContext.java:169)\n\t... 17 more\n"}],"type":"source"}

MySql does not have GTID mode enabled.

Thanks

Ritesh

[Gunnar Morling]

The root cause seems to be "Too many connections". So have you looked
into increasing the max. number of connections with your DB if there's
many other connections already indeed?

--Gunnar

Am Mi., 1. Juli 2020 um 12:26 Uhr schrieb Ritesh Singh
<ritesh...@gmail.com>:

> --
> You received this message because you are subscribed to the Google Groups "debezium" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/299b4745-518c-470c-8781-2dee73a80225o%40googlegroups.com.

[Gunnar Morling]

I've also logged DBZ-2300 for trying to re-connect in that case. It seems worth trying to do so for the sake of better connector resiliency, although the underlying issue of too many connections (by other apps supposedly) should be analyzed of course.

[Ritesh Singh]

Thanks Gunnar, it works as I am able to make DB connection now.

The only thing that confused me from the trace was "Looking at GTID mode"