Error in mariadb galera cluster

[최동주]

hello

After running Debezium on the DB composed of mariadb Galera cluster, the following ERROR message appears.

[2022-02-22 16:57:22,462] INFO Snapshot - Final stage (io.debezium.pipeline.source.AbstractSnapshotChangeEventSource:88)
[2022-02-22 16:57:22,463] ERROR Producer failure (io.debezium.pipeline.ErrorHandler:31)
io.debezium.DebeziumException: java.lang.RuntimeException: java.sql.SQLNonTransientConnectionException: No operations allowed after connection closed.
        at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:85)
        at io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:153)
        at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:135)
        at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:108)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.RuntimeException: java.sql.SQLNonTransientConnectionException: No operations allowed after connection closed.
        at io.debezium.relational.RelationalSnapshotChangeEventSource.rollbackTransaction(RelationalSnapshotChangeEventSource.java:520)
        at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:143)
        at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:76)
        ... 8 more
Caused by: java.sql.SQLNonTransientConnectionException: No operations allowed after connection closed.
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73)
        at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:73)
        at com.mysql.cj.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:1846)
        at io.debezium.relational.RelationalSnapshotChangeEventSource.rollbackTransaction(RelationalSnapshotChangeEventSource.java:517)
        ... 10 more
Caused by: com.mysql.cj.exceptions.ConnectionIsClosedException: No operations allowed after connection closed.
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
        at com.mysql.cj.NativeSession.checkClosed(NativeSession.java:762)
        at com.mysql.cj.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:568)
        at com.mysql.cj.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:1800)
        ... 11 more
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure

The last packet successfully received from the server was 2 milliseconds ago. The last packet sent successfully to the server was 56 milliseconds ago.
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
        at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
        at com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
        at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:520)
        at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:700)
        at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:639)
        at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:987)
        at com.mysql.cj.protocol.a.NativeProtocol.sendQueryString(NativeProtocol.java:933)
        at com.mysql.cj.NativeSession.execSQL(NativeSession.java:664)
        at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1174)
        at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:556)
        at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:499)
        at io.debezium.connector.mysql.MySqlSnapshotChangeEventSource.createSchemaEventsForTables(MySqlSnapshotChangeEventSource.java:371)
        at io.debezium.connector.mysql.MySqlSnapshotChangeEventSource.readTableStructure(MySqlSnapshotChangeEventSource.java:360)
        at io.debezium.connector.mysql.MySqlSnapshotChangeEventSource.readTableStructure(MySqlSnapshotChangeEventSource.java:47)
        at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:113)
        ... 9 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
        at com.mysql.cj.protocol.FullReadInputStream.readFully(FullReadInputStream.java:67)
        at com.mysql.cj.protocol.a.SimplePacketReader.readHeaderLocal(SimplePacketReader.java:81)
        at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:63)
        at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:45)
        at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:52)
        at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:41)
        at com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:54)
        at com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:44)
        at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:514)
        ... 21 more

After the corresponding ERROR occurs, the Galera Cluster Connection is disconnected. Any solution?

my Debezium execute body

{
"name" : "oneQuery",
"config" : {
"include.schema.changes":"true",
"snapshot.mode":"initial",
"connector.class" : "io.debezium.connector.mysql.MySqlConnector",

"database.hostname" : "ip",
"database.port" : "port",
"database.user":"root",
"database.password":"root",
"database.server.id":"91015",
"database.server.name":"klagoDev_add9",
"database.history.kafka.bootstrap.servers":"localhost:9092",
"database.history.kafka.topic":"history_000010",
"table.include.list":
"neos_1234.debezium_signal,neos_1234.t_co_emp,neos_1234.t_co_dept,neos.t_co_emp,dongju_neos.user,dongju_neos.debezium_signal,study.employee",
"database.serverTimezone":"Asia/Seoul",
"tasks.max":"1",
"transforms":"unwrap,route",
"transforms.unwrap.type":"io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.add.fields":"op,table,db,snapshot",
"transforms.unwrap.delete.handling.mode":"rewrite",
"transforms.route.type":"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex":"([^.]+)\\.([^.]+)\\.([^.]+)",
"transforms.route.replacement":"add9",
"decimal.handling.mode":"double",

"database.history.security.protocol":"SASL_PLAINTEXT",
"database.history.sasl.mechanism":"SCRAM-SHA-512",
"database.history.sasl.jaas.config":"config",

"database.history.producer.security.protocol":"SASL_PLAINTEXT",
"database.history.producer.sasl.mechanism":"SCRAM-SHA-512",
"database.history.producer.sasl.jaas.config":"config",

"database.history.consumer.security.protocol":"SASL_PLAINTEXT",
"database.history.consumer.sasl.mechanism":"SCRAM-SHA-512",
"database.history.consumer.sasl.jaas.config""config",

"signal.data.collection":"cdctool.debezium_signal"
}
}

[Gunnar Morling]

Hi,

Could you clarify, does this happen after a while only (i.e. at some point the connection gets lost, perhaps due to some networking issue), or can you actually not process any change events from MariaDB/Galera at all? In case of the former, the error seems worthy of automatically reconnecting; could you please log an issue at https://issues.redhat.com/browse/DBZ, and perhaps also send a PR for adjusting Debezium's MySqlErrorHandler class accordingly?

Note neither MariaDB nor Galera are officially supported at this point (though we are aware of many MariaDB users; not sure about Galera).

Thx,

--Gunnar