Debezium MySQL connector fails with MariaDB in useCursorFetch mode
697 views
Skip to first unread message
Omar Ghalawinji
Sep 22, 2021, 4:21:39 AM9/22/21
to debezium
Hello everyone!
I am trying to run Debezium MySQL connector on a MariaDB. Since we have very big tables, I had to set "snapshot.fetch.size" = "1000" in order not to compromise the performance of the DB during the initial snapshot. My understanding is that this setting will automatically enable useCursorFetch = true (changing the connection protocol from TextProtocol to BinaryProtocl). The connector starts successfully to snapshot few tables then fails with the following exception:
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped. at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:130) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: io.debezium.DebeziumException: java.lang.RuntimeException: Invalid length when read MySQL DATETIME value. BIN_LEN_DATETIME is 0 at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:78) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:113) ... 5 more Caused by: java.lang.RuntimeException: Invalid length when read MySQL DATETIME value. BIN_LEN_DATETIME is 0 at io.debezium.connector.mysql.MysqlBinaryProtocolFieldReader.readTimestampField(MysqlBinaryProtocolFieldReader.java:86) at io.debezium.connector.mysql.AbstractMysqlFieldReader.readField(AbstractMysqlFieldReader.java:39) at io.debezium.connector.mysql.MySqlConnection.getColumnValue(MySqlConnection.java:571) at io.debezium.jdbc.JdbcConnection.rowToArray(JdbcConnection.java:1470) at io.debezium.relational.RelationalSnapshotChangeEventSource.createDataEventsForTable(RelationalSnapshotChangeEventSource.java:355) at io.debezium.relational.RelationalSnapshotChangeEventSource.createDataEvents(RelationalSnapshotChangeEventSource.java:306) at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:136) at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:69) ... 6 more
Would you have any idea on how to work-around this error?
Many thanks
I am trying to run Debezium MySQL connector on a MariaDB. Since we have very big tables, I had to set "snapshot.fetch.size" = "1000" in order not to compromise the performance of the DB during the initial snapshot. My understanding is that this setting will automatically enable useCursorFetch = true (changing the connection protocol from TextProtocol to BinaryProtocl). The connector starts successfully to snapshot few tables then fails with the following exception:
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped. at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:130) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: io.debezium.DebeziumException: java.lang.RuntimeException: Invalid length when read MySQL DATETIME value. BIN_LEN_DATETIME is 0 at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:78) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:113) ... 5 more Caused by: java.lang.RuntimeException: Invalid length when read MySQL DATETIME value. BIN_LEN_DATETIME is 0 at io.debezium.connector.mysql.MysqlBinaryProtocolFieldReader.readTimestampField(MysqlBinaryProtocolFieldReader.java:86) at io.debezium.connector.mysql.AbstractMysqlFieldReader.readField(AbstractMysqlFieldReader.java:39) at io.debezium.connector.mysql.MySqlConnection.getColumnValue(MySqlConnection.java:571) at io.debezium.jdbc.JdbcConnection.rowToArray(JdbcConnection.java:1470) at io.debezium.relational.RelationalSnapshotChangeEventSource.createDataEventsForTable(RelationalSnapshotChangeEventSource.java:355) at io.debezium.relational.RelationalSnapshotChangeEventSource.createDataEvents(RelationalSnapshotChangeEventSource.java:306) at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:136) at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:69) ... 6 more
Would you have any idea on how to work-around this error?
Many thanks
Chris Cranford
Sep 22, 2021, 12:29:33 PM9/22/21
to debe...@googlegroups.com, Omar Ghalawinji
Hi Omar -
There was a bug reported, DBZ-3773 [1], but this had to deal explicitly with TIME data types. It seems in your case yours is TIMESTAMP but its the same problem. Could you please open a new issue [2], provide the stack trace and the table's DDL and link the issue to be related to DBZ-3773? We'll try and get this fixed for TIMESTAMP asap.
Thanks,
CC
[1]: https://issues.redhat.com/browse/DBZ-3773
[2]: https://issues.redhat.com/browse/DBZ
There was a bug reported, DBZ-3773 [1], but this had to deal explicitly with TIME data types. It seems in your case yours is TIMESTAMP but its the same problem. Could you please open a new issue [2], provide the stack trace and the table's DDL and link the issue to be related to DBZ-3773? We'll try and get this fixed for TIMESTAMP asap.
Thanks,
CC
[1]: https://issues.redhat.com/browse/DBZ-3773
[2]: https://issues.redhat.com/browse/DBZ
Many thanks --
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/96e86563-decd-42c8-9eee-0f5a4e895259n%40googlegroups.com.
Omar Ghalawinji
Sep 23, 2021, 5:15:47 AM9/23/21
to debezium
Hi Chris!
Thanks very much for your reply and for looking into the issue.. I created DBZ-4032 to track the bug
Cheers
Omar
Thanks very much for your reply and for looking into the issue.. I created DBZ-4032 to track the bug
Cheers
Omar
Omar Ghalawinji
Sep 27, 2021, 5:04:46 AM9/27/21
to debezium
Hi Chris,
Would you know whether this bug affects MySQL DBs as well? Or it only happens when using the MySQL connector with MariaDB?
Thanks
Omar
Would you know whether this bug affects MySQL DBs as well? Or it only happens when using the MySQL connector with MariaDB?
Thanks
Omar
Chris Cranford
Sep 27, 2021, 8:39:43 AM9/27/21
to debe...@googlegroups.com, Omar Ghalawinji
Hi Omar -
I'm not sure but if history is any indicator, it would either be MariaDB related, a very new release of MySQL related, or both. These reports only came up recently.
Chris
I'm not sure but if history is any indicator, it would either be MariaDB related, a very new release of MySQL related, or both. These reports only came up recently.
Chris
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/322f1681-1c17-4d53-9c06-0adcc29d2a77n%40googlegroups.com.
Omar Ghalawinji
Oct 28, 2021, 11:31:58 AM10/28/21
to debezium
Hi Chris,
I am running into a similar problem when triggering an adhoc incremental snapshot using signaling table.
I am running into a similar problem when triggering an adhoc incremental snapshot using signaling table.
Error
--------------------------------
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:366)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:855)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1125)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:973)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.debezium.DebeziumException: Error processing binlog event
... 7 more
Caused by: org.apache.kafka.connect.errors.ConnectException: Error while processing event at offset {transaction_id=null, ts_sec=1635434641, file=mysql-bin.000117, pos=78378556, incremental_snapshot_maximum_key=aced000570, gtids=3786cf53-f86d-11eb-afe2-42010a3c7032:1-7415958, row=1, server_id=2906499770, event=2, incremental_snapshot_collections=db.newsletter_configuration, incremental_snapshot_primary_key=aced000570}
at io.debezium.pipeline.EventDispatcher.dispatchDataChangeEvent(EventDispatcher.java:255)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$handleInsert$4(MySqlStreamingChangeEventSource.java:688)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleChange(MySqlStreamingChangeEventSource.java:745)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleInsert(MySqlStreamingChangeEventSource.java:686)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$16(MySqlStreamingChangeEventSource.java:831)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:349)
... 6 more
Caused by: java.lang.RuntimeException: Invalid length when read MySQL DATETIME value. BIN_LEN_DATETIME is 19
at io.debezium.connector.mysql.MysqlBinaryProtocolFieldReader.readTimestampField(MysqlBinaryProtocolFieldReader.java:101)
at io.debezium.connector.mysql.AbstractMysqlFieldReader.readField(AbstractMysqlFieldReader.java:39)
at io.debezium.connector.mysql.MySqlConnection.getColumnValue(MySqlConnection.java:595)
at io.debezium.jdbc.JdbcConnection.rowToArray(JdbcConnection.java:1484)
at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.lambda$readChunk$0(AbstractIncrementalSnapshotChangeEventSource.java:232)
at io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:645)
at io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:513)
at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.readChunk(AbstractIncrementalSnapshotChangeEventSource.java:228)
at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.addDataCollectionNamesToSnapshot(AbstractIncrementalSnapshotChangeEventSource.java:287)
at io.debezium.pipeline.signal.ExecuteSnapshot.arrived(ExecuteSnapshot.java:56)
at io.debezium.pipeline.signal.Signal.process(Signal.java:140)
at io.debezium.pipeline.signal.Signal.process(Signal.java:184)
at io.debezium.pipeline.EventDispatcher$2.changeRecord(EventDispatcher.java:229)
at io.debezium.relational.RelationalChangeRecordEmitter.emitCreateRecord(RelationalChangeRecordEmitter.java:78)
at io.debezium.relational.RelationalChangeRecordEmitter.emitChangeRecords(RelationalChangeRecordEmitter.java:46)
at io.debezium.pipeline.EventDispatcher.dispatchDataChangeEvent(EventDispatcher.java:218)
... 11 more
---------------------------------
Contrary to the old error, this time the invalid length is 19 not 0
Table DDL
------------------------------------------------------------------------
CREATE TABLE `newsletter_configuration` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`lang` varchar(5) COLLATE utf8_unicode_ci NOT NULL,
`recipient_email_addresses` longtext COLLATE utf8_unicode_ci NOT NULL COMMENT '(DC2Type:json_array)',
`shipper_organization_id` int(11) NOT NULL,
`carrier_organization_id` int(11) NOT NULL,
`created_at` datetime DEFAULT NULL COMMENT '(DC2Type:datetime_immutable)',
`updated_at` datetime DEFAULT NULL COMMENT '(DC2Type:datetime_immutable)',
PRIMARY KEY (`id`),
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
-----------------------------------------------------------------------
Would you have some idea regarding this error?
Thank you very much
Omar
Chris Cranford
Oct 28, 2021, 12:26:37 PM10/28/21
to debe...@googlegroups.com
Hi Omar -
Could you please raise a Jira issue with all this detail, including how you're triggering the incremental snapshots, the MySQL version, etc.
Thanks
Chris
Could you please raise a Jira issue with all this detail, including how you're triggering the incremental snapshots, the MySQL version, etc.
Thanks
Chris
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/23534ec5-0f5a-4328-a500-922f2783c99en%40googlegroups.com.
Omar Ghalawinji
Oct 29, 2021, 11:32:19 AM10/29/21
to debezium
Reply all
Reply to author
Forward
0 new messages