Debezium MySQL connector fails with MariaDB in useCursorFetch mode

264 views
Skip to first unread message

Omar Ghalawinji

unread,
Sep 22, 2021, 4:21:39 AM9/22/21
to debezium
Hello everyone!

I am trying to run Debezium MySQL connector on a MariaDB. Since we have very big tables, I had to set "snapshot.fetch.size" = "1000" in order not to compromise the performance of the DB during the initial snapshot. My understanding is that this setting will automatically enable useCursorFetch = true (changing the connection protocol from TextProtocol to BinaryProtocl). The connector starts successfully to snapshot few tables then fails with the following exception:

org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped. at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:130) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: io.debezium.DebeziumException: java.lang.RuntimeException: Invalid length when read MySQL DATETIME value. BIN_LEN_DATETIME is 0 at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:78) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:113) ... 5 more Caused by: java.lang.RuntimeException: Invalid length when read MySQL DATETIME value. BIN_LEN_DATETIME is 0 at io.debezium.connector.mysql.MysqlBinaryProtocolFieldReader.readTimestampField(MysqlBinaryProtocolFieldReader.java:86) at io.debezium.connector.mysql.AbstractMysqlFieldReader.readField(AbstractMysqlFieldReader.java:39) at io.debezium.connector.mysql.MySqlConnection.getColumnValue(MySqlConnection.java:571) at io.debezium.jdbc.JdbcConnection.rowToArray(JdbcConnection.java:1470) at io.debezium.relational.RelationalSnapshotChangeEventSource.createDataEventsForTable(RelationalSnapshotChangeEventSource.java:355) at io.debezium.relational.RelationalSnapshotChangeEventSource.createDataEvents(RelationalSnapshotChangeEventSource.java:306) at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:136) at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:69) ... 6 more

Would you have any idea on how to work-around this error?

Many thanks

Chris Cranford

unread,
Sep 22, 2021, 12:29:33 PM9/22/21
to debe...@googlegroups.com, Omar Ghalawinji
Hi Omar -

There was a bug reported, DBZ-3773 [1], but this had to deal explicitly with TIME data types.  It seems in your case yours is TIMESTAMP but its the same problem.  Could you please open a new issue [2], provide the stack trace and the table's DDL and link the issue to be related to DBZ-3773?  We'll try and get this fixed for TIMESTAMP asap.

Thanks,
CC

[1]: https://issues.redhat.com/browse/DBZ-3773
[2]: https://issues.redhat.com/browse/DBZ
Many thanks --
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/96e86563-decd-42c8-9eee-0f5a4e895259n%40googlegroups.com.

Omar Ghalawinji

unread,
Sep 23, 2021, 5:15:47 AM9/23/21
to debezium
Hi Chris!

Thanks very much for your reply and for looking into the issue.. I created DBZ-4032 to track the bug

Cheers
Omar

Omar Ghalawinji

unread,
Sep 27, 2021, 5:04:46 AM9/27/21
to debezium
Hi Chris,

Would you know whether this bug affects MySQL DBs as well? Or it only happens when using the MySQL connector with MariaDB?

Thanks
Omar

Chris Cranford

unread,
Sep 27, 2021, 8:39:43 AM9/27/21
to debe...@googlegroups.com, Omar Ghalawinji
Hi Omar -

I'm not sure but if history is any indicator, it would either be MariaDB related, a very new release of MySQL related, or both.  These reports only came up recently.

Chris

Omar Ghalawinji

unread,
Oct 28, 2021, 11:31:58 AM10/28/21
to debezium
Hi Chris,

I am running into a similar problem when triggering an adhoc incremental snapshot using signaling table.

Error
--------------------------------
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped. at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42) at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:366) at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:855) at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1125) at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:973) at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599) at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: io.debezium.DebeziumException: Error processing binlog event ... 7 more Caused by: org.apache.kafka.connect.errors.ConnectException: Error while processing event at offset {transaction_id=null, ts_sec=1635434641, file=mysql-bin.000117, pos=78378556, incremental_snapshot_maximum_key=aced000570, gtids=3786cf53-f86d-11eb-afe2-42010a3c7032:1-7415958, row=1, server_id=2906499770, event=2, incremental_snapshot_collections=db.newsletter_configuration, incremental_snapshot_primary_key=aced000570} at io.debezium.pipeline.EventDispatcher.dispatchDataChangeEvent(EventDispatcher.java:255) at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$handleInsert$4(MySqlStreamingChangeEventSource.java:688) at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleChange(MySqlStreamingChangeEventSource.java:745) at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleInsert(MySqlStreamingChangeEventSource.java:686) at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$16(MySqlStreamingChangeEventSource.java:831) at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:349) ... 6 more Caused by: java.lang.RuntimeException: Invalid length when read MySQL DATETIME value. BIN_LEN_DATETIME is 19 at io.debezium.connector.mysql.MysqlBinaryProtocolFieldReader.readTimestampField(MysqlBinaryProtocolFieldReader.java:101) at io.debezium.connector.mysql.AbstractMysqlFieldReader.readField(AbstractMysqlFieldReader.java:39) at io.debezium.connector.mysql.MySqlConnection.getColumnValue(MySqlConnection.java:595) at io.debezium.jdbc.JdbcConnection.rowToArray(JdbcConnection.java:1484) at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.lambda$readChunk$0(AbstractIncrementalSnapshotChangeEventSource.java:232) at io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:645) at io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:513) at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.readChunk(AbstractIncrementalSnapshotChangeEventSource.java:228) at io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.addDataCollectionNamesToSnapshot(AbstractIncrementalSnapshotChangeEventSource.java:287) at io.debezium.pipeline.signal.ExecuteSnapshot.arrived(ExecuteSnapshot.java:56) at io.debezium.pipeline.signal.Signal.process(Signal.java:140) at io.debezium.pipeline.signal.Signal.process(Signal.java:184) at io.debezium.pipeline.EventDispatcher$2.changeRecord(EventDispatcher.java:229) at io.debezium.relational.RelationalChangeRecordEmitter.emitCreateRecord(RelationalChangeRecordEmitter.java:78) at io.debezium.relational.RelationalChangeRecordEmitter.emitChangeRecords(RelationalChangeRecordEmitter.java:46) at io.debezium.pipeline.EventDispatcher.dispatchDataChangeEvent(EventDispatcher.java:218) ... 11 more
---------------------------------

Contrary to the old error, this time the invalid length is 19 not 0


Table DDL
------------------------------------------------------------------------
CREATE TABLE `newsletter_configuration` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `lang` varchar(5) COLLATE utf8_unicode_ci NOT NULL,
  `recipient_email_addresses` longtext COLLATE utf8_unicode_ci NOT NULL COMMENT '(DC2Type:json_array)',
  `shipper_organization_id` int(11) NOT NULL,
  `carrier_organization_id` int(11) NOT NULL,
  `created_at` datetime DEFAULT NULL COMMENT '(DC2Type:datetime_immutable)',
  `updated_at` datetime DEFAULT NULL COMMENT '(DC2Type:datetime_immutable)',
  PRIMARY KEY (`id`),
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
-----------------------------------------------------------------------


Would you have some idea regarding this error?
Thank you very much
Omar

Chris Cranford

unread,
Oct 28, 2021, 12:26:37 PM10/28/21
to debe...@googlegroups.com
Hi Omar -

Could you please raise a Jira issue with all this detail, including how you're triggering the incremental snapshots, the MySQL version, etc.

Thanks
Chris

Omar Ghalawinji

unread,
Oct 29, 2021, 11:32:19 AM10/29/21
to debezium
Chris thanks for your comment. I created: https://issues.redhat.com/browse/DBZ-4217
Cheers
Omar

Reply all
Reply to author
Forward
0 new messages