Ad-Hoc Snapshot trigger's but errors when attempting to read table

[Aidan]

Hi,

I have configured my debezium connector with a signal table to test triggering an ad-hoc snapshot. I am able to see that the snapshot has tried to trigger but it fails to actually process.

Please see both configuration and error message below:

curl -i -X PUT -H "Accept:application/json" \

  -H  "Content-Type:application/json" http://localhost:8083/connectors/test2/config \

  -d '{

      "connector.class"            : "io.debezium.connector.mysql.MySqlConnector",

      "database.hostname"          : "mysql",

      "database.user"              : "mysqluser",

      "database.port"              : "3306",

      "database.password"          : "mysqlpw",

      "database.server.name"       : "asgard",

      "table.include.list"         : "demo.test, demo.debezium_signal",

      "name"                       : "test2",

      "database.whitelist"         : "demo",

      "snapshot.mode"              : "initial",

      "tasks.max"                  : "1",

      "database.history.kafka.topic" : "asgard.dbhistory.demo",

      "database.history.kafka.bootstrap.servers": "kafka:29092",

      "signal.data.collection"     : "demo.debezium_signal"

      }'

"org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.

at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)

at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:366)

at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:842)at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1118)

at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:966)

at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:606)

at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:850)

at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: io.debezium.DebeziumException: Error processing binlog event

... 7 more

Caused by: org.apache.kafka.connect.errors.ConnectException: Error while processing event at offset {transaction_id=null, ts_sec=1628467431, file=binlog.000002, pos=1762, incremental_snapshot_maximum_key=aced000570, row=1, server_id=1, event=2, incremental_snapshot_collections=demo.test, incremental_snapshot_primary_key=aced000570}

at io.debezium.pipeline.EventDispatcher.dispatchDataChangeEvent(EventDispatcher.java:253)

at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$handleInsert$4(MySqlStreamingChangeEventSource.java:682)

at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleChange(MySqlStreamingChangeEventSource.java:732)

at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleInsert(MySqlStreamingChangeEventSource.java:681)

at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$16(MySqlStreamingChangeEventSource.java:818)

at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:349)

... 6 more

Caused by: io.debezium.DebeziumException: Database error while executing incremental snapshot

at io.debezium.pipeline.source.snapshot.incremental.SignalBasedIncrementalSnapshotChangeEventSource.readChunk(SignalBasedIncrementalSnapshotChangeEventSource.java:257)

at io.debezium.pipeline.source.snapshot.incremental.SignalBasedIncrementalSnapshotChangeEventSource.addDataCollectionNamesToSnapshot(SignalBasedIncrementalSnapshotChangeEventSource.java:273)

at io.debezium.pipeline.signal.ExecuteSnapshot.arrived(ExecuteSnapshot.java:64)

at io.debezium.pipeline.signal.Signal.process(Signal.java:136)

at io.debezium.pipeline.signal.Signal.process(Signal.java:180)

at io.debezium.pipeline.EventDispatcher$2.changeRecord(EventDispatcher.java:227)

at io.debezium.relational.RelationalChangeRecordEmitter.emitCreateRecord(RelationalChangeRecordEmitter.java:77)

at io.debezium.relational.RelationalChangeRecordEmitter.emitChangeRecords(RelationalChangeRecordEmitter.java:45)

at io.debezium.pipeline.EventDispatcher.dispatchDataChangeEvent(EventDispatcher.java:217)

... 11 more

Caused by: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Data too long for column 'id' at row 1

at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:104)

at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)

at com.mysql.cj.jdbc.ClientPreparedStatement.execute(ClientPreparedStatement.java:370)

at io.debezium.jdbc.JdbcConnection.prepareUpdate(JdbcConnection.java:764)

at io.debezium.pipeline.source.snapshot.incremental.SignalBasedIncrementalSnapshotChangeEventSource.emitWindowOpen(SignalBasedIncrementalSnapshotChangeEventSource.java:135)

at io.debezium.pipeline.source.snapshot.incremental.SignalBasedIncrementalSnapshotChangeE100  4165  100  4165    0     0   123k      0 --:--:-- --:--:-- --:--:--  123kpshotChangeEventSource.java:211)

... 19 more

[jiri.p...@gmail.com]

Hi,

could you please share the DDL of the table?

Thanks

J.

[Aidan]

Hi Jiri,

I am using debezium v.1.6.0.

Table that I am trying to source from:

 CREATE TABLE test (personid int, name varchar(20));

Debezium signal table:

CREATE TABLE debezium_signal (id VARCHAR(10) NOT NULL, type VARCHAR(32) NOT NULL, data VARCHAR(2048) NULL);

Message sent into debezium table:

INSERT INTO debezium_signal VALUES('ad-hoc', 'execute-snapshot', '{"data-collections": ["demo.test"]}');

Regards,

Aidan

[Aidan]

Issue was that the debezium signal(id) was not long enough  CREATE TABLE debezium_signal (id VARCHAR(50) NOT NULL, type VARCHAR(32) NOT NULL, data VARCHAR(2048) NULL); was successful