Encountered a change event for table whose schema isn't Known Debezium Mysql Connector

[Sanket Nagrale]

Hi

Recently added a new schema and new tables for that schema in our connector connector for Debezium Mysql Connector version 1.8.0.Final. After the traffic started of this new table - the connector failed with error

ERROR [debezium-mysql-connector-prod-payment-ntrp-14|task-0] Encountered change event 'Event{header=EventHeaderV4{timestamp=1719367936000, eventType=TABLE_MAP, serverId=613230571, headerLength=19, dataLength=133, nextPosition=76694821, flags=0}, data=TableMapEventData{tableId=25437, database='pynp14', table='pyn_pay_order_0', columnTypes=15, 15, 15, 15, 15, 15, 15, 15, 15, 8, 15, 15, 15, 15, 15, 15, 15, 15, 8, 15, 8, 15, 17, 17, 17, 15, 15, 15, 15, 17, 17, columnMetadata=256, 512, 512, 32, 128, 80, 64, 256, 256, 0, 12, 64, 128, 3060, 128, 256, 128, 128, 0, 12, 0, 12, 6, 6, 6, 256, 128, 128, 128, 6, 0, columnNullability={2, 8, 13, 14, 15, 16, 17, 18, 19, 20, 21, 24, 25, 27, 28}, eventMetadata=TableMapEventMetadata{signedness={}, defaultCharset=255, charsetCollations=null, columnCharsets=null, columnNames=null, setStrValues=null, enumStrValues=null, geometryTypes=null, simplePrimaryKeys=null, primaryKeysWithPrefix=null, enumAndSetDefaultCharset=null, enumAndSetColumnCharsets=null,visibility=null}}}' at offset {transaction_id=null, file=mysql-bin-changelog.035694, pos=76694322, server_id=613230571, event=3} for table pynp14.pyn_pay_order_0 whose schema isn't known to this connector. One possible cause is an incomplete database history topic. Take a new snapshot in this case.
Use the mysqlbinlog tool to view the problematic event: mysqlbinlog --start-position=76694669 --stop-position=76694821 --verbose mysql-bin-changelog.035694 (io.debezium.connector.mysql.MySqlStreamingChangeEventSource:649)
[2024-06-26 07:48:32,779] ERROR [debezium-mysql-connector-prod-payment-ntrp-14|task-0] Error during binlog processing. Last offset stored = {transaction_id=null, file=mysql-bin-changelog.035694, pos=76694322, server_id=613230571, event=3}, binlog reader near position = mysql-bin-changelog.035694/76694669 (io.debezium.connector.mysql.MySqlStreamingChangeEventSource:1072)

io.debezium.DebeziumException: Error processing binlog event
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:369)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:860)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1125)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:973)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.debezium.DebeziumException: Encountered change event for table pynp14.pyn_pay_order_0 whose schema isn't known to this connector
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:654)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:633)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$13(MySqlStreamingChangeEventSource.java:831)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:349)
... 6 more
[2024-06-26 07:48:32,779] INFO [debezium-mysql-connector-prod-payment-ntrp-14|task-0] Error processing binlog event, and propagating to Kafka Connect so it stops this connector. Future binlog events read before connector is shutdown will be ignored. (io.debezium.connector.mysql.MySqlStreamingChangeEventSource:374)
[2024-06-26 07:48:32,874] INFO [debezium-mysql-connector-prod-payment-ntrp-14|task-0] Keepalive thread is running (io.debezium.connector.mysql.MySqlStreamingChangeEventSource:942)
[2024-06-26 07:48:33,763] ERROR [debezium-mysql-connector-prod-payment-ntrp-14|task-0] WorkerSourceTask{id=debezium-mysql-connector-prod-payment-ntrp-14-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:196)
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:369)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:860)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1125)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:973)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.debezium.DebeziumException: Error processing binlog event
... 7 more
Caused by: io.debezium.DebeziumException: Encountered change event for table pynp14.pyn_pay_order_0 whose schema isn't known to this connector
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:654)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:633)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$13(MySqlStreamingChangeEventSource.java:831)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:349)
... 6 more


attempted recovery by changing property:

1) database.history.kafka.topic to something else
2) snapshot.mode to "schema_only_recovery

Connector starter running again successfully. I just wanted ask if there was any data loss in this case, as the history topic was changed and connector started again.

[jiri.p...@gmail.com]

Hi,

no there is no data loss in this situation.

Jiri