Debezium MySQL Connector failed due to DDL binlog of table not included in table.include.list

1,311 views

Skip to first unread message

shani.kim

unread,

Apr 28, 2021, 9:46:29 PM4/28/21

to debezium

Hello,!

My Debezium MySQL Connector failed due to DDL binlog of table not included in table.include.list.

I think it is inefficient for the Connector to be affected by changes of tables that are not included in the table.include.list.

I have two questions.

1. Can i skip DDL binlog of tables that are not included in table.include.list?

As intended in the config below, I would like to CDC test.test_outbox table only.

2. Why did Debezium connector fail to parse the problematic DDL?

I'd really appreciate it if you could answer my question.

I'll attach the relevant information.

Regards,

shani.kim

- Debezium version used:1.3

- Connector config:

{

"name": "shopper-connector",

"config": {

"connector.class": "io.debezium.connector.mysql.MySqlConnector",

"tasks.max": "1",

"database.hostname": "...",

"database.port": "...",

"database.user": "...",

"database.password": "...",

"database.server.id": "5555",

"database.server.name": "mysql_real",

"database.include.list": "test",

"table.include.list": "test.test_outbox",

"tombstones.on.delete": false,

"skipped.operations": "d,u",

"value.converter": "io.debezium.converters.CloudEventsConverter",

"message.key.columns": "test.test_outbox:aggregate_id",

"database.history.kafka.bootstrap.servers": "...",

"database.history.kafka.topic": "schema.test"

}

- Problematic DDL

'CREATE TABLE separated_shopper_sns ( \nshpr_id_no varchar(6) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '', \nsns_code varchar(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL DEFAULT '', \nsns_nick_name varchar(255) NULL DEFAULT NULL, \nsns_key varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL DEFAULT '', \nreg_date_time datetime(6) NOT NULL, \nsns_id varchar(50) NULL DEFAULT NULL, \nid bigint(20) unsigned unsigned'

- error log:

2021-04-12 18:15:03,569 ERROR MySQL|mysql_real|binlog Error during binlog processing. Last offset stored = {ts_sec=1618215502, file=mysql-bin.000001, pos=8233223, row=1, server_id=7297, event=8}, binlog reader near position = mysql-bin.000001/8239665 [io.debezium.connector.mysql.BinlogReader]

2021-04-12 18:15:03,570 ERROR MySQL|mysql_real|binlog Failed due to error: Error processing binlog event [io.debezium.connector.mysql.BinlogReader]

org.apache.kafka.connect.errors.ConnectException: no viable alternative at input 'CREATE TABLE separated_shopper_sns ( \nshpr_id_no varchar(6) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '', \nsns_code varchar(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL DEFAULT '', \nsns_nick_name varchar(255) NULL DEFAULT NULL, \nsns_key varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL DEFAULT '', \nreg_date_time datetime(6) NOT NULL, \nsns_id varchar(50) NULL DEFAULT NULL, \nid bigint(20) unsigned unsigned'

at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)

at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:207)

at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:604)

at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1100)

at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:951)

at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:594)

at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:838)

at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: io.debezium.text.ParsingException: no viable alternative at input 'CREATE TABLE separated_shopper_sns ( \nshpr_id_no varchar(6) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '', \nsns_code varchar(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL DEFAULT '', \nsns_nick_name varchar(255) NULL DEFAULT NULL, \nsns_key varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL DEFAULT '', \nreg_date_time datetime(6) NOT NULL, \nsns_id varchar(50) NULL DEFAULT NULL, \nid bigint(20) unsigned unsigned'

at io.debezium.antlr.ParsingErrorListener.syntaxError(ParsingErrorListener.java:40)

at org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:41)

at org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:544)

at org.antlr.v4.runtime.DefaultErrorStrategy.reportNoViableAlternative(DefaultErrorStrategy.java:310)

at org.antlr.v4.runtime.DefaultErrorStrategy.reportError(DefaultErrorStrategy.java:136)

at io.debezium.ddl.parser.mysql.generated.MySqlParser.sqlStatements(MySqlParser.java:1154)

at io.debezium.ddl.parser.mysql.generated.MySqlParser.root(MySqlParser.java:882)

at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.parseTree(MySqlAntlrDdlParser.java:68)

at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.parseTree(MySqlAntlrDdlParser.java:41)

at io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:80)

at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:326)

at io.debezium.connector.mysql.BinlogReader.handleQueryEvent(BinlogReader.java:807)

at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:587)

... 5 more

Caused by: org.antlr.v4.runtime.NoViableAltException

at org.antlr.v4.runtime.atn.ParserATNSimulator.noViableAlt(ParserATNSimulator.java:2026)

at org.antlr.v4.runtime.atn.ParserATNSimulator.execATN(ParserATNSimulator.java:467)

at org.antlr.v4.runtime.atn.ParserATNSimulator.adaptivePredict(ParserATNSimulator.java:393)

at io.debezium.ddl.parser.mysql.generated.MySqlParser.sqlStatements(MySqlParser.java:956)

... 12 more

2021-04-12 18:15:03,574 INFO MySQL|mysql_real|binlog Error processing binlog event, and propagating to Kafka Connect so it stops this connector. Future binlog events read before connector is shutdown will be ignored. [io.debezium.connector.mysql.BinlogReader]

2021-04-12 18:15:03,939 INFO || WorkerSourceTask{id=shopper-connector-0} Committing offsets [org.apache.kafka.connect.runtime.WorkerSourceTask]

2021-04-12 18:15:03,939 INFO || WorkerSourceTask{id=shopper-connector-0} flushing 0 outstanding messages for offset commit [org.apache.kafka.connect.runtime.WorkerSourceTask]

2021-04-12 18:15:03,939 ERROR || WorkerSourceTask{id=shopper-connector-0} Task threw an uncaught and unrecoverable exception [org.apache.kafka.connect.runtime.WorkerTask]