Max.request.size Error from Table Not in Whitelist

[Brad M]

Hi All

Not sure if this is a bug that I should log.

I am working on a POC on a fairly busy MariaDB database with many tables.

I am only interested in changes to a small number of these tables so I am using the table.whitelist configuration property of the Mysql Connector to specify regex for the tables to get changes for. This has been working well until the connector stopped with the following exception:

The message is 24735841 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.

The problem is that when I use the mysqlbinlog tool to inspect the record that caused the error, it is for a (legitimate) insert for a table that is not included by the table.whitelist regex. I am purposly not whitelisting tables like this that have large changes that are not suitable for CDC.

Any guideance would be greatly appreciated. Is there perhaps another whitelist configuration that I need to use in addition to table.whitelist  or does this look like a bug?

Kind Regards

Brad

[Brad M]

I have managed to resolve this error in my environment.

In case anyone else runs into this problem again in the future, here is what I did:

I noticed from the stack trace that the error was orginating from the kafka library when sending messages for DDL changes.

Looking at the implementation of MySqlSchema::applyDdl()  (https://github.com/debezium/debezium/blob/1.2/debezium-connector-mysql/src/main/java/io/debezium/connector/mysql/MySqlSchema.java line 365), the table filters from the whitelist are intentionally ignored if `storeOnlyMonitoredTablesDdl` is false.

So I changed the config for the connector to have the 'database.history.store.only.monitored.tables.ddl' property set to true.  Now the binlog entries for the non-whitelist tables with large changes that are no longer causing problems