Debezium does not include column made by HASH in schema

[Konrad Sokun]

Hi all,

I am using Debezium MySQL connector 2.6.1.Final to read a MySQL database table, but currently Debezium cannot read the binlog as it encounters an error that internal schema size does not match row size. The table being read is generating a column made by a a HASH which is not in the original schema but is included in event updates in the binling, which leads to the mismatch in row and schema size. I have attempted to restart the connector in various snapshot modes but had no success.

If there is a configuration that enables HASH columns to be included in the schema please let me know. 

Thank you in advance.

[Chris Cranford]

Hi Konrad

I'm afraid that isn't something the connector supports.  I'd suggest raising a Jira enhancement [1] request for that if this is something needed.

Thanks,
Chris

[1]: https://issues.redhat.com/projects/DBZ

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/ada1f4d6-d07f-4f47-8634-1ad5163ab08bn%40googlegroups.com.

[Konrad Sokun]

Hi Chris,

Thank you for letting me know, I've made a request like you suggested. Is there any way to get around this issue in the mean time, I've tried to use inconsistent.schema.handling to warn but that doesn't seem to work, perhaps I am misunderstanding its function? This HASH is not needed for my project, so if there is a way to ignore this column in the bin logs I'd love to know. Would modifying the schema in the schema registry manually resolve this issue?

Thanks in advance,

Konrad

[Chris Cranford]

Hi,

The inconsistent schema handling is specifically for when we detect a DML event for a table the connector does not yet know about. If we know about the table already but the schema mismatches between the event and the in-memory relational model, then this setting has no affect.  You could certainly try setting a column.exclude.list for this hash column and see if that may work; however, I wouldn't be surprised if you still encounter problems.

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/e02beb55-5421-455b-b42e-49156c5585cdn%40googlegroups.com.