EventDataDeserializationException - Debezium 3.1.2

[Shrey Ratna]

Hi, I have been getting the error below ever since my RDS autoscaled on storage and connectors auto-restarted.
Now I keep getting every single time I tried recovery mode but they throw the same error, even when i go initial snapshot, it fails at the end with same error and keeps running indefinitely.
RDS Version: MySQL RDS 8.0.35
Yes I tried changing net.read/write timeout, it didn't change a thing.

```
2025-07-04 05:38:48,253 ERROR  MySQL|eventila_new|binlog  Error during binlog processing. Last offset stored = {ts_sec=1751607528, file=mysql-bin-changelog.452526, pos=20145300, gtids=12bed93b-15ab-11ef-ae65-0affccd81c43:1-435756524, server_id=490987222, event=1}, binlog reader near position = mysql-bin-changelog.452526/20145379   [io.debezium.connector.binlog.BinlogStreamingChangeEventSource]
Jul 04, 2025 5:38:48 AM com.github.shyiko.mysql.binlog.BinaryLogClient$5 run
INFO: threadExecutor is shut down, terminating keepalive thread
2025-07-04 05:38:48,254 ERROR  MySQL|eventila_new|binlog  Producer failure   [io.debezium.pipeline.ErrorHandler]
io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4{timestamp=1751607390000, eventType=TABLE_MAP, serverId=490987222, headerLength=19, dataLength=297, nextPosition=20145695, flags=0}
at io.debezium.connector.binlog.BinlogStreamingChangeEventSource.wrap(BinlogStreamingChangeEventSource.java:1206)
at io.debezium.connector.binlog.BinlogStreamingChangeEventSource$ReaderThreadLifecycleListener.onCommunicationFailure(BinlogStreamingChangeEventSource.java:1252)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1110)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:657)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:959)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1751607390000, eventType=TABLE_MAP, serverId=490987222, headerLength=19, dataLength=297, nextPosition=20145695, flags=0}
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:343)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeTableMapEventData(EventDeserializer.java:315)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:239)
at io.debezium.connector.binlog.BinlogStreamingChangeEventSource$1.nextEvent(BinlogStreamingChangeEventSource.java:426)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1082)
... 3 more
Caused by: java.io.EOFException: Failed to read remaining 2 of 2 bytes from position 170. Block length: 0. Initial block length: 4.
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.fill(ByteArrayInputStream.java:115)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:105)
at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventMetadataDeserializer.readBooleanList(TableMapEventMetadataDeserializer.java:147)
at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventMetadataDeserializer.deserialize(TableMapEventMetadataDeserializer.java:129)
at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:49)
at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:29)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:337)
... 7 more
2025-07-04 05:38:48,254 INFO   MySQL|eventila_new|binlog  Stopped reading binlog after 136 events, last recorded offset: {ts_sec=1751607528, file=mysql-bin-changelog.452526, pos=20145300, gtids=12bed93b-15ab-11ef-ae65-0affccd81c43:1-435756524, server_id=490987222, event=1}   [io.debezium.connector.binlog.BinlogStreamingChangeEventSource]
2025-07-04 05:38:48,611 INFO   ||  WorkerSourceTask{id=streaming-logs-connector-0} Committing offsets for 1843 acknowledged messages   [org.apache.kafka.connect.runtime.WorkerSourceTask]
2025-07-04 05:38:48,614 ERROR  ||  WorkerSourceTask{id=streaming-logs-connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted   [org.apache.kafka.connect.runtime.WorkerTask]
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:67)
at io.debezium.connector.binlog.BinlogStreamingChangeEventSource$ReaderThreadLifecycleListener.onCommunicationFailure(BinlogStreamingChangeEventSource.java:1252)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1110)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:657)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:959)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4{timestamp=1751607390000, eventType=TABLE_MAP, serverId=490987222, headerLength=19, dataLength=297, nextPosition=20145695, flags=0}
at io.debezium.connector.binlog.BinlogStreamingChangeEventSource.wrap(BinlogStreamingChangeEventSource.java:1206)
... 5 more
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1751607390000, eventType=TABLE_MAP, serverId=490987222, headerLength=19, dataLength=297, nextPosition=20145695, flags=0}
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:343)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeTableMapEventData(EventDeserializer.java:315)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:239)
at io.debezium.connector.binlog.BinlogStreamingChangeEventSource$1.nextEvent(BinlogStreamingChangeEventSource.java:426)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1082)
... 3 more
Caused by: java.io.EOFException: Failed to read remaining 2 of 2 bytes from position 170. Block length: 0. Initial block length: 4.
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.fill(ByteArrayInputStream.java:115)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:105)
at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventMetadataDeserializer.readBooleanList(TableMapEventMetadataDeserializer.java:147)
at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventMetadataDeserializer.deserialize(TableMapEventMetadataDeserializer.java:129)
at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:49)
at com.github.shyiko.mysql.binlog.event.deserialization.TableMapEventDataDeserializer.deserialize(TableMapEventDataDeserializer.java:29)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:337)
... 7 more
2025-07-04 05:38:48,614 INFO   ||  Stopping down connector   [io.debezium.connector.common.BaseSourceTask]
2025-07-04 05:38:48,640 INFO   MySQL|eventila_new|streaming  Finished streaming   [io.debezium.pipeline.ChangeEventSourceCoordinator]
2025-07-04 05:38:48,641 INFO   ||  SignalProcessor stopped   [io.debezium.pipeline.signal.SignalProcessor]
2025-07-04 05:38:48,641 INFO   ||  Debezium ServiceRegistry stopped.   [io.debezium.service.DefaultServiceRegistry]
2025-07-04 05:38:48,643 INFO   ||  Connection gracefully closed   [io.debezium.jdbc.JdbcConnection]
2025-07-04 05:38:48,644 INFO   ||  Connection gracefully closed   [io.debezium.jdbc.JdbcConnection]
2025-07-04 05:38:48,644 INFO   ||  [Producer clientId=eventila_new-schemahistory] Closing the Kafka producer with timeoutMillis = 30000 ms.   [org.apache.kafka.clients.producer.KafkaProducer]
2025-07-04 05:38:48,646 INFO   ||  Metrics scheduler closed   [org.apache.kafka.common.metrics.Metrics]
2025-07-04 05:38:48,646 INFO   ||  Closing reporter org.apache.kafka.common.metrics.JmxReporter   [org.apache.kafka.common.metrics.Metrics]
2025-07-04 05:38:48,646 INFO   ||  Closing reporter org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter   [org.apache.kafka.common.metrics.Metrics]
2025-07-04 05:38:48,646 INFO   ||  Metrics reporters closed   [org.apache.kafka.common.metrics.Metrics]
2025-07-04 05:38:48,646 INFO   ||  App info kafka.producer for eventila_new-schemahistory unregistered   [org.apache.kafka.common.utils.AppInfoParser]
2025-07-04 05:38:48,646 INFO   ||  [Producer clientId=connector-producer-streaming-logs-connector-0] Closing the Kafka producer with timeoutMillis = 30000 ms.   [org.apache.kafka.clients.producer.KafkaProducer]
2025-07-04 05:38:48,648 INFO   ||  Metrics scheduler closed   [org.apache.kafka.common.metrics.Metrics]
2025-07-04 05:38:48,649 INFO   ||  Closing reporter org.apache.kafka.common.metrics.JmxReporter   [org.apache.kafka.common.metrics.Metrics]
2025-07-04 05:38:48,649 INFO   ||  Closing reporter org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter   [org.apache.kafka.common.metrics.Metrics]
2025-07-04 05:38:48,649 INFO   ||  Metrics reporters closed   [org.apache.kafka.common.metrics.Metrics]
2025-07-04 05:38:48,649 INFO   ||  App info kafka.producer for connector-producer-streaming-logs-connector-0 unregistered   [org.apache.kafka.common.utils.AppInfoParser]
2025-07-04 05:38:48,649 INFO   ||  App info kafka.admin.client for connector-adminclient-streaming-logs-connector-0 unregistered   [org.apache.kafka.common.utils.AppInfoParser]
2025-07-04 05:38:48,650 INFO   ||  Metrics scheduler closed   [org.apache.kafka.common.metrics.Metrics]
2025-07-04 05:38:48,650 INFO   ||  Closing reporter org.apache.kafka.common.metrics.JmxReporter   [org.apache.kafka.common.metrics.Metrics]
2025-07-04 05:38:48,650 INFO   ||  Metrics reporters closed   [org.apache.kafka.common.metrics.Metrics]
``` 

[Chris Cranford]

Hi  -

Can you dump the binlog and see the details, I am curious if when the auto-scale happened if that left the binlogs in an inconsistent/unexpected state.

-cc

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/ee8c5f7d-95bf-4074-946c-01fe409265a6n%40googlegroups.com.

[Shrey Ratna]

This issue was because of default charset and collation:
Query:
```ALTER TABLE multi_sku_assign ADD COLUMN processing_flow ENUM('CRON', 'EVENT') NOT NULL DEFAULT 'CRON';``` -> Broke binary logs, didn't really get the time to dig into binlogs

Fixed query:
```ALTER TABLE multi_sku_assign ADD COLUMN processing_flow ENUM('CRON', 'EVENT') NOT NULL DEFAULT 'CRON' CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;```