Re: Issue with Incremental Snapshot Not Being Triggered via Kafka Signal Topic in Debezium MySQL Connector 2.5.4

[jiri.p...@gmail.com]

Hi,

you need the source channel enable for non-read only incremental snapshots. See note in https://debezium.io/documentation/reference/3.0/configuration/signalling.html#debezium-signaling-enabling-kafka-signaling-channel

Jiri

On Saturday, March 15, 2025 at 12:29:06 PM UTC+1 suleiman...@1win.pro wrote:

I guess, this isse  maybe linked with source which I try to incremental snapshot.
My source is 10.11.10-MariaDB-log.
I see what debezium has status Technology Preview in feature  incremental snapshot for MariaDB due to documentation https://debezium.io/documentation//reference/stable/configuration/signalling.html#debezium-signaling-ad-hoc-incremental-snapshots

суббота, 15 марта 2025 г. в 13:24:47 UTC+3, Suleiman Bagirov:

Hello!

I am experiencing an issue with triggering an incremental snapshot using the Debezium MySQL Connector (version 2.5.4) with kafka signal channel. My connector is configured with the following parameters

{ ... "topic.prefix": "cdc",
 "snapshot.mode": "schema_only",
 "snapshot.chunk.size": "1000",
 "snapshot.fetch.size": "1000",
 "snapshot.max.threads": "1",
 "snapshot.lock.timeout.ms": "10000",
 "incremental.snapshot.chunk.size": "1000", "incremental.snapshot.allow.schema.changes": "true",
 "max.queue.size": "8192",
 "poll.interval.ms": "2000",
 "signal.enabled.channels": "kafka",
 "signal.kafka.topic": "connector-signal",
 "signal.kafka.bootstrap.servers": "b-3.bootstrap.eu-central-1.amazonaws.com:9098,b-2.bootstrap.eu-central-1.amazonaws.com:9098,b-1.bootstrap.eu-central-1.amazonaws.com:9098",
 "signal.consumer.security.protocol":"SASL_SSL",
 "signal.consumer.sasl.mechanism": "AWS_MSK_IAM", "signal.consumer.sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;",
 "signal.consumer.sasl.client.callback.handler.class": "software.amazon.msk.auth.iam.IAMClientCallbackHandler" }

I am sending a signal message to the topic connector-signal with the following command:

echo 'cdc|{"type":"execute-snapshot","data":{"data-collections":["db.table"],"type":"INCREMENTAL"}}' | \
./kafka-console-producer.sh --bootstrap-server b-1.bootstrap.eu-central-1.amazonaws.com:9098,b-2.bootstrap.eu-central-1.amazonaws.com:9098,b-3.bootstrap.eu-central-1.amazonaws.com:9098 \
--topic connector-signal \
--property "parse.key=true" --property "key.separator=|"

However, I receive the following log:

WARN [connector-name|task-0] Action execute-snapshot failed. The signal SignalRecord{id='null', type='execute-snapshot', data='{ "data-collections" : [ "db.table" ], "type" : "INCREMENTAL" }', additionalData={channelOffset=0}} may not have been processed. (io.debezium.pipeline.signal.SignalProcessor:202) io.debezium.DebeziumException: Incremental snapshot is not properly configured, either sinalling data collection is not provided or connector-specific snapshotting not set

[Chris Cranford]

Hi -

If you do not have GTIDs enabled (requirement for read-only incremental snapshots), then incremental snapshots will require the `signal.data.collection` to be specified in the configuration. This is used by non-read-only incremental snapshots to track watermark points in the binlog for deduplication.

Hope that helps.
-cc

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/7e0be1fa-f9b7-4bc1-942b-a982c412fbfbn%40googlegroups.com.