Any having experience replicating TB sized table in Oracle using Debezium?

[Shahul Nagoorkani]

The requirement for us is to perform an initial load of several TB sized tables in oracle. With the loads taking too much time to complete and the initial loads failing within few hours with ora-1555 snapshot too old error, the idea is to use schema_only refresh and do an incremental data load by issuing a signal into the source signal table/kafka topic. For some reason, both the Source Signal/Signal Kafka topic methods are not working as expected and not triggering the initial loads as expected?

Does anyone face similar issues?

[Chris Cranford]

Hi -

Can you please share both your connector configuration and the payload for the signal you're writing to the database table or the kafka signal topic?

Thanks,
-cc

On 9/11/25 7:20 PM, Shahul Nagoorkani wrote:

The requirement for us is to perform an initial load of several TB sized tables in oracle. With the loads taking too much time to complete and the initial loads failing within few hours with ora-1555 snapshot too old error, the idea is to use schema_only refresh and do an incremental data load by issuing a signal into the source signal table/kafka topic. For some reason, both the Source Signal/Signal Kafka topic methods are not working as expected and not triggering the initial loads as expected?

Does anyone face similar issues?

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/fcbdf9b4-1bfa-4bed-958d-590584a38d8dn%40googlegroups.com.

[Shahul Nagoorkani]

Hello Chris,

Currently I tried both source signal as well as the Kafka signal topic. None of them worked. Enabled the trace logs as well and did not see any useful information in the trace logs.

Here is the format of the message that i posted into the kafka signal topic with the key as the "topic.prefix".

{
"id": "Message8",
"type": "execute-snapshot",
"data": {
"data-collections": [
"<dbname>.<schemaname>.<tablename>"
],
"type": "INCREMENTAL"
}
}

The minute I publish the message, the message gets processed and show up in the logs but doesn't trigger any snapshot though.

{"stream":"stdout","timestamp":1757630724405,"log":{"method":"arrived","@timestamp":"2025-09-11T22:45:24.404Z","logger_name":"io.debezium.pipeline.signal.actions.snapshotting.ExecuteSnapshot","source_host":"kafka-connect-dba-ebiz-connect-0","line_number":"64","message":"Requested 'INCREMENTAL' snapshot of data collections '[schema.tablename]' with additional conditions '[]' and surrogate key 'PK of table will be used'","class":"io.debezium.pipeline.signal.actions.snapshotting.ExecuteSnapshot","file":"ExecuteSnapshot.java","@version":1.0,"level":"INFO","thread_name":"debezium-oracleconnector-ebiz_LT_wallet-SignalProcessor","mdc":{"dbz.connectorContext":"streaming","connector.context":"[debezium-connector-ebiz-walletload|task-0] ","dbz.connectorName":"ebiz_LT_connector","dbz.databaseName":"dbname","dbz.connectorType":"Oracle","dbz.taskId":"0"}}}

{"stream":"stdout","timestamp":1757630724406,"log":{"method":"addDataCollectionNamesToSnapshot","@timestamp":"2025-09-11T22:45:24.406Z","logger_name":"io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotContext","source_host":"kafka-connect-dba-ebiz-connect-0","line_number":"206","message":"Adding data collections names [schemaname.tablename] to snapshot","class":"io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotContext","file":"AbstractIncrementalSnapshotContext.java","@version":1.0,"level":"TRACE","thread_name":"debezium-oracleconnector-ebiz_LT_schemaname-SignalProcessor","mdc":{"dbz.connectorContext":"streaming","connector.context":"[debezium-connector-ebiz-schemanameload|task-0] ","dbz.connectorName":"ebiz_LT_schemaname","dbz.databaseName":"dbname","dbz.connectorType":"Oracle","dbz.taskId":"0"}}}

{"stream":"stdout","timestamp":1757630724406,"log":{"method":"addDataCollectionNamesToSnapshot","@timestamp":"2025-09-11T22:45:24.406Z","logger_name":"io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource","source_host":"kafka-connect-dba-ebiz-connect-0","line_number":"472","message":"Expanded data collections [schemaname.tablename]","class":"io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource","file":"AbstractIncrementalSnapshotChangeEventSource.java","@version":1.0,"level":"TRACE","thread_name":"debezium-oracleconnector-ebiz_LT_schemaname-SignalProcessor","mdc":{"dbz.connectorContext":"streaming","connector.context":"[debezium-connector-ebiz-schemanameload|task-0] ","dbz.connectorName":"ebiz_LT_schemaname","dbz.databaseName":"dbname","dbz.connectorType":"Oracle","dbz.taskId":"0"}}}

{"stream":"stdout","timestamp":1757630724406,"log":{"method":"addDataCollectionNamesToSnapshot","@timestamp":"2025-09-11T22:45:24.406Z","logger_name":"io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource","source_host":"kafka-connect-dba-ebiz-connect-0","line_number":"471","message":"Configured data collections [schemaname.tablename]","class":"io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource","file":"AbstractIncrementalSnapshotChangeEventSource.java","@version":1.0,"level":"TRACE","thread_name":"debezium-oracleconnector-ebiz_LT_schemaname-SignalProcessor","mdc":{"dbz.connectorContext":"streaming","connector.context":"[debezium-connector-ebiz-schemanameload|task-0] ","dbz.connectorName":"ebiz_LT_schemaname","dbz.databaseName":"dbname","dbz.connectorType":"Oracle","dbz.taskId":"0"}}}

Tried inserting into the Source table as well and no luck.

INSERT INTO debezium.debezium_wal_load_signal (id, type, data) VALUES ('snap-2025-09-10-03','execute-snapshot','{"type":"INCREMENTAL","data-collections":["schema.tablename"]}');

Here is the config.

snapshot.mode: no_data

incremental.snapshot.enabled: true

snapshot.max.threads: 4

# Incremental snapshot tuning

incremental.snapshot.chunk.size: 30000

snapshot.fetch.size: 60000

query.fetch.size: 60000

#Debezium Engine/Queing Settings

max.batch.size: 20000

max.queue.size: 200000

max.queue.size.in.bytes: 536870912 #512MB

#Oracle Driver Tuning

driver.defaultRowPrefetch: 60000

driver.oracle.net.SDU: 65535

database.hostname: <servername>

database.port: 1522

database.user: username

database.password: password

database.dbname: dbname

schema.include.list: schemaname

table.include.list: <tablename>

column.exclude.list: <column exclusionlist>

topic.prefix: ebiz_LT_wallet

request.timeout.ms: 60000

connections.max.idle.ms: 60000

topic.creation.enable : true

topic.creation.default.replication.factor: -1

topic.creation.default.partitions: -1

topic.creation.default.retention.ms: 1209600000

schema.history.internal.kafka.topic: ebiz_LT_wallet.schema_history

schema.history.internal.kafka.bootstrap.servers: XXXXXX

schema.history.internal.consumer.security.protocol: SASL_SSL

schema.history.internal.consumer.sasl.mechanism: SCRAM-SHA-512

schema.history.internal.consumer.sasl.jaas.config: 'org.apache.kafka.common.security.scram.ScramLoginModule required username=usen password=pwd'

schema.history.internal.producer.security.protocol: SASL_SSL

schema.history.internal.producer.sasl.mechanism: SCRAM-SHA-512

schema.history.internal.producer.sasl.jaas.config: 'org.apache.kafka.common.security.scram.ScramLoginModule required username=usen password=pwd'

# Signal configuration for incremental snapshots

signal.kafka.bootstrap.servers: <servernames>

signal.kafka.topic: ebiz_wallet_LT_signal

signal.enabled.channels: source,kafka

signal.data.collection: <dbname>.debezium.debezium_wal_load_signal

signal.consumer.security.protocol: SASL_SSL

signal.consumer.sasl.mechanism: SCRAM-SHA-512

signal.consumer.sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username=usen password=bbbb;

signal.producer.security.protocol: SASL_SSL

signal.producer.sasl.mechanism: SCRAM-SHA-512

signal.producer.sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username=usen password=bbb;

notification.enabled.channels: sink

notification.sink.topic.name: ebiz_wallet_LT_notification

#heartbeat parameters

heartbeat.interval.ms: 30000

heartbeat.topics.prefix: walletload_heartbeats

heartbeat.topic.partitions: 1

heartbeat.topic.replication.factor: -1

heartbeat.action.query: UPDATE CDC_HEARTBEAT_WAL_LT SET last_update=SYSTIMESTAMP WHERE id=1

Regards,

Shahul

[Chris Cranford]

Hi,

Can you share the full trace log where you sent the signal. I'd like to check a few more things in the log, if possible.

Thanks,
-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/8c3342b5-1a28-40bd-ade4-088aeb7fb9aan%40googlegroups.com.

[Shahul Nagoorkani]

Our logs gets uploaded to sumo logic  and the logs that i shared earlier are the ones that i got from sumo. Let me try downloading the logs around that time when we issued the signal. But for sure there is no useful information with the trace logs.

Regards,

Shahul