How to add new tables to a running Debezium Oracle connector (v2.5)?

[mohd fasil]

Hi,

I'm using Debezium 2.5 with Oracle 11g. My connector is running with snapshot.mode=initial and capturing changes from one table.

I’ve added a new table (TORRENT.PSTA) to table.include.list and restarted the connector. New inserts are captured, but existing records aren’t.

I tried using the signal table to trigger a snapshot:

INSERT INTO DBZUSER.DEBEZIUM_SIGNAL (id, type, data)
VALUES (
  'reshot-torrent-psta',
  'execute-snapshot',
  '{"data-collections": ["TORRENT.PSTA"]}'
);

  The signal appears in Kafka, but the snapshot doesn’t run.

Questions:

• How can I trigger a snapshot for the new table?

Thanks!

  

[mohd fasil]

just to be more clear,

I have to include new tables for the existing running connector and i performed below steps:

I’ve added new table  to the connector’s table.include.list and attempted to trigger a snapshot using the ad hoc method described  Debezium Connector for Oracle :: Debezium Documentation  .

The signal was inserted into DBZUSER.DEBEZIUM_SIGNAL and appeared in signaling topic Kafka, but the snapshot didn’t execute.

The connector is running with snapshot.mode=initial.

 Let me know if any additional config is needed to enable snapshot execution for newly added tables.  

Thanks in advance.

Best regards,

Fasil

[Chris Cranford]

Hi Fasil -

From what I can see in what you've shared, it perhaps is because the table name you've shared in the signal isn't fully qualified. The format should be `<db>.<schema>.<table>`. If adding the database name does not solve your issue, could you please share the connector logs so that we can see what error or warnings are present after the Kafka signal is received, along with your complete connector configuration.

Thanks,
-cc

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/e2ff3b8c-9ef1-441d-aea0-5a369beb2895n%40googlegroups.com.

[mohd fasil]

Hi Chris,

Thanks for the response.

I am using a fully qualified name under connector configuration as you mentioned  <db>.<schema>.<table>.  and added the whole config below.

I got a warning message after sending a signal. with below format from Debezium Connector for Oracle :: Debezium Documentation

 warn message : 

Please assist me here with further steps.

Thanks,

To view this discussion visit https://groups.google.com/d/msgid/debezium/e0914335-a9cf-477a-9f74-5aa6b4b62a03%40gmail.com.

--

Best regards,

Mohammed Fasil.

[Chris Cranford]

Hi Fasil -

Could you please enable DEBUG logging for `io.debezium.pipeline.signal` and please share the full logs. It would appear there is likely a problem with the 'data' column value.

Thanks,
-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/CAJMtcOiEHzXJ_wiD1Zv8-F_P3NKCuCm1sars%2BB9UkvXL9oRY%2Bg%40mail.gmail.com.

[mohd fasil]

Hi Chris,

I sent 
INSERT INTO DBZUSER.DEBEZIUM_SIGNAL (ID, TYPE, DATA)
VALUES (
  'INCREMENTAL-SNAPSHOT-15',
  'EXECUTE-SNAPSHOT',
  '{"data-collections": ["TORRENT.PSTA"], "type": "incremental"}'
);
COMMIT;

and 
WARN
app:
connect-cluster-dev
server:
0
clusterId:
cfk-dev
level:
WARN
logger:
io.debezium.pipeline.signal.SignalProcessor
message:
Signal 'INCREMENTAL-SNAPSHOT-15' has been received but the type 'EXECUTE-SNAPSHOT' is not recognized
host:
connect-cluster-dev-0
@timestamp:
2025-09-12T04:20:49.601Z
thread:
debezium-oracleconnector-dev.ismf.wus3.v5-change-event-source-coordinator
@version:
1

current snapshot mode is initial do i need to change mode ? as I am using debezium 2.5 version.

Please let me know further steps/ full steps for adding new tables.

Best regards,
Fasil

[mohd fasil]

Hi Chris,

Update when i use data collection format as schema.table

INSERT INTO DBZUSER.DEBEZIUM_SIGNAL (ID, TYPE, DATA)
VALUES (

  'ad-hoc-snapshot-17',
  'execute-snapshot',
  '{

    "data-collections": ["TORRENT.PSTA"],
    "type": "incremental"
  }'
);
COMMIT;

I am getting warn like :

WARN

• app:connect-cluster-dev
• server:0
• clusterId:cfk-dev
• level:WARN

• logger:io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource
• message:Schema not found for table 'TORRENT.PSTA', known tables [ISMFTEST.TORRENT.PSTA, ISMFTEST.DBZUSER.DEBEZIUM_SIGNAL, ISMFTEST.TORRENT.LTCT]
• host:connect-cluster-dev-0
• @timestamp:2025-09-12T04:49:52.787Z
• thread:debezium-oracleconnector-dev.ismf.wus3.v5-change-event-source-coordinator
• @version:1
  
  
  

and when using data collection format as dbname.schema.table

INSERT INTO DBZUSER.DEBEZIUM_SIGNAL (ID, TYPE, DATA)
VALUES (

  'ad-hoc-snapshot-20',
  'execute-snapshot',
  '{
    "data-collections": ["ISMFTEST.TORRENT.PSTA"],
    "type": "incremental"
  }'
);
COMMIT;

level:
WARN
logger:
io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource
message:
Incremental snapshot for table 'ISMFTEST.TORRENT.PSTA' skipped cause the table has no primary keys
host:
connect-cluster-dev-0
@timestamp:
2025-09-12T05:14:58.011Z
thread:
debezium-oracleconnector-dev.ismf.wus3.v5-change-event-source-coordinator
@version:
1

[mohd fasil]

Hi Chris,

This is sorted now—I’ve realized the table doesn’t have a primary key.

Quick question: what’s the recommended approach for triggering an ad-hoc snapshot on tables that don’t have a primary key defined?


regards,
Fasil

[Chris Cranford]

Hi Fasil -

One option is to use `message.key.columns` where you define a custom key configuration for the table that is used by all facets of Debezium. In other words, when events are emitted or when we do any logic around table keys, the column mapping you define will be used. This is one way for a keyless table to be present in a Kafka topic with keys, allowing you to use compaction.

Another option is to use a surrogate key in the signal. This is a layered approach where Incremental Snapshots will perform deduplication based on the surrogate key column you set. This is useful for tables that are keyless, allowing you to perform incremental snapshots on them or as an alternative to composite keys if the composite key performs poorly. The only requirement here is that the surrogate key column must be unique across all rows.

If neither of those options satisfy your needs, then rather than incremental snapshots, you could fall back to blocking snapshots.

-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/9536a9dc-1c5d-42d7-a74f-e9d7ef1f71f8n%40googlegroups.com.

[mohd fasil]

Hi Chris,

Thanks for your response.

Here I have few more questions.

We have 180 tables, some with PK and some without. In prod, what’s the best practice for handling snapshots on large non-PK tables?

Can we manage all 180 tables under one connector, or is it better to split them for performance and isolation?

What advice would you give for tables that don’t have a primary key? Should we even include such tables in Debezium streaming, or is it better to avoid them?

And these are large streaming tables. 

Best regards,

Fasil.

To view this discussion visit https://groups.google.com/d/msgid/debezium/990a2c1b-0e5b-4640-b9e1-32d2fcd70f40%40gmail.com.

[Chris Cranford]

Hi Fasil

Nothing we've discussed prevents using a single connector for all tables. The `message.key.columns` configuration property supports multiple tables, allowing you to define a custom key for one or more tables that are keyed/keyless based on your needs.

Now there is one important aspect when we talk about topics that contain events that do or do not have keys, and that is compaction. In topics where you have keys, if a row is deleted, then not only is a delete event written to the topic, but a tombstone is also written to signal that any data for that key can be removed during compaction if eligible, preventing the data for that key from staying in the topic until the log segments are removed, which is 7 days by default. But with keyless tables, all events in the topic have no key, and therefore those topics either should have compaction disabled or you should make sure you disable tombstones in the source connector to avoid compaction from removing all eligible events since they are written with no keys.

Now if you use `message.key.columns` for all your keyless tables, they effectively are keyed from the perspective of Kafka, so compaction will work as expected.

So when you begin to look at how do you horizontally scale your environment, there are many factors that can play into that decision:

    - Isolation by tenant or business function
    - Simplification of common configuration behavior, avoiding predicates or transforms
   
With Oracle in particular, scaling to more than one connector per cluster has some implications, which I would recommend you read the LogMining Strategy section [1] in the documentation. For example, you should not use the `redo_log_catalog` mode with multiple connectors to avoid issues and you will need to use `online_catalog` or `hybrid` only. In addition, multiple connectors add additional load to your Oracle system, and if your Oracle database is already operating near max capacity or not well tuned, performance can suffer greatly with multiple connectors due to redo subsystem contention.

In practice, we always recommend starting with one connector and only horizontally scale as needed.

Hope that helps.
-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/CAJMtcOg9zxhWWEUUF-w1o%2BAqecwDo4f9daNG0b__vhvLRu1WeA%40mail.gmail.com.