Oracle Snapshot question

22 views
Skip to first unread message

Donald Lyon

unread,
May 7, 2021, 9:27:11 AMMay 7
to debezium
I'm having an issue where the snapshot process is looking at all of the tables in my schema rather than just tables in my tables.include.list.  This has the side effect of erroring on a table that's not supported, and also requiring supplemental logging on every table in the schema.  Is there a way to limit the schema tables to only the one(s) I want monitored?

Version: 1.5.0.Final
Running EmbeddedEngine
Config:
return io.debezium.config.Configuration.create()
.with("name", "my-oracle-connector")
.with("connector.class", "io.debezium.connector.oracle.OracleConnector")
.with("offset.storage", "org.apache.kafka.connect.storage.FileOffsetBackingStore")
.with("offset.storage.file.filename", "/Users/don/debezium/my-offset.dat")
.with("offset.flush.interval.ms", 60000)
.with("database.server.name", DBHost+"-"+DBName)
.with("database.hostname", DBHost)
.with("database.port", DBPort)
.with("database.user", DBUserName)
.with("database.password", DBPassword)
.with("database.dbname", DBName)
.with("database.history", "io.debezium.relational.history.FileDatabaseHistory")
.with("database.history.file.filename", "/Users/don/debezium/dbhistory.dat")
.with("snapshot.mode", "schema_only")
.with("schema.include.list", "MYSCHEMA")
.with("schema.exclude.list", "MYSCHEMA.BAD_ORACLEAQ_TABLE_Q")
.with("table.include.list", "MYSCHEMA.TABLE1")
.build();

Donald Lyon

unread,
May 10, 2021, 5:15:25 PMMay 10
to debezium

Specifically, in LogMinerHelper.checkSupplementalLogging():402.  Is there a way to limit the check for supplemental logging to only be tables that are in the tables.include.list?

Donald Lyon

unread,
May 11, 2021, 2:13:18 AMMay 11
to debezium
Some more information.  It appears that the combination of schema.include.list and schema.exclude.list was causing the schema tables to include the additional tables.  When I comment out the schema.excludes.list, it gets to LogMinerHelper.checkSupplementalLogging() with the single table I specified.  However, I wasn't able to complete the initial snapshotting without that setting since BAD_ORACLEAQ_TABLE_Q was not supported.  Is there a different way to exclude tables from being included in the schema structure snapshot?

I'm also now getting an issue in the supplemental logging check.  It fails because tableId.table() is lower case and the SQL isn't case insensitive.  I'm not sure if database version plays a role in that, but I'm using Oracle 11.2.0.4.

Chris Cranford

unread,
May 11, 2021, 8:52:45 AMMay 11
to debe...@googlegroups.com, Donald Lyon
Hi Donald,

Regarding the case, for LogMiner you need to specify "database.tablename.case.insensitive=false" in your connector configuration.  That should resolve the problem with the tableId.table() case comparison failure you're experiencing. 

With respect to include/exclude list configuration, those settings regardless of connector are mutually exclusive.  In your connector configuration below, you can't combine schema.include.list with schema.exclude.list as that can lead to unexpected behavior and you've encountered.  The documentation for these options indicate this mutual exclusivity.  If you're only interested in a single table in your database, specifying just the "table.include.list" with that table or the small list of monitored tables is more than sufficient for the connector to only check those tables for logging configurations.

HTH,
Chris
--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/da467e04-04e2-46ac-9b83-114244add03cn%40googlegroups.com.

Reply all
Reply to author
Forward
0 new messages