Oracle LogMiner limitations

[Alexander Kashintsev]

Hi!

This document (page 9)

https://www.oracle.com/a/ocom/docs/techpaper-goldengate-advantage-july2021.pdf

states that Oracle LogMiner has some limitations. For example:

-  Tables larger than 100 GB can't be backfilled.

-  Events have a size limitation of 3 MB.

Can anyone confirm or deny this information?

[Chris Cranford]

Hi Alexander -

So LogMiner does have its own set of limitations, but GoldenGate has similar ones as well.  There are some hurdles that GoldenGate can overcome that standalone LogMiner may not because Oracle owns the proprietary formats and may choose to not expose specific detail in a consumable way through other methods.  Furthermore, this document prefaces these limitations as "this is what one CDC-vendor found were limitations", but I would argue that's solely from that vendor's perspective and isn't entirely accurate nor should be taken at face value.

I am not aware of any scenario where the table size and its ability to be backfilled has been a problem.  We do have users using the CDC implementation on Oracle RAC in large installations, generating 100k+ changes per minute and it would seem reasonable they would have tables at or near such capacity and they haven't reported any issues that I've seen.

As for event sizes, I guess there could technically be an internal limitation to Oracle LogMiner based on numbers of columns in a table paired with its data.  But Oracle LogMiner already supports a continuation signal in its events, which signals to the consumer that the event is materialized as a series of multiple rows.  Debezium has officially supported this continuation mechanism since LogMiner support was added, and again I haven't been made aware of any SQL for which we've had failures with that had any type of size limitation.

Looking at the other points in the list, I must say context is really key.

For example, the point about CDB/PDB multi-tenant architecture is speaking from the perspective that the Oracle LogMiner view does not impose any strict visibility rules on what a user can see.  In other words, the logs added to the LogMiner session are read and all entries are loaded into this view.  This means someone with access to LogMiner can see changes in another tenants, but that's actually not really a bad thing when you look at this from an overall architecture perspective in terms of the CPU/IO costs to read/load the logs.  You do this once, disseminate the information, and then apply permissive rules on the exposed data.

Furthermore, the one about "events for tables without a primary key" is true, but only if you do not enable full supplemental logging for the table.  If you enable full supplemental logging, as Debezium's installation guide suggests, you get all columns and you can then use Debezium's automatic unique-index to primary key lifting as the alternative for key-less table's event keys or map your own using "message.key.columns". 

Lastly, and one that I find completely false is the notion of CLOB, BLOB, NCLOB, and XMLTYPE being unsupported.  Oracle LogMiner does indeed support these column types, but it requires extra effort to mine these values from LogMiner.  We introduced LOB support in Debezium 1.6 as an incubating feature and we continue to improve and refine this support.  We intend to expose LOB support to cover XMLTYPE in the not to distant future.

So if there are any specific points you are concerned about, let me know and I'll gladly help define where those limitations are or are not in terms of Debezium.

Thanks,
Chris

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/7406cffc-8e49-44e5-ba81-dec77219a7e1n%40googlegroups.com.