Debezium Oracle LogMiner connector failing with Java heap OOM
Sri Rezeki
Hi
We are seeing a recurring production issue with our Debezium Oracle LogMiner connector reading from an on-prem Oracle database.
The connector runs for a period, then fails with the following pattern:
• java.lang.OutOfMemoryError: Java heap space
• LogMiner session stopped due to an error
• java.sql.SQLRecoverableException: IO Error: Socket read interrupted
• Producer failure
• The task then fails to shut down cleanly and becomes unresponsive
• Restarting the connector/task temporarily restores processing
This is impacting downstream data freshness.
We would like to understand whether this is a known Debezium Oracle LogMiner behaviour, especially in scenarios involving large transactions, high redo volume, LogMiner buffering, or Kafka producer/backpressure issues.
Current relevant debezium settings:
{
"connector.class": "io.debezium.connector.oracle.OracleConnector",
"snapshot.mode": "no_data",
"tasks.max": "1",
"log.mining.strategy": "online_catalog",
"log.mining.query.filter.mode": "in",
"log.mining.archive.log.only.mode": "false",
"log.mining.archive.log.hours": "72",
"archive.destination.name": "<sanitized-oracle-archive-destination>",
"lob.enabled": "true",
"max.iteration.transactions": "1000",
"include.schema.changes": "false",
"schema.history.internal.store.only.captured.tables.ddl": "true",
"decimal.handling.mode": "string",
"tombstones.on.delete": "false",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable": "false",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
"transforms": "Reroute",
"transforms.Reroute.type": "io.debezium.transforms.ByLogicalTableRouter",
"transforms.Reroute.key.enforce.uniqueness": "false",
"incremental.snapshot.chunk.size": "102400"
}
Could you please advise what Debezium-specific areas we should investigate or tune to reduce heap usage and avoid the connector becoming unresponsive?
Thanks
Sri
This message has been sent from Foodstuffs North Island Limited
The information contained in this email is intended only for the person or entity to which it is addressed. If you received this message in error, please contact the sender immediately by return email and delete this message and your reply. If you received this message in error you are prohibited from using any information in this email in any other way.
Chris Cranford
With the Oracle connector, an OutOfMemoryError is almost always a reflection of one or more of the following scenarios:
- A single large transaction with many rows
- A very large number of concurrent transactions (tens-of / hundreds-of thousands)
- A transaction with very large row payloads (large CLOB or BLOB column types)
Oracle writes uncommitted transactions to the transaction logs, so any CDC consumer has no idea when a transaction finishes until the stream emits a COMMIT or a ROLLBACK. Until that point in time, the Oracle connector must buffer those transactions, and the default is to use JVM memory for that.
Looking at your configuration, I do see that you have `lob.enabled` set to `true`, which would indicate at least the last point is potentially plausible. So your JVM memory for Kafka Connect needs to be able to sustain the large CLOB or BLOB payloads for the number of rows within a window of concurrent transactions on your captured tables or you may want to evaluate using one of the off-heap cache stores like Infinispan or Ehcache so that there isn't JVM heap pressure at play.
Without a heap dump and JMX metrics to identify which could be the true cause, this is the most general information we can supply at this time.
-cc
--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/SY0P300MB0742623DFB0204723D16F05AADEF2%40SY0P300MB0742.AUSP300.PROD.OUTLOOK.COM.