Debezium Oracle LogMiner connector failing with Java heap OOM
Sri Rezeki
Hi
We are seeing a recurring production issue with our Debezium Oracle LogMiner connector reading from an on-prem Oracle database.
The connector runs for a period, then fails with the following pattern:
• java.lang.OutOfMemoryError: Java heap space
• LogMiner session stopped due to an error
• java.sql.SQLRecoverableException: IO Error: Socket read interrupted
• Producer failure
• The task then fails to shut down cleanly and becomes unresponsive
• Restarting the connector/task temporarily restores processing
This is impacting downstream data freshness.
We would like to understand whether this is a known Debezium Oracle LogMiner behaviour, especially in scenarios involving large transactions, high redo volume, LogMiner buffering, or Kafka producer/backpressure issues.
Current relevant debezium settings:
{
"connector.class": "io.debezium.connector.oracle.OracleConnector",
"snapshot.mode": "no_data",
"tasks.max": "1",
"log.mining.strategy": "online_catalog",
"log.mining.query.filter.mode": "in",
"log.mining.archive.log.only.mode": "false",
"log.mining.archive.log.hours": "72",
"archive.destination.name": "<sanitized-oracle-archive-destination>",
"lob.enabled": "true",
"max.iteration.transactions": "1000",
"include.schema.changes": "false",
"schema.history.internal.store.only.captured.tables.ddl": "true",
"decimal.handling.mode": "string",
"tombstones.on.delete": "false",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable": "false",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
"transforms": "Reroute",
"transforms.Reroute.type": "io.debezium.transforms.ByLogicalTableRouter",
"transforms.Reroute.key.enforce.uniqueness": "false",
"incremental.snapshot.chunk.size": "102400"
}
Could you please advise what Debezium-specific areas we should investigate or tune to reduce heap usage and avoid the connector becoming unresponsive?
Thanks
Sri
This message has been sent from Foodstuffs North Island Limited
The information contained in this email is intended only for the person or entity to which it is addressed. If you received this message in error, please contact the sender immediately by return email and delete this message and your reply. If you received this message in error you are prohibited from using any information in this email in any other way.
Chris Cranford
With the Oracle connector, an OutOfMemoryError is almost always a reflection of one or more of the following scenarios:
- A single large transaction with many rows
- A very large number of concurrent transactions (tens-of / hundreds-of thousands)
- A transaction with very large row payloads (large CLOB or BLOB column types)
Oracle writes uncommitted transactions to the transaction logs, so any CDC consumer has no idea when a transaction finishes until the stream emits a COMMIT or a ROLLBACK. Until that point in time, the Oracle connector must buffer those transactions, and the default is to use JVM memory for that.
Looking at your configuration, I do see that you have `lob.enabled` set to `true`, which would indicate at least the last point is potentially plausible. So your JVM memory for Kafka Connect needs to be able to sustain the large CLOB or BLOB payloads for the number of rows within a window of concurrent transactions on your captured tables or you may want to evaluate using one of the off-heap cache stores like Infinispan or Ehcache so that there isn't JVM heap pressure at play.
Without a heap dump and JMX metrics to identify which could be the true cause, this is the most general information we can supply at this time.
-cc
--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/SY0P300MB0742623DFB0204723D16F05AADEF2%40SY0P300MB0742.AUSP300.PROD.OUTLOOK.COM.
Sri Rezeki
Thank you for the explanation.
One additional detail we have identified: an Oracle supplementary purge activity occurred on Saturday, 20 June at around 5:00pm NZT, shortly before the connector started failing repeatedly as I explained in the previous email.
We are currently running Debezium 3.4.3.Final. We had previously upgraded from 3.4.0 following guidance from an earlier Debezium thread regarding a LogMiner-related regression.
Given your comments around Debezium buffering Oracle transactions in JVM memory until COMMIT or ROLLBACK, we would like your guidance on the recommended Debezium-side solution or mitigation for this scenario.
Could this type of Oracle supplementary/log-related purge activity cause the LogMiner connector to process or retain a much larger redo/log window, resulting in heap pressure even on 3.4.3.Final?
Since we are already on 3.4.3.Final, could you please advise what the recommended Debezium-side fix, configuration change, or architecture change would be to prevent the connector from becoming unresponsive after heap pressure?
We are looking for the best-practice Debezium recommendation for handling this scenario.
Thanks
Sri
Chris Cranford
Before I answer, can you clarify exactly what you mean by "supplementary purge activity"? Did you delete data, did you remove old archive logs, or something else? Without knowing exactly what you mean, it's not feasible to give you a precise answer, I'm afraid.
-cc
To view this discussion visit https://groups.google.com/d/msgid/debezium/5867faec-e893-4f25-9aca-0075e668668cn%40googlegroups.com.
Sri Rezeki
Thanks, and apologies I should have clarified what we meant by “supplementary purge” earlier. We have now confirmed the details.
The primary purge is the standard application purge process. It purges data based on the normal/load-based retention rules.
The supplementary purge is an additional application-level purge process. It cleans up data that is not covered by the primary purge. It affects more tables and broader/longer date ranges than the primary purge.
To clarify, the supplementary purge was not a purge or removal of Oracle archive logs or redo logs. It was an application-level data purge in the Oracle source database.
More specifically:
- It deleted application/business/audit data from a wider set of Oracle tables.
- It included tables that are captured by Debezium.
- It covered longer date ranges than the primary purge.
- It generated a large number of database deletes, transactions, and commits.
- It generated Oracle redo/archive logs as a side effect of the data changes, but it did not delete or remove Oracle log files.
We were advised that the supplementary purge on 20 June generated approximately 50 GB of Oracle logs.
For comparison, a later primary purge on 27 June generated approximately 100 GB of Oracle logs, but it did not appear to cause the same connector impact. One possible difference is that the supplementary purge affected more Debezium-captured tables / different table sets.
Based on this clarification, could you please advise what Debezium would recommend for handling this type of purge workload on captured Oracle tables?
We would appreciate your guidance on the most appropriate Debezium-side configuration, tuning, or architecture changes to make the connector more resilient for future purge runs.
Thanks
Sri
Chris Cranford
There are many factors that can influence how well such a purge activity can impact the connector.
The most notable is whether there were any in-flight transactions just before the purge. When this happens, and you have a transaction XYZ start at SCN A the purge happens and then the transaction XYZ commits at SCN A + 100 million, then its highly possible the connector will re-read those purge sections on multiple passes in order to eventually arrive at the commit associated with that transaction. Depending on the volume of the purge, the `log.mining.batch.size.*` settings, and with the fact this continuous re-read is IO, CPU, and RAM intensive, all database processes compete for the same resources. If there were no in-flight transactions at the time the purge started, you minimize and avoid a lot of this problem, but trying to time this in-between an in-flight transaction is near impossible.
This is precisely why Debezium 3.6 (released today), includes a new mining algorithm that no longer relies on `log.mining.batch.size.*` settings and instead uses `log.mining.log.count.min`, which defaults to 2. Instead of trying to slice logs by SCN ranges, logs are read in single passes, which has shown significant improvements during high burst activity periods, like what you describe.
So using your example of 50GB (assuming 5GB logs) is 10 archive logs. The `log.mining.batch.size.*` settings, depending on how large the increment and max settings were, could result in dozens or even hundreds of mining passes to read that 50GB of data. With the new 3.6 algorithm, it would have taken approximately 5 read passes, depending on how quickly the logs filled up and moved to the archive in-between mining passes.
But without JMX metrics during those periods, all we can do is speculate. But given the change in 3.6, I would recommend updating ASAP once we've officially announced the release.
Thanks,
-cc
To view this discussion visit https://groups.google.com/d/msgid/debezium/8eee9a7a-e50d-4af6-9fef-d80e2e660336n%40googlegroups.com.