Oracle / XStream: ORA-26824: user-defined XStream callback error

[Ajay Bhatnagar]

Hello Folks,

We are seeing the following error when using Debezium (3.2.7.Final) with Oracle 19c. The connector streamed CDC events successfully for a but now is stuck with this issue:

org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
  at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:67)
  at io.debezium.connector.oracle.xstream.XstreamStreamingChangeEventSource.execute(XstreamStreamingChangeEventSource.java:142)
  at io.debezium.connector.oracle.xstream.XstreamStreamingChangeEventSource.execute(XstreamStreamingChangeEventSource.java:46)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:326)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:207)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:147)
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: oracle.streams.StreamsException: ORA-26824: user-defined XStream callback error
KNJLCRDeliverLCR.exportLCR


  at oracle.streams.XStreamOut.XStreamOutReceiveLCRCallbackNative(Native Method)
  at oracle.streams.XStreamOut.receiveLCRCallback(Unknown Source)
  at io.debezium.connector.oracle.xstream.XstreamStreamingChangeEventSource.execute(XstreamStreamingChangeEventSource.java:116)
  ... 9 more

Connector Config

connector.class=io.debezium.connector.oracle.OracleConnector
errors.log.include.messages=true
topic.creation.default.partitions=-1
transforms.outbox.type=com.ontada.kafka.outbox.OutboxOracleRouter
schema.history.internal.consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='${env:KF_USERNAME}' password='${env:KF_PASSWORD}';
transforms=outbox
schema.include.list=EMR
transforms.outbox.dltTopicName=schema-changes.eventoutbox.dlt
transforms.outbox.table.fields.event.topic=topic
include.schema.changes=false
database.sslmode=require
schema.history.internal.producer.security.protocol=SASL_SSL
topic.creation.default.replication.factor=-1
lob.enabled=true
heartbeat.topics.prefix=__debezium-heartbeat
errors.log.enable=true
schema.history.internal.producer.sasl.mechanism=PLAIN
errors.retry.timeout=600000
database.dbname=${env:ORACLE_DB_NAME}
database.user=${env:ORACLE_DB_USER}
heartbeat.interval.ms=10000
schema.history.internal.kafka.bootstrap.servers=${env:KF_BOOTSTRAP_SERVER}
transforms.outbox.route.by.field=topic
transforms.outbox.table.fields.event.payload=payload
schema.history.internal.skip.unparseable.ddl=true
transforms.outbox.table.fields.event.key=key
database.password=${env:ORACLE_DB_PASSWORD}
schema.history.internal.consumer.sasl.mechanism=PLAIN
schema.history.internal.producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='${env:KF_USERNAME}' password='${env:KF_PASSWORD}';
snapshot.mode=schema_only
schema.history.internal.consumer.security.protocol=SASL_SSL
tasks.max=1
schema.history.internal.recovery.poll.interval.ms=2000
schema.history.internal.store.only.captured.tables.ddl=true
topic.prefix=outbox
decimal.handling.mode=string
schema.history.internal.kafka.topic=schema-changes.eventoutbox-qa2
transforms.outbox.route.topic.replacement=${routedByValue}
database.out.server.name=DBZOUT_EVENTBOX
database.connection.adapter=xstream
database.url=${env:ORACLE_DATABASE_URL}
errors.retry.delay.max.ms=30000
transforms.outbox.table.fields.event.timestamp=createddate
database.port=1521
schema.history.internal.recovery.attempts=4
topic.creation.enabled=false
database.hostname=${env:ORACLE_DB_HOSTNAME}
transforms.outbox.table.fields.event.headers=headers
table.include.list=EMR.EVENTOUTBOX
transforms.outbox.table.fields.event.id=id

Thanks,

Ajay

[Aqhil Mohammad]

Hello Ajay,

The connector is crashing due to a problematic CLOB record from the payload column in EMR.EVENTOUTBOX. Please:

Verify ALL COLUMNS supplemental logging is enabled on EMR.EVENTOUTBOX

Add the following to the connector config:

Properties

event.processing.failure.handling.mode=warn

max.batch.size=500

max.queue.size=4096

max.queue.size.in.bytes=209715200

This will prevent the connector from crashing on bad CLOB chunks and instead log and continue and restart the connector. 

Thanks,

Mohammad Akhil.

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/8fe206c3-1126-4fee-b1d1-6693652daa6en%40googlegroups.com.

[Chris Cranford]

Hi Ajay -

This may be related to this pr [1].

-cc

[1]: https://github.com/debezium/debezium/pull/7374

To view this discussion visit https://groups.google.com/d/msgid/debezium/CAHCJhW-JzQrdo2469pyke17%2Bzsu5_NVNqyNviYz-Y8B4GpNOgA%40mail.gmail.com.

[Ajay Bhatnagar]

Thanks Mohammad Akhil: I will give that a try. However I would like to understand the root cause of the bad CLOB record and rectify it instead of skipping the record.

Chris: That PR seems relevant only if application is using DBMS_LOB package, and I am pretty sure that is not the case for us. Any other suggestions?

Thanks

You received this message because you are subscribed to a topic in the Google Groups "debezium" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/debezium/evKELPlGP0w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/2837bdb5-a948-4ce7-b4b9-acebece3c532%40gmail.com.

[Aqhil Mohammad]

Hello Ajay, 

The bad CLOB most likely came during the data migration writing incomplete or empty CLOBs. Identifying the exact record via SCN from the connector error logs is the fastest way to root cause.

Few steps to find out and fix it.

1. Find the exact SCN from connector error logs

         

2. Identify the bad record using SCN or timestamp

         

3. Check CLOB size and content.

         

4. Fix at data migration level:

   - Re-insert the bad record correctly

   - Or delete and re-insert with proper CLOB write

         

5. Restart connector from the corrected SCN

Thanks,

Mohammad Akhil.

To view this discussion visit https://groups.google.com/d/msgid/debezium/CADifRRXz9SyfYOJEn-Z73Cr47L-1qpseMG2vUtjCvALP%2BansGQ%40mail.gmail.com.