Missing transactions with Oracle connector after the restarts

[Shahul Nagoorkani]

Hello Debezium Experts,

We are noticing a peculiar problem where our logminer based debezium oracle connector is loosing transactions whenever the connector is getting restarted. We picked up few heavy traffic tables and did a hourly record counts from snowflake consumer, we noticed that logminer is dropping the transactions. Are there any known issues with logminer missing the transactions on a busy OLTP environment?

These sections are from debezium logs during oracle connector restarts at various times. From the below restart snippets with the scns, are there any clues present to tell why we are loosing data upon restarts?

Offsets as shutdown: OracleOffsetContext [scn=9430661332325, txId=35001a00420d4700, txSeq=14, commit_scn=["9430661332325:1:35001a00420d4700"], lcr_position=null] Found previous offset OracleOffsetContext [scn=9430661330770, txId=35001a00420d4700, txSeq=14, commit_scn=["9430661332322:1:1a000100cb1f3a01"], lcr_position=null] Snapshot ended with SnapshotResult [status=COMPLETED, offset=OracleOffsetContext [scn=9430662135664, commit_scn=[], lcr_position=null]]
Offsets as shutdown: OracleOffsetContext [scn=9430663516607, txId=14001a00b26a3701, txSeq=10, commit_scn=["9430663516612:1:14001a00b26a3701"], lcr_position=null] Found previous offset OracleOffsetContext [scn=9430663514922, txId=14001a00b26a3701, txSeq=10, commit_scn=["9430663516609:1:2800120026e8d000"], lcr_position=null] Snapshot ended with SnapshotResult [status=COMPLETED, offset=OracleOffsetContext [scn=9430665098253, commit_scn=[], lcr_position=null]] Offsets as shutdown: OracleOffsetContext [scn=9430666741794, txId=1800060015013501, txSeq=1, commit_scn=["9430666741794:1:01001500c5d5c200"], lcr_position=null] Found previous offset OracleOffsetContext [scn=9430666740584, txId=1800060015013501, txSeq=1, commit_scn=["9430666741788:1:3500060058fd4600"], lcr_position=null] Snapshot ended with SnapshotResult [status=COMPLETED, offset=OracleOffsetContext [scn=9430668015294, commit_scn=[], lcr_position=null]] Offsets as shutdown: OracleOffsetContext [scn=9430689002349, txId=0b0010005678ee00, txSeq=1, commit_scn=["9430689002349:1:18001000afd73401"], lcr_position=null] Found previous offset OracleOffsetContext [scn=9430689001605, txId=0b0010005678ee00, txSeq=1, commit_scn=["9430689002324:1:0d00140023fffa00"], lcr_position=null] Snapshot ended with SnapshotResult [status=COMPLETED, offset=OracleOffsetContext [scn=9430690709803, commit_scn=[], lcr_position=null]] 

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi -

The log entries appear to show that the same offsets that were flushed at shutdown are used on start-up, and so far no one has reported this issue. So I'm afraid there isn't much I can provide right now.

Could you please raise a GitHub Issue [1] with the following:

    - Connector Configuration
    - Connector Version
    - Oracle database version (including if its standalone or RAC)
    - TRACE logs

With trace los, please use this logging configuration:

    io.debezium.connector.oracle=TRACE
    io.debezium.connector.oracle.OracleValueConverters=INFO

With the logging configuration, run your tests where you capture changes, stop and restart the connector where you observe the issues, and share the full, complete trace logs, including the list of transactions that you believe were missed and not replicated. 

In addition to this, if you could also clone our Oracle query tool [2] and build it, and run the `list-changes` command. In this case, you'd want to record your offset SCN before you perform the above test, and then record the database current scn after the test finishes.  Then use the offset scn as the start scn and the current scn as the end scn in the `list-changes` command.  This will export all the changes from LogMiner to a file that would be helpful for us to compare what Debezium saw to what is in the transaction logs themselves.

If you have any questions, don't hesitate to ask.

Thanks,
Chris

[1]: https://github.com/debezium/dbz
[2]: https://github.com/Naros/debezium-oracle-query-tool

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/434c87a0-fcb0-4640-9245-32745c9a5b5dn%40googlegroups.com.

[Shahul Nagoorkani]

Thanks for your response, Chris.

Sure will raise an issue. We are also looking at trying out the latest connector to see if it helps.

Meanwhile enabling the trace was spitting out critical data into the logs and hence we certainly don't want to enable the trace logging in prod. Is there a way to restrict the data spilling over to the logs?

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi -

You could try

    io.debezium.connector.oracle=TRACE
    io.debezium.connector.oracle.OracleValueConverters=INFO
    io.debezium.util.Loggings=DEBUG

But be aware that if the data becomes something that is necessary, we'll have to go through this process again.

Thanks,
-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/78ba2e13-9ee0-4302-834a-ddcc486e16dan%40googlegroups.com.

[Shahul Nagoorkani]

Hello Chris,

Since we noticed the data loss, we created a table "APPS.APPS_HEARTBEAT_CUSTOM" and updated it every 2 seconds through a cronjob with the following update statement.

UPDATE APPS.APPS_HEARTBEAT_CUSTOM SET last_update=SYSTIMESTAMP,counter=counter+1 WHERE id=1;

Stopped the connector around: 2/2/2026 14:43 MST (21:43 UTC)

Connector took about 15 minutes to come backup as the schema history refresh took really long time as it captures the entire database schema.

From the ingested data into snowflake, we found that nearly 13 minutes of data based on the missing counter from APPS.APPS_HEARTBEAT_CUSTOM which is around the same time the connector was down.

Logs are attached for the entire time duration.

Meanwhile we created a XStream connector for the same set of tables and observed similar behavior too with the data loss. I will share the logs from the XStreams connector in the next thread.

So irrespective of the connector mode, logminer or Xstreams, we are noticing the data loss.

Please review the logs and let us know if we miss anything from the configuration stand point.

Regards,

Shahul Nagoorkani

[Shahul Nagoorkani]

Part 2 of the log attached.

Regards,

Shahul Nagoorkani

[Shahul Nagoorkani]

Hi Chris,

I am attaching similar logs(2 parts) for the XStreams connector which also have the data loss problem.

Regards,

Shahul Nagoorkani

[Shahul Nagoorkani]

Part 2 of the XStreams connector logs.

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi -

The issue with the unbuffered implementation is a regression, which has an open PR [1], which we are currently discussing and hopefully will be included in 3.5.0.Alpha2. My suggestion would be for LogMiner, please use Debezium 3.4.0.Final (not 3.4.1.Final) with the "LogMiner" adapter and not the unbuffered LogMiner implementation. 

As for XStream, I suspect this is most likely similarly related to the ongoing issue with restarting incremental snapshots and the tests are failing, so this is most likely a bug. Now the question is whether or not this is a bug in Oracle or Debezium, which we'd need to investigate. For this, could you please raise a GitHub Issue [2].

Thanks,
-cc

[1]: https://github.com/debezium/debezium/pull/7039
[2]: https://github.com/debezium/dbz/issues


So it would appear that you're using the unbuffered LogMiner adapter, and not the buffered (default implementation). The unbuffered implementation works very similar to that of XStream, where if there is an active in-progress transaction, this can 

--

You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/debezium/40dc5368-04cb-429c-8e90-75e0a02110ban%40googlegroups.com.

[Shahul Nagoorkani]

Hi Chris,

Our connectors were using the buffered logminer mode(default) and faced similar issue which caused us to move to Xstreams testing where we faced similar data loss as well.

So the issue is across the board with logminer buffered, logminer unbuffered and XStreams.

Regards,

Shahul Nagoorkani

[Chris Cranford]

The logs you shared were for unbuffered, so I would kindly ask if yo could please produce TRACE level logs with the following log configuration using just the buffered (default) implementation:

    io.debezium.connector.oracle=TRACE
    io.debezium.util.Loggings=TRACE
    io.debezium.connector.oracle.OracleValueConverters=INFO

Please leave all other log configurations to their defaults of INFO, rerun your test and share the complete logs.

Thanks,
-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/e1f1af07-579f-40dd-b306-59643b6bab5an%40googlegroups.com.

[Shahul Nagoorkani]

Thanks Chris. Will upload the logs with the connection adapter mode changes to logminer instead of logminer_unbuffered shortly.

Could you please shed some light on the portion of the logs which proves that we are loosing the data with both unbuffered and xStreams connector? It will be useful for us to monitor for such patterns in our logs in the future.

Thanks.

Shahul Nagoorkani

[Shahul Nagoorkani]

Hello Chris,

Please find the trace logs attached after converting the connector to default logminer mode. We still notice the data loss upto 13 minutes.

We will try the 3.4.0.Final connector as well and see if we are able to reproduce the error and post the logs.

Regards,

Shahul Nagoorkani

[Shahul Nagoorkani]

Hello Chris,

Connector restart after updating the image to 3.4.0 yielded same results with data loss of upto 15 minutes.

Logs for these changes is attached.

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi -

Would you please share your connector configuration, it does not appear to be included in your logs, and I suspect this is critical. It appears to me that when you stop & restart your connector, it re-takes a snapshot, which indicates you've either implemented a custom configuration-based snapshot configuration setup, implemented your own custom snapshotter, or you have set `snapshot.mode` to `always`.

Thanks,
-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/d0508a71-7c74-4b47-b490-fd1116341ed3n%40googlegroups.com.

[Shahul Nagoorkani]

Hello Chris,

Here is the configuration and have attached the config for both logminer adapter as well as xstreams adapter.

We just use:

snapshot.mode: configuration_based

snapshot.mode.configuration.based.snapshot.schema: true

snapshot.mode.configuration.based.snapshot.data: false

snapshot.mode.configuration.based.start.stream: true

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi

So this explains the behavior, by setting `snapshot.mode.configuration.based.snapshot.schema`, the schema will be snapshot on each restart, which means we need to reset the read position. This is basically like setting `snapshot.mode` with `always` combined with `no_data`. That would explain why you see part of the transaction sent to your Kafka topics and why the remaining events would not appear. 

If you want to snapshot the schema but not the data, and you want to have the connector resume where it left off after being stopped, you should remove all `snapshot.mode.configuration.based.*` settings and set `snapshot.mode` to `no_data`. That should solve your issue.

-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/3ec91611-0d98-48b0-9e3d-92ece2f01081n%40googlegroups.com.

[Shahul Nagoorkani]

Hi Chris,

Since we have so many huge tables in our environment, our thought process of using the configuration based connector was to:

1. Start the connector with capturing the schema

2. Do not perform full load at the start
3. Start the CDC immediately
4. Perform the initial data load using the incremental signal for most of the tables under 100gb and perform a initial load as parquet using another utility for any tables greater than 100gb.

So our understanding is that configuration based connector start the connector without the initial load but with the offsets present already in the offset topics, it will go for last scn available and pick up from it left off during the connector restarts. Is our understanding wrong?

 If we set the snapshot.mode to "no_data", the connector would not capture the CDC going forward right? With us want the CDC to be captured after the connector restart from where it left off, what would be the ideal configuration that we should use?

Regards,

Shahul Nagoorkani

[Shahul Nagoorkani]

To add to my previous conversation, our environment goes through constant DDL changes and if we want our connectors compatible with the future ddls changes, what would be the ideal settings?

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi,

So lets address the data load using incremental snapshots. What you described in steps 1-4 is exactly and precisely what `snapshot.mode=no_data` does. The approach you're taking with the configuration_based approach always re-snapshots the schema and overwrites the offsets. I believe that was the intent, but I have opened a discussion with my colleagues on Zulip. So for now, it's best you use`snapshot.mode=no_data` as I suggested so that you use so that you observe the expected behavior.

With regard to DDL changes, this will depend entirely on the mining strategy you're using.

With `online_catalog`, this strategy is designed to be efficient and fast, but it expects that for DDL changes to be captured successfully and without DML failures, the DDL events need to be captured in a very specific way called lock-step. We discuss this in the documentation [1], but the expected steps are as follows:

    1. Stop allowing data changes on the table.
    2. Wait until all changes for that table are consumed.
    3. Apply the DDL change and wait for it to be consumed by Debezium.
    4. Resume data changes

But as you might image, this may not always be possible in every environment, and that is why the `hybrid` mining strategy was introduced. The `hybrid` strategy builds on top of the performance of `online_catalog` but with the resiliency of DDL change handling if you were using `redo_log_catalog`. So the net effect is you get the best of both worlds when using the `hybrid` strategy.

However with `hybrid`, there is one limitation that is driven by LogMiner itself, and that is you cannot pair it with `lob.enabled` set to `true`. Unfortunately LogMiner does not provide sufficient information in the event stream that would allow the connector to reconstruct the LOB data when a DDL event is seen mixed with DML events. But this corner case is one reason we made some adjustments in Debezium 3.1. If you find you need to use `hybrid` and you need to capture LOB data, there is one configuration pattern that can be used

    1. Set `log.mining.strategy` to `hybrid`
    2. Leave `lob.enabled` set to `false` (the default)
    3. Enable the Reselect Columns post processor, setting reselecting the unavailable values as `true` (leave reselecting null values as false)

If you have any questions, let us know.
-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/33972679-125f-45be-9dfb-2d17676e5476n%40googlegroups.com.

[Shahul Nagoorkani]

Thanks Chris for your prompt response. We just tried with no_data mode and did not see any data loss. 

Going by the debezium oracle connector documentation (https://debezium.io/documentation/reference/stable/connectors/oracle.html#oracle-property-configuration-based-snapshot-schema), certainly it doesn't gave any clue that setting it to true would overwrite the offsets leading to data loss. Even when we did the research with chat gpt and other AI tools, it never came up with a warning about the data loss. It's better to make the documentation bit clear on this particular scenario. 

We will check on log_mining_strategy as well as the post processor further and let you know if we have any questions. 

Meanwhile we have same configuration based settings for other postgres, sql server, mysql connectors as well and in production for a while. Should we be worried about other connectors as well about the data loss or do they behave differently? With us know the issue with oracle connectors, we will validate other "configuration-based" connectors from our end.

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi -

I fully agree with you in reference to the documentation. This is why I raised the internal team discussion, because either the behavior is unintended or we should better articulate the expected behavior in the documentation.
https://debezium.zulipchat.com/#narrow/channel/302533-dev/topic/Snapshot.20Configuration.20Based.20Mode/with/571969908

In terms of the other connectors and configuration_based setups, I would expect the behavior to be the same. All relational database connectors use a centralized snapshot implementation.

Thanks,
-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/f7adaab6-bfd1-4658-8f4c-f00745a71dd6n%40googlegroups.com.

[Shahul Nagoorkani]

Hello Chris,

We just tested the data loss scenario with the same kind of test that we did with Oracle against a Aurora Postgres database and it did not loose any data with the "configuration-based" connector.

We will test with other rdbms and record the the findings here.

So with this test, we confirm that "configuration-based" connector doesn't loose data whereas oracle loose the data.

Regards,

Shahul Nagoorkani

[Chris Cranford]

I'd be interested to see your findings with MySQL or SQL Server. Unfortunately comparing PostgreSQL to any other relational connector is a bit of an Apples-V-Oranges situation because PostgreSQL does not maintain a schema history.

To view this discussion visit https://groups.google.com/d/msgid/debezium/e9dbf22a-95ee-45cc-9394-6dd61f4daeecn%40googlegroups.com.

[Shahul Nagoorkani]

Sure, Chris. That's what we thought that any databases which uses schema history may be dealing with the data loss with the "configuration-based" connector. We will keep posted on our test results.

Thanks,

Shahul Nagoorkani

[Shahul Nagoorkani]

Hello Chris,

With the following settings:

snapshot.mode: no_data
log.mining.strategy: hybrid
include.schema.changes: true
lob.enabled: false (default and not present in our config)
schema.refresh.interval.ms: 10000

We made a DDL change:

alter table APPS.APPS_HEARTBEAT_CUSTOM ADD (comments_clob clob)

We did not see the DDL changes replicated to schema history/Avro registry. Is this the expected behavior? We also tried restarting the connector but hitting the same issue.

Subsequent update to the clob failed with the following error:

 - id: 0
      state: FAILED
      trace: "org.apache.kafka.connect.errors.ConnectException: An exception occurred
        in the change event producer. This connector will be stopped.\n\tat io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:67)\n\tat
        io.debezium.connector.oracle.logminer.AbstractLogMinerStreamingChangeEventSource.execute(AbstractLogMinerStreamingChangeEventSource.java:217)\n\tat
        io.debezium.connector.oracle.logminer.AbstractLogMinerStreamingChangeEventSource.execute(AbstractLogMinerStreamingChangeEventSource.java:88)\n\tat
        io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:339)\n\tat
        io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:217)\n\tat
        io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:148)\n\tat
        java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat
        java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat
        java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat
        io.debezium.util.Threads$3.lambda$newThread$0(Threads.java:314)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused
        by: io.debezium.connector.oracle.logminer.parser.DmlParserException: DML statement
        couldn't be parsed. Please open a Jira issue with the statement 'update \"APPS\".\"APPS_HEARTBEAT_CUSTOM\"
        set \"COMMENTS_CLOB\" = EMPTY_CLOB() where \"ID\" = '1' and \"LAST_UPDATE\"
        = TO_TIMESTAMP('2026-02-09 19:48:48.449300000') and \"COUNTER\" = '318419'
        and \"COMMENTS\" = 'Jay Nair';'.\n\tat io.debezium.connector.oracle.logminer.AbstractLogMinerStreamingChangeEventSource.parseDmlStatement(AbstractLogMinerStreamingChangeEventSource.java:1463)\n\tat
  io.debezium.connector.oracle.logminer.AbstractLogMinerStreamingChangeEventSource.dispatchDataChangeEventInternal(AbstractLogMinerStreamingChangeEventSource.java:1325)\n\tat
        io.debezium.connector.oracle.logminer.AbstractLogMinerStreamingChangeEventSource.handleDataChangeEvent(AbstractLogMinerStreamingChangeEventSource.java:615)\n\tat
        io.debezium.connector.oracle.logminer.AbstractLogMinerStreamingChangeEventSource.processEvent(AbstractLogMinerStreamingChangeEventSource.java:481)\n\tat
        io.debezium.connector.oracle.logminer.AbstractLogMinerStreamingChangeEventSource.executeAndProcessQuery(AbstractLogMinerStreamingChangeEventSource.java:404)\n\tat
        io.debezium.connector.oracle.logminer.buffered.BufferedLogMinerStreamingChangeEventSource.process(BufferedLogMinerStreamingChangeEventSource.java:291)\n\tat
        io.debezium.connector.oracle.logminer.buffered.BufferedLogMinerStreamingChangeEventSource.executeLogMiningStreaming(BufferedLogMinerStreamingChangeEventSource.java:167)\n\tat
        io.debezium.connector.oracle.logminer.AbstractLogMinerStreamingChangeEventSource.execute(AbstractLogMinerStreamingChangeEventSource.java:212)\n\t...
        10 more\nCaused by: io.debezium.connector.oracle.logminer.parser.DmlParserException:
        Failed to parse update DML: 'update \"APPS\".\"APPS_HEARTBEAT_CUSTOM\" set
        \"COMMENTS_CLOB\" = EMPTY_CLOB() where \"ID\" = '1' and \"LAST_UPDATE\" =
        TO_TIMESTAMP('2026-02-09 19:48:48.449300000') and \"COUNTER\" = '318419' and
        \"COMMENTS\" = 'Jay Nair';'\n\tat io.debezium.connector.oracle.logminer.parser.LogMinerDmlParser.parseUpdate(LogMinerDmlParser.java:173)\n\tat
        io.debezium.connector.oracle.logminer.parser.LogMinerDmlParser.parse(LogMinerDmlParser.java:82)\n\tat
        io.debezium.connector.oracle.logminer.AbstractLogMinerStreamingChangeEventSource.parseDmlStatement(AbstractLogMinerStreamingChangeEventSource.java:1444)\n\t...
        17 more\nCaused by: io.debezium.DebeziumException: No column 'COMMENTS_CLOB'
        found in table 'DB3Q51.APPS.APPS_HEARTBEAT_CUSTOM'\n\tat io.debezium.connector.oracle.logminer.LogMinerHelper.getColumnIndexByName(LogMinerHelper.java:26)\n\tat
        io.debezium.connector.oracle.logminer.parser.LogMinerDmlParser.getColumnIndexByName(LogMinerDmlParser.java:98)\n\tat
        io.debezium.connector.oracle.logminer.parser.LogMinerDmlParser.parseSetClause(LogMinerDmlParser.java:526)\n\tat
        io.debezium.connector.oracle.logminer.parser.LogMinerDmlParser.parseUpdate(LogMinerDmlParser.java:148)\n\t...
        19 more\n"
      worker_id: kafka-connect-dba-ebiz-apps-connect-0.kafka-connect-dba-ebiz-apps-connect.dba-debezium.svc:8083

Please advise.

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi -

Was the DDL change executed by the SYS or SYSTEM user? 

-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/86f15cfe-88c8-4978-b468-0595b2201b7an%40googlegroups.com.

[Shahul Nagoorkani]

Hello Chris,

The column was added as SYSTEM user. Does it makes any difference with ddl propagation? Even with connector restarts, the schema refresh never happened with the clob changes and we eventually had to place the connector in "recovery" mode which captured the ddl changes and resolved the issue.

We have done similar tests with Xstreams using the same SYSTEM user and never faced these issues. 

Another question around configuration-based connector:

What if we want to go with configuration based connector but have "snapshot.mode.configuration.based.snapshot.schema" set to false? What are the implications of doing this?

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi

If the DDL was performed by the SYS or SYSTEM users, it will not be captured by the connector. This behavior is specific to LogMiner. This can be customized by setting `internal.log.mining.schema_changes.username.exclude.list` with an empty value in the connector configuration.  If this is functionality that the broader community would like to see changed, we're open to changing this behavior; but due to the extra traffic that standard Oracle does automatically by the SYS user, this could create more connector traffic than desired.

For your last question, setting both the capture of schemas and data as false will lead to skipping the snapshot. 

Thanks,
-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/c4a82f75-b610-4e7b-8f58-1060e0658efcn%40googlegroups.com.

[Shahul Nagoorkani]

Thanks Chris. We tested a change using the schema owner when the log.mining.strategy set to "online_catalog" without following the lock-step method and faced the issue:

      io.debezium.util.Threads$3.lambda$newThread$0(Threads.java:314)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused

        by: io.debezium.DebeziumException: Oracle LogMiner is unable to re-construct the SQL for 'UPDATE' event with SCN 9310218384464\n\tat

So is there a way to recover a connector which is in "online_catalog" mode when the DDL changes are not done in the lock-step method without any data loss?

Regards,

Shahul Nagoorkani

[Chris Cranford]

As long as that is the only DDL change performed, you could use `snapshot.mode` set to `recovery` to recover the history topic. But if there have been other schema changes since that DDL, I'm afraid a re-snapshot is required. This is the benefit of using `hybrid` rather than `online_catalog`.

To view this discussion visit https://groups.google.com/d/msgid/debezium/ea1a0e5f-bb35-4d00-ac02-b8981785a636n%40googlegroups.com.

[Shahul Nagoorkani]

Hello Chris,

In some of these cases, even setting the 'snapshot.mode' to recovery did not help and the connector kept failing at the same point. Since this is just our test connector, we had to just clear the offset and move on. 

Agree that hybrid would solve some of these issues but complexity around the lobs seems like negates these advantages.

So if we loose the data by resetting the offsets or starting the connector using the "config-based" with schema capture, is there a way to go back to the known SCN (in this case, say start.scn: 9310218384464) where the connector broke and instruct the connector to start mining from the last known broken scn? Once the connector catches up to the latest scn, we would then comment the "start.scn" parameter to allow the connector to just proceed from the last scn recorded. We looked around for parameters in the documentation and obviously did not find any.

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi -

What complexities with LOBs and Hybrid? I provided a 3 step solution to using it below, did that not work?

As for starting the connector from a given SCN offset, there is no official way to do this via configuration. There is an open GitHub Issue to introduce a signal for this behavior universally for all connectors.

-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/6489b0fb-4d4e-4623-acb0-1613e97fd464n%40googlegroups.com.

[Shahul Nagoorkani]

Hello Chris,

We did not get a chance to test the Post processors with hybrid option. We decided to go with XStreams as it seems more reliable. But we will test Post processors too and share our experience.

Meanwhile we tested out the data loss scenario in MS SQL/MYSQL databases with configuration-based connectors and was able to replicate the issue in MSSQL that it lost data after the restart but MYSQL was perfectly OK.

So it seems the configuration-based connectors have the data loss problem only with Oracle/MSSQL. 

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi -

I'm afraid I am not seeing the same in my local tests when `snapshot.mode.configuration.based.snapshot.schema` is set to `true`. In all instances, the offset position is reset to the latest write position in the database, and we begin to stream from that position. 

As for the data loss, when the offset position is reset, anything that happened between the old read position in the offsets and the current database write position would only be present if you retook a full snapshot of the historical data, which you've explicitly disabled by setting `snapshot.mode.configuration.based.snapshot.data` as `false`. 

What you're attempting to accomplish with this setup is the equivalent of `snapshot.mode` set to `no_data`, and ideally that should be what you use if you wish for the connector to resume from where it left off after a connector restart.

If you believe there is a bug here, you're more than welcomed to open a GitHub Issue [1] with all the supporting evidence, and steps using our tutorial to reproduce the problem.

Thanks,
-cc

[1]: https://github.com/debezium/dbz/issues

To view this discussion visit https://groups.google.com/d/msgid/debezium/6c3773e2-f9ae-4ba7-9571-37ad037dafb4n%40googlegroups.com.

[Shahul Nagoorkani]

Hi Chris,

Do you mean to say that you see the data loss behavior in MySQL databases too with `snapshot.mode.configuration.based.snapshot.schema` is set to `true`? If that's the case, we will do further testing on our end and we will also move to "no_data" mode. 

But if there are consistencies in the behavior between Oracle/MSSQL vs MySQL wrt `snapshot.mode.configuration.based.snapshot.schema`, then it needs to be looked at certainly and raise a bug.

So with the XStreams connector that we started in the last few hours, we noticed the capture latency keeps creeping up and is around 1+ hour now. Are there any known issues around slow capture performance?

SELECT
  d.current_scn                            AS db_current_scn,
  c.captured_scn                           AS capture_captured_scn,
  (d.current_scn - c.captured_scn)         AS db_minus_capture_scn,
  scn_to_timestamp(d.current_scn)          AS db_time,
  scn_to_timestamp(c.captured_scn)         AS capture_time,
  ROUND(
    ( CAST(scn_to_timestamp(d.current_scn) AS DATE)
    - CAST(scn_to_timestamp(c.captured_scn) AS DATE)
    ) * 86400
  )/60 AS db_minus_capture_minutes
FROM v$database d
JOIN dba_capture c
  ON c.capture_name = '<capture name>'
JOIN v$xstream_outbound_server x
  ON x.server_name = '<outbound server name>';

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi -

A number of tuning things we recommend when using Debezium LogMiner adapter also applies to XStream.

    - Increase capture process memory if possible to reduce IO
    - Verify capture process has adequate CPU, as the CAP processes are extensively CPU intensive
    - Ensure redo/archive logs are on SSD (fast) disks, as disk IO is expensive
    - Increase redo log sizes if you have more than 6 log switches per hour
    - Make sure capture filters are set to only capture the tables you absolutely need
    - In parallel capture servers as more parallelism can speed up redo log mining, though scaling this requires sufficient CPU

In addition, have the DBA check the xstreams metrics to see if there are any bottlenecks or unintended latch waits or locks that could impede performance.

If none of those help, please contact Oracle Support for guidance to tune and scale your Oracle database based on your load.

Thanks,
-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/d07a68c4-d2d4-4d12-9181-c20dd1981f8en%40googlegroups.com.

[Shahul Nagoorkani]

Hello Chris,

Not sure what was holding up but when we created the connector, we had couple of network issues preventing the pods to be created. Eventually the network issues were resolved and the connector came up for the first time and it was too slow. We were trying to understand the reason behind the slowness, we just attempted restarting the connector and the capture latency drained immediately.

We will create more Xstreams connectors and see if this issue is reproducible.

Thanks.

Regards,

Shahul Nagoorkani

[Shahul Nagoorkani]

Hello Chris,

We were able to kind of reproduce the issue that it is too slow during the first time the connector comes up but a subsequent restart fix the issue. We faced similar issues in the last couple of days where a simple restart of the connector fixed the issue. We may need to look into the reason behind this issue.

Also the Xstreams connector keept failing couple of times in the last few days and a simple restart fixed the issue. Anyone reported similar kind of issues?

      state: FAILED
      trace: "org.apache.kafka.connect.errors.ConnectException: An exception occurred
        in the change event producer. This connector will be stopped.\n\tat io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:67)\n\tat

        io.debezium.connector.oracle.xstream.XstreamStreamingChangeEventSource.execute(XstreamStreamingChangeEventSource.java:142)\n\tat
        io.debezium.connector.oracle.xstream.XstreamStreamingChangeEventSource.execute(XstreamStreamingChangeEventSource.java:46)\n\tat
        io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:329)\n\tat
        io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:207)\n\tat
        io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:147)\n\tat

        java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat
        java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat
        java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat

        java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: oracle.streams.StreamsException:
        ORA-26824: user-defined XStream callback error\nXStreamOut DeliverLCR: StreamsException
        thrown in processLCR().\n\n\n\tat oracle.streams.XStreamOut.XStreamOutReceiveLCRCallbackNative(Native
        Method)\n\tat oracle.streams.XStreamOut.receiveLCRCallback(Unknown Source)\n\tat
        io.debezium.connector.oracle.xstream.XstreamStreamingChangeEventSource.execute(XstreamStreamingChangeEventSource.java:116)\n\t...
        9 more\n"
      worker_id: kafka-connect-dba-ebiz-abs-xstreams-connect-0.kafka-connect-dba-ebiz-abs-xstreams-connect.dba-debezium.svc:8083

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi, can you please share the full connector logs.

To view this discussion visit https://groups.google.com/d/msgid/debezium/c7f2b345-19d6-4728-8e1e-5eca5681b87an%40googlegroups.com.

[Shahul Nagoorkani]

Hello Chris,

Logs are attached for the recent such event but this time, the connector did not even come backup. Several restart attempts just failed and though we don't see any detailed logs in our DataDog, I did notice this error in the pod logs:

at io.debezium.pipeline.EventDispatcher.handleEventProcessingFailure(EventDispatcher.java:355)
at io.debezium.pipeline.EventDispatcher.dispatchDataChangeEvent(EventDispatcher.java:347)
at io.debezium.connector.oracle.xstream.LcrEventHandler.dispatchDataChangeEvent(LcrEventHandler.java:241)
at io.debezium.connector.oracle.xstream.LcrEventHandler.resolveAndDispatchCurrentChunkedRow(LcrEventHandler.java:420)
at io.debezium.connector.oracle.xstream.LcrEventHandler.processChunk(LcrEventHandler.java:368)
at oracle.streams.XStreamOut.XStreamOutReceiveLCRCallbackNative(Native Method)
at oracle.streams.XStreamOut.receiveLCRCallback(Unknown Source)
at io.debezium.connector.oracle.xstream.XstreamStreamingChangeEventSource.execute(XstreamStreamingChangeEventSource.java:116)
at io.debezium.connector.oracle.xstream.XstreamStreamingChangeEventSource.execute(XstreamStreamingChangeEventSource.java:46)
at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:329)
at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:207)
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:147)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.IllegalArgumentException: Received event of unexpected command type: DefaultRowLCR:
Header:
source database name: DB3P05
command type: LOB WRITE
object owner: ECHO
object name: SYMEMAILPAYLOAD
tag: []
position: [0 0 8 ffffff94 3a ffffffc1 ffffffac 74 0 0 0 1 0 0 0 1 0 0 8 ffffff94 3a ffffffc1 ffffffac 6d 0 0 0 e 0 0 0 1 2]
transaction id: 26.18.10222908
source time: 2026-02-17 06:28:01
scn: null
commit scn: null
lcr flags:
Statement LCR: false
IDKeyColumnsOnly LCR: false
Sequence LCR: false
Attribute[1]
(0): ROOT_NAME : DB3P05
Old column values (0):
New column values (13):
(0): DefaultColumnValue(csid:0, tde:false, type:0, name:NOTIFICATIONID, data:xxxxxx)
(1): DefaultColumnValue(csid:0, tde:false, type:0, name:CAMPAIGNNAME, data:xxxxx)
(2): DefaultColumnValue(csid:0, tde:false, type:0, name:TEMPLATETYPE, data:xxxxxx)
(3): DefaultChunkColumnValue(offset:1, size:0, chunkType:1, emptyChunk:false, lastChunk:true, xmlDiff:false, endOfRow:false, charSetId:0, DefaultColumnValue(csid:0, tde:false, type:1, name:PAYLOAD, data:null))
(4): DefaultColumnValue(csid:0, tde:false, type:0, name:STATUS, data:N)
(5): DefaultColumnValue(csid:0, tde:false, type:0, name:BATCHID, data:0)
(6): DefaultColumnValue(csid:0, tde:false, type:0, name:DATACENTER, data:xxxx)
(7): DefaultColumnValue(csid:0, tde:false, type:0, name:CREATEDDATE, data:2026-02-17 06:28:01.947)
(8): DefaultColumnValue(csid:0, tde:false, type:0, name:CREATEDBY, data:xxxx)
(9): DefaultColumnValue(csid:0, tde:false, type:0, name:LASTUPDATEDDATE, data:2026-02-17 06:28:01.947)
(10): DefaultColumnValue(csid:0, tde:false, type:0, name:LASTUPDATEDBY, data:xxxx)
(11): DefaultColumnValue(csid:0, tde:false, type:1, name:ISUNDELIVERABLE, data:null)
(12): DefaultColumnValue(csid:0, tde:false, type:1, name:COMMENTS, data:null)

at io.debezium.connector.oracle.xstream.XStreamChangeRecordEmitter.getOperation(XStreamChangeRecordEmitter.java:51)
at io.debezium.relational.RelationalChangeRecordEmitter.emitChangeRecords(RelationalChangeRecordEmitter.java:43)
at io.debezium.pipeline.EventDispatcher.dispatchDataChangeEvent(EventDispatcher.java:299)
... 15 more

Looks like the XStreams has some issues capturing the CLOB column PAYLOAD.

We don't see any XStreams limitations with CLOB in the documentation except for this section:

XStream and DBMS_LOB

Oracle provides a database package called DBMS_LOB that consists of a collection of programs to operate on BLOB, CLOB, and NCLOB columns. Most of these programs manipulate the LOB column in totality, however, one program, WRITEAPPEND, is capable of manipulating a subset of the LOB data buffer.

When using XStream, WRITEAPPEND emits a logical change record (LCR) event for each invocation of the program. These LCR events are not combined into a single change event as they are when using the Oracle LogMiner adapter. As a result, consumers of the topic might receive events with partial column values. This diverged behavior is captured in DBZ-4741 and will be addressed in a future release. 

Regards,

Shahul Nagoorkani

[Shahul Nagoorkani]

Hello Chris,

We were able to reproduce the issue in non-prod XStreams connector as well using a simple "piece-wise" lob update script and the connector failed immediately. Is this an expected behavior?

declare
  l_clob   CLOB;
  l_buf    VARCHAR2(32767);
  p_times  integer:=5;
BEGIN
  SELECT jay_clob INTO l_clob
    FROM APPS.APPS_HEARTBEAT_XSTREAM_CUSTOM
   WHERE id = 1
   FOR UPDATE;

  FOR i IN 1 .. p_times LOOP
    l_buf := RPAD('append-' || i || '-', 5000, 'y');
    DBMS_LOB.WRITEAPPEND(l_clob, LENGTH(l_buf), l_buf);
  END LOOP;

  COMMIT;
END;
/

    tasks:

    - id: 0
      state: FAILED
      trace: "org.apache.kafka.connect.errors.ConnectException: An exception occurred
        in the change event producer. This connector will be stopped.\n\tat io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:67)\n\tat

        io.debezium.connector.oracle.xstream.XstreamStreamingChangeEventSource.execute(XstreamStreamingChangeEventSource.java:142)\n\tat
        io.debezium.connector.oracle.xstream.XstreamStreamingChangeEventSource.execute(XstreamStreamingChangeEventSource.java:46)\n\tat
        io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:329)\n\tat
        io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:207)\n\tat
        io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:147)\n\tat
        java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat
        java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat
        java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat
        java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: oracle.streams.StreamsException:

        ORA-26824: user-defined XStream callback error\nXStreamOut DeliverChunk: exception
        thrown in processChunk callback.\n\n\n\tat oracle.streams.XStreamOut.XStreamOutReceiveLCRCallbackNative(Native

        Method)\n\tat oracle.streams.XStreamOut.receiveLCRCallback(Unknown Source)\n\tat
        io.debezium.connector.oracle.xstream.XstreamStreamingChangeEventSource.execute(XstreamStreamingChangeEventSource.java:116)\n\t...
        9 more\n"

      worker_id: kafka-connect-dba-ebiz-appsxstreams-connect-0.kafka-connect-dba-ebiz-appsxstreams-connect.dba-debezium.svc:8083

Regards,

Shahul Nagoorkani

[Chris Cranford]

I didn't see anything in your logs, but let me check out this code snippet and get back to you.

To view this discussion visit https://groups.google.com/d/msgid/debezium/303976d3-965a-46b9-b975-b94fd902c127n%40googlegroups.com.

[Shahul Nagoorkani]

Hello Chris,

LOB issue happened with one more database too with XStreams connector and we had to exclude that table for the connector to recover. Log around that time frame is attached. 

Did you get a chance to check on the query that I provided to reproduce the issue? Thanks.

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi -

I did, and currently using the DBMS_LOB package to directly manipulate LOB data is unsupported. It's expected that you should use DML operations (Insert, Update, and Delete) operations to manipulate the fields, currently. 

I've logged dbz#1634 [1]. You can track the status of this change via the GitHub Issue.  

Thanks for the report.
-cc

[1]: https://github.com/debezium/dbz/issues/1634

To view this discussion visit https://groups.google.com/d/msgid/debezium/49c95de0-865f-4e2b-97c1-f64593cfe266n%40googlegroups.com.

[Shahul Nagoorkani]

Hello Chris,

Thanks for the confirmation and creating an issue to address this in future versions. Any ETA on when these fixes will be available? Also it will be useful to update the documentation as well to mention about this XStreams limitation that it's a hard failure with DBMS_LOB operations rather than the partial lob updates.

Regards,

Shahul Nagoorkani

[Shahul Nagoorkani]

Hello Chris, 

We faced another peculiar issue with XStreams connector. The issue originated from a query executed by Oracle Default Maintenane Jobs - Segment Advisor.

The XStreams connector crashed due to the following DDL statement:

java.base\/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base\/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base\/java.lang.Thread.run(Thread.java:840)\nCaused by: io.debezium.text.ParsingException: DDL statement couldn't be parsed. Please open a Jira issue with the statement 'alter index \"SYMPAY\".\"IDX_SERVICE_RESULT_CDATE\" modify subpartition \"SYS_SUBP2495985\" shrink space CHECK;'\nmismatched input 'shrink' expecting {'ALLOCATE', 'DEALLOCATE', 'UNUSABLE'}\n\tat io.debezium.antlr.ParsingErrorListener.syntaxError(ParsingErrorListener.java:43)\n\tat org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:41)\n\tat org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:543)\n\tat org.antlr.v4.runtime.DefaultErrorStrategy.reportInputMismatch(DefaultErrorStrategy.java:327)\n\tat

We had no choice other than to introduce the parameter to get through this issue:

schema.history.internal.skip.unparseable.ddl: true

Since these kind of queries may originate from Segment Advisor jobs in the future and break the connector, we left this parameter permanently for now in our connector configuration.

Is this a known issue with Oracle XStreams connectors? Did anyone reported this issue in the past? Is it ok to leave this parameter permanently in our configuration or do we have to disable it and enable only when we hit similar kind of DDL Parsing issues in the future? Please advise.

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi, I left a comment on the issue. The fix is fairly invasive, so I don't have an ETA at this time.

To view this discussion visit https://groups.google.com/d/msgid/debezium/56c28119-0c1f-485a-91da-1a0be395441cn%40googlegroups.com.

[Chris Cranford]

This isn't common to XStream, this would be a problem if you used any of the Oracle adapters. When you see these sorts of errors, its helpful to report these in our issue tracker.  We tend to fix DDL parser errors relatively quickly. As for the parameter, I would suggest only leaving it set temporarily to get past the problem DDL since this is index-related and these are not replicated to the schema history topic.

I'll raise a GH issue and we'll get this fixed in 3.5.0.Alpha2 and 3.4.2.Final.

-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/c1b61a95-bbbb-4a12-8e86-13cf6694bc90n%40googlegroups.com.

[Shahul Nagoorkani]

Thanks Chris. Since these ddl parsers are originating from the Segment Advisor, this crash might happen almost everyday when the Segment Advisor executes similar kind of queries. Disabling/enabling the  schema.history.internal.skip.unparseable.ddl everytime would be a maintenance nightmare for sure. It happened again today with another database as well causing the connector to crash in middle of the night.

Another issue noticed especially with XStreams connector while doing an incremental data load which we never faced with logminer mode:

    trace: "org.apache.kafka.connect.errors.ConnectException: An exception occurred
        in the change event producer. This connector will be stopped.\n\tat io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:67)\n\tat

        io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:154)\n\tat

        java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat
        java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat
        java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat

        java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: io.debezium.DebeziumException:
        Database error while executing incremental snapshot for table 'CollectionId{id=DB3P04.SYMPAY.SYMPAY_INTF_LINE_ITEM_RD,
        additionalCondition=, surrogateKey=}'\n\tat io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.readChunk(AbstractIncrementalSnapshotChangeEventSource.java:349)\n\tat
 io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.init(AbstractIncrementalSnapshotChangeEventSource.java:236)\n\tat
        io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$initStreamEvents$7(ChangeEventSourceCoordinator.java:352)\n\tat
        java.base/java.util.Optional.ifPresent(Optional.java:178)\n\tat io.debezium.pipeline.ChangeEventSourceCoordinator.initStreamEvents(ChangeEventSourceCoordinator.java:352)\n\tat
        io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:319)\n\tat
        io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:207)\n\tat
        io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:147)\n\t...
        5 more\nCaused by: java.lang.NegativeArraySizeException: -867801888\n\tat
        oracle.jdbc.driver.T2CStatement.t2cDefineFetch(Native Method)\n\tat oracle.jdbc.driver.T2CStatement.doDefineFetch(T2CStatement.java:1598)\n\tat
        oracle.jdbc.driver.T2CStatement.executeForRows(T2CStatement.java:1223)\n\tat
        oracle.jdbc.driver.OracleStatement.prepareDefineBufferAndExecute(OracleStatement.java:1291)\n\tat
        oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1149)\n\tat
        oracle.jdbc.driver.OracleStatement.executeSQLSelect(OracleStatement.java:1661)\n\tat
        oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1470)\n\tat
        oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:2055)\n\tat
        oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:394)\n\tat
        io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:653)\n\tat
        io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:520)\n\tat
        io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource.readChunk(AbstractIncrementalSnapshotChangeEventSource.java:280)\n\t...

Connector is now stuck in the loop. Trying to stop the signal by inserting a stop signal doesn't work as well as this happens way before the signal processor going to the signal table. Everytime the connector comes up and thinks that there is a previous incremental in progress, kick starts the incremental data load and crashes immediately.

Any ideas how we could get through this issue and what's causing this issue? The table doesn't have any offending data types like CLOBS/BLOB/LONG/RAW and etc but has lot of standard VARCHAR columns.

Regards,

Shahul Nagoorkani

[Chris Cranford]

A few questions:

    - What is the incremental chunk size in the configuration?
    - What is the query fetch size in the configuration?
    - How many columns are in DB3P04.SYMPAY.SYMPAY_INTF_LINE_ITEM_RD table, can you provide the table's DDL?

Thanks,
-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/b227e505-9a68-49e0-88ff-ddd5a5dfb3f5n%40googlegroups.com.

[Shahul Nagoorkani]

Hello Chris,

I reduced the incremental chunk size to even 125 and query fetch size to 150 per gpt suggestion but it did not help.

  CREATE TABLE "SYMPAY"."SYMPAY_INTF_LINE_ITEM_RD"
   ( "ITEM_INTF_ID" NUMBER(19,0) NOT NULL ENABLE,
"FK_RD_ORDER_INTF_ID" NUMBER(19,0),
"ITEM_ID" NUMBER(19,0),
"ORDER_ID" NUMBER(19,0),
"ROS_LINE_NUMBER" VARCHAR2(256 CHAR),
"ITEM_TYPE_CODE" VARCHAR2(256 CHAR),
"PRODUCT_SKU" VARCHAR2(256 CHAR),
"SKU_PRICE_LIST_NAME" VARCHAR2(256 CHAR),
"SKU_AMOUNT" VARCHAR2(256 CHAR),
"ITEM_QUANTITY" VARCHAR2(256 CHAR),
"LINE_ITEM_AMOUNT" VARCHAR2(256 CHAR),
"LINE_ITEM_TAX_AMOUNT" VARCHAR2(256 CHAR),
"COUNTRY_TAX_AMOUNT" VARCHAR2(256 CHAR),
"STATE_TAX_AMOUNT" VARCHAR2(256 CHAR),
"CITY_TAX_AMOUNT" VARCHAR2(256 CHAR),
"COUNTY_TAX_AMOUNT" VARCHAR2(256 CHAR),
"DISTRICT_TAX_AMOUNT" VARCHAR2(256 CHAR),
"SHIPTO_PREFIX_NAME" VARCHAR2(256 CHAR),
"SHIPTO_FIRST_NAME" VARCHAR2(256 CHAR),
"SHIPTO_MIDDLE_NAME" VARCHAR2(256 CHAR),
"SHIPTO_LAST_NAME" VARCHAR2(256 CHAR),
"SHIPTO_SUFFIX_NAME" VARCHAR2(256 CHAR),
"SHIPTO_EMAIL" VARCHAR2(256 CHAR),
"SHIPTO_COMPANY_NAME" VARCHAR2(256 CHAR),
"SHIPPING_ADDRESS_LINE_1" VARCHAR2(256 CHAR),
"SHIPPING_ADDRESS_LINE_2" VARCHAR2(256 CHAR),
"SHIPPING_ADDRESS_LINE_3" VARCHAR2(256 CHAR),
"SHIPPING_ADDRESS_LINE_4" VARCHAR2(256 CHAR),
"SHIPPING_CITY" VARCHAR2(256 CHAR),
"SHIPPING_STATE" VARCHAR2(256 CHAR),
"SHIPPING_POSTAL_CODE" VARCHAR2(256 CHAR),
"SHIPPING_COUNTRY_CODE" VARCHAR2(256 CHAR),
"SHIPPING_COUNTY" VARCHAR2(256 CHAR),
"ERR_DESC" VARCHAR2(1024 CHAR),
"SALES_CHANNEL" VARCHAR2(256 CHAR),
"PARTNER" VARCHAR2(255 CHAR),
"SKU_ENTITLEMENT_START_DATE" TIMESTAMP (6),
"SKU_ENTITLEMENT_END_DATE" TIMESTAMP (6),
"BUNDLE_ID" VARCHAR2(256 CHAR),
"GL_DEBIT_ACCOUNT" VARCHAR2(60 CHAR),
"GL_CREDIT_ACCOUNT" VARCHAR2(120 CHAR),
"GROUP_ID" VARCHAR2(100 CHAR),
"SETTLEMENT_CURRENCY_CODE" VARCHAR2(3 CHAR),
"CONTEXT1" VARCHAR2(256 CHAR),
"CONTEXT2" VARCHAR2(256 CHAR),
"CONTEXT3" VARCHAR2(256 CHAR),
"CONTEXT4" VARCHAR2(256 CHAR),
"CONTEXT5" VARCHAR2(256 CHAR),
"CONTEXT6" VARCHAR2(256 CHAR),
"CONTEXT7" VARCHAR2(256 CHAR),
"CONTEXT8" VARCHAR2(256 CHAR),
"CONTEXT9" VARCHAR2(256 CHAR),
"CONTEXT10" VARCHAR2(256 CHAR),
"CONTEXT11" VARCHAR2(256 CHAR),
"CONTEXT12" VARCHAR2(256 CHAR),
"CONTEXT13" VARCHAR2(256 CHAR),
"CONTEXT14" VARCHAR2(256 CHAR),
"CONTEXT15" VARCHAR2(256 CHAR),
"CONTEXT16" VARCHAR2(256 CHAR),
"CONTEXT17" VARCHAR2(256 CHAR),
"CONTEXT18" VARCHAR2(256 CHAR),
"CONTEXT19" VARCHAR2(256 CHAR),
"CONTEXT20" VARCHAR2(256 CHAR),
"ATTRIBUTE1" VARCHAR2(256 CHAR),
"ATTRIBUTE2" VARCHAR2(256 CHAR),
"ATTRIBUTE3" VARCHAR2(256 CHAR),
"ATTRIBUTE4" VARCHAR2(256 CHAR),
"ATTRIBUTE5" VARCHAR2(256 CHAR),
"ATTRIBUTE6" VARCHAR2(256 CHAR),
"ATTRIBUTE7" VARCHAR2(256 CHAR),
"ATTRIBUTE8" VARCHAR2(256 CHAR),
"ATTRIBUTE9" VARCHAR2(256 CHAR),
"ATTRIBUTE10" VARCHAR2(256 CHAR),
"ATTRIBUTE11" VARCHAR2(256 CHAR),
"ATTRIBUTE12" VARCHAR2(256 CHAR),
"ATTRIBUTE13" VARCHAR2(256 CHAR),
"ATTRIBUTE14" VARCHAR2(256 CHAR),
"ATTRIBUTE15" VARCHAR2(256 CHAR),
"ATTRIBUTE16" VARCHAR2(256 CHAR),
"ATTRIBUTE17" VARCHAR2(256 CHAR),
"ATTRIBUTE18" VARCHAR2(256 CHAR),
"ATTRIBUTE19" VARCHAR2(256 CHAR),
"ATTRIBUTE20" VARCHAR2(256 CHAR),
"CREATED_BY" VARCHAR2(40 CHAR) NOT NULL ENABLE,
"CREATED_DATE" TIMESTAMP (6) NOT NULL ENABLE,
"LAST_MODIFIED_BY" VARCHAR2(40 CHAR) NOT NULL ENABLE,
"LAST_MODIFIED_DATE" TIMESTAMP (6) NOT NULL ENABLE,
"REFUND_TYPE" VARCHAR2(40 CHAR),
"TAX_COUNTRY" VARCHAR2(3 CHAR),
"PHYSICAL_ITEM_DELIVERY_DATE" TIMESTAMP (6),
"PRODUCT_SERIAL_NUMBER" VARCHAR2(256 CHAR),
"REPLACED_PRODUCT" NUMBER(1,0),
"PROMO_CODE" VARCHAR2(256 CHAR),
"PROMO_CODE_VERSION" VARCHAR2(256 CHAR),
"ORIGINAL_ENTITLEMENT_END_DATE" TIMESTAMP (6),
"ORIGINAL_AMOUNT" VARCHAR2(256 CHAR),
"LINE_ITEM_TYPE" VARCHAR2(40 CHAR),
"EXCHANGE_RATE" VARCHAR2(256 CHAR),
"REV_ATTRIBUTION" VARCHAR2(1024 CHAR),
 PRIMARY KEY ("ITEM_INTF_ID")
)

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi -

Can you please share your full configuration for the entire connector. This error is from the Oracle OCI (thick client) API, which is known to have a 32-bit signed (2GB) limit on data fetches, unlike Oracle JDBC (thin client) API which is 64-bit signed (9223PB). Because your table has an extremely large number of varchar fields, the actual row size paired with the fetch size used is causing a buffer overflow in the Oracle driver.

Thanks,
-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/bbc9f932-3a93-4226-90d7-2edf0a7a5ef8n%40googlegroups.com.

[Shahul Nagoorkani]

Hi Chris,

Yeah seems like the thick client that XStreams connector use may be the issue here. But I even tried with incremental chunk size set to 1 and fetch size set to 10, the connector still failed.

Shared both the kafka connector as well as kafkaconnect configuration. We gave lots of memory/cpu to the pods.

Regards,

Shahul Nagoorkani

[Chris Cranford]

Hi -

Can you please try by removing `driver.defaultRowPrefetch` and use `query.fetch.size` and `incremental.snapshot.chunk.size` only to control the size of the data set. When you set the value at the driver level, this will be used by the OCI thick client.  

To look at the math:

    60,000 rows at 86,000 bytes/row = 5.16GB (which is greater than the 2.1GB limit)

-cc

To view this discussion visit https://groups.google.com/d/msgid/debezium/595572ea-40d1-400e-b4ae-cd458aecb2bcn%40googlegroups.com.

[Shahul Nagoorkani]

Thanks for all your responses on a weekend, Chris. 

Disabled:

#driver.defaultRowPrefetch: 60000

Reduced them to following helped with the connector recovery and performing CDC at this point.

# Incremental snapshot tuning

incremental.snapshot.chunk.size: 1

snapshot.fetch.size: 10

query.fetch.size: 10

 But inserting a new signal to trigger incremental signal doesn't seems to getting picked up and throwing messages like the following but don't see any counts increasing in the signal table:

Received request to open window with id = '677ad350-3799-4c7a-b777-9c02dd002dc5-open', expected = '5a1e3717-96d9-45f2-83aa-96cbaae7214a', request ignored

Any ideas? Thanks again.

Regards,

Shahul Nagoorkani

[Shahul Nagoorkani]

Hi Chris,

With the connector being down for last two days, it had to do the catch work before getting to the signals and when the capture became current without any lags, we notice the signal got picked up with incremental snapshot chunk size 1 and proceeding further. 

Will try to increase the chunk size later when the incremental load for this table completes.

Regards,

Shahul Nagoorkani