Oracle Debezium Connector cannot startup due to failing incremental snapshot

[Per Dahlén]

This is really URGENT because it is in our production environment! 

Oracle Debezium Connector version 2.5.3.Final.

When I restart the connector it immediately fails with ORA-17041: Missing IN or OUT parameter at index:3.

Check the attached log.

Any tips on workarounds?

[Per Dahlén]

I started an incremental snapshot of the PARTSALESDESCR table at around 11:17Z according to the log:

2024-05-20 11:17:42,461 INFO [vivaldi-cdc-connector|task-0] Requested 'INCREMENTAL' snapshot of data collections '[VIVP.EHDA.PARTSALESDESCR]' with additional conditions '[]' and surrogate key 'PK of table will be used' (io.debezium.pipeline.signal.actions.snapshotting.ExecuteSnapshot) [debezium-oracleconnector-dbzvivaldi-change-event-source-coordinator]

It finished around 13:13Z:

2024-05-20 13:13:42,291 INFO [vivaldi-cdc-connector|task-0] No data returned by the query, incremental snapshotting of table 'VIVP.EHDA.PARTSALESDESCR' finished (io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource) [debezium-oracleconnector-dbzvivaldi-change-event-source-coordinator]

At 14:27Z I initiated an incremental snapshot of the A_HICOHEAD table by issuing the following SQL:

INSERT INTO CDC_DBZ.DBZ_SIGNAL
(
    ID,
    TYPE,
    DATA
)
VALUES
(
    '11',
    'execute-snapshot',
    '{"data-collections":["VIVP.EHDA.A_HICOHEAD"],"type":"incremental"}'
);

According to the log the incremental snapshot started:

2024-05-20 14:27:58,083 INFO [vivaldi-cdc-connector|task-0] Requested 'INCREMENTAL' snapshot of data collections '[VIVP.EHDA.A_HICOHEAD]' with additional conditions '[]' and surrogate key 'PK of table will be used' (io.debezium.pipeline.signal.actions.snapshotting.ExecuteSnapshot) [debezium-oracleconnector-dbzvivaldi-change-event-source-coordinator]

2024-05-20 14:27:58,325 INFO [vivaldi-cdc-connector|task-0] Incremental snapshot for table 'VIVP.EHDA.A_HICOHEAD' will end at position [91999999] (io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource) [debezium-oracleconnector-dbzvivaldi-change-event-source-coordinator]

After close to two hours I can see the following in the log:

2024-05-20 16:24:41,620 INFO [vivaldi-cdc-connector|task-0] 2093031 records sent during previous 01:59:52.998, last recorded offset of {server=dbzvivaldi} partition is {commit_scn=505285677872:0:0086000200564112-010200090021962d-0103000d0021c288-01300001001eefe6,510490745935:1:000a00010162390c, transaction_id=null, incremental_snapshot_correlation_id=11, incremental_snapshot_maximum_key=aced0005757200135b4c6a6176612e6c616e672e4f626a6563743b90ce589f1073296c020000787000000001737200146a6176612e6d6174682e426967446563696d616c54c71557f981284f0300024900057363616c654c0006696e7456616c7400164c6a6176612f6d6174682f426967496e74656765723b787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000737200146a6176612e6d6174682e426967496e74656765728cfc9f1fa93bfb1d030006490008626974436f756e744900096269744c656e67746849001366697273744e6f6e7a65726f427974654e756d49000c6c6f776573745365744269744900067369676e756d5b00096d61676e69747564657400025b427871007e0004fffffffffffffffffffffffefffffffe00000001757200025b42acf317f8060854e0020000787000000004057bceff7878, snapshot_scn=504034060474, incremental_snapshot_collections=[{"incremental_snapshot_collections_id":"VIVP.EHDA.PARTSALESDESCR","incremental_snapshot_collections_additional_condition":null,"incremental_snapshot_collections_surrogate_key":null}], incremental_snapshot_primary_key=aced0005757200135b4c6a6176612e6c616e672e4f626a6563743b90ce589f1073296c020000787000000001737200146a6176612e6d6174682e426967446563696d616c54c71557f981284f0300024900057363616c654c0006696e7456616c7400164c6a6176612f6d6174682f426967496e74656765723b787200106a6176612e6c616e672e4e756d62657286ac951d0b94e08b020000787000000000737200146a6176612e6d6174682e426967496e74656765728cfc9f1fa93bfb1d030006490008626974436f756e744900096269744c656e67746849001366697273744e6f6e7a65726f427974654e756d49000c6c6f776573745365744269744900067369676e756d5b00096d61676e69747564657400025b427871007e0004fffffffffffffffffffffffefffffffe00000001757200025b42acf317f8060854e0020000787000000004017d02d27878, scn=510472376317} (io.debezium.connector.common.BaseSourceTask) [task-thread-vivaldi-cdc-connector-0]

There is something seriously wrong here...

I have attached a file with records from the connect offset topic. Check out the record with offset 163436 in particular...

[Per Dahlén]

We got the connector running again by removing the Kafka Connect cluster, producing a "fake" record to the Kafka Connect offset topic and then start the Kafka Connect cluster and the connector.

Not really something you should do, but it looks like it solved our problem for the moment.

We'll try running blocking snapshots of the newly added tables instead and see if that works better.

//Per Dahlén