PostgreSQL source connector - behaviour when replication slot is removed

Vladyslav Lysenko

unread,

Oct 24, 2025, 6:26:48 AMOct 24

to debezium

Hello!

Have a question about the case:

1. Connector and replication slot are active

2. Connector is turned off (for example because of k8s rearranges pods)

3. While connector is turned off - postgres replication slot for connector is removed for some reason

4. Connector starts again

-> In this case connector creates new replication slot and silently starts. There is a message in logs:

Creating replication slot with command CREATE_REPLICATION_SLOT "debezium" LOGICAL pgoutput ( FAILOVER )

But there is a problem:

IF connector has snapshot.mode=initial - some messages might be lost (which were added to db in period when connector was )

IF connector has snapshot.mode=when_needed - all messages from db table will be published again.

Ideal scenario which I want to achieve:

"Connector starts again and fails (f.e because of some config parameter for that case), then I delete from db table records that were already published to kafka and start connector with when_needed -> no messages lost and no duplicates in kafka topics.

But I have't found any parameter that fails the container in such case.

Do you have any idea for that case?
Thanks.

Chris Cranford

unread,

Oct 28, 2025, 2:42:01 AMOct 28

to debe...@googlegroups.com

Hi,

What version of Debezium are you working with, because the latest builds should already have this baked in.

Now, when connectors start and offsets exist but the log position check validation fails, for PostgreSQL means that the slot does not exist or the offset position is before the lowest position in the replication slot, the connector will fail to start and throws an error when using `initial`. You should see an error that reads:

The connector is trying to read change stream starting at <position>, but this is no longer available on the server.
Reconfigure the connector to use a snapshot mode when needed.

If you are using when needed, this error is not thrown but instead the offsets are reset, and the snapshot is retaken as you described.

Thanks,
-cc

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/cd6d113e-af95-4256-a39d-2dfd6c2d50c0n%40googlegroups.com.

Vladyslav Lysenko

unread,

Oct 28, 2025, 8:54:51 AMOct 28

to debezium

I use 3.1.2 connector version.
This version is the latest there https://www.confluent.io/hub/debezium/debezium-connector-postgresql - I use `confluentinc/cp-kafka-connect-base` as a base image

вівторок, 28 жовтня 2025 р. о 08:42:01 UTC+2 Chris Cranford пише:

Vladyslav Lysenko

unread,

Oct 28, 2025, 11:22:03 AMOct 28

to debezium

I've checked logs:
for `initial` snapshot mode, after restart I see:
`WARN [postgres-payment-service|task-0] The connector is trying to read redo log starting at PostgresOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.postgresql.Source:STRUCT}, sourceInfo=source_info[server='local'db='platform_payment_service', lsn=LSN{10C/924EC930}, txId=731544, messageType=UPDATE, lastCommitLsn=LSN{10C/924EC930}, timestamp=2025-10-24T10:05:39.927559Z, snapshot=FALSE, schema=, table=], lastSnapshotRecord=false, lastCompletelyProcessedLsn=LSN{10C/924EC930}, lastCommitLsn=LSN{10C/924EC930}, streamingStoppingLsn=null, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], incrementalSnapshotContext=IncrementalSnapshotContext [windowOpened=false, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]], but this is no longer available on the server. Reconfigure the connector to use a snapshot when needed if you want to recover. If not the connector will streaming from the last available position in the log (io.debezium.connector.common.BaseSourceTask:147)`

for `when_needed` snapshot mode, after restart I see: `Snapshot ended with SnapshotResult [status=COMPLETED` In both cases connector starts successfully. For initial some messages might be lost. But I need to have task error with the initial mode in this case.

вівторок, 28 жовтня 2025 р. о 14:54:51 UTC+2 Vladyslav Lysenko пише:

Chris Cranford

unread,

Oct 28, 2025, 2:56:02 PMOct 28

to debe...@googlegroups.com

Hi -

In Debezium v3.1.x it logged a warning, and in v3.2.x we changed it to throw an exception. If possible, transition to v3.3.1.Final to get all the latest improvements.

-cc