Postgres | Replication Slot | WAL Size | How to re-start it from last point

[Mukesh Kumar]

Hello All,

I am using Postgres Debezium Source Connector for the source database on AWS RDS. Sometimes, my server ( the one hosting Debezium) shutdown and the replication slot size grows. To avoid this, I delete the replication slots so I do not run into storage issues as this setup is currently in dev only. 

I need to know how this should be approached once I put this thing in prod, as deleting the replication slot is just a workaround.

Another query is, if I create a new source connector, how can I ensure it should take changes from the time when the last connector left?

Regards

Mukesh

[Chris Cranford]

Hi,

In production, you ideally want to set-up some type of retry/back-off error handling in Kafka Connect or utilize some monitoring mechanism to identify when a connector is not active and address the problem within a reasonable time to avoid WAL issues.  Once you restart the connector, Debezium will read from where we left off and the WAL size will begin to shrink as the connector catches up.

To your last question, this really depends on several of factors.  For example, if you don't remove the offsets from the offset topic and redeploy the connector with the same name, in Kafka Connect the connector will simply start based on the position in the offsets.  If you redeploy with a new connector name, then the offset lookup will determine there are no offsets for that connector name and will be treated as if its a brand new connector.

Chris

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/4e2e327a-e1bb-4b72-8a1f-031060e30858n%40googlegroups.com.