PostgreSQL 12.17 Replication Slot Lag Persistent Growth

[Leon Kozlowski]

Configuration:
PostgreSQL 12.17
Debezium 2.1.1.Final (Using Kinesis Connector)
Offset Store: Redis

We have been running CDC replication with Debezium on this database for over 1 year, however as of recently, the replication slot lag has grown to over 2TB and doesn't seem to be trending down much.

We have tried tuning the `max.bach.size` and `max.queue.size` parameters to keep up with the volume of the database, but this has not helped in combatting the replication slot lag growth.

The database is relatively high volume, with 8 million new records per day across all tables we are replicating.

This is a 8 month graph of the replication slot lag over time

We can see the slot lag growing since mid Feb - as we zoom in we can see hourly growth below

We have tried increasing `work_mem` on the database to allocate more memory to WAL process, however nothing has yielded results as of yet.

Looking for any guidance on tuning/techniques to combat this issue.

Thanks,
Leon