Postgres txlogs are not cleaned up

[Anoop Krishnakumar]

Hello,

My application runs in AWS and I am using debezium to stream change events from Postgres RDS instance. After starting postgres connector the tx logs are not getting cleaned resulting in increased disk usage (monitored for 4 days). If I stop the connector and drop the replication slot the txlogs gets deleted and disk usage goes down. Am I am missing anythings that is resulting in this behaviour?

The configuration file is :

name=testdb
connector.class=io.debezium.connector.postgresql.PostgresConnector
tasks.max=1
plugin.name=wal2json_rds
database.hostname=testhost
database.port=5432
database.dbname=test_db
database.user=user_replication
database.password=test123
database.server.name=test
table.whitelist=public.audit_(.*)
column.blacklist=public.audit(.*).full_log
snapshot.mode=never
transforms=Reroute
transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter
transforms.Reroute.topic.regex=(.*)audit_(.*)
transforms.Reroute.topic.replacement=$1audit

-anoop

[Jiri Pechanec]

Hi,

what are the numbers you have in pg_current_xlog_flush_location(), pg_current_xlog_insert_location() and pg_current_xlog_location() and in pg_replication_slots?

Thanks

J.

[Anoop Krishnakumar]

Postgres version is 10.3

testdb=> select pg_current_wal_flush_lsn();

 pg_current_wal_flush_lsn
--------------------------
 99/E3000060
(1 row)

testdb=> select pg_current_wal_insert_lsn();
 pg_current_wal_insert_lsn
---------------------------
 99/E303D318
(1 row)

testdb=> select pg_current_wal_lsn();
 pg_current_wal_lsn
--------------------
 99/E303D350
(1 row)

testdb=> SELECT *  FROM pg_replication_slots;
 slot_name |  plugin  | slot_type | datoid | database | temporary | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn
-----------+----------+-----------+--------+----------+-----------+--------+------------+------+--------------+-------------+---------------------
 debezium  | wal2json | logical   |  16390 | testdb | f         | t      |       3084 |      |      2335327 | 99/E20D69F0 | 99/E20E5C79

-anoop

[Jiri Pechanec]

Could you please monitor those values over some time? Is is possible you monitor a low traffic database and there are also high-traffic databases on the same postgres instance?

Thanks

J.

[Anoop Krishnakumar]

Sure will monitor the values. Currently I don't have high traffic instance?

-anoop

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To post to this group, send email to debe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/38fadfbf-7945-48f5-9a07-922d5089830c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ori Popowski]

Hi,

We are experiencing the same issue. I've opened a bug here: https://issues.jboss.org/browse/DBZ-926

For now, our disk space is constantly increasing and it causes downtime.