BARMAN slow archiving of WALS on NFS

[Domagoj Batinic]

Hello,

I am using barman 3.3 with postgres 14. For replication I am using repmgr.

I am backuping ever growing DB.

Everything is working OK, but as DB grows I can see that backups more often and often are WAITING FOR WALLS.

[barman ~]$ barman list-backup Zabbix-DB
Zabbix-DB 20240127T033005 - Sat Jan 27 07:31:02 2024 - Size: 832.4 GiB - WAL Size: 0 B - WAITING_FOR_WALS
Zabbix-DB 20240126T033005 - Fri Jan 26 07:14:36 2024 - Size: 738.0 GiB - WAL Size: 0 B - WAITING_FOR_WALS
Zabbix-DB 20240125T033005 - Thu Jan 25 07:15:20 2024 - Size: 835.8 GiB - WAL Size: 0 B - WAITING_FOR_WALS
Zabbix-DB 20240124T033004 - Wed Jan 24 07:09:01 2024 - Size: 852.9 GiB - WAL Size: 1.1 TiB
Zabbix-DB 20240123T033003 - Tue Jan 23 05:14:17 2024 - Size: 753.9 GiB - WAL Size: 1.1 TiB
Zabbix-DB 20240116T033007 - Tue Jan 16 09:17:22 2024 - Size: 1.4 TiB - WAL Size: 957.9 GiB
Zabbix-DB 20240115T033006 - Mon Jan 15 08:55:26 2024 - Size: 1.4 TiB - WAL Size: 418.4 GiB
Zabbix-DB 20240114T033007 - Sun Jan 14 09:11:45 2024 - Size: 1.4 TiB - WAL Size: 147.9 GiB
Zabbix-DB 20240113T033006 - Sat Jan 13 08:28:28 2024 - Size: 1.4 TiB - WAL Size: 1.4 TiB

The backup at the some point receives all the WALs.

Here is more info:

[barman@lxnmszbxdbv3p ~]$ barman check Zabbix-DB
Server Zabbix-DB:
        PostgreSQL: OK
        superuser or standard user with backup privileges: OK
        wal_level: OK
        PostgreSQL server is standby: OK
        Primary server is not a standby: OK
        Primary and standby have same system ID: OK
        directories: OK
        retention policy settings: OK
        backup maximum age: OK (no last_backup_maximum_age provided)
        backup minimum size: OK (832.4 GiB)
        wal maximum age: OK (no last_wal_maximum_age provided)
        wal size: OK (0 B)
        compression settings: OK
        failed backups: OK (there are 0 failed backups)
        minimum redundancy requirements: OK (have 6 backups, expected at least 0)
        ssh: OK (PostgreSQL server)
        systemid coherence: OK
        archive_mode: OK
        archive_command: OK
        continuous archiving: OK
        archiver errors: OK

barman@lxnmszbxdbv3p barman.d]$ cat Zabbix-DB.conf
[Zabbix-DB]
description = Zabbix-DB
#DB1 192.168.251.4 je trenutni standby sa kojeg uzimamo backup
ssh_command = ssh -q  post...@192.168.251.4
conninfo = host=192.168.251.4 user=postgres port=5432
#DB2 192.168.251.5 je trenutni primary na kojem je VIP 192.168.251.7
primary_conninfo = host=192.168.251.5 user=postgres port=5432
retention_policy_mode = auto
retention_policy = RECOVERY WINDOW OF 14 days
wal_retention_policy = main
archiver = on
backup_method = rsync
reuse_backup = link
backup_options = concurrent_backup
incoming_wals_directory = /var/lib/barman/Zabbix-DB/incoming/

In barman logs you can see differance in received and archived log:

2024-01-27 12:52:02,197 [247395] barman.server INFO: Another cron process is already running on server Zabbix-DB. Skipping to the next server
2024-01-27 12:52:02,200 [247394] barman.server INFO: Another archive-wal process is already running on server Zabbix-DB. Skipping to the next server
2024-01-27 12:52:15,354 [2668] barman.wal_archiver INFO: Archiving segment 4205 of 6117 from file archival: Zabbix-DB/000000120000B6C600000004
2024-01-27 12:52:40,563 [2668] barman.wal_archiver INFO: Archiving segment 4206 of 6117 from file archival: Zabbix-DB/000000120000B6C600000005
2024-01-27 12:52:59,282 [247455] barman.server INFO: Received file '000000120000B93D00000003' with checksum '621e726d1df1f5096d535b2e425390c6' by put-wal for server 'Zabbix-DB' (SSH host: 192.168.251.5)

Barman data DIR is mounted and NFS

//172.17.130.13/bxdbv3p_cifs     728T  169T  560T  24% /var/lib/barman/Zabbix-DB

How to speed up WAL archiving?

If I move Baraman on faster storage, would that help?

Regards.

Domagoj