Passive Server Keeping Additional Backup

[hu...@whtc.ca]

We backup an approximately 1TB database weekly, retaining 2 base backups and all required WALs. A passive server at another location keeps a copy of the backup.

The primary and passive servers have the same size archive volumes. Recently, the passive server has been reporting excessive disk space usage. Upon investigating, it seems the passive server is keeping 3 backups, and only considers the first two as "valid," with the latest showing a status of syncing. Yet everything seems to be up to date. I can't find anything about this "SYNCING" status in the manuals.

Primary Backup:

barman list-backups mydb
mydb 20220619T060004 - Sun Jun 19 09:57:59 2022 - Size: 822.5 GiB - WAL Size: 69.8 GiB
mydb 20220612T060004 - Sun Jun 12 11:56:02 2022 - Size: 825.4 GiB - WAL Size: 355.5 GiB

barman status mydb
Server mydb:
       Description: REDACTED
       Active: True
       Disabled: False
       PostgreSQL version: 11.13
       Cluster state: in production
       pgespresso extension: Not available
       Current data size: 814.2 GiB
       PostgreSQL Data directory: /var/lib/postgresql/11/main
       Current WAL segment: 0000000200002835000000F1
       PostgreSQL 'archive_command' setting: barman-wal-archive -c /etc/barman.conf -U mcbackup archive.local mydb %p
       Last archived WAL: 0000000200002835000000EF, at Mon Jun 20 17:46:18 2022
       Failures of WAL archiver: 1217 (000000020000262600000059 at Sat May 28 08:00:15 2022)
       Server WAL archiving rate: 213.97/hour
       Passive node: False
       Retention policies: enforced (mode: auto, retention: REDUNDANCY 2, WAL retention: MAIN)
       No. of available backups: 2
       First available backup: 20220612T060004
       Last available backup: 20220619T060004
       Minimum redundancy requirements: satisfied (2/2)

Secondary:
As you can see, the June 5 and 12 backups are considered the valid ones, whereas the primary server only has the June 12 and 19 backups.

barman list-backups mydb        
mydb 20220619T060004 - SYNCING
mydb 20220612T060004 - Sun Jun 12 11:56:02 2022 - Size: 825.4 GiB - WAL Size: 425.2 GiB
mydb 20220605T060003 - Sun Jun  5 09:21:57 2022 - Size: 814.1 GiB - WAL Size: 365.6 GiB

barman status mydb

Server mydb:

       Description: REDACTED
       Active: True
       Disabled: False
       Passive node: True
       Retention policies: enforced (mode: auto, retention: REDUNDANCY 2, WAL retention: MAIN)
       No. of available backups: 2
       First available backup: 20220605T060003
       Last available backup: 20220612T060004
       Minimum redundancy requirements: satisfied (2/2)
       SSH command to primary server: REDACTED

Can anyone tell me why this is happening, and how we can avoid it? We consider having two base backups, and all the WALS to recover from either to be sufficient. We are running barman 2.17 on PostgreSQL 11.13

Thank you,

Hugh Ranalli

[Michael Wallace]

Hi Hugh,

Apologies for not responding to this earlier - this mail ended up in my spam folder for unknown reasons.

The SYNCING status should only ever occur while a backup is being copied from the primary Barman server to the passive Barman server. If that copy process succeeds then the backup should move to a DONE state and if it fails it should move to a FAILED state. If you're seeing the backup remain in the SYNCING state for a long time then either:

1. The sync is ongoing and is taking a long time.

2. The process is completing and somehow the status of the backup is not being updated properly (this would indicate a bug in Barman).

Take a look for a sync-backup process on your passive Barman server, something like:

    $ ps -eaf | grep [s]ync-backup
    barman   11961     1  0 10:57 ?        00:00:00 /usr/bin/python3 /usr/local/bin/barman -c /etc/barman.conf -q sync-backup SERVER_NAME BACKUP_ID

If there is an active sync-backup process then the sync is still running, though perhaps it has stalled - try searching the Barman log file (usually /var/log/barman/barman.log) and look for logs which include the process ID from the above command. Messages at INFO level starting "Synchronising with server" will tell you the stage of syncing the process has reached.

If there is no active process then something has likely gone wrong and the status has not been updated correctly - if that's the case there should be an ERROR level log in the Barman log file related to the sync-backup process. If that's the case then please share the logs here and I can take a deeper look.

Hope this helps,

Mike

--
--
You received this message because you are subscribed to the "Barman for PostgreSQL" group.
To post to this group, send email to pgba...@googlegroups.com
To unsubscribe from this group, send email to
pgbarman+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---
You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/ac5354e6-0f73-41e9-bcc5-ecea5baeea59n%40googlegroups.com.

[hu...@whtc.ca]

Hi Mike,

Thank you very much. This is very helpful. I was away for a few days,  but when I took a look this morning, our secondary server was all in sync with Sunday's backup (of course). Although our primary and secondary archives are in different data centres, we shouldn't have large delays (e.g. days to catch up) between the two. However, perhaps there was some sort of network anomaly on those occasions, creating a backlog. The next time this happens, I will follow the steps you gave, and see what I can find.

Thank you again for responding with such detail; it is greatly appreciated.

All the best,

Hugh