Barman's use of pg_basebackup and the WAITING_FOR_WALS state

[JK Laiho]

Hi all,

I'd like to confirm my understanding of how Barman operates pg_basebackup.

I've just started doing streaming backups and WAL streaming of a low-traffic Postgres 17 cluster. I took my second ever full backup today with barman backup prod-db and got the warning about the backup being in WAITING_FOR_WALS state. After 5-10 minutes and with barman cron running every minute, it was still in that state. I then ran barman switch-wal, and the barman cron run following that finally made the backup show as being completed.

pg_basebackup defaults to -X stream, which should handle transmission of WALs accumulated during the backup over a second streaming connection. But the docs for the command state that using -X none on a low-traffic server can exhibit delays due to a WAL segment filling up slowly, which seems to match what happened for me. So does Barman use -X none, relying on pg_receivewal to eventually fill in the needed WALs? I guess this would make sense, since otherwise I guess you'd have two connections streaming WALs from the same source: pg_receivewal and the second connection that pg_basebackup -X stream opens, which could be problematic(?)

Do I have this right? If yes, I'll add barman switch-wal to my cron job after barman backup to make sure that the WAITING_FOR_WALS state doesn't take too long. During quiet periods, it could easily take hours for the WAL switch to happen on its own.

-- JK

[Israel Barth]

Hi JK,

Yes, your understanding is correct.

When backups are taken using backup_method = postgres, Barman invokes pg_basebackup with the --wal-method=none option (i.e., -X none). You can see this in the source code here:
https://github.com/EnterpriseDB/barman/blob/master/barman/command_wrappers.py#L962-L969.

In this setup, Barman relies on either the archive_command or pg_receivewal (depending on your configuration) to archive WAL files from the Postgres server to the Barman server.

Periodically, barman cron runs and checks whether all the WALs required for backup consistency have been received. Once they are, Barman changes the backup status from WAITING_FOR_WALS to DONE (see https://github.com/EnterpriseDB/barman/blob/master/barman/backup.py#L1847-L1852).

As you noticed, if there's little or no activity on the PostgreSQL server, it can take some time for the final WAL segment (needed to complete the backup) to be switched, delaying the status update in Barman.

If you prefer to complete the backup sooner, it’s safe to manually run barman switch-wal at the end of your backup process. This can help ensure the backup is marked as DONE without waiting for natural WAL activity in Postgres.

Best regards,

Israel.