WARNING: still waiting for all required WAL segments to be archived

[Bob Van]

Hello, after getting barman setup with streaming connection on ubuntu servers, I ran a backup of a small dev db which started but seemed to run indefinitley. There's a message that I can cancel safely, but there's no cancel command, so I did ctrl + c.

The 'barman check' is all ok so am not sure why the log says it's waiting for segments and to check my archive command. I'm not using wal file archive command, just streaming.

2024-01-11 00:28:02,395 [198399] barman.wal_archiver INFO: No xlog segments found from streaming for pg.
2024-01-11 00:28:15,013 [197196] barman.backup_executor INFO: WARNING:  still waiting for all required WAL segments to be archived (7680 seconds elapsed)
2024-01-11 00:28:15,013 [197196] barman.backup_executor INFO: HINT:  Check that your archive_command is executing properly.  You can safely cancel this backup, but the database backup will not be usable without all the WAL segments.

[Abhijit Menon-Sen]

On Thu, Jan 11, 2024 at 9:31 PM Bob Van <overm...@gmail.com> wrote:
>
> The 'barman check' is all ok so am not sure why the log says it's waiting for segments and to check my archive command.

The check results aren't going to tell you if a particular backup is
OK or not, but you can use "barman show-backup <server> latest" or
similar to see the backup status.

> I'm not using wal file archive command, just streaming.

Well, then is streaming working and actually archiving WAL? To
complete a backup, you need both a copy of the data directory (which
your "barman backup" did create) and a copy of the WAL segments
generated during the process, which Barman is not yet convinced about
(hence, WAITING_FOR_WALS).

-- Abhijit

[Martin Marques]

Hello,

> > I'm not using wal file archive command, just streaming.
>
> Well, then is streaming working and actually archiving WAL? To
> complete a backup, you need both a copy of the data directory (which
> your "barman backup" did create) and a copy of the WAL segments
> generated during the process, which Barman is not yet convinced about
> (hence, WAITING_FOR_WALS).

I would add to what Abhijit says here that you need to make sure that
`barman cron` is running. Else you end up with a pile of WAL files in
the `streaming` directory which are never archived to the final
destination.

[Bob Van]

Thanks for the response. I wasn't able to get back to it until today. I setup a 2nd config for a smaller db, since I thought the backup might be hanging up. I ran 'barman cron' and now it says:

Starting WAL archiving for server intata
Starting WAL archiving for server pg

Now when i trigger backups, they finish without hanging...

barman@SRV:~$ barman switch-xlog --force --archive intata
The WAL file 000000010000002900000057 has been closed on server 'intata'
Waiting for the WAL file 000000010000002900000057 from server 'intata' (max: 30 seconds)
Processing xlog segments from streaming for intata
        000000010000002900000057


barman@SRV:~$ barman switch-xlog --force --archive pg
The WAL file 000000010000002900000058 has been closed on server 'pg'
Waiting for the WAL file 000000010000002900000058 from server 'pg' (max: 30 seconds)
Processing xlog segments from streaming for pg
        000000010000002900000058

Check for latest returns an error:

barman@SRV:~$ barman show-backup pg latest
ERROR: Unknown backup 'latest' for server 'pg'

I assume backups are working now, but there's a learning curve on how to use and manage via barman. Now when I run check I get:

Server pg:
        PostgreSQL: OK
        superuser or standard user with backup privileges: OK
        PostgreSQL streaming: FAILED (connection to server at "192.168.1.251", port 5432 failed: FATAL:  number of requested standby connections exceeds max_wal_senders (currently 2))
        wal_level: OK
        replication slot: OK
        directories: OK
        retention policy settings: OK
        backup maximum age: OK (no last_backup_maximum_age provided)
        backup minimum size: OK (0 B)
        wal maximum age: OK (no last_wal_maximum_age provided)
        wal size: OK (0 B)
        compression settings: OK
        failed backups: FAILED (there are 1 failed backups)
        minimum redundancy requirements: OK (have 0 backups, expected at least 0)
        pg_basebackup: OK
        pg_basebackup compatible: FAILED (PostgreSQL version: None, pg_basebackup version: 16.1)
        pg_basebackup supports tablespaces mapping: OK
        systemid coherence: OK
        pg_receivexlog: OK
        pg_receivexlog compatible: FAILED (PostgreSQL version: None, pg_receivexlog version: 16.1)
        receive-wal running: OK
        archiver errors: OK

I'm guessing maybe an open repl slot needs to be available in order to run checks, so can increase it.

How would I stop or cancel a backup as the earlier error said?

[Bob Van]

It seems I didn't trigger backups but ran the 'verify wal archiving' command. I increased repl slots to 4 so can run the barman check now.

I used 'barman delete' to delete the bad backup, so that message went away.

I try to recheck streaming connection and it prompts for a password which doesn't make sense because it was working and the barman check passes.

~$ psql -U streaming_barman -h 192.168.1.251 -c "IDENTIFY_SYSTEM" replication=1
Password for user streaming_barman:

~$ barman check pg

Server pg:
        PostgreSQL: OK
        superuser or standard user with backup privileges: OK

        PostgreSQL streaming: OK

        wal_level: OK
        replication slot: OK
        directories: OK
        retention policy settings: OK
        backup maximum age: OK (no last_backup_maximum_age provided)
        backup minimum size: OK (0 B)
        wal maximum age: OK (no last_wal_maximum_age provided)
        wal size: OK (0 B)
        compression settings: OK

        failed backups: OK (there are 0 failed backups)

        minimum redundancy requirements: OK (have 0 backups, expected at least 0)
        pg_basebackup: OK

        pg_basebackup compatible: OK

        pg_basebackup supports tablespaces mapping: OK
        systemid coherence: OK
        pg_receivexlog: OK

        pg_receivexlog compatible: OK

        receive-wal running: OK
        archiver errors: OK

I saw that there are no backups so ran one, but it doesn't seem to finish:

~$ barman backup intata
Starting backup using postgres method for server intata in /var/lib/barman/intata/base/20240116T143420
Backup start at LSN: 29/6036A0D0 (000000010000002900000060, 0036A0D0)
Starting backup copy via pg_basebackup for 20240116T143420

2024-01-16 14:34:02,196 [279795] barman.wal_archiver INFO: No xlog segments found from streaming for intata.
2024-01-16 14:34:02,247 [279796] barman.wal_archiver INFO: No xlog segments found from streaming for pg.

2024-01-16 14:34:20,268 [279809] barman.backup INFO: Starting backup using postgres method for server intata in /var/lib/barman/intata/base/20240116T143420
2024-01-16 14:34:20,280 [279809] barman.backup_executor INFO: Backup start at LSN: 29/6036A0D0 (000000010000002900000060, 0036A0D0)
2024-01-16 14:34:20,280 [279809] barman.backup_executor INFO: Starting backup copy via pg_basebackup for 20240116T1434202024-01-16 14:34:20,354 [279809] barman.backup_executor INFO: pg_basebackup: initiating base backup, waiting for checkpoint to complete
2024-01-16 14:34:20,478 [279358] barman.command_wrappers INFO: intata: pg_receivewal: finished segment at 29/61000000 (timeline 1)
2024-01-16 14:34:20,478 [279360] barman.command_wrappers INFO: pg: pg_receivewal: finished segment at 29/61000000 (timeline 1)
2024-01-16 14:34:20,545 [279809] barman.backup_executor INFO: pg_basebackup: checkpoint completed
2024-01-16 14:35:01,992 [279823] barman.wal_archiver INFO: Found 1 xlog segments from streaming for intata. Archive all segments in one run.
2024-01-16 14:35:01,992 [279823] barman.wal_archiver INFO: Archiving segment 1 of 1 from streaming: intata/000000010000002900000060
2024-01-16 14:35:02,056 [279824] barman.wal_archiver INFO: Found 1 xlog segments from streaming for pg. Archive all segments in one run.
2024-01-16 14:35:02,056 [279824] barman.wal_archiver INFO: Archiving segment 1 of 1 from streaming: pg/000000010000002900000060
2024-01-16 14:35:26,082 [279360] barman.command_wrappers INFO: pg: pg_receivewal: finished segment at 29/62000000 (timeline 1)
2024-01-16 14:35:26,082 [279358] barman.command_wrappers INFO: intata: pg_receivewal: finished segment at 29/62000000 (timeline 1)
2024-01-16 14:35:31,991 [279809] barman.backup_executor INFO: NOTICE:  base backup done, waiting for required WAL segments to be archived
2024-01-16 14:36:02,028 [279834] barman.wal_archiver INFO: Found 1 xlog segments from streaming for intata. Archive all segments in one run.
2024-01-16 14:36:02,028 [279834] barman.wal_archiver INFO: Archiving segment 1 of 1 from streaming: intata/000000010000002900000061
2024-01-16 14:36:02,065 [279835] barman.wal_archiver INFO: Found 1 xlog segments from streaming for pg. Archive all segments in one run.
2024-01-16 14:36:02,065 [279835] barman.wal_archiver INFO: Archiving segment 1 of 1 from streaming: pg/000000010000002900000061
2024-01-16 14:36:26,043 [279809] barman.backup_executor INFO: WARNING:  still waiting for all required WAL segments to be archived (60 seconds elapsed)
2024-01-16 14:36:26,043 [279809] barman.backup_executor INFO: HINT:  Check that your archive_command is executing properly.  You can safely cancel this backup, but the database backup will not be usable without all the WAL segments.

2024-01-16 14:36:41,255 [279839] barman.wal_archiver INFO: No xlog segments found from streaming for intata.
2024-01-16 14:36:41,305 [279840] barman.wal_archiver INFO: No xlog segments found from streaming for pg.

For the running backup that doesn't seem to finish, a ctrl + c seems to have killed it, but makes the backup bad.

^CERROR: Backup failed copying files.
DETAILS: KeyboardInterrupt

I ran 'barman cron' again since have no way of knowing if it was still running or if i need to have a console session running just for that.

 barman cron

Starting WAL archiving for server intata
Starting WAL archiving for server pg

There should be a sort of usage document for steps to go through once the set up is done.