archive command failed with exit code 2?

[Bob Van]

Hello, after getting basic streaming archiving & backup working I added rsync/ssh archive_command with 'barman-wal-archive' as recommended.

Backup mostly works from 'Backup start at LSN' / 'Starting backup copy via rsync/SSH' through 8 'copy steps' and finishing with 'Copy finished' / 'Copy done', but then hangs on 'Asking PostgreSQL server to finalize the backup'.

The barman log has:

ERROR: Error executing ssh: [Errno 32] Broken pipe
python3.10/tarfile.py self.fileobj.write(self.buf)
ValueError: write to closed file
archive command failed with exit code 2
The failed archive command was: barman-wal-archive backup pg pg_wal/000000010000002A00000099

My barman server has 2 database configs. One for postgres 'pg' and one for a test 'db1'.
My backup is of 'db1', not 'pg', but I assume any dbase backup uses that 'pg' connection in barman.
Is this correct? That I can backup a particular database on the server, or should I just do 'postgres' which includes all dbs on the server?
The database server has no reference to 'pg' so am not sure if that's the issue.
What have I setup wrong or how can I troubleshoot this error?

[Luca Ferrari]

On Mon, Feb 12, 2024 at 11:04 AM Bob Van <overm...@gmail.com> wrote:
> The barman log has:
>
> ERROR: Error executing ssh: [Errno 32] Broken pipe
> python3.10/tarfile.py self.fileobj.write(self.buf)
> ValueError: write to closed file
> archive command failed with exit code 2

Look at the postgresql log, they are probably telling you which
command is failing, and you have to execute to understand what is
happening.
Most probably is something wrong with the ssh authentatication.

Luca

[Bob Van]

Hello, I responded to my own question but it was blocked by moderator. The issue is that I needed to replace 'barman' in the archive command with the server IP, an easy one for barman experts. And there's the 'dummy' test command which should be used to verify before testing. I had additional errors so had to add 'archiver=on' to server conf, and 'backup_option' to barman conf. Then with 'verify backup', I got 'backup_manifest no such file' so found I could generate manifest, then verify. It's not clear yet why it just doesn't generate as part of the backup process, but might figure it out at some point.

[Bob Van]

After reviewing docs and other examples I see that 'backup' in the command isn't known and needs to be replaced by the barman server ip address and that I needed to test the connection first with 'dummy':

  postgres@dev:~$
  barman-wal-archive --test 192.168.1.250 pg DUMMY
  Ready to accept WAL files for the server pg

So I updated the archive_command in postgres conf and backups with ssh/streaming are working now.


After this, 'barman check' has a new 'fail':

  empty incoming directory: FAILED ('/var/lib/barman/pg/incoming' must be empty when archiver=off)

The docs refer to 'archiver' and 'archiver = on' for new versions, but mine isn't on or off but I went ahead and added the lines to my server conf files and the 'fail' went away in the barman checks.

Also for 'WARNING: No backup strategy set for server ...', it's not clear where to add the 'backup_option' but an old post mentions 'setting' in 'global/server configuration'. I took this to mean the main 'barman.conf' conf, which doesn't have the line, so I added it which seems to fix it now.


Now that a backup over ssh completes, and my 'verify' util is fixed, I get an error that 'backup_manifest' doesn't exist, which it doesn't. I read how 'backup_manifest' was adding in psql13, so I should have it, so would appear that pg_basebackup isn't working right, which isn't a barman issue.

barman verify-backup db1 latest

Verifying backup 'latest' on server ['db1']
Calling pg_verifybackup
ERROR: verify backup failure on directory '/var/lib/barman/db1/base/20240212T193258/data'
ERROR: pg_verifybackup: error: could not open file "/var/lib/barman/db1/base/20240212T193258/data/backup_manifest": No such file or directory

[Luca Ferrari]

On Wed, Feb 21, 2024 at 3:13 PM Bob Van <overm...@gmail.com> wrote:
>
> After reviewing docs and other examples I see that 'backup' in the command isn't known and needs to be replaced by the barman server ip address and that I needed to test the connection first with 'dummy':
>
> postgres@dev:~$
> barman-wal-archive --test 192.168.1.250 pg DUMMY
> Ready to accept WAL files for the server pg
>

This is a DNS/lookup problem. I told you to test the exact command
PostgreSQL was executing, and which is _always_ reported in the logs
in case of failure, so you would have figured out immediately.

> After this, 'barman check' has a new 'fail':
>
> empty incoming directory: FAILED ('/var/lib/barman/pg/incoming' must be empty when archiver=off)
>

Sounds to me you played with archive on/off, but I've never
experienced this error so far.

> Also for 'WARNING: No backup strategy set for server ...', it's not clear where to add the 'backup_option' but an old post mentions 'setting' in 'global/server configuration'. I took this to mean the main 'barman.conf' conf, which doesn't have the line, so I added it which seems to fix it now.
>

This is a warning, not an error. From the docs
<https://docs.pgbarman.org/release/3.10.0/>:
"Barman will now use concurrent backups if neither concurrent_backup
nor exclusive_backup are specified in backup_options. "
so you should not have to set up such variable at all.

>
> Now that a backup over ssh completes, and my 'verify' util is fixed, I get an error that 'backup_manifest' doesn't exist, which it doesn't. I read how 'backup_manifest' was adding in psql13, so I should have it, so would appear that pg_basebackup isn't working right, which isn't a barman issue.
>

Uhm...strange. Any chance you can nuke the backup repository and start
over on a fresh environment to see what happens?
Which barman and PostgreSQL are you using?

Luca

[Kashif Hafeez]

I had the similar problem which I managed to fix today.

It's to do with ssh connectivity.

To fix this problem you need to add barman user in the /etc/ssh/sshd_config files.

Example:

[root@barman ~]# grep AllowGroup /etc/ssh/sshd_config
AllowGroups postgres root Fabric barman

Kashif

--
--
You received this message because you are subscribed to the "Barman for PostgreSQL" group.
To post to this group, send email to pgba...@googlegroups.com
To unsubscribe from this group, send email to
pgbarman+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---
You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/CAKoxK%2B4hOncBaGzePmxUeGKWfioAXhUXVXa_7GhUc2iBnEXVUw%40mail.gmail.com.

[Bob Van]

No I mean after I posted the question, I researched more and found how to fix as well as test the command, so re-posted my own answer, but it was delayed.

I don't know about dns lookup but don't think the server would be found by 'barman' so used the IP. I suppose FQDN server name could work, if all the dns is set up right, but am ok using ip at this stage.