what do the erros upon barman check [server] mean?

[dulh...@mailbox.org]

I am getting a ton of FAILED upon barman check. I can't make anything from this really because it's unclear, what those items refer to. Not even whether a FAIL is caused by the barman backup server or the PostgreSQL server.

:~> barman check pg
Server pg:
WAL archive: FAILED (please make sure WAL shipping is setup)
PostgreSQL: FAILED
directories: OK
retention policy settings: OK
backup maximum age: OK (no last_backup_maximum_age provided)
compression settings: OK
failed backups: OK (there are 0 failed backups)
minimum redundancy requirements: OK (have 0 backups, expected at least 0)
pg_basebackup: OK
pg_basebackup compatible: FAILED (PostgreSQL version: None, pg_basebackup version: 12.6)
systemid coherence: OK (no system Id available)
pg_receivexlog: OK
pg_receivexlog compatible: FAILED (PostgreSQL version: None, pg_receivexlog version: 12.6)
receive-wal running: FAILED (See the Barman log file for more details)
archiver errors: OK

Is there any sort of Glossary which explains these items available anywhere? That would be a starting point for troubleshooting

[dulh...@mailbox.org]

I am still struggling with getting barman going.

barman check pg continues to leave me with WAL archived FAILED still.

:~> barman check pg
Server pg:
WAL archive: FAILED (please make sure WAL shipping is setup)

As far as my understanding goes I can not see anything wrong in my setup.

postgresql.conf

listen_addresses = '*'
max_connections = 100

shared_buffers = 512MB
work_mem = 16MB
dynamic_shared_memory_type = posix
wal_level = replica
checkpoint_timeout = 5min
max_wal_size = 1GB
min_wal_size = 80MB

archive_mode = on

archive_command = 'ssh bar...@barman.server.intern "test ! -f /opt/barman/pg/incoming/%f" && scp %p barman@barman@barman.server.intern:/opt/barman/pg/incoming/%f'

archive_timeout = 60

max_wal_senders = 2
max_replication_slots = 2

log_destination = 'stderr,syslog'

logging_collector = on

log_rotation_size = 0

log_checkpoints = on
log_connections = on
log_disconnections = on
log_hostname = on
log_statement = 'ddl'
log_timezone = 'Europe/Berlin'

track_io_timing = on
track_functions = all

log_autovacuum_min_duration = 100
datestyle = 'iso, dmy'
timezone = 'Europe/Berlin'
lc_messages = 'en_US.UTF-8'
lc_monetary = 'de_DE.UTF-8'
lc_numeric = 'de_DE.UTF-8'
lc_time = 'de_DE.UTF-8'

default_text_search_config = 'pg_catalog.german'

shared_preload_libraries = 'pg_stat_statements'

pg_hba.conf

host replication streaming_barman  10.6.255.209/22  md5
host postgres    barman            10.6.255.209/22  md5

barman configuration

[pg]
description = "Example of PostgreSQL Database (Streaming-Only)"
conninfo = host=postgres.server.intern user=barman dbname=postgres
streaming_conninfo = host=postgres.server.intern user=streaming_barman dbname=postgres
backup_method = postgres
streaming_archiver = on
slot_name = barman
create_slot = auto

All connections test run as they should.

What is it I am obviously missing?

Is there anything I need to do else then barman backup pg on the barman server in order to get this running?

--
--
You received this message because you are subscribed to the "Barman for PostgreSQL" group.
To post to this group, send email to pgba...@googlegroups.com
To unsubscribe from this group, send email to
pgbarman+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---
You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/1307760610.312833.1638526338570%40office.mailbox.org.

[Luca Ferrari]

On Mon, Dec 20, 2021 at 4:58 PM dulhaver via Barman, Backup and
Recovery Manager for PostgreSQL <pgba...@googlegroups.com> wrote:
> archive_command = 'ssh bar...@barman.server.intern "test ! -f /opt/barman/pg/incoming/%f" && scp %p barman@barman@barman.server.intern:/opt/barman/pg/incoming/%f'

I suspect this command is wrong, probably you have warning on the
PostgreSQL logs.

Luca

[dulh...@mailbox.org]

thx @ Luca

I have made progress with the archive_command. Not sure whether it's me or the format of the documentation ... I seem to have serious problems to identify the things that are important in there. I changed the archive_command to archive_command = 'barman-wal-archive backup pg %p'

which makes the FAILED disappear upon barman check. That's epic progress !

Maybe I need to read everything again to understand how the actual backup is going to happen then.

just running the assumed backup command barman backup pg looks good at the beginning but never seems to pass the Starting backup copy via pg_basebackup for 20211220T181559 state

barman@barman0180:~> barman backup pg

Starting backup using postgres method for server pg in /opt/barman/pg/base/20211220T181559

Backup start at LSN: 0/3A000110 (00000001000000000000003A, 00000110)

Starting backup copy via pg_basebackup for 20211220T181559

so maybe I am not understanding yet how this is intended to be run. My test psql instance has practically no data, so I would not assume it is just taking that long (more then 2 hours now).

On 12/20/2021 5:30 PM Luca Ferrari <fluc...@gmail.com> wrote:

On Mon, Dec 20, 2021 at 4:58 PM dulhaver via Barman, Backup and

Recovery Manager for PostgreSQL <pgba...@googlegroups.com> wrote:

archive_command = 'ssh bar...@barman.server.intern "test ! -f /opt/barman/pg/incoming/%f" && scp %p barman@bar...@barman.server.intern:/opt/barman/pg/incoming/%f'

I suspect this command is wrong, probably you have warning on the

PostgreSQL logs.

Luca

--

--

You received this message because you are subscribed to the "Barman for PostgreSQL" group.

To post to this group, send email to pgba...@googlegroups.com

To unsubscribe from this group, send email to

pgbarman+u...@googlegroups.com

For more options, visit this group at

http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---

You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+u...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/CAKoxK%2B46gLKtSQKzEYkNXzLkd6F3c_8mgot4x-BGS431i%3DsLfA%40mail.gmail.com.

[Luca Ferrari]

On Mon, Dec 20, 2021 at 8:32 PM dulhaver via Barman, Backup and
Recovery Manager for PostgreSQL <pgba...@googlegroups.com> wrote:
> barman@barman0180:~> barman backup pg
> Starting backup using postgres method for server pg in /opt/barman/pg/base/20211220T181559
> Backup start at LSN: 0/3A000110 (00000001000000000000003A, 00000110)
> Starting backup copy via pg_basebackup for 20211220T181559

This is really strange. One thing I noted is that you are asking for
md5 authentication on streaming connection while you did not provide
any password in your configuration. Is that correct or have you
stripped of the password to not show it to us?
Can you run pg_basebackup from the barman host manually and see what
it does? Something like:

pg_basebackup -X stream -R -r 100M -D /tmp/pgtest -l "TEST" -P -d
"host=postgres.server.intern user=streaming_barman dbname=postgres"
--verbose

and see if the output provides some hint?

Luca

[dulh...@mailbox.org]

On 12/21/2021 8:10 AM Luca Ferrari <fluc...@gmail.com> wrote:

On Mon, Dec 20, 2021 at 8:32 PM dulhaver via Barman, Backup and

Recovery Manager for PostgreSQL <pgba...@googlegroups.com> wrote:

barman@barman0180:~> barman backup pg

Starting backup using postgres method for server pg in /opt/barman/pg/base/20211220T181559

Backup start at LSN: 0/3A000110 (00000001000000000000003A, 00000110)

Starting backup copy via pg_basebackup for 20211220T181559

This is really strange. One thing I noted is that you are asking for md5 authentication on streaming connection while you did not provide

any password in your configuration. Is that correct or have you stripped of the password to not show it to us?

I have a .pgpass file in place holding the password

Can you run pg_basebackup from the barman host manually and see what it does? Something like:

pg_basebackup -X stream -R -r 100M -D /tmp/pgtest -l "TEST" -P -d

"host=postgres.server.intern user=streaming_barman dbname=postgres"

--verbose

that goes through without any problem

">barman@barman0180:~> pg_basebackup -X stream -R -r 100M -D /tmp/pgtest -l "TEST" -P -d “host=postgres.server.intern user=streaming_barman dbname=postgres”

--verbose

pg_basebackup: initiating base backup, waiting for checkpoint to complete

pg_basebackup: checkpoint completed

pg_basebackup: write-ahead log start point: 0/48000028 on timeline 1

pg_basebackup: starting background WAL receiver

pg_basebackup: created temporary replication slot "pg_basebackup_376301"

43464/43464 kB (100%), 1/1 tablespace

pg_basebackup: write-ahead log end point: 0/48000100

pg_basebackup: waiting for background process to finish streaming ...

pg_basebackup: syncing data to disk ...

pg_basebackup: base backup completed

repeating barman backup pg afterwards again starts a new backup, which seems to freeze up

barman@barman0180:~> barman backup pg

Starting backup using postgres method for server pg in /opt/barman/pg/base/20211221T122157
Backup start at LSN: 0/49000060 (000000010000000000000049, 00000060)
Starting backup copy via pg_basebackup for 20211221T122157

barman@barman0180:~> barman list-backup pg
pg 20211221T122157 - STARTED
barman@barman0180:~> barman show-backup pg 20211221T122157
Backup 20211221T122157:
Server Name : pg
System Id : 7024417304355768690
Status : STARTED

barman check-backup (which I would expect to deliver more detail on a given backup) does not return anything (maybe another hint that backup never finishes).

the backup dir is being populated ovousely

:~> less /opt/barman/pg/base/20211221T122157/backup.info

backup_label=None
begin_offset=96
begin_time=2021-12-21 12:21:57.568256+01:00
begin_wal=000000010000000000000049
begin_xlog=0/49000060
config_file=/opt/db/data/postgres/data/postgresql.conf
copy_stats=None
deduplicated_size=None
end_offset=None
end_time=None
end_wal=None
end_xlog=None
error=None
hba_file=/opt/db/data/postgres/data/pg_hba.conf
ident_file=/opt/db/data/postgres/data/pg_ident.conf
included_files=['/opt/db/data/postgres/data/postgresql.conf.d/01_barman.conf']
mode=postgres
pgdata=/opt/db/data/postgres/data
server_name=pg
size=None
status=STARTED
systemid=7024417304355768690
tablespaces=None
timeline=1
version=120006
xlog_segment_size=16777216
/opt/barman/pg/base/20211221T122157/backup.info (END)

[Luca Ferrari]

On Tue, Dec 21, 2021 at 12:39 PM dulhaver via Barman, Backup and
Recovery Manager for PostgreSQL <pgba...@googlegroups.com> wrote:
>
> I have a .pgpass file in place holding the password

If I remember correctly, there must be two entries in the pgpass file,
one for queries, one for replication. Could you please try hard coding
the password in the barman configuration instead? I still suspect the
process is hanging.

> repeating barman backup pg afterwards again starts a new backup, which seems to freeze up

Does barman diagnose provide any hint?
Have you tried to look at pg_stat_replication on the pg side to see if
the replication is going on?

Luca

[dulh...@mailbox.org]

On 12/21/2021 1:23 PM Luca Ferrari <fluc...@gmail.com> wrote:

On Tue, Dec 21, 2021 at 12:39 PM dulhaver via Barman, Backup and

Recovery Manager for PostgreSQL <pgba...@googlegroups.com> wrote:

>

I have a .pgpass file in place holding the password

If I remember correctly, there must be two entries in the pgpass file,

one for queries, one for replication. Could you please try hard coding

the password in the barman configuration instead? I still suspect the

process is hanging.

yes, two users, passwords. For now it is pretty permissive which might not be the final version.

bar...@barman.intern:~> cat .pgpass

postgres.server.intern:*:*:streaming_barman:somepasswd

postgres.server.intern:*:*:barman:somepasswd

repeating barman backup pg afterwards again starts a new backup, which seems to freeze up

Does barman diagnose provide any hint?

"pg": {
"backups": {
"20211221T122157": {
"backup_id": "20211221T122157",
"backup_label": null,
"begin_offset": 96,
"begin_time": "Tue Dec 21 12:21:57 2021",
"begin_wal": "000000010000000000000049",
"begin_xlog": "0/49000060",
"config_file": "/opt/db/data/postgres/data/postgresql.conf",
"copy_stats": null,
"deduplicated_size": null,
"end_offset": null,
"end_time": null,
"end_wal": null,
"end_xlog": null,
"error": "failure copying files (KeyboardInterrupt)",
"hba_file": "/opt/db/data/postgres/data/pg_hba.conf",
"ident_file": "/opt/db/data/postgres/data/pg_ident.conf",
"included_files": [
"/opt/db/data/postgres/data/postgresql.conf.d/01_barman.conf"
],
"mode": "postgres",
"pgdata": "/opt/db/data/postgres/data",
"server_name": "pg",
"size": null,
"status": "FAILED",
"systemid": "7024417304355768690",
"tablespaces": null,
"timeline": 1,
"version": 120006,
"xlog_segment_size": 16777216
}
},

the keyboard interrupt error comes from me pressind Ctrl +c after ~30 mins without progress I guess

Have you tried to look at pg_stat_replication on the pg side to see if the replication is going on?

I see that I am missing some basic understanding of PostgreSQL because these terms don't tell me much really

postgres=# \d pg_stat_replication
View "pg_catalog.pg_stat_replication"
Column           | Type                     | Collation | Nullable | Default
-----------------+--------------------------+-----------+----------+--------
pid              | integer                  | | |
usesysid         | oid                      | | |
usename          | name                     | | |
application_name | text                     | | |
client_addr      | inet                     | | |
client_hostname  | text                     | | |
client_port      | integer                  | | |
backend_start    | timestamp with time zone | | |
backend_xmin     | xid                      | | |
state            | text                     | | |
sent_lsn         | pg_lsn                   | | |
write_lsn        | pg_lsn                   | | |
flush_lsn        | pg_lsn                   | | |
replay_lsn       | pg_lsn                   | | |
write_lag        | interval                 | | |
flush_lag        | interval                 | | |
replay_lag       | interval                 | | |
sync_priority    | integer                  | | |
sync_state       | text                     | | |
reply_time       | timestamp with time zone | | |

[Luca Ferrari]

On Wed, Dec 22, 2021 at 9:19 AM dulhaver via Barman, Backup and
Recovery Manager for PostgreSQL <pgba...@googlegroups.com> wrote:
> I see that I am missing some basic understanding of PostgreSQL because these terms don't tell me much really
>
> postgres=# \d pg_stat_replication

I'm running out of ideas, in any case, p0lease do a select * from
pg_stat_replication to see, while the backup is "going" if there is
some activity. Also inspect PostgreSQL logs to see if there's a
problem with archiving in any way.
I suspect someone of the Barman team should give some advice at this point.

Luca

[Michael Wallace]

This is certainly puzzling - if pg_basebackup is succeeding when run directly then it's worth checking the exact invocation Barman is using, e.g.:

* Check the output of `ps -eaf | grep barman` while the barman backup is running - is there a pg_basebackup process belonging to the barman backup process?

* If there is, what happens when you run that command directly (you will need to modify the --pgdata argument to point to an empty directory)?

One key difference is that Barman will append `--wal-method none` to the pg_basebackup command which will cause it to wait until the WALs have been successfully archived by PostgreSQL, so if there are any issues with archive_command completing successfully then that can stop backups from completing (the invocation of pg_basebackup used earlier in this thread will stream the WALs during the backup so is not dependent on the success of WAL archiving).

A few other questions:

* Is there anything other than the backup.info in the directory for the failed backup, specifically is there a `data` directory and does it have any content?

* Are there any errors in the PostgreSQL logs around the time of the backup?

* Is anything being logged in the Barman log file around the same time?

--
--
You received this message because you are subscribed to the "Barman for PostgreSQL" group.
To post to this group, send email to pgba...@googlegroups.com
To unsubscribe from this group, send email to
pgbarman+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---
You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+u...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/CAKoxK%2B6Qefcjv6ixDzTEne0JALKeNEx9-a32b__8AwEdacGs0A%40mail.gmail.com.

[dulh...@mailbox.org]

I managed to get over this issue on a new couple of servers. I can not tell exactly with is different as it is really difficult to understand what is what.

thx anyway for the suggestions made. I hope I will raise to some sort of true understanding someday :-).

I will come back with more questions down the line for sure. I am not sure whether it is me or the documentation which makes this so difficult to wrap my head around. It feels I am somewhat at war with the this