Barman up and running, but confused about the mechanics of its operation

[Jarkko Laiho]

Hi all,

While setting up Barman 2.12 replication of a PostgreSQL 9.5 server to a backup server, I managed to get it running with very little trouble. Having read the Barman docs all the way through a few times, I still have gaps in my understanding though. (What I'm about to ask may be covered in there, but for some reason I haven't been able to properly absorb it.)

First, our relevant configuration. On the PostgreSQL side:

wal_level = hot_standby

max_wal_senders = 6

archive_mode = on

archive_command = 'barman-wal-archive -U someuser backupserver projectname %p'

max_replication_slots = 6

On the backup server side, our Barman config:

[barman]

barman_home = /backup

barman_user = someuser

log_file = /backup/barman.log

minimum_redundancy = 1

[projectname]

description = "Example project"

archiver = on

conninfo = host=dbserver user=barman dbname=postgres

backup_method = postgres

streaming_archiver = on

streaming_conninfo = host=dbserver user=barman dbname=postgres

slot_name = barman

create_slot = auto

retention_policy = RECOVERY WINDOW OF 7 DAYS

retention_policy_mode = auto

In addition, barman cron runs every minute.

So, we're using the hybrid approach of archive_mode/archive_command and WAL streaming, as recommended by the Barman manual. Everything seems to be running smoothly based on the Barman logs and the output of barman check projectname.

Some questions:

1. Is manually running barman backup necessary at all for the functioning of barman? I ran it once after initial setup when barman check complained about the minimum redundancy requirement of 1 (which I set myself) not being fulfilled, but would that requirement have been fulfilled on its own after some time had passed? (This is mainly terminology confusion about what a "backup" is in terms of all of Barman's functionality; man 5 barman just says "Minimum number of backups to be retained" for minimum_redundancy)

2. Had I never run barman backup, I presume that at any time in the future, I would have still been able to recover to any point in time 7 days back from that future time with the WAL-based methods, due to retention_policy?

3. Does running barman backup interact with the archival and streaming backups in any way, or is merely a tool for creating frozen points in time that you might want to return to beyond your normal recovery window?

4. When using the combination of archive_command and streaming, you (at least in theory) gain robustness, but is there any real downside, like more disk space used on the backup server?

5. In the manual, under Catalog Management Features, there is this warning: "Make sure that your retention policy settings do not collide with minimum redundancy requirements. Regularly check Barman's log for messages on this topic." Given my setup, under what circumstances – if any – would this collision likely occur?

6. Finally, are there some clear omissions or illogical bits in the configuration as shown?

Thank you!

[Luca Ferrari]

My thoguths...

On Tue, Feb 2, 2021 at 3:20 PM Jarkko Laiho <jarkko...@taiste.com> wrote:
> 1. Is manually running barman backup necessary at all for the functioning of barman? I ran it once after initial setup when barman check complained about the minimum redundancy requirement of 1 (which I set myself) not being fulfilled, but would that requirement have been fulfilled on its own after some time had passed? (This is mainly terminology confusion about what a "backup" is in terms of all of Barman's functionality; man 5 barman just says "Minimum number of backups to be retained" for minimum_redundancy)

If you don't run a backup, barman will do nothing. Barman creates a
"frozen" backup and merges it with incoming wals, but if you don't
have a backup you cannot do a restore.
Usually you schedule "backup" via cron or tools like that.

>
> 2. Had I never run barman backup, I presume that at any time in the future, I would have still been able to recover to any point in time 7 days back from that future time with the WAL-based methods, due to retention_policy?

Retention policy decides when a backup can be expired, that in your
case is a backup older than 7 days can be pruned. So, the quick answer
to your question is NO.
Moreover, consider what happens if you stay with only one backup
(assuming no retention policy is in place): you can move to any point
in time during a recovery, but to do a ful recovery you need to apply
a lot of wals from the time you took your original backup.

>
> 3. Does running barman backup interact with the archival and streaming backups in any way, or is merely a tool for creating frozen points in time that you might want to return to beyond your normal recovery window?
>

It issues a pg_start_backup and pg_stop_backup, as far as I know. In
turn, the above, do no interact very much with the cluster.

> 4. When using the combination of archive_command and streaming, you (at least in theory) gain robustness, but is there any real downside, like more disk space used on the backup server?

Don't know this, never used. I suspect in the long run you will not
have a doubled storage occupation, probably in the mid term you will
have because WALs are incoming as different sources.

>
> 5. In the manual, under Catalog Management Features, there is this warning: "Make sure that your retention policy settings do not collide with minimum redundancy requirements. Regularly check Barman's log for messages on this topic." Given my setup, under what circumstances – if any – would this collision likely occur?
>

retention policy decides when a backup can be expired, mimimum
redundancy how many you must keep. In your case, having a single
backup satisfies the latter, but could not be good for the former
after a week has elapsed.
I think the collision the documentation is referring to is something
like doing a backup a day, having a window of 7 days and a mimimum
backup policy of 14: this will clearly accumulate more backups than
you need.

> 6. Finally, are there some clear omissions or illogical bits in the configuration as shown?

I would say no, but clearly this is hard to say without knowing your aim.

Luca

[Jarkko Laiho]

Thank you, Luca. It seems I had severely misunderstood the role and functionality of barman backup, so I'm forced to rethink things a bit. In my defense, the docs aren't super clear on how it should be used in different kinds of backup scenarios. (For example, nowhere in the docs does it say that barman backup needs to be run periodically, and neither is that mentioned in many of the third-party tutorials, like the Severalnines one.)

What I'm actually trying to achieve is something like this: at any future point in time, I want to be able to recover the database to any point in time up to 7 days in the past, counting back from the moment of starting the recovery.

What do I have to do to get there? What settings do I need to have, and how often should I run barman backup as a cron job? Do I need to delete old backups manually with another cron job, or is barman cron handling it based on some settings?

It seems that I'm unfortunately not able to piece together all the documentation into a clear understanding about how the various seemingly related options (retention_policy, minimum_redundancy, others) actually interact with this goal.

- JL

[Luca Ferrari]

On Wed, Feb 3, 2021 at 1:23 PM Jarkko Laiho <jarkko...@taiste.com> wrote:
>
> Thank you, Luca. It seems I had severely misunderstood the role and functionality of barman backup, so I'm forced to rethink things a bit. In my defense, the docs aren't super clear on how it should be used in different kinds of backup scenarios. (For example, nowhere in the docs does it say that barman backup needs to be run periodically, and neither is that mentioned in many of the third-party tutorials, like the Severalnines one.)

I do agree that the documentation, even if very precise, is not very clear.

>
> What I'm actually trying to achieve is something like this: at any future point in time, I want to be able to recover the database to any point in time up to 7 days in the past, counting back from the moment of starting the recovery.
>

Things are these: you can do a single backup and let the barman system
archive the wals, then you will be able to recover to any point in
time from the backup instant, including seven days back from now. The
tragedy is that if your only backup is 1 year old, you will need to
wait for PostgreSQL to restore the backup (at one year later) and come
thru the wals until it hits seven days back (or the instant you
asked). This is why you need to get a regular backup, thus reducing
the recvery time required (and also improving reliability, etc).

> What do I have to do to get there? What settings do I need to have, and how often should I run barman backup as a cron job? Do I need to delete old backups manually with another cron job, or is barman cron handling it based on some settings?

I would schedule a backup twice a week, and keep 4 backups (i.e., two
weeks) that will give you the ability to go back 7 days. This could be
a starting point, but it depends on your infrastructure, size of the
backup, bandwidth, etc.

Hope it helps.
Luca