Disk usage keeps growing, even when reuse_backup is set to "link"

[Lisandro]

I'm using pgbarman for the first time. I've managed to setup the backup scenario, and it's been working ok for the last two or three days.

However, I've noticed that the spaced used for the backups grows exponentially. 

The cluster I'm backing up has 6.4gb, and every time pgbarman makes a new backup, I loose another 6.4gb of free space.

I've read the documentation about the reuse_backup parameter, and of course it's set to link. However, every new backup eats 6.4gb of free space.

This is my pgbarman configuration:

[barman]
barman_home = /var/lib/barman
barman_user = barman
log_file = /var/log/barman/barman.log
compression = gzip
reuse_backup = link
basebackup_retry_times = 3
basebackup_retry_sleep = 30
last_backup_maximum_age = 1 DAYS


[mycluster]
description = "My Cluster"
ssh_command = ssh postgres@myclusterip
conninfo = host=myclusterip user=postgres password=mypassword
retention_policy_mode = auto
retention_policy = RECOVERY WINDOW OF 1 days
wal_retention_policy = main

Then, the cron configuration for the barman user is this:

* * * * * /usr/bin/barman cron
*/60 * * * * /usr/bin/barman backup mycluster

The barman list-backup command outputs this:

mycluster 20160327T080001 - Sun Mar 27 08:01:47 2016 - Size: 6.5 GiB - WAL Size: 5.9 MiB
mycluster 20160327T070001 - Sun Mar 27 07:01:52 2016 - Size: 6.5 GiB - WAL Size: 25.2 MiB
mycluster 20160327T060001 - Sun Mar 27 06:01:50 2016 - Size: 6.5 GiB - WAL Size: 24.4 MiB
mycluster 20160327T050002 - Sun Mar 27 05:01:59 2016 - Size: 6.5 GiB - WAL Size: 22.6 MiB
mycluster 20160327T040001 - Sun Mar 27 04:01:49 2016 - Size: 6.5 GiB - WAL Size: 31.0 MiB
mycluster 20160327T030001 - Sun Mar 27 03:01:53 2016 - Size: 6.5 GiB - WAL Size: 38.7 MiB
mycluster 20160327T020001 - Sun Mar 27 02:01:51 2016 - Size: 6.5 GiB - WAL Size: 68.9 MiB
mycluster 20160327T010002 - Sun Mar 27 01:01:54 2016 - Size: 6.5 GiB - WAL Size: 30.4 MiB
mycluster 20160327T000002 - Sun Mar 27 00:01:54 2016 - Size: 6.5 GiB - WAL Size: 30.2 MiB
mycluster 20160326T230001 - Sat Mar 26 23:01:50 2016 - Size: 6.5 GiB - WAL Size: 24.2 MiB
mycluster 20160326T220002 - Sat Mar 26 22:01:57 2016 - Size: 6.5 GiB - WAL Size: 29.7 MiB
mycluster 20160326T210002 - Sat Mar 26 21:01:53 2016 - Size: 6.5 GiB - WAL Size: 29.8 MiB
mycluster 20160326T200002 - Sat Mar 26 20:01:50 2016 - Size: 6.5 GiB - WAL Size: 36.8 MiB
mycluster 20160326T190002 - Sat Mar 26 19:01:50 2016 - Size: 6.5 GiB - WAL Size: 30.3 MiB
mycluster 20160326T180001 - Sat Mar 26 18:07:48 2016 - Size: 6.4 GiB - WAL Size: 29.8 MiB
mycluster 20160326T170002 - Sat Mar 26 17:01:53 2016 - Size: 6.4 GiB - WAL Size: 30.2 MiB
mycluster 20160326T160002 - Sat Mar 26 16:01:52 2016 - Size: 6.4 GiB - WAL Size: 30.8 MiB
mycluster 20160326T150002 - Sat Mar 26 15:01:51 2016 - Size: 6.4 GiB - WAL Size: 31.1 MiB
mycluster 20160326T140002 - Sat Mar 26 14:01:53 2016 - Size: 6.4 GiB - WAL Size: 37.0 MiB
mycluster 20160326T130002 - Sat Mar 26 13:01:49 2016 - Size: 6.4 GiB - WAL Size: 44.7 MiB
mycluster 20160326T120002 - Sat Mar 26 12:01:48 2016 - Size: 6.4 GiB - WAL Size: 44.2 MiB
mycluster 20160326T110001 - Sat Mar 26 11:01:53 2016 - Size: 6.4 GiB - WAL Size: 50.3 MiB
mycluster 20160326T100001 - Sat Mar 26 10:01:50 2016 - Size: 6.4 GiB - WAL Size: 50.2 MiB
mycluster 20160326T090002 - Sat Mar 26 09:01:56 2016 - Size: 6.4 GiB - WAL Size: 49.6 MiB
mycluster 20160326T080002 - Sat Mar 26 08:01:52 2016 - Size: 6.4 GiB - WAL Size: 50.1 MiB

And if I check my disk space:

Filesystem          Size  Used Avail Use% Mounted on
/dev/dm-0           245G  164G   69G  71% /

The server is a fresh install, and in a couple of days disk usage is already in 71%.

Am I missing something? Maybe I missinterpreted the docs about reuse_backup parameter?

Any tip will be appreciated. Thanks in advance!

Regards,

Lisandro.

[Giulio Calacoci]

Hi Lisandro,

there is a reason for tacking backups so frequently?
Could you please provide the output of the command 'barman diagnose' ?

Regards and Happy Easter
Giulio

> --
> --
> You received this message because you are subscribed to the "Barman for
> PostgreSQL" group.
> To post to this group, send email to pgba...@googlegroups.com
> To unsubscribe from this group, send email to
> pgbarman+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/pgbarman?hl=en?hl=en-GB
>
> ---
> You received this message because you are subscribed to the Google Groups
> "Barman, Backup and Recovery Manager for PostgreSQL" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pgbarman+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Giulio Calacoci - 2ndQuadrant Italia
PostgreSQL Training, Services and Support
giulio....@2ndQuadrant.it | www.2ndQuadrant.it

[Lisandro]

Hi Giulio,

The barman diagnose command outputs an error:

barman@backup:~$ barman diagnose

usage: barman [-h] [-v] [-c CONFIG] [-q] [-f {console}]

              {cron,list-server,show-server,status,check,backup,list-backup,show-backup,list-files,recover,delete,rebuild-xlogdb}

              ...

barman: error: invalid choice: 'diagnose' (choose from 'cron', 'list-server', 'show-server', 'status', 'check', 'backup', 'list-backup', 'show-backup', 'list-files', 'recover', 'delete', 'rebuild-xlogdb')

After that, I checked the version, and I'm using version 1.3.0, but I can see that the latest is 1.6.0.

I've installed barman with apt-get in Ubuntu 14.04, I will try to install latest version from source.

The reason I'm doing a backup every 60 minutes is because I would like to have a 1 or 2 days recovery window, and I would like to have the possibility to choose any point in time within that period.

For example, if something goes wrong at 2.30pm, I have the 2pm backup to recover from.

But if I do one daily backup at 12am, if something goes wrong at 2pm, I would loose the changes from 12am to 2pm.

Or maybe I've missinterpreted the docs.

I thought that using reuse_backup=link the first backup would take much space, but the subsequent backups would only need very little space.

So that's why I thouht it would be ok to run a backup every hour.

But again, maybe I'm missing some points.

Any tip or suggestion will be appreciated!

Regards,

Lisandro.

 giulio.calacoci@2ndQuadrant.it | www.2ndQuadrant.it

[Lisandro]

I've installed latest pgbarman 1.6.0 from source.

Now the command barman diagnose works ok, however the space usage still grows.

This is the barman diagnose output:

{

    "global": {

        "config": {

            "barman_home": "/var/lib/barman", 

            "barman_user": "barman", 

            "basebackup_retry_sleep": "30", 

            "basebackup_retry_times": "3", 

            "compression": "gzip", 

            "errors_list": [], 

            "immediate_checkpoint": "true", 

            "last_backup_maximum_age": "1 DAYS", 

            "log_file": "/var/log/barman/barman.log", 

            "reuse_backup": "link"

        }, 

        "system_info": {

            "barman_ver": "1.6.0", 

            "kernel_ver": "Linux backup 3.13.0-83-generic #127-Ubuntu SMP Fri Mar 11 00:25:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux", 

            "python_ver": "Python 2.7.6", 

            "release": "Distributor ID:\tUbuntu\nDescription:\tUbuntu 14.04.4 LTS\nRelease:\t14.04\nCodename:\ttrusty", 

            "rsync_ver": "rsync  version 3.1.0  protocol version 31", 

            "ssh_ver": "OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6, OpenSSL 1.0.1f 6 Jan 2014"

        }

    }, 

    "servers": {
        "medios": {

            "backups": {

                "20160328T104346": {

                    "backup_id": "20160328T104346", 

                    "backup_label": null, 

                    "begin_offset": 36, 

                    "begin_time": "Mon Mar 28 13:43:46 2016", 

                    "begin_wal": "000000010000000D00000009", 

                    "begin_xlog": "D/9000024", 

                    "config_file": "/etc/postgresql/9.3/main/postgresql.conf", 

                    "deduplicated_size": 6984988471, 

                    "end_offset": 5529192, 

                    "end_time": "Mon Mar 28 13:46:12 2016", 

                    "end_wal": "000000010000000D00000009", 

                    "end_xlog": "D/9545E68", 

                    "error": null, 

                    "hba_file": "/etc/postgresql/9.3/main/pg_hba.conf", 

                    "ident_file": "/etc/postgresql/9.3/main/pg_ident.conf", 

                    "included_files": null, 

                    "mode": "default", 

                    "pgdata": "/var/lib/postgresql/9.3/main", 

                    "server_name": "medios", 

                    "size": 6984988471, 

                    "status": "DONE", 

                    "tablespaces": null, 

                    "timeline": 1, 

                    "version": 90311

                },
                "20160328T114218": {

                    "backup_id": "20160328T114218", 

                    "backup_label": null, 

                    "begin_offset": 36, 

                    "begin_time": "Mon Mar 28 14:42:19 2016", 

                    "begin_wal": "000000010000000D00000012", 

                    "begin_xlog": "D/12000024", 

                    "config_file": "/etc/postgresql/9.3/main/postgresql.conf", 

                    "deduplicated_size": 5732192033, 

                    "end_offset": 5767784, 

                    "end_time": "Mon Mar 28 14:45:12 2016", 

                    "end_wal": "000000010000000D00000012", 

                    "end_xlog": "D/12580268", 

                    "error": null, 

                    "hba_file": "/etc/postgresql/9.3/main/pg_hba.conf", 

                    "ident_file": "/etc/postgresql/9.3/main/pg_ident.conf", 

                    "included_files": null, 

                    "mode": "default", 

                    "pgdata": "/var/lib/postgresql/9.3/main", 

                    "server_name": "medios", 

                    "size": 6987708218, 

                    "status": "DONE", 

                    "tablespaces": null, 

                    "timeline": 1, 

                    "version": 90311

                }

            },
            "config": {

                "active": true, 

                "archiver": true, 

                "backup_directory": "/var/lib/barman/medios", 

                "backup_method": "rsync", 

                "backup_options": "exclusive_backup", 

                "bandwidth_limit": null, 

                "barman_home": "/var/lib/barman", 

                "barman_lock_directory": "/var/lib/barman", 

                "basebackup_retry_sleep": 30, 

                "basebackup_retry_times": 3, 

                "basebackups_directory": "/var/lib/barman/medios/base", 

                "compression": "gzip", 

                "conninfo": "host=myhostip user=postgres password=mypassword", 

                "custom_compression_filter": null, 

                "custom_decompression_filter": null, 

                "description": "Database Server", 

                "disabled": false, 

                "errors_directory": "/var/lib/barman/medios/errors", 

                "immediate_checkpoint": true, 

                "incoming_wals_directory": "/var/lib/barman/medios/incoming", 

                "last_backup_maximum_age": "1 day", 

                "minimum_redundancy": 0, 

                "msg_list": [], 

                "name": "medios", 

                "network_compression": false, 

                "path_prefix": null, 

                "post_archive_retry_script": null, 

                "post_archive_script": null, 

                "post_backup_retry_script": null, 

                "post_backup_script": null, 

                "pre_archive_retry_script": null, 

                "pre_archive_script": null, 

                "pre_backup_retry_script": null, 

                "pre_backup_script": null, 

                "recovery_options": "", 

                "retention_policy": "window 1 d", 

                "retention_policy_mode": "auto", 

                "reuse_backup": "link", 

                "ssh_command": "ssh postgres@myhostip", 

                "streaming_archiver": false, 

                "streaming_conninfo": "host=myhostip user=postgres password=mypassword", 

                "streaming_wals_directory": "/var/lib/barman/medios/streaming", 

                "tablespace_bandwidth_limit": null, 

                "wal_retention_policy": "simple-wal 1 d", 

                "wals_directory": "/var/lib/barman/medios/wals"

            }, 

            "status": {

                "archive_command": "rsync -a %p barman@mybackupip:/var/lib/barman/medios/incoming/%f", 

                "archive_mode": "on", 

                "config_file": "/etc/postgresql/9.3/main/postgresql.conf", 

                "current_size": 6983012004.0, 

                "current_xlog": "000000010000000D00000014", 

                "data_directory": "/var/lib/postgresql/9.3/main", 

                "hba_file": "/etc/postgresql/9.3/main/pg_hba.conf", 

                "ident_file": "/etc/postgresql/9.3/main/pg_ident.conf", 

                "last_archived_wal": "000000010000000D00000013", 

                "pgespresso_installed": false, 

                "server_txt_version": "9.3.11", 

                "wal_level": "archive"

            }, 

            "system_info": {

                "kernel_ver": "Linux medios 4.4.0-x86_64-linode63 #2 SMP Tue Jan 19 12:43:53 EST 2016 x86_64 x86_64 x86_64 GNU/Linux", 

                "python_ver": "Python 2.7.6", 

                "release": "Distributor ID:\tUbuntu\nDescription:\tUbuntu 14.04.4 LTS\nRelease:\t14.04\nCodename:\ttrusty", 

                "rsync_ver": "rsync  version 3.1.0  protocol version 31", 

                "ssh_ver": "OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6, OpenSSL 1.0.1f 6 Jan 2014"

            }

        }

    }

}

However, te problem still persist (maybe it's not a problem and I'm missinterpreting pgbarman).

I did a first backup, then a subsequent backup, and this is barman list-backup output:

medios 20160328T114218 - Mon Mar 28 14:45:12 2016 - Size: 6.5 GiB - WAL Size: 0 B

medios 20160328T104346 - Mon Mar 28 13:46:12 2016 - Size: 6.5 GiB - WAL Size: 52.8 MiB

Then, I check used disk space:

barman@backup:~$ df -h

Filesystem          Size  Used Avail Use% Mounted on

/dev/dm-0           245G   14G  220G   6% /

What am I missing?

[Lisandro]

I've done the following test:

I've deleted all my previous backups, and did a new fresh first backup.

barman@backup:~$ barman backup medios

Starting backup for server medios in /var/lib/barman/medios/base/20160329T104511

Backup start at xlog location: D/B9000084 (000000010000000D000000B9, 00000084)

This is the first backup for server medios

Copying files.

Copy done.

Asking PostgreSQL server to finalize the backup.

Backup size: 6.5 GiB. Actual size on disk: 6.5 GiB (-0.00% deduplication ratio).

Backup end at xlog location: D/B9590D24 (000000010000000D000000B9, 00590D24)

Backup completed

Processing xlog segments from file archival for medios

000000010000000D000000B9

000000010000000D000000B9.00000084.backup

Then, inmediatelly after, I did another backup:

barman@backup:~$ barman backup medios

Starting backup for server medios in /var/lib/barman/medios/base/20160329T105014

Backup start at xlog location: D/BB002570 (000000010000000D000000BB, 00002570)

Copying files.

Copy done.

Asking PostgreSQL server to finalize the backup.

Backup size: 6.5 GiB. Actual size on disk: 4.8 GiB (-26.90% deduplication ratio).

Backup end at xlog location: D/BB893E54 (000000010000000D000000BB, 00893E54)

Backup completed

I've noticed that the "Backup size" line, where it says "Actual size on disk".

Could someone clarify on this? 

I don't understand why the second backup uses so much space, if I'm using reuse_backup=link and the two backups were made with very little time difference between them.

Then, the barman list-backup outputs this:

barman@backup:~$ barman list-backup medios

medios 20160329T105014 - Tue Mar 29 13:53:01 2016 - Size: 6.5 GiB - WAL Size: 0 B

medios 20160329T104511 - Tue Mar 29 13:47:46 2016 - Size: 6.5 GiB - WAL Size: 10.5 MiB

I will appreciate any clarification on this.

I want to estimate how much rapidly the used space will grow, given the frequency of the backups and the retention period.

Thanks in advance!

[Gabriele Bartolini]

Hi Lisandro,

2016-03-27 20:54 GMT+02:00 Lisandro <rostagno...@gmail.com>:

The reason I'm doing a backup every 60 minutes is because I would like to have a 1 or 2 days recovery window, and I would like to have the possibility to choose any point in time within that period.

That comes for free with WAL archiving. You do not need to do frequent full backups. Normally weekly full backups are more than enough (especially in the case of a very small database like yours).

Ciao,

Gabriele

--
 Gabriele Bartolini - 2ndQuadrant Italia - Director

 PostgreSQL Training, Services and Support

 gabriele....@2ndQuadrant.it | www.2ndQuadrant.it

[Lisandro]

Hi Gabriele, thank you for your answer.

So I was missinterpreting the docs. I was using a retention_policy of just 1 day, because I was doing a full backup each hour.

I guess then that the "barman cron" takes care of getting wal files and keeping the backup synchronized, in order to allow me to restore to any point of time.

I will schedule a full backup just once every week. 

Thanks!

Hi Lisandro,

 gabriele.bartolini@2ndQuadrant.it | www.2ndQuadrant.it