WAL files not removed from incoming directory after compression

[Hal Martin]

Hi everyone,

I have configured barman to backup a postgresql 9.6 database. Barman is using replication to stream updates from the database, and postgresql.conf on the database server is also configured to ship WAL logs to the barman server's incoming directory.

However, I have noticed that WAL files are never deleted or removed from the incoming directory on the barman server, despite the fact that they have been processed successfully by "barman cron"

This is rather confusing and seems to go against the barman documentation, which states:

Usually, the best approach (especially in a local network), is to let Barman compress these 16MB files on the backup server end (you can use the “compression” option for this purpose). This operation is performed jointly with archival by the “barman cron” command, which goes through every file in the incoming directory, performs compression (where applicable) and permanently moves the WAL file in the WAL archive for that server.

https://blog.2ndquadrant.com/management-wal-archive-barman/

Has anyone else noticed this? The incoming directory has several months of WAL files, and has grown to over 50GB. For comparison, the base backup and compressed WAL files since the last backup in March for the postgres server is only 20GB.

How can I safely clean up the incoming directory to remove the old uncompressed WAL files which have already been processed via "barman cron"?

Thanks,
Hal Martin

[Marcin K.]

Hi,

Same problem here - configured barman to use both streaming WAL and postgres to ship logs via archive_command. Logs are not being removed from incoming dir. Paths are certainly correct.

-- Marcin

[Marco Nenciarini]

Il 06/04/17 18:38, Marcin K. ha scritto:

> Hi,
>
> Same problem here - configured barman to use both streaming WAL and
> postgres to ship logs via archive_command. Logs are not being removed
> from incoming dir. Paths are certainly correct.
>

Hi, could you please attach the `barma diagnose` output?

Regards,
Marco

--
Marco Nenciarini - 2ndQuadrant Italy
PostgreSQL Training, Services and Support
marco.ne...@2ndQuadrant.it | www.2ndQuadrant.it

[Marcin K.]

Hi, could you please attach the `barma diagnose` output?

Hi Marco,

Unfortunately this is not possible, corporate server with no network access. I understand that it may cause the problem much difficult to diagnose - sorry about that. If it's any help, I can quickly write more important parts of the config:

compression = gzip

immediate_checkpoint = false

conninfo = ...

streaming_conninfo = ...

backup_method = postgres

streaming_archiver = on

slot_name = ...

and that's about it. postgresql.conf archive_command configured (which works, because I can see files in incoming_wals_directory)

Is there anything I can copy from diagnose what would help?

Is this expected to work? I mean, using barman with two sources of WALs - streaming and shipping from postgres. Is barman suppose to deal with it and remove unnecessary logs (which I have twice - in incoming and streaming dir)? Logs from streaming dir get compressed and moved to wals/ - this is fine.

Thanks,

-- Marcin

[Hal Martin]

Our server is also on a corporate network, but I might be able to censor enough identifying information to submit the output of barman diagnose.

We have our barman instance configured the same as Marcin: postgres is shipping WAL logs and barman is simultaneously streaming from postgres using a replication slot.

I've already confirmed that WAL files are being correctly compressed and archived into the appropriate server directory. They're likely coming from the streaming directory as well as that directory isn't constantly increasing in size, unlike the incoming directory.

Thanks,

Hal

--
--
You received this message because you are subscribed to the "Barman for PostgreSQL" group.
To post to this group, send email to pgba...@googlegroups.com
To unsubscribe from this group, send email to
pgbarman+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---
You received this message because you are subscribed to a topic in the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pgbarman/oCZKl9mu4Lk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pgbarman+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marcin K.]

Hi All,

I think I've found a solution.

"archiver = on" needs to be added to the config. Marco, can you please confirm that after adding this to my existing config file (with streaming_archiver and streaming_conninfo settings), my WAL backups are redundant and in if either method fails, the other one still does the job?

Thanks 

[Gabriele Bartolini]

Hi Marcin,

  if both archiver and streaming_archiver are on, during archiving Barman will process those files that arrive first in the backup server. The second one is checked through MD5 and if the checksum is identical to the already processed one, it is discarded.

  This technology has been rigorously tested and it is in production with large installations over the world, at least by us through our 2ndQuadrant support service. However, bugs can always be behind the corner. ;)

  I hope you solved the issue and ... enjoy your espresso backup! ;)

Ciao,

Gabriele

--
 Gabriele Bartolini - 2ndQuadrant Italia - Director

 PostgreSQL Training, Services and Support

 gabriele....@2ndQuadrant.it | www.2ndQuadrant.it

--

--
You received this message because you are subscribed to the "Barman for PostgreSQL" group.
To post to this group, send email to pgba...@googlegroups.com
To unsubscribe from this group, send email to
pgbarman+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---

You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+unsubscribe@googlegroups.com.

[Marco Nenciarini]

Il 06/04/17 18:52, Marcin K. ha scritto:

>
> Hi, could you please attach the `barma diagnose` output?
>
>
> Hi Marco,
>
> Unfortunately this is not possible, corporate server with no network
> access. I understand that it may cause the problem much difficult to
> diagnose - sorry about that. If it's any help, I can quickly write more
> important parts of the config:
> compression = gzip
> immediate_checkpoint = false
> conninfo = ...
> streaming_conninfo = ...
> backup_method = postgres
> streaming_archiver = on
> slot_name = ...
>
> and that's about it. postgresql.conf archive_command configured (which
> works, because I can see files in incoming_wals_directory)
>

Well, it looks like you don't have 'archiver = on' in your list.

> Is there anything I can copy from diagnose what would help?
>

The diagnose is only a sum of 'barman check', 'barman show-server',
'barman status' and 'barman list-backup' for every server. It is useful
to have the status of the whole server in one shot. For this issue the
most important part is the output of 'barman show-server'.

> Is this expected to work? I mean, using barman with two sources of WALs
> - streaming and shipping from postgres. Is barman suppose to deal with
> it and remove unnecessary logs (which I have twice - in incoming and
> streaming dir)? Logs from streaming dir get compressed and moved to
> wals/ - this is fine.
>

Logs from the two sources are identical if everything is correctly
configured, so barman verify them and simply removes the second
identical file. In case they will not match, usually due to some
mistake, it will put the second file in the errors directory and the
'barman check' command will start to complain.