pgbarman rsync link_stat "/var/lib/pgsql/data/base/pgsql_tmp/pgsql_tmp14255.0" failed: no such file

[Tjeerd]

Hello,

We are using pgbarman to backup our PostgreSQL-database and quite often we get the following error message in the pgbarman-logs:

pgbarman rsync link_stat "/var/lib/pgsql/data/base/pgsql_tmp/pgsql_tmp14255.0" failed: no such file or directory (2)
file has vasnished: "/var/lib/pgsql/data/base/pgsql_tmp/pgsql_tmp14255.1"
file has vasnished: "/var/lib/pgsql/data/base/pgsql_tmp/pgsql_tmp14255.2"

As I understand this is ´perfectly fine´, but in the status report of pgbarman this is reported as `FAILED`, which shows up quite often and doesn´t give confidence in a correct backup. Although when we restored the backup everything was fine.

Is there any way to prevent these error messages from happening? Is this happening because someone is running queries on the database while pgbarman is happening and some sort of temporary files are created and removed?

Can these folders be excluded somehow? Or is there an other procedure we have to follow to get rid of these messages?

Thanks in advance,

Tjeerd

[Giulio Garau]

Hi Tjeed, 

the rsync done during barman backup fails because it tries to save vanished temp files, you can solve this moving your stats_temp_directory (defined in postgresql.conf) out of /var/lib/pgsql/data directory. 

Be aware that the archive_command you specified is NOT saving your wals.

Assuming your barman is NOT in the same host of your postgres the correct syntax should be something like
archive_command = 'rsync -a %p barman@backup:INCOMING_WALS_DIRECTORY/%f'

If unsure, you che retrive the INCOMING_WALS_DIRECTORY with the command   "barman diagnose | grep incoming"

kr

Giulio

On Thu, Sep 22, 2016 at 11:07 AM, Tjeerd <t.w....@gmail.com> wrote:

I did some research and it seems that it is possible to set archive commands which should be picked up by the rsync command:

archive_command = 'rsync -a %p --exclude /var/lib/pgsql/data/base/pgsql_tmp'

Restarted the server/database and when I do a 'SHOW ALL' command in PostgreSQL I can clearly see that the option has been set:

SHOW ALL;
name    setting    description
archive_command    rsync -a %p --exclude /var/lib/pgsql/data/base/pgsql_tmp  Sets the shell command that will be called to archive a WAL file.

But it still gives the same error that the /var/lib/pgsql/data/base/pgsql_tmp/ folder doesn't exist anymore and the backup reports a "FAILED" message.

It almost looks like the added parameters are not being picked up by Pgbarman, is there anything I am still missing?

Op woensdag 31 augustus 2016 15:48:09 UTC+2 schreef Tjeerd:

--
--
You received this message because you are subscribed to the "Barman for PostgreSQL" group.
To post to this group, send email to pgba...@googlegroups.com
To unsubscribe from this group, send email to
pgbarman+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---
You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tjeerd]

Hello Giulio,

Thank you for your quick response. By purpose I left out the last barman@backup-server part which you noticed, so that's correct, but in our environment we have it there.

Your solution about moving the temporary folder to another location will be taken into consideration and looks like an interesting approach.

Still makes me curious, can you tell me why the exclude path isn't being picked up by Pgbarman?

Kind regards,

Tjeerd



Op donderdag 22 september 2016 11:45:47 UTC+2 schreef Giulio Garau:

pgbarman+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---
You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+u...@googlegroups.com.

[Giulio Garau]

Hi Tjeerd, 

the --exclude you specified is used only  by postgres archive_command, not by barman.

as I said be aware that your current archive_command configuration IS NOT sending completed wals to barman incoming wals directory.

using archive_command  = 'rsync -a %p --exclude /var/lib/pgsql/data/base/pgsql_tmp'

you are saying to your postgres to do nothing with completed wals 

kr 

Giulio 

pgbarman+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---
You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+unsubscribe@googlegroups.com.

[Adam Mejalli]

Hi Giulio,

It seems that I have similar ERROR on my barman backup, however, it's a bit different. Every now and then, we get error in the barman.log that backup had failed, see log out put below:

ERROR: Backup failed copying files.
DETAILS: data transfer failure
rsync error:
rsync: link_stat "/pgdata/data/9.6/base/16409/18037166" failed: No such file or directory (2)
rsync: link_stat "/pgdata/data/9.6/base/16409/18037167.1" failed: No such file or directory (2)
rsync: link_stat "/pgdata/data/9.6/base/16409/18037169" failed: No such file or directory (2)
rsync: link_stat "/pgdata/data/9.6/base/16409/18037171.1" failed: No such file or directory (2)
rsync: link_stat "/pgdata/data/9.6/base/16409/18037198.2" failed: No such file or directory (2)
rsync: link_stat "/pgdata/data/9.6/base/16409/18037224" failed: No such file or directory (2)
rsync: link_stat "/pgdata/data/9.6/base/16409/18037236.1" failed: No such file or directory (2)
rsync: link_stat "/pgdata/data/9.6/base/16409/18037260.2" failed: No such file or directory (2)
rsync: link_stat "/pgdata/data/9.6/base/16409/18037272" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1505) [receiver=3.0.6]

In the issue that Tjeerd has, it was part of some type of temp table/space which, in my opinion is easy to solve by moving ~/pgsql_temp somewhere else.

I know you will ask me what are these objects? We use in our report process a real tables to pull data and then join it for a complete report. It's not the best process but that's how the DEV team have developed this report.

So the question is, how I can force rsync to ignore these missing objects and mark my backup as successful?

Thanks,
Ahmed.