barman backup failing with another backup process is running error

[vs...@nodalexchange.com]

All,

I am trying to take a barman backup of a server but fails with `ERROR: Another backup process is running` error. 

Per other articles on the internet, I have made sure that there is no other backup running on the server itself and the barman check command also shows no error for the server. 

Can anyone suggest what might be the issue?

I am using barman 2.10 and PostgreSQL 10.12

Thank you in Advance

[Luca Ferrari]

If you are really sure that no process is running, there could be a
lock file lying somewhere. If you did not setup lock_file_directory
you should find a loc file (.lock) in barman home directory, otherwise
you need to dig into lock_file_directory.

Luca

[vs...@nodalexchange.com]

Hello Luca,

Thank you for your reply. I see many lock files for the server in the barman home directory. Should they be deleted once the backup is taken? I understand barman creates a lock file when it is trying to take a backup but it looks like barman doesn't delete the lock file once the backup is complete. 

[Luca Ferrari]

On Mon, Aug 3, 2020 at 3:52 PM vs...@nodalexchange.com
<vs...@nodalexchange.com> wrote:
>
> Hello Luca,
>
> Thank you for your reply. I see many lock files for the server in the barman home directory. Should they be deleted once the backup is taken? I understand barman creates a lock file when it is trying to take a backup but it looks like barman doesn't delete the lock file once the backup is complete.

As far as I know, the lock file should disappear once the backup has finished.
It seems to me that barman tries to acquire the lock, and in the case
there's a failure, it does not delete it if it thinks there is another
process running. At least, that's what this suggests to me
<https://github.com/2ndquadrant-it/barman/blob/master/barman/server.py#L1097>,
but I would get some info from the developers about.
If the above is true, barman thinks there is still a process somewhere.

I would say a clean reboot could help fixing the process problem, but
again I'm not sure if this is a safe situation with regard to backups.
Are you really sure there are no barman processes running?

Luca

[vs...@nodalexchange.com]

Hi Luca,

I do not believe the lock file disappears once the backup is finished. I see lock files going back two years on my server even though the retention window is two weeks. So that is unrelated. I ran the select pg_is_in_backup() on the server to make sure any backup is in progress. It returned false. Although I was able to figure out the issue. There were orphan processes on the barman server that was still looking to back the server even though no backup was in progress. 

I killed those orphan process and voila!, I was able to take the backup now. 

Thank you for your help with this!

Best,
Viral

[Luca Ferrari]

On Mon, Aug 3, 2020 at 5:30 PM vs...@nodalexchange.com
<vs...@nodalexchange.com> wrote:
>
> Hi Luca,
>
> I do not believe the lock file disappears once the backup is finished. I see lock files going back two years on my server even though the retention window is two weeks. So that is unrelated. I ran the select pg_is_in_backup() on the server to make sure any backup is in progress. It returned false. Although I was able to figure out the issue. There were orphan processes on the barman server that was still looking to back the server even though no backup was in progress.
>
> I killed those orphan process and voila!, I was able to take the backup now.

Could you keep an eye on the lock files and see if they increase in
number as you take new backups?
I'm curious about.

Luca

[Frederic KAPP]

Hello

This lock files are not deleted once the backup is finished but inside there is the process of the PID : So if you have this kind of error have a look on the PID in the lock file

then ps -ef |grep NNN where NNN is the number you found and kill this process (zombie or not) if it is still present.

[vs...@nodalexchange.com]

Luca,

Like Frederic mentioned, I do not see lock files getting disappeared. I still have lock files from 2019 on my barman server. It would be nice if barman get rid of them automatically along with the backup once it goes out of the retention window. I had raised this issue in our group last year. Link here

Frederic,

Thank you for the explanation. I did not know the lock file keeps the PID of the backup process. I used ps -ef | grep backup to figure out the pid and then killed it. 

Viral

[Luca Ferrari]

On Tue, Aug 4, 2020 at 5:41 PM vs...@nodalexchange.com
<vs...@nodalexchange.com> wrote:
>
> Luca,
>
> Like Frederic mentioned, I do not see lock files getting disappeared. I still have lock files from 2019 on my barman server. It would be nice if barman get rid of them automatically along with the backup once it goes out of the retention window. I had raised this issue in our group last year. Link here

Uhm...I suspect there is something wrong here, because it does not
make any sense to last the PID of a succesfully terminated process.
Try opening an issue <https://github.com/2ndquadrant-it/barman/issues>.

Luca

[Savita Pandey]

Hi 

I am receiving the exact same barman issue 

Starting Incremental-backup

Wed Aug  5 04:08:18 GMT 2020
Another archive-wal process is already running on server <server-name>. Skipping to the next server 

Thanks

Savita pandey

--
--
You received this message because you are subscribed to the "Barman for PostgreSQL" group.
To post to this group, send email to pgba...@googlegroups.com
To unsubscribe from this group, send email to
pgbarman+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/pgbarman?hl=en?hl=en-GB

---
You received this message because you are subscribed to the Google Groups "Barman, Backup and Recovery Manager for PostgreSQL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pgbarman+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pgbarman/CAKoxK%2B7kCxo%3D%3D7LhFdtuOV0zTFmnWgxMJLg96ToGrtmU%3D51XJQ%40mail.gmail.com.

--

Thanks

Savita Pandey

469537803 

--

Thanks

Savita Pandey

469537803 

[Frederic KAPP]

I don't think there is something wrong here: it is usual when using a lock file to record inside it the PID of the new / running process : this lock file is not updated or deleted when the process is terminated that's all.

It seems that there is already an issue opened : huge amount of lock files #289

Frédéric

[Frederic KAPP]

Hello

Could you try to locate the lock file, see the PID inside it, ps -ef | grep PID-found and kill -9 PID-found

Frédéric