New PostgreSQL plugin search volunteers for testing

[Bruno Friedmann]

Hi community, users and customers,

As we're moving towards Bareos 23, we could use some help testing the all new PostgreSQL plugin which is needed because of changes happened in PostgreSQL 15.

Main features compared to the old one:

- use non exclusive mode
- support all clusters starting from version 10
- support clusters with tablespace
- support non standard port
- support password parameter
- better debugging
- renew of parameters names and variables
- renewed documentation

We're looking for a few early adopters to work with directly. If you'd like to participate please contact https://bareos.com/contact with a small description of your environment to have multiple environments as test scenarios.

And "For all others who'd like to test without early adapters program the testing packages are now available
at this address https://download.bareos.org/experimental/PR-1541/
The corresponding documentation is also available on
https://download.bareos.org/experimental/PR-1541/BareosMainReference/TasksAndConcepts/Plugins.html#postgresql-plugin

The ongoing development PR is available here https://github.com/bareos/bareos/pull/1541

We would really appreciate comments, remarks, and tests.
Better to use github comments if you want to participate, and help.

The new plugin aims to replace the old one, and will be delivered in the same package 

bareos-filedaemon-postgresql-python-plugin
Notice The old plugin (deprecated) will still also be delivered in 23, but we recommend (not sure if we will force) its usage to only restore previous backups done with, until you migrate to the new one.

Thanks and all the best

[Dennis Benndorf]

Hi Bruno,

we tested your new plugin with a quite large database with a running application using it.

When doing a full backup we ran into:

JobId 5116734: Fatal error: bareosfd: Traceback (most recent call last):
File "/usr/lib64/bareos/plugins/BareosFdWrapper.py", line 61, in start_backup_file
return bareos_fd_plugin_object.start_backup_file(savepkt)
File "/usr/lib64/bareos/plugins/bareos-fd-postgresql.py", line 703, in start_backup_file
return super().start_backup_file(savepkt)
File "/usr/lib64/bareos/plugins/BareosFdPluginLocalFilesBaseclass.py", line 118, in start_backup_file
mystatp.st_mode = statp.st_mode
UnboundLocalError: local variable 'statp' referenced before assignment

JobId 5116734: Error: python3-fd-mod: Could net get stat-info for file /db/pgsql/pgsql-15/pg_subtrans/00BF: "[Errno 2] No such file or directory: '/db/pgsql/pgsql-15/pg_subtrans/00BF'" 

It seems that the plugin fails when it wants to backup a subtransaction that has been commited and removed from that directory in the meantime.

Is it save to exclude the pg_subtrans directory?

With kind regards,

Dennis

-------- Ursprüngliche Nachricht --------

Von: Bruno Friedmann <bruno.f...@bareos.com>

An: bareos-users <bareos...@googlegroups.com>

Betreff: [bareos-users] New PostgreSQL plugin search volunteers for testing

Datum: 13.09.2023 09:46:08

--
You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/8d4432a2-4561-4adf-9fa6-b8aff31da6f0n%40googlegroups.com.

[Bruno Friedmann (bruno-at-bareos)]

Hello Dennis, 

Nice you already tried the fresh PG16 meat, after checking the new documentation about subtrans we can see that upstream has added that directory as shouldn't be saved.

```The contents of the directories pg_dynshmem/, pg_notify/, pg_serial/, pg_snapshots/, pg_stat_tmp/, and pg_subtrans/ (but not the directories themselves) can be omitted from the backup as they will be initialized on postmaster startup.```

Will apply that to the plugin by default before release.

[Bruno Friedmann (bruno-at-bareos)]

Sorry in fact it is in 15 doc too.

[Dennis Benndorf]

Hi Bruno,

its me once again. The next run with excluded pg_subtrans dir fails on the base directory:

JobId 5116772: Fatal error: bareosfd: Traceback (most recent call last):

File "/usr/lib64/bareos/plugins/BareosFdWrapper.py", line 61, in start_backup_file
return bareos_fd_plugin_object.start_backup_file(savepkt)
File "/usr/lib64/bareos/plugins/bareos-fd-postgresql.py", line 703, in start_backup_file
return super().start_backup_file(savepkt)
File "/usr/lib64/bareos/plugins/BareosFdPluginLocalFilesBaseclass.py", line 118, in start_backup_file
mystatp.st_mode = statp.st_mode
UnboundLocalError: local variable 'statp' referenced before assignment

JobId 5116772: Error: python3-fd-mod: Could net get stat-info for file /db/pgsql/pgsql-15/base/16386/2117113: "[Errno 2] No such file or directory: '/db/pgsql/pgsql-15/base/16386/2117113'"

It is a active database. Any thoughts on this?

With kind regards,

Dennis

-------- Ursprüngliche Nachricht --------

Von: Bruno Friedmann (bruno-at-bareos) <bruno.f...@bareos.com>

An: bareos-users <bareos...@googlegroups.com>

Betreff: Re: [bareos-users] New PostgreSQL plugin search volunteers for testing

Datum: 18.09.2023 16:04:42

[Bruno Friedmann (bruno-at-bareos)]

Would you mind to try to run the job with a debug level set to 150 + timestamp. and attach the job log.

I was trying to fix that case, and just emit a warning when the database has removed and changed a file during the backup.

[Dennis Benndorf]

Hi Bruno,

we increased the debug level but the load on the database decreased so it ran successful.

Will come back to this when there is more load and the problem happens again. 

Another thing I found is that Increments seem to not work as I would expect them:

The Full which ended at 2023-09-19 09:32:39 contains:

/backupserver/postgresdumps/wal_archive/
/backupserver/postgresdumps/wal_archive/0000000100000135000000A1
/backupserver/postgresdumps/wal_archive/0000000100000135000000A2
/backupserver/postgresdumps/wal_archive/0000000100000135000000A2.00000028.backup
/backupserver/postgresdumps/wal_archive/0000000100000135000000A3
/backupserver/postgresdumps/wal_archive/0000000100000135000000A4

In those archive dir at 10:08 a new file 0000000100000135000000A5 was created.

But the increment at(10:18:41) after that timestamp does not include the file:

*list files jobid=5118974
*

The log says:

		JobId 5118974: python3-fd-mod: Same LSN 00000135/A6517EE8 as last time - nothing to do
		JobId 5118974: python3-fd-mod: Current LSN 00000135/A6517EE8, last LSN: 135/A42EF7F0
		JobId 5118974: python3-fd-mod: Connected to PostgreSQL version 150004
		JobId 5118974: Spooling data ...
		JobId 5118974: ACL support is enabled
		JobId 5118974: Extended attribute support is enabled
		JobId 5118974: Connected Storage daemon at bareos:9103, encryption: None
		JobId 5118974: Encryption: None
		JobId 5118974: python3-fd-mod: Got pg major version 15 from restore object of job 5118972
		JobId 5118974: python3-fd-mod: Got lastLSN 135/A42EF7F0 from restore object of job 5118972
		JobId 5118974: python3-fd-mod: Got last_backup_stop_time 1695108740 from restore object of job 5118972

With kind regards,

Dennis

-------- Ursprüngliche Nachricht --------

Von: Bruno Friedmann (bruno-at-bareos) <bruno.f...@bareos.com>

An: bareos-users <bareos...@googlegroups.com>

Betreff: Re: [bareos-users] New PostgreSQL plugin search volunteers for testing

Datum: 18.09.2023 17:39:01

--
You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/c8072468-d608-4c1d-87a5-7bf07a15a3fbn%40googlegroups.com.

[Dennis Benndorf]

Hi Bruno,

I think we found the cause:

In the function start_backup_job there is a if-statement

if self.switch_wal and current_lsn > self.lastLSN:

We added some additional information to the else-clause to come closer to that problem:

        else: 
               # Nothing has changed since last backup - only send ROP this time 
               bareosfd.JobMessage( 
                   bareosfd.M_INFO, 
                   f"Same LSN {current_lsn} as last time - nothing to do; Switch_wal: {self.switch_wal} LastLSN: {self.lastLSN} \n", 
               ) 
               return bareosfd.bRC_OK

And found out that current_lsn and lastLSN are different in their format which leads the plugin to choose the else condition.

*root@bareos:~# python3
Python 3.8.10 (default, May 26 2023, 14:05:08)  
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> "00000135/B5371B08" > "135/A42EF7F0"
False
>>> "135/B5371B08" > "135/A42EF7F0"
True
>>>

So the fix should be to remove leading zeros or to fill it with leading zeros on the other var.

With kind regards,

Dennis

-------- Ursprüngliche Nachricht --------

Von: Dennis Benndorf <dennis....@googlemail.com>

An: Bruno Friedmann (bruno-at-bareos) <bruno.f...@bareos.com>, bareos-users <bareos...@googlegroups.com>

Betreff: Re: [bareos-users] New PostgreSQL plugin search volunteers for testing

Datum: 19.09.2023 10:33:33

[bareos-users]

Thanks Dennis, this will help the fixes, the new lsn has to be formatted 

I will create the fix for that.

In the meantime, I'm working in improving our systemtests to create load and activity in the cluster during the backup

to be able to reproduce and fix for all that case.

Stay tuned, I will post when the patches will be in.

[Bruno Friedmann (bruno-at-bareos)]

Dear community, it has been a long time since we were able to release a new set of binaries to test the new bareos-fd-postgresql plugin.
This long awaited release is now over, and you can get the updated binaries directly from the experimental project
https://download.bareos.org/experimental/PR-1541/

For those who have already installed and tested it, you should be able to simply update your test system with your operating system command
like dnf, apt, zypper, etc.

The plugin is now able to backup any PostgreSQL cluster from version 10 to 16, it will no longer complain if file/dir changes during the
during the backup (as these changes are recorded in wal and saved at the end).

We improved our system test to cover the case with cluster having activity or not, using tablespace, etc.

We add to rename some configuration parameters to make the whole more coherent, check the documentation or code if you need to adapt.

Reminder: the whole new documentation chapter about the plugin is available at
https://download.bareos.org/experimental/PR-1541/BareosMainReference/TasksAndConcepts/Plugins.html#postgresql-plugin

Thanks for any test you can do in your own environment, please report failure/success here on the mailing list,
and any code improvements in the github comments here

https://github.com/bareos/bareos/pull/1541/

Happy testing.