bareos server - disaster recovery - catalog restore

[Martin Krämer]

Hi All,

I am currently testing my disaster recovery strategy and already struggling with getting my bareos server instance back. :/

Based on instructions from https://docs.bareos.org/TasksAndConcepts/CatalogMaintenance.html#backing-up-your-bareos-database 
I have the following available during disaster recovery:

1. Backup volume (file volume), which contains (only 1 job)
1. /etc/bareos configuration files
2. ASCII dump of bareos sql database / catalog
2. Bootstrap file of the corresponding backup job created via Write Bootstrap
3. A newly setup bareos server (debian 10 - Bareos 19.2.6 - Postgresql 11.7.0)
   More of less equivalent to old "crushed" server (maybe small differences in minor patch version).

To restore I try to follow instructions from:

https://docs.bareos.org/Appendix/DisasterRecoveryUsingBareos.html#restoring-a-bareos-server

https://docs.bareos.org/Appendix/Troubleshooting.html#section-restorecatalog - instructions of "Solution with a catalog backup"

Unfortunately I end up with errors and without any idea how to further proceed.
(I did not try to manually extracting my files via bextract at the moment as what I understood this is not the intended way of disaster recovery...correct?)

TRY 1 - failure description:

First I mount my external storage and "re-initialize" the database by running drop_bareos_tables & make_bareos_tables.

I found that additionally (not within instructions) I had to run grant_bareos_privileges to ensure my bareos user has access to the database.
Furthermore I had to reload console once to ensure the default jobs like RestoreFiles are visible again.
After this as described I try to restore directly using my bootstrap file. - Unfortunately I end up with error 'No FileSet found for client "bareos-000001.example.corp".'

TRY 2 - failure description:

I follow exact same procedure as on TRY 1. Never the less i do not try to restore directly via boostrap file.

Instead (due to the failure of TRY 1) I first reload the backup job into current catalog using bscan and the bootstrap file.
Running now the RestoreFiles job I unfortunately end up with error 'No Full backup before 2020-04-12 07:34:08 found.'

Any Ideas what I am doing wrong?

I am really thankful for any hint.

Below are my console logs for more details.

Thank you in advance for any help.

TRY 1 - console log:

root@bareos-000001:~# export PGPASSWORD="MyP@$$w0rd" #root password for database

root@bareos-000001:~# /usr/lib/bareos/scripts/drop_bareos_tables -U postgres -h 127.0.0.1

Dropping postgresql tables

DROP TABLE

DROP TABLE

....

Dropping of Bareos PostgreSQL tables succeeded.

root@bareos-000001:~# /usr/lib/bareos/scripts/make_bareos_tables -U postgres -h 127.0.0.1

Making postgresql tables

CREATE TABLE

ALTER TABLE

CREATE INDEX

....

INSERT 0 1

DELETE 0

INSERT 0 1

Creation of Bareos PostgreSQL tables succeeded.

root@bareos-000001:~# /usr/lib/bareos/scripts/grant_bareos_privileges -U postgres -h 127.0.0.1

Granting postgresql tables

psql:/tmp/grants.sql.15159:1: ERROR:  role "bareos" already exists

GRANT

GRANT

....

Privileges for user bareos_usr granted ON database bareos.

root@bareos-000001:~# bconsole <<< "reload"

Connecting to Director localhost:9101

 Encryption: TLS_CHACHA20_POLY1305_SHA256

1000 OK: bareos-dir Version: 19.2.6 (11 February 2020)

You are connected using the default console

Enter a period (.) to cancel a command.

reload

reloaded

root@bareos-000001:~# bconsole <<< "restore client=bareos-000001.example.corp restoreclient=bareos-000001.example.corp restorejob=RestoreFiles bootstrap='/var/lib/bareos/storage/disk/8-bareos-000001.example.corp-srv_bareos-backup.bsr' Where=/tmp/bareos-restores select all done yes"

Connecting to Director localhost:9101

 Encryption: TLS_CHACHA20_POLY1305_SHA256

1000 OK: bareos-dir Version: 19.2.6 (11 February 2020)

You are connected using the default console

Enter a period (.) to cancel a command.

restore client=bareos-000001.example.corp restoreclient=bareos-000001.example.corp restorejob=RestoreFiles bootstrap='/var/lib/bareos/storage/disk/8-bareos-000001.example.corp-srv_bareos-backup.bsr' Where=/tmp/bareos-restores select all done yes

Using Catalog "MyCatalog"

The defined FileSet resources are:

Selection list for "FileSet" is empty!

No FileSet found for client "bareos-000001.example.corp".

TRY 2 - console log:

root@bareos-000001:~# bscan -s -v -b "/var/lib/bareos/storage/disk/8-bareos-000001.example.corp-srv_bareos-backup.bsr" -B postgresql -n bareos -u bareos -P "MyP@$$w0rd" -h 127.0.0.1 disk0

bscan: stored/butil.cc:288-0 Using device: "disk0" for reading.

12-Apr 07:32 bscan JobId 0: Ready to read from volume "bareos-000001.example.corp-srv_bareos-backup.20200411.124618-F" on device "disk0" (/var/lib/bareos/storage/disk).

bscan: stored/bscan.cc:370-0 Using Database: bareos, User: bareos_usr

12-Apr 07:32 bscan JobId 0: Forward spacing Volume "bareos-000001.example.corp-srv_bareos-backup.20200411.124618-F" to file:block 0:271.

bscan: stored/bscan.cc:1252-0 Created Client record for Client: bareos-000001.example.corp

bscan: stored/bscan.cc:1342-0 Created new JobId=1 record for original JobId=8

bscan: stored/bscan.cc:1280-0 Created FileSet record "srv_bareos"

bscan: stored/bscan.cc:1408-0 Updated Job termination record for JobId=1 Level=Full TermStat=T

bscan: stored/bscan.cc:1487-0 Could not create JobMedia record. ERR=cats/sql_create.cc:151 Update Media record UPDATE Media SET EndFile=0, EndBlock=275648 WHERE MediaId=0 failed: ERR=

12-Apr 07:32 bscan JobId 0: End of Volume at file 0 on device "disk0" (/var/lib/bareos/storage/disk), Volume "bareos-000001.example.corp-srv_bareos-backup.20200411.124618-F"

bscan (0): stored/bscan.cc:409-0 ========== JobId=0 ========

bscan: stored/bscan.cc:1195-0 Could not update media record. ERR=cats/sql_update.cc:380 Update failed: affected_rows=0 for UPDATE Media SET VolJobs=1,VolFiles=0,VolBlocks=0,VolBytes=275210,VolMounts=0,VolErrors=0,VolWrites=0,MaxVolBytes=0,VolStatus='',Slot=0,InChanger=0,VolReadTime=0,VolWriteTime=0,LabelType=0,StorageId=0,PoolId=0,VolRetention=0,VolUseDuration=0,MaxVolJobs=0,MaxVolFiles=0,Enabled=0,LocationId=0,ScratchPoolId=0,RecyclePoolId=0,RecycleCount=0,Recycle=0,ActionOnPurge=0,MinBlocksize=0,MaxBlocksize=0 WHERE VolumeName=''

12-Apr 07:32 bscan JobId 0: End of all volumes.

bscan: stored/bscan.cc:1195-0 Could not update media record. ERR=cats/sql_update.cc:380 Update failed: affected_rows=0 for UPDATE Media SET VolJobs=1,VolFiles=0,VolBlocks=0,VolBytes=275210,VolMounts=1,VolErrors=0,VolWrites=0,MaxVolBytes=0,VolStatus='',Slot=0,InChanger=0,VolReadTime=0,VolWriteTime=0,LabelType=0,StorageId=0,PoolId=0,VolRetention=0,VolUseDuration=0,MaxVolJobs=0,MaxVolFiles=0,Enabled=0,LocationId=0,ScratchPoolId=0,RecyclePoolId=0,RecycleCount=0,Recycle=0,ActionOnPurge=0,MinBlocksize=0,MaxBlocksize=0 WHERE VolumeName=''

bscan: stored/bscan.cc:815-0 End of all Volumes. VolFiles=0 VolBlocks=0 VolBytes=275,210

Records added or updated in the catalog:

      0 Media

      0 Pool

      1 Job

    153 File

      0 RestoreObject

12-Apr 07:32 bscan JobId 0: Releasing device "disk0" (/var/lib/bareos/storage/disk).

root@bareos-000001:~# bconsole <<< "restore client=bareos-000001.example.corp restoreclient=bareos-000001.example.corp restorejob=RestoreFiles bootstrap='/var/lib/bareos/storage/disk/8-bareos-000001.example.corp-srv_bareos-backup.bsr' Where=/tmp/bareos-restores select all done yes"

Connecting to Director localhost:9101

 Encryption: TLS_CHACHA20_POLY1305_SHA256

1000 OK: bareos-dir Version: 19.2.6 (11 February 2020)

You are connected using the default console

Enter a period (.) to cancel a command.

restore client=bareos-000001.example.corp restoreclient=bareos-000001.example.corp restorejob=RestoreFiles bootstrap='/var/lib/bareos/storage/disk/8-bareos-000001.example.corp-srv_bareos-backup.bsr' Where=/tmp/bareos-restores select all done yes

Using Catalog "MyCatalog"

Automatically selected FileSet: srv_bareos

No Full backup before 2020-04-12 07:34:08 found.

[Oleg Volkov]

I wrote memo for me for that purpose Disaster Recovery using bareos

It is about v17 and quite outdated, but main flow still the same.

[Oleg Volkov]

Also, here is Bare Metal Restore (BMR) using bareos file level backup

[Martin Krämer]

Thanks for your replies.

As I can see you are using bextract here.
I was aware that this is technically possible, while I just was not fully sure this is the "bareos intended way" to do.
Never the less doing (more or less) the same procedure as you do I am fully able to restore my bareos server.

Thanks for your help :)