Volume mangagement best practices

[Ludwig BaLu]

Hi,

I am very happy about bareos. It is very nice to configure and awesome with usabillity. But there are some things, I couldn't understand.

For my setup, I would like to backup all my machines. Every week a full backup and every day incremental. For this setup, my full backups should not be deleted for 1 month, but my incremental should be deleted after 1 week.

Here are my configs

#
# Full Pool definition
#
Pool {
Name = Full
Pool Type = Backup
Recycle = yes # Bareos can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
Volume Retention = 1 month # How long should the Full Backups be kept? (#06)
Label Format = "Full-" # Volumes will be labeled "Full-<volume-id>"
}

Pool {
Name = Incremental
Pool Type = Backup
Recycle = yes # Bareos can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
Label Format = "Incremental-" # Volumes will be labeled "Incremental-<volume-id>"
Volume Retention = 7 days # How long should the Full Backups be kept? (#06)
#Maximum Volumes = 32 # Limit number of Volumes in Pool
#Maximum Volume Jobs = 10
#Volume Retention = 7d # How long should the Incremental Backups be kept? (#12)
#Maximum Volume Bytes = 1G # Limit Volume size to something reasonable
}

Schedule {
Name = "WeeklyCycle"
Run = Level=Full Pool=Full on 1st at 2:00 # (#04)
Run = Level=Incremental Pool=Incremental on 2-31 at 02:00 # (#10)
}

Now I have over 100 volumes in my folder. But in my process, there should be max 8 for full and 6 or 7 for incremental. Every volume is in state 'Used', but there should be 'pruned', shouldn it?

What did I wrong in my configuration? Can anyone give me a hint for this?

Thank you very much,
Ludwig

[Bruno Friedmann]

If you want to run a full each week then a day should be inserted here

> Run = Level=Full Pool=Full on 1st at 2:00 # (#04)

Run = Level=Full Pool=Full on sun at 2:00 # (#04)

Then every sunday at 2 (I would avoid the 2-3 range due to DST) a full is run.

As advise, I would always have at least 2 full backups, if for any reasons the
last run failed you still have one.


--

Bruno Friedmann
Ioda-Net Sàrl www.ioda-net.ch

openSUSE Member & Board, fsfe fellowship
GPG KEY : D5C9B751C4653227
irc: tigerfoot

[Ludwig BaLu]

Hi Bruno,

thank you for your answer and for your suggestions. You are right, but how can I avoid, that bareos uses so much volumes? It does not prune the used volumes. All of my harddisks are running out of space, because alle volumes are in state 'used'.

Can you advise, what to do?

Thanks very much,
Ludwig

[Bruno Friedmann]

> Hi Bruno,
>
> thank you for your answer and for your suggestions. You are right, but how can I avoid, that bareos uses so much volumes? It does not prune the used volumes. All of my harddisks are running out of space, because alle volumes are in state 'used'.
>
> Can you advise, what to do?
>
> Thanks very much,
> Ludwig
>
>

If your volume are not purged then you have another problem somewhere.
Typically if you change the pool retention time afterward, the first volumes there
need to be adjusted to have their time aligned with the new retention.

The purge and retention system works once mastered, but without a bit more of your
configuration it will be hard to guess for us.
list media etc ...

[Ludwig BaLu]

Hi Bruno,

yes but I don't know, where to search. What I have found is a problem with the "Use Once" directive. If a add this to my pool definition, the storage is blocked and new volumes aren't created. Here is one Client configured in my system:

##################
# Bareos Dir
##################

Client {
Name = franky-fd
Address = 1.2.3.4
Password = "{secretpw}" # password for FileDaemon
FDPort = 9102
}

Job {
Name = "franky-ink"
JobDefs = "DefaultJob-franky"
Level = Incremental
Client = franky-fd
FileSet = "franky" # selftest fileset (#13)
}

JobDefs {
Name = "DefaultJob-franky"
Type = Backup
Storage = franky-sd
Schedule = "WeeklyCycle"
Messages = Standard
Pool = Incremental
Priority = 10
Full Backup Pool = Full
Incremental Backup Pool = Incremental
Maximum Concurrent Jobs = 10
Write Bootstrap = "/var/lib/bareos/%c.bsr"
}



FileSet {
Name = "franky"
Include {
Options {
Signature = MD5 # calculate md5 checksum per file
}
File = "/root"
File = "/etc"
File = "/var/www"
File = "/var/www"
File = "/var/lib/pgsql"

}
}

Pool {
Name = Full
Pool Type = Backup
Recycle = yes # Bareos can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes

VolumeRetention = 1 month # How long should the Full Backups be kept? (#06)

Label Format = "Full-" # Volumes will be labeled "Full-<volume-id>"

Use Volume Once = yes

}

Pool {
Name = Incremental
Pool Type = Backup
Recycle = yes # Bareos can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
Label Format = "Incremental-" # Volumes will be labeled "Incremental-<volume-id>"

VolumeRetention = 2 h # How long should the Full Backups be kept? (#06)
Use Volume Once = yes
Maximum Volumes = 75 # Limit number of Volumes in Pool

}

Schedule {
Name = "WeeklyCycle"
Run = Level=Full Pool=Full on 1st at 2:00 # (#04)
Run = Level=Incremental Pool=Incremental on 2-31 at 02:00 # (#10)
}

##################
# Bareos SD
##################
Device {
Name = FrankyStorage
Media Type = File2
Archive Device = /backup/bareos
LabelMedia = yes; # lets Bareos label unlabeled media
Random Access = yes;
AutomaticMount = yes; # when device opened, read it
RemovableMedia = yes;
AlwaysOpen = no;
}

##################
# Bareos FD
##################
FileDaemon { # this is me
Name = franky-fd
Maximum Concurrent Jobs = 20
Plugin Directory = /usr/lib64/bareos/plugins
compatible = no
}


Can you take a look please? I would love to understand, what I have missed here.

Best regards,
Ludwig

[Bruno Friedmann]

hey I've be forced to recheck the doc for the Volume Once :-)
Use Volume Once =<yesjno>
Please note!
This directive is deprecated.
Use
Maximum Volume Jobs Dir Pool = 1 instead.

to discover it is deprecated ...


There's nothing absolutely wrong with your definitions.
I don't understand how you can expect to have a good pruning with the Incremental lifetime set to 2 hours.
That seems wired to me.

For example a typical pool I'm using for current data (Full and Inc goes there)
# pool ioda data
Pool {
Name = "ioda"

Pool Type = Backup
Recycle = yes # Bareos can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes

Volume Retention = 12 days # 12 days (We keep 2 weeks on it)
Maximum Volumes = 14
Maximum Volume Jobs = 1
FileRetention = 12 days
JobRetention = 12 days
LabelFormat = "ioda_"
}


a list media will give this result
Using Catalog "MyCatalog"
+---------+------------+-----------+---------+-----------------+----------+--------------+---------+------+-----------+-----------+---------------------+
| mediaid | volumename | volstatus | enabled | volbytes | volfiles | volretention | recycle | slot | inchanger | mediatype | lastwritten |
+---------+------------+-----------+---------+-----------------+----------+--------------+---------+------+-----------+-----------+---------------------+
| 1 | ioda_w1d0 | Used | 1 | 818,352,185 | 0 | 1,036,800 | 1 | 0 | 0 | ioda | 2015-09-01 22:34:42 |
| 2 | ioda_w1d1 | Used | 1 | 875,680,688 | 0 | 1,036,800 | 1 | 0 | 0 | ioda | 2015-09-02 22:35:21 |
| 3 | ioda_w1d2 | Used | 1 | 645,773,913 | 0 | 1,036,800 | 1 | 0 | 0 | ioda | 2015-09-03 22:34:49 |
| 4 | ioda_w1d3 | Used | 1 | 70,159,941,478 | 16 | 1,036,800 | 1 | 0 | 0 | ioda | 2015-09-04 23:01:01 |
| 5 | ioda_w1d4 | Used | 1 | 5,450,775,870 | 1 | 1,036,800 | 1 | 0 | 0 | ioda | 2015-09-05 22:39:45 |
| 6 | ioda_w1d5 | Used | 1 | 313,255,514,901 | 72 | 1,036,800 | 1 | 0 | 0 | ioda | 2015-09-07 01:17:51 |
| 7 | ioda_w1d6 | Used | 1 | 1,138,057,547 | 0 | 1,036,800 | 1 | 0 | 0 | ioda | 2015-08-24 22:32:24 |
| 8 | ioda_w2d0 | Used | 1 | 667,068,234 | 0 | 1,036,800 | 1 | 0 | 0 | ioda | 2015-08-25 22:33:07 |
| 9 | ioda_w2d1 | Used | 1 | 821,980,363 | 0 | 1,036,800 | 1 | 0 | 0 | ioda | 2015-08-27 22:33:56 |
| 10 | ioda_w2d2 | Used | 1 | 678,748,967 | 0 | 1,036,800 | 1 | 0 | 0 | ioda | 2015-08-26 22:32:47 |
| 11 | ioda_w2d3 | Used | 1 | 606,238,740 | 0 | 1,036,800 | 1 | 0 | 0 | ioda | 2015-08-28 22:33:30 |
| 12 | ioda_w2d4 | Used | 1 | 4,731,533,720 | 1 | 1,036,800 | 1 | 0 | 0 | ioda | 2015-08-29 22:37:49 |
| 13 | ioda_w2d5 | Used | 1 | 323,105,264,797 | 75 | 1,036,800 | 1 | 0 | 0 | ioda | 2015-08-31 00:56:25 |
| 14 | ioda_w2d6 | Used | 1 | 721,784,763 | 0 | 1,036,800 | 1 | 0 | 0 | ioda | 2015-08-31 22:33:20 |
+---------+------------+-----------+---------+-----------------+----------+--------------+---------+------+-----------+-----------+---------------------+

What happen there : I'm keeping 2 fulls (so one get replace every sunday)
and the following day are incremental.
I've setup the file and job retention a bit lower, in case a sunday backup end later
(or if I miss something). Bareos will not touch my backup file until a new job
is starting to write to it.

Have a look at the volume retention its really what I have in my pools definition.
12d = 3600*24*12 = 1 036 800 seconds.

What about your media in the 2 pools : are they having the right retention time you expect ?

You can update all the media of the pool by its definition with bconsole in case they diverge too much

*update
Using Catalog "MyCatalog"
Update choice:
1: Volume parameters
2: Pool from resource
3: Slots from autochanger
4: Long term statistics
Choose catalog item to update (1-4): 1
Parameters to modify:
1: Volume Status
2: Volume Retention Period
3: Volume Use Duration
4: Maximum Volume Jobs
5: Maximum Volume Files
6: Maximum Volume Bytes
7: Recycle Flag
8: Slot
9: InChanger Flag
10: Volume Files
11: Pool
12: Volume from Pool
13: All Volumes from Pool
14: All Volumes from all Pools
15: Enabled
16: RecyclePool
17: Action On Purge
18: Done
Select parameter to modify (1-18): 13
The defined Pool resources are:
1: Catalog
2: Catalog_External
3: Scratch
4: Default
5: ioda
6: computers
7: External_Months
8: External_Years
9: External_Archives
Select Pool resource (1-9): 5
All Volume defaults updated from "ioda" Pool record.

Keep hope, sometimes the setup can be a bit tricky, but once you find a way that fit your needs
(cause there's several way to do things :-) it will last for years.