Help with retention

[Yariv Hazan]

Hello,
My retention is pretty simple(?) I have only full backups and I need to keep backups for
Last 6 daily backups for a week
Last 4 weekly backups for a month
Last 6 monthly backups for 6 months.

But:
1. All backups are kept for much longer without being pruned.
2. A daily backup volume is created every day but older daily backup volume are used instead.

I run version 22.1.1~pre26.eeec2501e without any changes to defaults.

Here is an examples of my configuration:

Job {
  Name = "lpsoar01_job_D"
  JobDefs = "DailyJobDefs"
  FileSet = "lpsoar01_fileset"
  Schedule = "DailyFullCycle"
}

JobDefs {
  Name = "DailyJobDefs"
  Type = Backup
  Level = Full
  Client = bareos-fd
  Schedule = "DailyFullCycle"
  Storage = File
  Messages = Standard
  Pool = DailyFullCyclePool
  Priority = 10
  Write Bootstrap = "/var/lib/bareos/%c.bsr"
  Full Backup Pool = DailyFullCyclePool                 # write Full Backups into "Full-Pool" Pool
  Differential Backup Pool = Differential  # write Diff Backups into "Differential" Pool
  Incremental Backup Pool = Incremental    # write Incr Backups into "Incremental" Pool
}

Pool {
  Name = DailyFullCyclePool
  Pool Type = Backup
  Recycle = yes
  AutoPrune = yes
  Volume Retention = 7 days
  Maximum Volume Jobs = 100
  Label Format = Daily-
  Maximum Volumes = 40
}

What am I doing wrong here?

Thanks,

Yariv

[Miguel Santos]

You need 3 different pools to do this.

One that keeps your:

- 6 daily backups for a week
- 4 weekly backups for a month
- 6 monthly backups for 6 months.

From there you can decide to either:

* create different jobs to write to different pools (simpler)

* make a migration/copy of the data to the pool (a bit more complicated)

It will not work with your current configuration because the retention works at the pool level.

Good luck.

--
You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/b1913656-a493-4671-bb44-f6aaf325806cn%40googlegroups.com.

[Yariv Hazan]

Thank you!

To simplify (for me L ) please look on the daily only.

Per configuration below I do 6 backups for a week.

My retention is on the pool level (Volume Retention = 7 days) right? On what level it should be? Or how I should define it in the pool level that I’m not doing correctly now?

Thanks,

Yariv

[Miguel Santos]

This is what I would do.

Job {
  Name = "lpsoar01_job_D"
  JobDefs = "DailyJobDefs"
  FileSet = "lpsoar01_fileset"

  Schedule = "CustomCycle"

}

JobDefs {
  Name = "DailyJobDefs"
  Type = Backup
  Level = Full
  Client = bareos-fd
  Schedule = "DailyFullCycle"
  Storage = File
  Messages = Standard
  Pool = DailyFullCyclePool
  Priority = 10
  Write Bootstrap = "/var/lib/bareos/%c.bsr"
  Full Backup Pool = DailyFullCyclePool                 # write Full Backups into "Full-Pool" Pool
  Differential Backup Pool = Differential  # write Diff Backups into "Differential" Pool
  Incremental Backup Pool = Incremental    # write Incr Backups into "Incremental" Pool
}

Pool {
  Name = Daily

  Pool Type = Backup
  Recycle = yes
  AutoPrune = yes
  Volume Retention = 7 days
  Maximum Volume Jobs = 100
  Label Format = Daily-

  Maximum Volumes = 7
}

Pool {
  Name = Weekly

  Pool Type = Backup
  Recycle = yes
  AutoPrune = yes

  Volume Retention = 31 days

  Maximum Volume Jobs = 100

  Label Format = Weekly-
  Maximum Volumes = 5
}

Pool {

  Name = Monthly

  Pool Type = Backup
  Recycle = yes
  AutoPrune = yes

  Volume Retention = 181 days

  Maximum Volume Jobs = 100

  Label Format = Monthly-
  Maximum Volumes = 6
}

Schedule {

  Name = CustomCycle

  Run = Level=Full Pool=Daily 1st sun at 22:00

  Run = Level=Full Pool=Weekly 2nd-5th sun at 22:00

  Run = Level=Full Pool=Monthly mon-sat at 22:00

}

I hope you are able to see the bold text, but basically you create 3 pools and a custom schedule that will take care of writting to the specific pool you need. I have no tested this configuration so it is at your discretion.

Good luck.

[Miguel Santos]

Forgot to change Max Volume Jobs, I will set it to 1, not 100.

[Bruno Friedmann (bruno-at-bareos)]

And if you already have volumes, don't forget to update them all.

bconsole -> update -> volume parameter -> 

13: All Volumes from Pool
or

14: All Volumes from all Pools

[Yariv Hazan]

Hi Guys,

I did as per your help and created the files per your examples thank you so much.

I can see the bold letering and also updated the volumes, thanks!

Seems backups are being saved to an old volume Monthly-0006 , meaning they are not written to a new Daily-XXXX that is supposed to be created.

Please see my configuration:

 

Job {

  Name = "lpsoar01_job_D"

  JobDefs = "DailyJobDefs"

  FileSet = "lpsoar01_fileset"

  Schedule = "CustomCycle"

}

 

Job {

   Name = "lpsyslog01_job_D"

   JobDefs = "DailyJobDefs"

   FileSet = "lpsyslog01_fileset"

   Schedule = "CustomCycle"

}

 

 

JobDefs {

  Name = "DailyJobDefs"

  Type = Backup

  Level = Full

  Client = bareos-fd

  Schedule = "CustomCycle"   ß--------------I’ve changed this line

  Storage = File

  Messages = Standard

  Pool = DailyFullCyclePool

  Priority = 10

  Write Bootstrap = "/var/lib/bareos/%c.bsr"

  Full Backup Pool = DailyFullCyclePool    # write Full Backups into "Full-Pool" Pool

  Differential Backup Pool = Differential  # write Diff Backups into "Differential" Pool

  Incremental Backup Pool = Incremental    # write Incr Backups into "Incremental" Pool

}

 

Pool {

  Name = DailyFullCyclePool

  Pool Type = Backup

  Recycle = yes

  AutoPrune = yes

  Volume Retention = 7 days

  Maximum Volume Jobs = 1

  Label Format = Daily-

  Maximum Volumes = 7

}

 

Pool {

  Name = WeeklyFullCyclePool

  Pool Type = Backup

  Recycle = yes

  AutoPrune = yes

  Volume Retention = 31 days

  Maximum Volume Jobs = 100

  Label Format = Weekly-

  Maximum Volumes = 5

}

 

Pool {

  Name = MonthlyFullCyclePool

  Pool Type = Backup

  Recycle = yes

  AutoPrune = yes

  Volume Retention = 181 days

  Maximum Volume Jobs = 100

  Label Format = Monthly-

  Maximum Volumes = 6

}

 

Schedule {

  Name = CustomCycle

  Run = Level=Full Pool=DailyFullCyclePool 1st sun at 22:00                          ßChanged this line according to my existing pool

  Run = Level=Full Pool=WeeklyFullCyclePool 2nd-5th sun at 22:00             ß Changed this line according to my existing pool

  Run = Level=Full Pool=MonthlyFullCyclePool mon-sat at 22:00                  ß Changed this line according to my existing pool

[Miguel Santos]

I think I have seen this before and probably it may be a bug, but I have not had the initiative to look further.

So, what I believe is happening is that the Pool is taken from either the defaults.

Can you try to change the schedule so it ends up like this?

Schedule {

  Name = CustomCycle

  Run = Level=Full Pool=DailyFullCyclePool FullPool=DailyFullCyclePool  1st sun at 22:00

  Run = Level=Full Pool=WeeklyFullCyclePool FullPool=WeeklyFullCyclePool  2nd-5th sun at 22:00

  Run = Level=Full Pool=MonthlyFullCyclePool FullPool=MonthlyFullCyclePool  mon-sat at 22:00

}

Just adding FullPool to the schedule.

Good luck.

[Yariv Hazan]

This looks OK, but maybe its the naming is the issue.

What pool should hold which backups?

DailyFullCyclePool should hold mon-sat at 22:00

MonthlyFullCyclePool  should hold 1st sun at 22:00

Right?

[Miguel Santos]

Sorry about that, yes, the name of the pools is all mixed up since the beginning when I sent the first message.

But the fix should be self explanatory.

  Run = Level=Full Pool=DailyFullCyclePool FullPool=DailyFullCyclePool  mon-sat at 22:00

  Run = Level=Full Pool=WeeklyFullCyclePool FullPool=WeeklyFullCyclePool  2nd-5th sun at 22:00

  Run = Level=Full Pool=MonthlyFullCyclePool FullPool=MonthlyFullCyclePool  1st sun at 22:00

[Yariv Hazan]

Great so fixed and let it run see how it goes.

[Yariv Hazan]

I'm not sure its related but using bconsole I have deleted an empty unused volume restarted director and now I get this error running jobs:

bareos-sd JobId 663: Job wpcoreopswat01_job_D.2023-11-09_11.42.35_06 is waiting. Cannot find any appendable volumes.
Please use the "label" command to create a new Volume for:
Storage: "FileStorage" (/var/lib/bareos/storage)
Pool: DailyFullCyclePool
Media type: File

Runing label in bconsole and creating a new volume in  DailyFullCyclePool does not seem to solve the problem.

[Miguel Santos]

Not related, but what most likely happened is that it cannot find the volume you deleted and bareos had plans to write into it. Even though it was yet unused, the record of that media was in the DB. (I am speculating here based on what you describe)

The easiest option is to create a volume with the same name.

Do a "status storage" in bconsole, then select the storage name and it most likely will tell you what the storage daemon is expecting to write to. Create a volume in that pool with the same name.

If you deleted volumes, please make sure you also delete volumes from your DB by issuing:

"delete volume" in bconsole.

[Yariv Hazan]

Please see below some of the output I get running "status storage" in any case I get the "*" prompt so nothing to choose from:

There are WARNINGS for this storagedaemon's configuration!
See output of 'bareos-sd -t' for details.

Running Jobs:
Writing: Full Backup job wpcoreopswat01_job_D JobId=663 Volume=""
    pool="DailyFullCyclePool" device="FileStorage" (/var/lib/bareos/storage)
    spooling=0 despooling=0 despool_wait=0
    Files=0 Bytes=0 AveBytes/sec=0 LastBytes/sec=0
    FDReadSeqNo=6 in_msg=6 out_msg=4 fd=6

Device status:

Device "FileStorage" (/var/lib/bareos/storage) is not open.
    Device is BLOCKED waiting to create a volume for:

       Pool:        DailyFullCyclePool
       Media type:  File

[Miguel Santos]

Something is wrong between memory, filesystem and database.

bareos-sd -t to see any potential issues with the configuration. Hopefully it will tell you what's wrong.

Restart all daemons after you have fixed the issue.

[Yariv Hazan]

This is what I get, does not seem relevant.

Will reboot the system and check thanks.

There are configuration warnings:
 * Device FileStorage: unlimited (0) 'Maximum Concurrent Jobs' (the default) reduces the restore peformance.
 * Device FileStorage: the default value for 'Maximum Concurrent Jobs' will change from 0 (unlimited) to 1 in Bareos 23.

[Yariv Hazan]

After reboot I've created a volume manually using label and job seems to be working.

Does this mean issue is resolved and further volumes will be created automatically as needed?

bareos-dir JobId 664: Max Volume jobs=1 exceeded. Marking Volume "NewVol1" as Used.
bareos-sd JobId 664: Wrote label to prelabeled Volume "NewVol1" on device "FileStorage" (/var/lib/bareos/storage)

[Yariv Hazan]

Seems bareos stopped at some point using the newly manually created volume and does not create new volumes as needed.

What do I need to do to restore the volumes creations?

bareos-sd JobId 666: Job thclpcvprox03_job_D.2023-11-09_21.00.00_47 is waiting. Cannot find any appendable volumes.

Please use the "label" command to create a new Volume for:
Storage: "FileStorage" (/var/lib/bareos/storage)
Pool: DailyFullCyclePool
Media type: File