Guide for parameters and job types

[Jon Schewe]

I've been using bareos for quite some time, mostly with the standard Full, Differential, Incremental backup scheme. The parameters used for this scheme seem to be fairly clear. 

Recently I have had a need to use the Always Incremental scheme and Virtual Full backups for offsite storage. It seems that the parameters used for these are sometimes the same as the standard scheme and sometimes aren't. It also appears that new parameters have been added in the past few years to help with these new schemes, however it hasn't been very clear to me which parameters matter. Is there a good guide somewhere of what parameters are used for each scheme?

I found the documentation of the Always Incremental scheme helpful. That seems pretty good. I find the most trouble with Virtual Full backups for offsites.

My questions are:

- What pool is used for the source of a Virtual Full backup? Or is that a pointless question because bareos doesn't source from a pool for such a backup, but rather considers all backup jobs for the particular client that aren't archive jobs and uses whatever pool those volumes are in.

- What pool is used for the destination of the Virtual Full backup?

  - Pool -> Next Pool?

  - Job -> Next Pool?

  - Job -> Virtual Full Backup Pool? This one appears to have been added since I first looked at Virtual Full backups about 5 years ago and is perhaps the correct one to use.

I have not used Copy or Migration jobs, although feel like I would have similar questions for those jobs as I do for Virtual Full backups.

[Jon Schewe]

Well, I made changes to my configuration before last weekend to use "job -> Virtual Full Backup Pool" to specify the destination for virtual full backups and got errors that the pool resource didn't have "next pool" set, so the backups failed. What's the correct configuration for setting up an archive job from a set of onsite jobs?

My onsite jobs consist of 3 patterns:

1. Always Incremental to file

2. Standard Full, Differential, Incremental to file

3. Standard Full, Differential, Incremental to tape

For each of these I'd like to do a Virtual Full backup every 2 weeks to a pool of tapes that will go offsite. I know how to automatically mark the offsite jobs as archive jobs using RunScript so that they aren't considered for standard restores or as sources of the Virtual Full backups. The question is what are the right parameters to use for these jobs? I had thought that the Virtual Full backup would not need a source pool directive because they just find all Backup jobs for the specified job and consolidate them into a full backup. Given that I would have expected to only need to specify Virtual Full Backup Pool. However as I pointed out above, this isn't working.

[Bruno Friedmann (bruno-at-bareos)]

Hi Jon

For example in your AI you should have a full/consolidate pool with is the target of where incremental are stored.

You extend that pool with next pool -> pointing to your tapes archives pool (that one need to have its storage definition)

Here is how I use it @home to export once a month my ai_jobs

ai consoliated pool

Pool {
  Name = "ai_consolidated"
  Description = "AI consolidated"
  ActionOnPurge = Truncate
  MaximumVolumes = 48
  MaximumVolumeJobs = 8
  VolumeRetention = 3 months
  VolumeUseDuration = 23 hours
  NextPool = "vf_archives"
  Storage = "ai_storage"
  RecyclePool = "scratch-ai"
  ScratchPool = "scratch-ai"
  FileRetention = 3 months
  JobRetention = 3 months
}

*show pool=vf_archives
Pool {
  Name = "vf_archives"
  Description = "VF Archives Virtual Fulls"
  LabelFormat = "vf_archives-"
  MaximumVolumes = 16
  MaximumVolumeBytes = 1 t 776 g
  VolumeRetention = 3 months 2 weeks 1 days
  VolumeUseDuration = 1 days
  Storage = "vf_archives"
  FileRetention = 4 months
  JobRetention = 4 months
}

One of my VF job looks like 

*show job=vf_month
Job {
  Name = "vf_month"
  Type = Backup
  Level = VirtualFull
  Messages = "standard"
  Pool = "ai_consolidated"
  VirtualFullBackupPool = "vf_archives"
  Client = "client-fd"
  FileSet = "fileset"
  Schedule = "vf_archives_month"
  JobDefs = "default_jobdef"
  Priority = 12
  RunScript {
    Console = "update jobid=%i jobtype=A"
    RunsWhen = "after"
  }
  Accurate = Yes
  AllowDuplicateJobs = No
  CancelLowerLevelDuplicates = Yes
  CancelQueuedDuplicates = Yes
}

Which is scheduled by a normal schedule first sunday at 3am.

Hope this will help you to glue the last bytes of your configuration.

[Jon Schewe]

So I see you have both "Pool -> Next Pool" and "Job -> Virtual Full Backup Pool" defined. Are both required? It appears to be working only with "Pool -> Next Pool". Why both parameters?

Back to my original questions:

- What pool is used for the source of a Virtual Full backup? Or is that a pointless question because bareos doesn't source from a pool for such a backup, but rather considers all backup jobs for the particular client that aren't archive jobs and uses whatever pool those volumes are in.

- What pool is used for the destination of the Virtual Full backup?

  - Pool -> Next Pool?

  - Job -> Next Pool?

  - Job -> Virtual Full Backup Pool?

[Sebastian Sura]

If i recall correctly, both `Pool->NextPool` as well as `Job->NextPool`
are being used as the destination.  Normally what you write in the `run`
command has the highest priority, then comes your Job configuration and
then the place where it is normally configured.

So if you write `run ... NextPool=xyz` then the job will use `xyz` as
the next pool; otherwise it will use `Job->NextPool` if that is defined
and lastly it will use `Pool->NextPool`.

`Job->VirtualFullBackupPool` is currently not used, but the idea is that
it will take precedence over your next pool configuration if set.

Am 23.01.25 um 14:05 schrieb Jon Schewe:

>
> - What pool is used for the destination of the Virtual Full backup?
>   - Pool -> Next Pool?
>   - Job -> Next Pool?
>   - Job -> Virtual Full Backup Pool?

Regarding your other question: Iirc a virtual full backup will first
build a backup chain for the used job, and then simply read from all
volumes that are used in that backup chain.  It still has a pool defined
but this is mostly used for the `Pool->NextPool` configuration, etc.

--
Sebastian Sura sebasti...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221 630693-0
https://www.bareos.com
Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz

[Jon Schewe]

Sebastian,

Thank you for that clarification. Having parameters in the documentation that aren't really used is quite frustrating.

Here's an example of the documentation that I would find useful to have in a single place. Some of the terms I'm using may not be appropriate, but I expect you'll get the idea.

# Creating a Virtual Full backup

This backup type creates a new Full backup by walking the backup chain (Full, Differential, Incremental) for a Job that contains all changes since the previous Full backup.

This is useful for creating long term backups such as off-site backups without needing to go back to the backup client to create the backup. It can be used with both a standard backup scheme and the always incremental backup scheme.

The source is determined by the jobs in the backup chain used to create a full backup. This can span multiple volumes and media types depending on how the backup chain (Full, Differential, Incremental) is setup. There is no specific configuration parameter to specify the source.

The destination is specified by the Next Pool parameter. This parameter is first checked in the run line, then the Job Resource (or JobDefs Resource) and finally the Pool Resource. The "Pool" parameter for the job is used to find the "Next Pool" parameter, otherwise it is not used.

Link to any existing documentation about Virtual Full backups

# Creating a standard backup scheme

This backup type uses Full, Differential and Incremental backups. A Full backup contains all data from the host at the specified time. The Differential contains all changes since the previous Full backup. The Incremental contains all changes since the previous Incremental or Differential or Full backup, whichever is most recent.

The source is the backup client.

The destination is the determined by the Pool parameter. This parameter is first checked in the run line, then the Job Resource (or JobDefs Resource).

Link to any existing documentation about standard backups

# Creating an Always Incremental backup scheme

See the existing documentation for Always Incremental backups

# Creating a Copy backup

... documentation about why this would be used and what the source and destination are

# Creating a Migration backup

... documentation about why this would be used and what the source and destination are

[Bruno Friedmann (bruno-at-bareos)]

it might be a good idea to try to start a PR to update the documentation with what you would like to see there.

If it is here, it may stay here :-/ 

BTW the parameter on the run schedule line are the one that overwrite previously defined ;-)