Always Incremental with One Tape Drive

[Russell Harmon]

Hi there,

I see the note in https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html#storages-and-pools about "at least two storages are needed" for Always Incremental, but is there a way to make this work with a *temporary* secondary storage?

I have just one tape drive, and while I can temporarily use local disk as a secondary storage, I don't have enough for a full backup on disk... only enough for one tape's worth of data.

Is there any way to spool a consolidated job to disk, then let me swap in a tape for the consolidated pool (therefore removing the incremental pool's tape), despool it, then resume consolidating?

Thanks,

Russ Harmon

[Brock Palen]

I do this and it works but it has some bugs that are reported but not had progress in few years.

My setup is:

Disk Pools:
AI-Incremental
AI-Consolidated

Tape Pool:
LTO
Offsite

I run the AI setup on the disk pools but I use a Migration Job to migrate AI-Consolidated jobs to Offsite.
This is where one bug shows up ’sometimes’ Bareos will correctly understand that the job it needs is on tape but if you run parallel jobs it won’t correctly wait for the tape drive to free up if already busy. It will instead fail. It’s easy to work around you just run it again when that happens.

Sometimes it also gets hungup with multiple disk ‘devices’ where it won’t swap the needed disk volume even though it’s not being used. You can avoid all of these by forcing serial jobs, and all ‘bugs’ are more inconvenient than show stoppers.


I also use the Offsite pool (which for me is a second tape drive but not part of an autoloader) to write monthly offsite copies using VirutalFulls
https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html#long-term-storage-of-always-incremental-jobs

This is not AI related, but regular copy jobs are not reocmended because the consolidate pulls in those jobs. So this is more of an ‘emergency get it back no more than a month old’ offsite copy.


The migration job runs a script that checks and truncates all pruned volumes to free disk space rather than waiting for them to expire by age. Again this setup I find requires few TB of disk (need to run full backups to disk) and requires some watching.


#!/bin/bash
POOL=$1
for x in `echo "list volumes pool=${POOL}" | bconsole | grep -v "list volumes" | grep $POOL | awk -F\| '{print $3}'`
do
echo "prune volume=$x yes"
done | bconsole

# actaully free up disk space
echo "truncate volstatus=Purged pool=$POOL yes" \
| bconsole


Pool {
Name = AI-Incremental
Pool Type = Backup
Recycle = yes # Bareos can automatically recycle Volumes
Auto Prune = yes # Prune expired volumes
Volume Retention = 6 months # How long should jobs be kept?
Maximum Volume Bytes = 10G # Limit Volume size to something reasonable
Label Format = "AI-Incremental-"
Volume Use Duration = 7d
Storage = File
Next Pool = AI-Consolidated # consolidated jobs go to this pool
Action On Purge=Truncate
Migration High Bytes = 500G
Migration Low Bytes = 300G
}

Pool {
Name = AI-Consolidated
Pool Type = Backup
Recycle = yes # Bareos can automatically recycle Volumes
Auto Prune = yes # Prune expired volumes
Volume Retention = 6 months # How long should jobs be kept?
Maximum Volume Bytes = 50G # Limit Volume size to something reasonable
Label Format = "AI-Consolidated-"
Volume Use Duration = 2 days
Storage = File
Next Pool = Longterm # copy jobs write to this pool
Action On Purge=Truncate
Migration Time = 7 days
Migration High Bytes = 600G
Migration Low Bytes = 300G
}


Job {
Name = "Migrate-To-Offsite-AI-Consolidated-size"
Client = myth-fd
Type = Migrate
Purge Migration Job = yes
Pool = AI-Consolidated
Level = Full
Next Pool = LTO
Schedule = WeeklyCycleAfterBackup
Allow Duplicate Jobs = no
Priority = 4 #before catalog dump
Messages = Standard
Selection Type = PoolOccupancy
Spool Data = No
Selection Pattern = "."
RunAfterJob = "sudo /usr/local/bin/prune.sh AI-Consolidated"
Enabled = no
}

Brock Palen
bro...@mlds-networks.com
www.mlds-networks.com
Websites, Linux, Hosting, Joomla, Consulting

> --
> You received this message because you are subscribed to the Google Groups "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/6addbe69-f3a4-4bb4-b10d-849d713d5723n%40googlegroups.com.

[Russell Harmon]

On Sat, Dec 23, 2023 at 10:49 Brock Palen <bro...@mlds-networks.com> wrote:

I do this and it works but it has some bugs that are reported but not had progress in few years.

My setup is:

Disk Pools:
AI-Incremental
AI-Consolidated

Tape Pool:
LTO
Offsite

I run the AI setup on the disk pools but I use a Migration Job to migrate AI-Consolidated jobs to Offsite.
This is where one bug shows up ’sometimes’ Bareos will correctly understand that the job it needs is on tape but if you run parallel jobs it won’t correctly wait for the tape drive to free up if already busy. It will instead fail.  It’s easy to work around you just run it again when that happens.

Sometimes it also gets hungup with multiple disk ‘devices’ where it won’t swap the needed disk volume even though it’s not being used.  You can avoid all of these by forcing serial jobs, and all ‘bugs’ are more inconvenient  than show stoppers.


I also use the Offsite pool (which for me is a second tape drive but not part of an autoloader) to write monthly offsite copies using VirutalFulls 
https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html#long-term-storage-of-always-incremental-jobs

This is not AI related, but regular copy jobs are not reocmended because the consolidate pulls in those jobs.  So this is more of an ‘emergency  get it back no more than a month old’  offsite copy.


The migration job runs a script that checks and truncates all pruned volumes to free disk space rather than waiting for them to expire by age.  Again this setup I find requires few TB of disk (need to run full backups to disk)  and requires some watching.

To make sure I understand, this requires enough disk space for a full backup? I can't write out each tape's worth of data to tape while the full backup is running?

[Brock Palen]

Correct. Because when you run your consolidate with a full it has to read the old full likely from your tape drive. So it has to write to disk. 

Sent from my iPhone

Brock Palen

On Dec 23, 2023, at 3:43 PM, Russell Harmon <eatnu...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/CA%2BzrezSxi_eLzbQ%2BZvaikD69%3DFcF2Vtv%2BBRrLawBNvJYk5jXcw%40mail.gmail.com.

[Russell Harmon]

On Sat, Dec 23, 2023 at 17:21 Brock Palen <bro...@mlds-networks.com> wrote:

Correct. Because when you run your consolidate with a full it has to read the old full likely from your tape drive. So it has to write to disk. 

What if I flip things around: use disk for incrementals and tape for full? Would I then just need to make sure I run a consolidate job before I run out of disk?

[Brock Palen]

Doesn’t work,

During the consolidation of a full, you have to read back from teh full pool and write to the full pool. So you always need to have 2 working devices, one to read, one to write to the AI-Consolidate pool.

That’s why in my setup tape is really a pool to ‘migrate to make space’ so I can read back from it (Bareos often correctly switches to read from that pool). But the AI-Consolidate pool is on disk and where all the shuffling happens.

There is no way to do AI without two devices, and enough disk to at least create one full backup for whatever you are backing up.

Brock Palen
bro...@mlds-networks.com
www.mlds-networks.com
Websites, Linux, Hosting, Joomla, Consulting