Automatic recycling of tapes does not work as I expect it

[Carsten Pache]

Hi all,

I have problems in understanding how automatic recycling of tapes works.

My configuration uses a single tape drive and ten tapes. Five tapes are used for every work day of an even week (labeled MON-E, TUE-E... FRI-E), the other five for odd weeks (MON-O, TUE-O... FRI-O).
There are two pools, one for even weeks and one for odd weeks, both configures with "Volume Retention = 8 days" and "Volume Use Duration = 23h".

Backups works fine. However, if after two weeks the tapes are to be reused, I need to prune each tape manually - otherwise Bareos asks to mount a volume or label a new one.

Parts of my bareos-dir.conf:

#########################

Director {
 Name = dasi-dir
 QueryFile = "/usr/lib/bareos/scripts/query.sql"
 Maximum Concurrent Jobs = 5
 Password = "director-password"
 Messages = Daemon
}


# JobDefs

JobDefs {
 Name = "DefaultJob"
 Type = Backup
 Level = Incremental
 Client = dasi-fd
 FileSet = "Full Set"
 Schedule = "Even-Uneven-Schedule"
 Storage = LTO-3
 Messages = Standard
 Pool = Default
 Priority = 10
 Write Bootstrap = "/var/lib/bareos/%c.bsr"
}

JobDefs {
 Name = "Backup Windows Server"
 Type = Backup
 Level = Full
 Storage = LTO-3
 SpoolData = yes
 Messages = Standard
 Pool = Default
 Priority = 10
 Write Bootstrap = "/var/lib/bareos/%c.bsr"
 Maximum Concurrent Jobs = 5
}


# Jobs

# Backup the catalog database (after the nightly save)
Job {
 Name = "BackupCatalog"
 JobDefs = "DefaultJob"
 Level = Full
 FileSet = "Catalog"
 Schedule = "Even-Uneven-Schedule"
 RunBeforeJob = "/usr/lib/bareos/scripts/make_catalog_backup bareos"
 RunAfterJob = "/etc/bareos/end_of_backup.sh"
 Write Bootstrap = "/var/lib/bareos/BackupCatalog.bsr"
 Priority = 50
}

# Backup the DATA-SERVER.
Job {
 Name = "SRV-DATA"
 JobDefs = "Backup Windows Server"
 Client = SRV-DATA-fd
 FileSet = "SRV-DATA-FileSet"
 Pool = Default
 Schedule = "Even-Uneven-Schedule"
}


# FileSets

# Files to backup SRV-DATA.
FileSet {
 Name = "SRV-DATA-FileSet"
 Enable VSS = yes
 Include {
   Options {
     Signature = SHA1
     IgnoreCase = yes
   }
   File = "D:/Personal"
   Exclude Dir Containing = .nobackup
 }
}


# Backup catalog.
FileSet {
 Name = "Catalog"
 Include {
   Options {
     signature = SHA1
   }
   File = "/var/lib/bareos/bareos.sql"
   File = "/etc/bareos"
 }
}


# List of files to be backed up
FileSet {
 Name = "Full Set"
 Include {
   Options {
     signature = MD5
   }
   File = /usr/sbin
 }

 Exclude {
   File = /var/lib/bareos
   File = /var/lib/bareos/storage
   File = /proc
   File = /tmp
   File = /.journal
   File = /.fsck
 }
}


# Schedules

Schedule {
 Name = "WeeklyCycle"
 Run = Full 1st sun at 23:05
 Run = Differential 2nd-5th sun at 23:05
 Run = Incremental mon-sat at 23:05
}

# This schedule does the catalog. It starts after the WeeklyCycle
Schedule {
 Name = "WeeklyCycleAfterBackup"
 Run = Full sun-sat at 23:10
}

Schedule {
 Name = "Even-Uneven-Schedule"
 Run = Level=Full Pool="EvenWeek"   sat w02,w04,w06,w08,w10,w12,w14,w16,w18,w20,w22,w24,w26,w28,w30,w32,w34,w36,w38,w40,w42,w44,w46,w48,w50,w52 at 19:30
 Run = Level=Full Pool="UnEvenWeek" sat w03,w05,w07,w09,w11,w13,w15,w17,w19,w21,w23,w25,w27,w29,w31,w33,w35,w37,w39,w41,w43,w45,w47,w49,w51,w53 at 19:30
 Run = Level=Incremental Pool="EvenWeek"   mon-thu w02,w04,w06,w08,w10,w12,w14,w16,w18,w20,w22,w24,w26,w28,w30,w32,w34,w36,w38,w40,w42,w44,w46,w48,w50 at 19:30
 Run = Level=Incremental Pool="UnEvenWeek" mon-thu w03,w05,w07,w09,w11,w13,w15,w17,w19,w21,w23,w25,w27,w29,w31,w33,w35,w37,w39,w41,w43,w45,w47,w49,w51 at 19:30
}


# Clients

# Client (File Services) to backup
Client {
 Name = dasi-fd
 Address = dasi.our.lan
 Password = "filedaemon-password"    # password for FileDaemon
 File Retention = 30 days
 Job Retention = 2 months
 AutoPrune = yes
}

Client {
 Name = SRV-DATA-fd
 Address = SRV-DATA.our.lan
 FDPort = 9102
 Password = "SRV-DATA-FileDaemon-XXX"
 File Retention = 30 days
 Job Retention = 30 days
 AutoPrune = yes
}


# Storage

# - LTO-3 (Tape)
Storage {
 Name = LTO-3
 Address = dasi.our.lan
 SDPort = 9103
 Password = "storagedaemon-password"
 Device = LTO-3
 Media Type = LTO-3
 Maximum Concurrent Jobs = 5
}


# Catalog

# Generic catalog service
Catalog {
 Name = MyCatalog
 dbdriver = "postgresql"
 dbname = "bareos"
 dbuser = "bareos"
 dbpassword = ""
}


# Pools

# Default pool definition
Pool {
 Name = Default
 Pool Type = Backup
 Recycle = yes
 AutoPrune = yes
 Volume Retention = 12 days
}

# Pool for even week numbers.
Pool {
 Name = EvenWeek
 Pool Type = Backup
 Recycle = yes
 AutoPrune = yes
 Volume Retention = 8 days
 Volume Use Duration = 23h
}

# Pool for uneven week numbers.
Pool {
 Name = UnEvenWeek
 Pool Type = Backup
 Recycle = yes
 AutoPrune = yes
 Volume Retention = 8 days
 Volume Use Duration = 23h
}

# Scratch pool definition
Pool {
 Name = Scratch
 Pool Type = Backup
}

#########################

Is there something missing or wrong?

Regards
Carsten

[Carsten Pache]

[Philipp Storz]

On 03.03.2016 11:22, Carsten Pache wrote:
> Hi all,
>
> I have problems in understanding how automatic recycling of tapes works.
>
> My configuration uses a single tape drive and ten tapes. Five tapes are used for every work day of an even week (labeled MON-E, TUE-E... FRI-E), the other five for odd weeks (MON-O, TUE-O... FRI-O).
> There are two pools, one for even weeks and one for odd weeks, both configures with "Volume Retention = 8 days" and "Volume Use Duration = 23h".
>
> Backups works fine. However, if after two weeks the tapes are to be reused, I need to prune each tape manually - otherwise Bareos asks to mount a volume or label a new one.
>

What is the output of the job when the no tape can be loaded?
What are the values of last written and volretention (use "list volumes")?

--
Mit freundlichen Grüßen

Philipp Storz philip...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221 63 06 93-92
http://www.bareos.com Fax: +49 221 63 06 93-10

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Geschäftsführer: Stephan Dühr, M. Außendorf,
J. Steffens, P. Storz, M. v. Wieringen

[Carsten Pache]

> On 03.03.2016 11:22, Carsten Pache wrote:

>> Hi all,

>> I have problems in understanding how automatic recycling of tapes works.

>> My configuration uses a single tape drive and ten tapes. Five tapes are used for every work day of an even week (labeled MON-E, TUE-E... FRI-E), the other five for odd weeks (MON-O, TUE-O... FRI-O).
>> There are two pools, one for even weeks and one for odd weeks, both configures with "Volume Retention = 8 days" and "Volume Use Duration = 23h".

>> Backups works fine. However, if after two weeks the tapes are to be reused, I need to prune each tape manually - otherwise Bareos asks to mount a volume or label a new one.

> What is the output of the job when the no tape can be loaded?

Example: On a Monday morning, I took out the tape labeled "FRI-O-yell" and loaded the tape "MON-O":

29-Feb 19:30 dasi-sd JobId 73: Warning: Director wanted Volume "FRI-O-yell".
  Current Volume "MON-O" not acceptable because:
  1998 Volume "MON-O" catalog status is Used, but should be Append, Purged or Recycle.
29-Feb 19:30 dasi-sd JobId 73: Please mount append Volume "FRI-O-yell" or label a new one for:
  Job:          SRV-DATA.2016-02-29_19.30.00_05
  Storage:      "LTO-3" (/dev/nst0)
  Pool:         UnEvenWeek
  Media type:   LTO-3
29-Feb 19:34 dasi-sd JobId 73: Please mount append Volume "FRI-O-yell" or label a new one for:
  Job:          SRV-DATA.2016-02-29_19.30.00_05
  Storage:      "LTO-3" (/dev/nst0)
  Pool:         UnEvenWeek
  Media type:   LTO-3

> What are the values of last written and volretention (use "list volumes")?

Here are my pools (well, the two I use):

Pool: EvenWeek
+---------+------------+-----------+---------+-----------------+----------+--------------+---------+------+-----------+-----------+---------------------+
| mediaid | volumename | volstatus | enabled | volbytes        | volfiles | volretention | recycle | slot | inchanger | mediatype | lastwritten         |
+---------+------------+-----------+---------+-----------------+----------+--------------+---------+------+-----------+-----------+---------------------+
|       9 | MON-E      | Used      |       1 | 192,362,760,965 |       42 |      691,200 |       1 |    0 |         0 | LTO-3     | 2016-02-23 01:33:15 |
|      10 | TUE-E      | Used      |       1 |  54,943,970,785 |       14 |      691,200 |       1 |    0 |         0 | LTO-3     | 2016-02-23 21:36:20 |
|      11 | WED-E      | Used      |       1 |  53,543,841,915 |       14 |      691,200 |       1 |    0 |         0 | LTO-3     | 2016-02-24 21:33:29 |
|      12 | THU-E      | Used      |       1 |  89,963,920,488 |       21 |      691,200 |       1 |    0 |         0 | LTO-3     | 2016-02-25 22:30:11 |
|      13 | FRI-E-yell | Append    |       1 | 564,303,401,987 |      115 |      691,200 |       1 |    0 |         0 | LTO-3     | 2016-02-28 13:26:13 |
|      14 | FRI-E-grey | Append    |       1 |             199 |        0 |      691,200 |       1 |    0 |         0 | LTO-3     |                     |
+---------+------------+-----------+---------+-----------------+----------+--------------+---------+------+-----------+-----------+---------------------+
Pool: UnEvenWeek
+---------+------------+-----------+---------+----------------+----------+--------------+---------+------+-----------+-----------+---------------------+
| mediaid | volumename | volstatus | enabled | volbytes       | volfiles | volretention | recycle | slot | inchanger | mediatype | lastwritten         |
+---------+------------+-----------+---------+----------------+----------+--------------+---------+------+-----------+-----------+---------------------+
|       2 | TUE-O      | Used      |       1 | 53,936,459,196 |       14 |      691,200 |       1 |    0 |         0 | LTO-3     | 2016-03-01 21:32:34 |
|       3 | WED-O      | Append    |       1 | 53,636,317,532 |       14 |      691,200 |       1 |    0 |         0 | LTO-3     | 2016-03-02 21:29:00 |
|       7 | FRI-O-yell | Append    |       1 |            201 |        0 |      691,200 |       1 |    0 |         0 | LTO-3     |                     |
|       8 | THU-O      | Append    |       1 |            196 |        0 |      691,200 |       1 |    0 |         0 | LTO-3     |                     |
|      15 | MON-O      | Used      |       1 | 93,720,685,680 |       22 |      691,200 |       1 |    0 |         0 | LTO-3     | 2016-02-29 22:45:14 |
+---------+------------+-----------+---------+----------------+----------+--------------+---------+------+-----------+-----------+---------------------+

Hope this helps!

Regards
Carsten

[David Westfall]

Hi, I am new to Bareos but have been using Bacula for years.

Try adding to your Pools the following:

File Retention = 8 days
Job Retention = 8 days
Recycle Oldest Volume = yes

Recycle Oldest Volume = yesno
This directive instructs the Director to search for the oldest used Volume in the Pool when another Volume is requested by the Storage daemon and none are available. The catalog is then pruned respecting the retention periods of all Files and Jobs written to this Volume. If all Jobs are pruned (i.e. the volume is Purged), then the Volume is recycled and will be used as the next Volume to be written. This directive respects any Job, File, or Volume retention periods that you may have specified, and as such it is much better to use this directive than the Purge Oldest Volume.

This directive can be useful if you have a fixed number of Volumes in the Pool and you want to cycle through them and you have specified the correct retention periods.

However, if you use this directive and have only one Volume in the Pool, you will immediately recycle your Volume if you fill it and Bacula needs another one. Thus your backup will be totally invalid. Please use this directive with care. The default is no.

http://www.bacula.org/5.2.x-manuals/en/main/main/Configuring_Director.html

Hope this helps.

Dave W

[Philipp Storz]

The right url to look at is of course:

http://doc.bareos.org/master/html/bareos-manual-main-reference.html#directiveDirPoolRecycle%20Oldest%20Volume

[Carsten Pache]

Now two weeks later, I see that the automatic recycling of tapes still does not work with my configuration.

Two weeks ago I added

Recycle Oldest Volume = yes

to my pools, reloaded the configuration and relabeled the tapes. Today, the first tape was in turn again but it was not recycled automatically - I had to prune and relabel it manually.

Is there something else that I can do?

Regards
Carsten

>> Try adding to your Pools the following:

>> File Retention = 8 days
>> Job Retention = 8 days
>> Recycle Oldest Volume = yes

>> Recycle Oldest Volume = yesno
>>  This directive instructs the Director to search for the oldest used Volume in the Pool when another Volume is requested by the Storage daemon and none are available. The catalog is then pruned respecting the retention periods of all Files and Jobs written to this Volume. If all Jobs are pruned (i.e. the volume is Purged), then the Volume is recycled and will be used as the next Volume to be written. This directive respects any Job, File, or Volume retention periods that you may have specified, and as such it is much better to use this directive than the Purge Oldest Volume.

>>  This directive can be useful if you have a fixed number of Volumes in the Pool and you want to cycle through them and you have specified the correct retention periods.

>>  However, if you use this directive and have only one Volume in the Pool, you will immediately recycle your Volume if you fill it and Bacula needs another one. Thus your backup will be totally invalid. Please use this directive with care. The default is no.

[Carsten Pache]

Dear folks,

sorry that I bring this issue up again.

I still have to prune the tapes manually. Well, this is true for the MON/TUE/WED/THU tapes but not for the FRE tapes. In fact I use two FRE tapes for even weeks (FRE-E-1, FRE-E-2) and two FRE tapes for odd weeks (FRE-O-1, FRE-O-2). Each FRE tape is used every forth even/odd week, whereas the MON/TUE/WED/THU tapes are used every second even/odd week. One more difference: The FRE tapes do not need to be pruned manually - they get pruned automatically. But why?

I already tried to lower "Volume Retention" to 4 days and relabeled all tapes, but this did not help.

Regards
Carsten

[nuesc...@gmail.com]

could you post your pool settings to check?

[Carsten Pache]

Hi Nuescht1982,

these are my pool definitions:

--
Regards
Carsten

[nuesc...@gmail.com]

Hi Carsten.

how much time is past, until the Friday copy to tape was made and the next job starts? was the tape unmounted within this time-period?
in my pools i had defined the "Recycle Pool = Scratch". Every expired Tape
will pushed from bareos in that pool
i can't see any use of your scratch pool. why not?
i also set the actiononpurge parameter in my file pool.
are you backup only one computer with the tapedrive connected on it?

[Carsten Pache]

Hi Nuescht1982,

on 31.05.2016 at 13:59 you wrote:

> how much time is past, until the Friday copy to tape was made and
> the next job starts?

After - for example - the Even-Week-Friday copy to tape has finished (usually on saturday morning), the Even-Week-Friday tape is ejected from the drive. The next tape that will be used is the Uneven-Week-Monday tape which will be put into the drive during monday.

>  was the tape unmounted within this time-period?

Yes, after the last backup job it gets ejected (which implies an unmount). But this is intended.

> in my pools i had defined the "Recycle Pool = Scratch". Every expired Tape
> will pushed from bareos in that pool
> i can't see any use of your scratch pool. why not?

I did not know that I need to use the scratch pool. Do I have to?

> i also set the actiononpurge parameter in my file pool.
> are you backup only one computer with the tapedrive connected on it?

The computer with the tape drive is only used to backup six other servers to tape (with data spooling enabled). After the backup of the servers, a BackupCatalog is performed, and when that job finishes, the tape is ejected.

--
Regards
Carsten

[nuesc...@gmail.com]

hi,

i get it working with the scratch pool. look in the manual at page 123
for further information. you'll only need the para Recycle Pool in the tape
ressource. okay, so you are running a disk to tape backup. my config is a
little bit different to yours, cause i'm running a disc 2 disc to tape backup.