Always incremental looking in wrong device

[Jan Hebler]

Hi

I try to implement an always Incremental Scheme, i follow the documentation guide as much as possible (the documentation how to configure multiple
Storages points to an documentation how to configure multiple devices to one storage) but finally i have the following Problem:





03-Apr 10:38 bareos-dir JobId 164: Start Virtual Backup JobId 164, Job=backup-desaster-fd.2022-04-02_17.53.41_11
03-Apr 10:38 bareos-dir JobId 164: Consolidating JobIds 9,13,17,21,25,29,33,37,41,45,53,57,61,65,69,73,77,81,85,89,93,97,101,105,109,113,117,121,125,129
03-Apr 10:38 bareos-dir JobId 164: Bootstrap records written to /var/lib/bareos/bareos-dir.restore.3.bsr
03-Apr 10:38 bareos-dir JobId 164: Connected Storage daemon at esmeralda:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
03-Apr 10:38 bareos-dir JobId 164: Using Device "Incremental" to read.
03-Apr 10:38 bareos-dir JobId 164: Using Device "Consolidated" to write.
03-Apr 10:38 DiskStorage JobId 164: Volume "AI-Consolidated-0005" previously written, moving to end of data.
03-Apr 10:38 DiskStorage JobId 164: Ready to append to end of Volume "AI-Consolidated-0005" size=26770517881
03-Apr 10:38 DiskStorage JobId 164: Warning: stored/acquire.cc:284 Read open device "Incremental" (/var/lib/bareos/storage/incremental) Volume "AI-Consolidated-0001" failed: ERR=stored/dev.cc:715 Could not open: /var/lib/bareos/storage/incremental/AI-Consolidated-0001, ERR=File or directory not found

IE: For consolidating bareos looks for Volumes in the wrong Device.

Regards, Jan

Configuration:

bareos-dir.d/storage/esmeralda.conf:
Storage {
Name = IncrementalStorage
Address = esmeralda
Password = "*"
Device = Incremental
Media Type = File
}
Storage {
Name = ConsolidatedStorage
Address = esmeralda
Password = "*"
Device = Consolidated
Media Type = File
}
Storage {
Name = LongtermStorage
Address = esmeralda
Password = "*"
Device = Longterm
Media Type = File
}

bareos-dir.d/jobdefs/DefaultJob.conf
JobDefs {
Name = "AiDefault"
Accurate = yes
Always Incremental = yes
Always Incremental Job Retention = 14
Always Incremental Keep Number = 7
Always Incremental Max Full Age = 21

Pool = AI-Incremental
Messages = Standard
Type = Backup
Full Backup Pool = AI-Consolidated
}

bareos-dir.d/job/backup-desaster-fd.conf
Job {
Name = "backup-desaster-fd"
Client = "desaster-fd"
Schedule = "DailyAI"
FileSet = "LinuxServer"
JobDefs = "AiDefault"
}

bareos-dir.d/pool/AI-*
Pool {
Name = AI-Incremental
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 360 days
Maximum Volume Bytes = 10G
Label Format = "AI-Incremental-"
Storage = IncrementalStorage
Next Pool = AI-Consolidated
}
Pool {
Name = AI-Consolidated
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 360 days
Maximum Volume Bytes = 50G
Label Format = "AI-Consolidated-"
Storage = ConsolidatedStorage
Next Pool = AI-Longterm
}
Pool {
Name = AI-Longterm
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 10 years
Maximum Volume Bytes = 50G
Label Format = "AI-Longterm-"
Storage = LongtermStorage
}

[Andreas Rogge]

Hi Jan,

You have two different Disk storages with the same Media Type.
This will not work, as is described in the docs[1]:

"If you want to tie Bareos to using a single Storage daemon or drive,
you must specify a unique Media Type for that drive. This is an
important point that should be carefully understood. Note, this applies
equally to Disk Volumes. If you define more than one disk Device
resource in your Storage daemon’s conf file, the Volumes on those two
devices are in fact incompatible because one can not be mounted on the
other device since they are found in different directories. For this
reason, you probably should use two different Media Types for your two
disk Devices (even though you might think of them as both being File
types)."

Hope that helps!

Best Regards,
Andreas

[1]
https://docs.bareos.org/Configuration/Director.html#config-Dir_Storage_MediaType
--
Andreas Rogge andrea...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221-630693-86
http://www.bareos.com

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: S. Dühr, M. Außendorf, J. Steffens, Philipp Storz

[Jan Hebler]

Am Mittwoch, 6. April 2022, 19:11:58 CEST schrieb Andreas Rogge:
> Hi Jan,
Hi Andreas

>
> You have two different Disk storages with the same Media Type.
> This will not work, as is described in the docs[1]:
>

i reconfigured the Storages as you advised, that looks way better now, Thank
you. Maybe it is worth to mention this in https://docs.bareos.org/
TasksAndConcepts/VolumeManagement.html#section-multiplestoragedevices
As there is only an example for multiple devices using the same directory
which uses (in that case correct) the same media Type.

Regards, Jan

>
> [1]
> https://docs.bareos.org/Configuration/Director.html#config-Dir_Storage_Media
> Type

[Andreas Rogge]

Hi Jan,

Am 08.04.22 um 13:10 schrieb Jan Hebler:

> Maybe it is worth to mention this in https://docs.bareos.org/
> TasksAndConcepts/VolumeManagement.html#section-multiplestoragedevices
> As there is only an example for multiple devices using the same directory
> which uses (in that case correct) the same media Type.

you're right, and I would be really happy if it was not so easy to get
it wrong.
However, you're referring to a subsection of the "Concurrent Disk Jobs"
chapter. The third paragraph in that chapter is completely devoted to
the very problem you were running into.

If you have any idea on how and where to document this so people don't
run into that problem all the time, we're really open to suggestions.

Best Regards,
Andreas

[Jan Hebler]

Hi Andreas

Am Freitag, 8. April 2022, 15:55:36 CEST schrieb Andreas Rogge:
> Hi Jan,
>
> Am 08.04.22 um 13:10 schrieb Jan Hebler:
> > Maybe it is worth to mention this in https://docs.bareos.org/
> > TasksAndConcepts/VolumeManagement.html#section-multiplestoragedevices
> > As there is only an example for multiple devices using the same directory
> > which uses (in that case correct) the same media Type.
>
> you're right, and I would be really happy if it was not so easy to get
> it wrong.
> However, you're referring to a subsection of the "Concurrent Disk Jobs"
> chapter. The third paragraph in that chapter is completely devoted to
> the very problem you were running into.

You are right, there is an example on how to implement this, unfortunally one
section above and under the line "concurrent disk jobs". I for myself was not
aware that this will helps with the different pools...

>
> If you have any idea on how and where to document this so people don't
> run into that problem all the time, we're really open to suggestions.

In the Chapter Always incremental there are an example configuration with
three Pools which uses the Storage File1 and File2 (btw: why uses the
longterm-pool file1 as Storage?) If the documentation under "Using multiple
devices" would keep these Names, that would help a lot. Or if ALL the Disk-
related examples would use an continuous Scheme. Like this:

You start with 3 Pools: incremental, differential and full. here you could
explain the configuration of the Pools, the Storage, show how to use different
devices in one storage to split the pools in separate directories, including
the device type and so on.
Next you could explain an conventional Setup with the usual incremental,
differential and full backups.
Next you could explain an Always incremental Backup, using differental for the
consolidated and full for the Longterm Backups. That way, you could keep the
Storage-definitions. There are only changes to the Pools required, if i got
anything right.
Finally you could show how to made concurrent Disk Jobs by adding devices to
the Incremental and the other example by adding an second incremental pool.

At least for me that would help a lot. But, then again, i'm not an Backup nor
Bacula-Expert and maybe i'ts just my miserable english :-)

Regards, Jan