Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

Bacula, Bacularis, and parallelization of backups

44 views
Skip to first unread message

Romain Delmas

unread,
Sep 26, 2024, 10:23:42 AM9/26/24
to bacularis
Hello Marcin,

I hope this message finds you as well as possible.

Here's what I trying to do :
- 1 client (toto.io)
- 2 different jobs (DATAS_toto.io ; DATABASES_toto.io)
- 1 volum for 1 job (volum DATAS_toto.io only contain toto.io datas job and volum DATABASES_toto.io only contain toto.io databases job)
- 5 Gb limit for each volum
- 5 jobs at the same time

So, in these conditions, a 12 GB job, DATAS_toto.io, will produce 3 volumes/files : DATAS_toto.io_1,  DATAS_toto.io_2 and  DATAS_toto.io_3, which contain only datas of toto.io). And i could easily find them on the disk with their names.

So, to achieve this, I activated :
- Director :   MaximumConcurrentJobs = 5
- Client :   MaximumConcurrentJobs = 5
- Storage :   MaximumConcurrentJobs = 5
- Job:   MaximumConcurrentJobs = 5

At first i have one pool for several jobs with theses conditions :
- Pool:   MaximumVolumeJobs = 1
              MaximumVolumeBytes = 5000000000

In this case I have a set of files/volumes whose data are those of the client alone (no sharing of volumes between clients), and the backups are well split into several 5G files.

However, when I run 5 jobs at the same time, they don't run at the same time... but one after the other...

I tried to create specific pool for each job with :
  MaximumVolumeBytes = 5000000000
   MaximumVolumeJobs = 0

But here again, jobs are launched one after the other (the next job is not launched until the previous one has finished)...

parallel.png

what am i missing?

Thank you in advance for your enlightenment,
Kind regards,

Romain

Marcin Haba

unread,
Sep 26, 2024, 10:44:08 AM9/26/24
to Romain Delmas, bacularis
On Thu, 26 Sept 2024 at 16:23, Romain Delmas <romain...@5group.fr> wrote:
> But here again, jobs are launched one after the other (the next job is not launched until the previous one has finished)...
>
> what am i missing?

Hello Romain,

The "status dir" bconsole command is capable to tell us why jobs are
waiting. I would propose to run this "status dir" command and see what
is the message for the waiting jobs.

In Bacularis it is possible to see in:

[Main menu] => [Page: Director] => [Tab: Actions] => [Button: Status director]

Probably you will need to enable the AllowMixedPriority directive for
jobs, but let's see first what the Status director will show. Maybe
there will be something other to change. We will see.

Best regards,
Marcin Haba (gani)

Romain Delmas

unread,
Sep 26, 2024, 11:27:02 AM9/26/24
to bacularis
Hi,

Thanks for your answer, but it seems i had misread the documentation...

"Now suppose you want to use multiple Pools, which means multiple Volumes, or suppose you want each client to have its own Volume and perhaps its own directory such as /home/bacula/client1 and /home/bacula/client2 ... With the single Storage and Device definition above, neither of these two is possible. Why? Because Bacula disk storage follows the same rules as tape devices. Only one Volume can be mounted on any Device at any time. If you want to simultaneously write multiple Volumes, you will need multiple Device resources in your bacula-sd.conf file, and thus multiple Storage resources in your bacula-dir.conf."

So, this would mean that I would have to create as much storage (bacula-dir) and as many resources (bacula-sd) as I have pools and therefore jobs...
For example, if i want to launch 2 jobs, DATA_toto and DATA_youplaboum in a same time (and therefore the data must be entered simultaneously) :
In bacula-dir.conf :

Storage {
  Name = "STORAGE_toto"
  Address = "FQDN_STORAGE_SERVER1"
  Password = "***"
  Device = "DEVICE_toto"
  MediaType = "File"
  MaximumConcurrentJobs = 5
  TlsEnable = yes
  TlsRequire = yes
  TlsCaCertificateFile = "path/ca.crt"
  TlsCertificate = "path/STORAGE_toto.crt"
  TlsKey = "path/STORAGE_toto.key"
}

Storage {
  Name = "STORAGE_youplaboum"
  Address = "FQDN_STORAGE_SERVER1"
  Password = "***"
  Device = "DEVICE_youplaboum"
  MediaType = "File"
  MaximumConcurrentJobs = 5
  TlsEnable = yes
  TlsRequire = yes
  TlsCaCertificateFile = "path/ca.crt"
  TlsCertificate = "path/STORAGE_youplaboum.crt"
  TlsKey = "path/STORAGE_youplaboum.key"
}

Pool {
  Name = "Full_datas_toto"
  PoolType = "Backup"
  LabelFormat = "${Level}_database_${Client}_${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}_${Hour:p/2/0/r}:${Minute:p/2/0/r}:${Second:p/2/0/r}"
  LabelType = "Bacula"
  UseVolumeOnce = no
  PurgeOldestVolume = yes
  ActionOnPurge = "Truncate"
  RecycleOldestVolume = no
  MaximumVolumeBytes = 5000000000
  VolumeRetention = 3888000
  MigrationTime = 2592000
  Storage = "STORAGE_toto"
  Recycle = no
  Catalog = "MyCatalog"
  FileRetention = 3888000
  JobRetention = 3888000
}

Pool {
  Name = "Full_datas_youplaboum"
  PoolType = "Backup"
  LabelFormat = "${Level}_database_${Client}_${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}_${Hour:p/2/0/r}:${Minute:p/2/0/r}:${Second:p/2/0/r}"
  LabelType = "Bacula"
  UseVolumeOnce = no
  PurgeOldestVolume = yes
  ActionOnPurge = "Truncate"
  RecycleOldestVolume = no
  MaximumVolumeBytes = 5000000000
  VolumeRetention = 3888000
  MigrationTime = 2592000
  Storage = "STORAGE_youplaboum"
  Recycle = no
  Catalog = "MyCatalog"
  FileRetention = 3888000
  JobRetention = 3888000
}

Pool {
  Name = "Incremental_datas_toto"
  PoolType = "Backup"
  LabelFormat = "${Level}_database_${Client}_${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}_${Hour:p/2/0/r}:${Minute:p/2/0/r}:${Second:p/2/0/r}"
  LabelType = "Bacula"
  UseVolumeOnce = no
  PurgeOldestVolume = yes
  ActionOnPurge = "Truncate"
  RecycleOldestVolume = no
  MaximumVolumeBytes = 5000000000
  VolumeRetention = 3888000
  MigrationTime = 2592000
  Storage = "STORAGE_toto"
  Recycle = no
  Catalog = "MyCatalog"
  FileRetention = 3888000
  JobRetention = 3888000
}

Pool {
  Name = "Incremental_datas_youplaboum"
  PoolType = "Backup"
  LabelFormat = "${Level}_database_${Client}_${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}_${Hour:p/2/0/r}:${Minute:p/2/0/r}:${Second:p/2/0/r}"
  LabelType = "Bacula"
  UseVolumeOnce = no
  PurgeOldestVolume = yes
  ActionOnPurge = "Truncate"
  RecycleOldestVolume = no
  MaximumVolumeBytes = 5000000000
  VolumeRetention = 3888000
  MigrationTime = 2592000
  Storage = "STORAGE_youplaboum"
  Recycle = no
  Catalog = "MyCatalog"
  FileRetention = 3888000
  JobRetention = 3888000
}

Job {
  Name = "DATAS_toto"
  Type = "Backup"
  Messages = "Standard"
  Storage = "DATAS_toto"
  Pool = "Full_datas_toto"
  FullBackupPool = "Full_datas_toto"
  IncrementalBackupPool = "Incremental_datas_toto"
  Client = "DATAS_toto"
  Fileset = "DATAS_toto"
  Schedule = "DATAS_toto"
  MaximumConcurrentJobs = 5
  Priority = 10
}

Job {
  Name = "DATAS_youplaboum"
  Type = "Backup"
  Messages = "Standard"
  Storage = "DATAS_youplaboumo"
  Pool = "Full_datas_youplaboum"
  FullBackupPool = "Full_datas_youplaboum"
  IncrementalBackupPool = "Incremental_Full_datas_youplaboum"
  Client = "Full_datas_youplaboum"
  Fileset = "Full_datas_youplaboum"
  Schedule = "Full_datas_youplaboum"
  MaximumConcurrentJobs = 5
  Priority = 10
}


Et dans mon bacula-sd :

Storage {
  Name = "STORAGE_toto"
  WorkingDirectory = "/opt/bacula/working"
  PidDirectory = "/opt/bacula/working"
  PluginDirectory = "/opt/bacula/plugins"
  MaximumConcurrentJobs = 20
  TlsEnable = yes
  TlsRequire = yes
  TlsCaCertificateFile = "/etc/ssl/bacula/ca.crt"
  TlsCertificate = "/etc/ssl/bacula/STORAGE_toto.crt"
  TlsKey = "/etc/ssl/bacula/STORAGE_toto.key"
}

Storage {
  Name = "STORAGE_youplaboum"
  WorkingDirectory = "/opt/bacula/working"
  PidDirectory = "/opt/bacula/working"
  PluginDirectory = "/opt/bacula/plugins"
  MaximumConcurrentJobs = 20
  TlsEnable = yes
  TlsRequire = yes
  TlsCaCertificateFile = "/etc/ssl/bacula/ca.crt"
  TlsCertificate = "/etc/ssl/bacula/STORAGE_youplaboum.crt"
  TlsKey = "/etc/ssl/bacula/STORAGE_youplaboum.key"
}

Device {
  Name = "DEVICE_toto"
  MediaType = "File"
  DeviceType = "Cloud"
  ArchiveDevice = "local/path/toto"
  RemovableMedia = no
  RandomAccess = yes
  AutomaticMount = yes
  LabelMedia = yes
  AlwaysOpen = no
  MaximumVolumeSize = 5000000000
  MaximumFileSize = 1000000000
  MaximumConcurrentJobs = 5
  MaximumPartSize = 5000000000
  Cloud = "S3Cloud_scw"
}

Device {
  Name = "DEVICE_youplaboum"
  MediaType = "File"
  DeviceType = "File"
  ArchiveDevice = "local/path/youplaboum"
  RemovableMedia = no
  RandomAccess = yes
  AutomaticMount = yes
  LabelMedia = yes
  MaximumVolumeSize = 5000000000
  MaximumFileSize = 1000000000
  MaximumConcurrentJobs = 5
  MaximumPartSize = 5000000000



Do you understand the same thing I do ?
This would mean to create several storages on the same server ...

Romain Delmas

unread,
Sep 26, 2024, 11:46:04 AM9/26/24
to bacularis
I think i made a mistake on the previous files :


bacula-dir

Storage {
  Name = "STORAGE_toto"
  Address = "FQDN_STORAGE_SERVER1"
  Password = "***"
  Device = "DEVICE_toto"
  MediaType = "File"
  MaximumConcurrentJobs = 5
  TlsEnable = yes
  TlsRequire = yes
  TlsCaCertificateFile = "path/ca.crt"
  TlsCertificate = "path/server.crt"
  TlsKey = "path/server.key"

}

Storage {
  Name = "STORAGE_youplaboum"
  Address = "FQDN_STORAGE_SERVER1"
  Password = "***"
  Device = "DEVICE_youplaboum"
  MediaType = "File"
  MaximumConcurrentJobs = 5
  TlsEnable = yes
  TlsRequire = yes
  TlsCaCertificateFile = "path/ca.crt"
  TlsCertificate = "path/server.crt"
  TlsKey = "path/server.key"
  Storage = "STORAGE_toto"

  Pool = "Full_datas_toto"
  FullBackupPool = "Full_datas_toto"
  IncrementalBackupPool = "Incremental_datas_toto"
  Client = "DATAS_toto"
  Fileset = "DATAS_toto"
  Schedule = "DATAS_toto"
  MaximumConcurrentJobs = 5
  Priority = 10
}

Job {
  Name = "DATAS_youplaboum"
  Type = "Backup"
  Messages = "Standard"
  Storage = "STORAGE_youplaboum"

  Pool = "Full_datas_youplaboum"
  FullBackupPool = "Full_datas_youplaboum"
  IncrementalBackupPool = "Incremental_Full_datas_youplaboum"
  Client = "Full_datas_youplaboum"
  Fileset = "Full_datas_youplaboum"
  Schedule = "Full_datas_youplaboum"
  MaximumConcurrentJobs = 5
  Priority = 10
}


bacula-sd :

Storage {
  Name = "DISKS_STORAGE"

  WorkingDirectory = "/opt/bacula/working"
  PidDirectory = "/opt/bacula/working"
  PluginDirectory = "/opt/bacula/plugins"
  MaximumConcurrentJobs = 20
  TlsEnable = yes
  TlsRequire = yes
  TlsCaCertificateFile = "/etc/ssl/bacula/ca.crt"
  TlsCertificate = "/etc/ssl/bacula/server.crt"
  TlsKey = "/etc/ssl/bacula/server.key"

}

Device {
  Name = "DEVICE_toto"
  MediaType = "File"
  DeviceType = "File"

  ArchiveDevice = "local/path/toto"
  RemovableMedia = no
  RandomAccess = yes
  AutomaticMount = yes
  LabelMedia = yes
  AlwaysOpen = no
  MaximumVolumeSize = 5000000000
  MaximumFileSize = 1000000000
  MaximumConcurrentJobs = 5
  MaximumPartSize = 5000000000
}

Device {
  Name = "DEVICE_youplaboum"
  MediaType = "File"
  DeviceType = "File"
  ArchiveDevice = "local/path/youplaboum"
  RemovableMedia = no
  RandomAccess = yes
  AutomaticMount = yes
  LabelMedia = yes
  MaximumVolumeSize = 5000000000
  MaximumFileSize = 1000000000
  MaximumConcurrentJobs = 5
  MaximumPartSize = 5000000000
}

Marcin Haba

unread,
Sep 26, 2024, 12:25:03 PM9/26/24
to Romain Delmas, bacularis
On Thu, 26 Sept 2024 at 17:46, Romain Delmas <romain...@5group.fr> wrote:
>
> I think i made a mistake on the previous files :

Hi Romain,

Thanks for providing more details.

That part of documentation is about using one volume by one device at
the same time. Yes, it is a base rule for using volumes by Bacula
device.

I see that your storage is single file storage. It can cause that the
jobs are waiting (to check in status director). An solution for this
case could be to use multi-device storage or keeping current storage
but increasing in Pool:

MaximumVolumeJobs = X

(and after - updating pool and volumes)

to be able to write more jobs at the same time using your current
devices (multiplexing).

If you use Bacularis 4.1.0, it provides a wizard for creating
multi-device file autochanger:

https://www.youtube.com/watch?v=OFOJ5P14jFI

Romain Delmas

unread,
Sep 30, 2024, 12:39:12 PM9/30/24
to bacularis
Hi,

Thanks for your answer.

I prefer using  multi-device storages, i find it clearer (including on my server).
I I've separated my different clients into different storage areas (with different MediaType), and I've planned three pools for each (full, incremental and s3).
These pools determine retention policies, and the full one included the "next pool" directive for s3 cold storage.
With this architecture, all clients jobs/datas are separated and jobs are performed synchronously, at the same time.

I know i can update a volume from a pool to another one with the "update volume=xxxxx pool=xxxx" but it does not include the MediaType directive, or i need this information to link correctly job/volume/data.
So, without having to intervene directly in the database by hand, do you know if i can update this directive on volume with bconsole ?

Kind regards,
Romain

Marcin Haba

unread,
Sep 30, 2024, 1:18:08 PM9/30/24
to Romain Delmas, bacularis
Hi Romain,

On Mon, 30 Sept 2024 at 18:39, Romain Delmas <romain...@5group.fr> wrote:
> I prefer using multi-device storages, i find it clearer (including on my server).
> I I've separated my different clients into different storage areas (with different MediaType), and I've planned three pools for each (full, incremental and s3).
> These pools determine retention policies, and the full one included the "next pool" directive for s3 cold storage.
> With this architecture, all clients jobs/datas are separated and jobs are performed synchronously, at the same time.

Great. Sounds solid :-)

> I know i can update a volume from a pool to another one with the "update volume=xxxxx pool=xxxx" but it does not include the MediaType directive, or i need this information to link correctly job/volume/data.
> So, without having to intervene directly in the database by hand, do you know if i can update this directive on volume with bconsole ?

I am not sure if I understood your question well. This media type is a
volume property which informs storage that it is able to use this
volume or not (if it supports a given volume or not). Moving the
volume between pools does not change the media type. Why would you
like to have a mediatype directive in the update command?

BTW: If you would like to move all volumes from pool AAA to pool BBB,
in Bacularis it can be done using the 'Assign volumes' function
available in the pool view page. I am attaching a screenshot. An
option to move a single volume from pool AAA to pool BBB can be done
on the volume view page (screenshot attached).
Bacularis-Bacula-Web-Interface-assign-volumes.png
Bacularis-Bacula-Web-Interface-update-volume.png

Romain Delmas

unread,
Oct 2, 2024, 6:10:36 AM10/2/24
to bacularis
Hello Marcin,

Sorry for the delay.

I was asking about MediaType because of its registration in the bacula database to associate volume/storage.
If i want to restore a backup, the MediaType on pool must match with the volume entry... 
However, in order to be able to parallelize jobs and retrieve the right files in the event of a problem, now i have as many MediaType as I have devices, and therefore as many pools.
So, if i move a volume from the ancient pool to the new one, and MediaType between volume and pool do not match, this could pose a problem.
That is why i wondered if i could change the MediaType of a volume to reassociate the path (in the event that I physically change the backups location ofc).

Kind to you,
Romain

Marcin Haba

unread,
Oct 2, 2024, 8:08:43 AM10/2/24
to Romain Delmas, bacularis
On Wed, 2 Oct 2024 at 12:10, Romain Delmas <romain...@5group.fr> wrote:
>
> Hello Marcin,
>
> Sorry for the delay.
>
> I was asking about MediaType because of its registration in the bacula database to associate volume/storage.
> If i want to restore a backup, the MediaType on pool must match with the volume entry...
> However, in order to be able to parallelize jobs and retrieve the right files in the event of a problem, now i have as many MediaType as I have devices, and therefore as many pools.
> So, if i move a volume from the ancient pool to the new one, and MediaType between volume and pool do not match, this could pose a problem.
> That is why i wondered if i could change the MediaType of a volume to reassociate the path (in the event that I physically change the backups location ofc).

Hello Romain,

Thanks for the response and for describing us how you see the media
type. Now I see why you wanted to have a media type in the 'update'
command. Let me try to share with you how I understand it.

You wrote:
"If i want to restore a backup, the MediaType on pool must match with
the volume entry..."

I think in this sentence are three wrong assumptions:

1) Media type is not a pool property. It is assigned to volumes as a
volume property.

2) We can have volumes with different media types in the same pool and
it is supported. This media type has been introduced to Bacula exactly
for this purpose, so that when the storage wants to use a volume it
checks if its media type is the same as a volume media type. This way
the storage takes in the pool only volumes that it is able to use and
work with.

3) Pools do not have meaning in restore. It is useful for backup,
copy, migration for groupping volumes in sets of volumes for a given
purpose, but for restore pools are not used. In the restore job
resource we have a pool defined but it is because the Job resource
requires it but it is not used in the restore process.

Please let me know what you think about it? Thanks for your opinions.

Romain Delmas

unread,
Oct 2, 2024, 8:59:34 AM10/2/24
to bacularis
Hey,

OK, i understand.

I had this archiecture :

bacula-dir.conf

Job {
  Storage = "STORAGE_client1"
  Pool = "POOL_client1"
}
Pool {
  Storage = "STORAGE_client1"
}
Storage {
  Device = "DEVICE_client1"
  MediaType = "MEDIA_client1"
}

bacula-sd.conf

Device {
  MediaType = "MEDIA_client1"
}


But i haven't considered the directive "Storage" in the job part of the conf file ; so i had this precedence Job > Pool > Storage > Device. In fact, you're right, of course, with this directive on job directive, it seems quite independent.
So the job itself registers the media type, and is therefore the most important low-level information for identifying volume and its physical part on server.

As a result, I can rebuild my pools, as long as I leave the storage directives in bacula-dir.conf and its associated device in bacula-sd; no matter where my volumes are on pools, they will always find their local paths.

Thanks, it's getting better and better.

Thanks to you

Marcin Haba

unread,
Oct 2, 2024, 9:21:51 AM10/2/24
to Romain Delmas, bacularis
On Wed, 2 Oct 2024 at 14:59, Romain Delmas <romain...@5group.fr> wrote:
>
> Hey,
>
> OK, i understand.
>
> I had this archiecture :
>
> bacula-dir.conf
>
> Job {
> Storage = "STORAGE_client1"
> Pool = "POOL_client1"
> }
> Pool {
> Storage = "STORAGE_client1"
> }
> Storage {
> Device = "DEVICE_client1"
> MediaType = "MEDIA_client1"
> }
>
> bacula-sd.conf
>
> Device {
> MediaType = "MEDIA_client1"
> }
>
> But i haven't considered the directive "Storage" in the job part of the conf file ; so i had this precedence Job > Pool > Storage > Device. In fact, you're right, of course, with this directive on job directive, it seems quite independent.
> So the job itself registers the media type, and is therefore the most important low-level information for identifying volume and its physical part on server.

Hello Romain,

Thanks. Now I fully understand.

This Storage directive in the Pool resource has been introduced for
copy/migration jobs that are realized on the pool level. Since it is
in Bacula, sometimes users use it for backup as well and it works, of
course.

> As a result, I can rebuild my pools, as long as I leave the storage directives in bacula-dir.conf and its associated device in bacula-sd; no matter where my volumes are on pools, they will always find their local paths.

Great. The JobDefs resources might be useful to store the Storage
directive as well or also Schedule resource if needed. But generally
it is defined, as you wrote, in the Job resource.

> Thanks, it's getting better and better.

Good luck!

Romain Delmas

unread,
Oct 2, 2024, 11:18:40 AM10/2/24
to bacularis
Hi Marcin,

One last thing, to be sure, by changing pool, I'm changing retention policy, I'm also changing migration policy.

If a volume is changed of pool, it takes the new migration policy.

The MediaType that determines the physical location causes no problem, as you said, when restoring data, but in this particular case, as this affects the pool level (and I understand that the MediaType still stays identifiable as long as the elements remain in the bacula-dir/bacula-sd directives), what about the migration ? Will the volume information (MediaType) be sufficient for temporary local passage (the directory where the file transits before the s3 push) and final migration, even if informations differs between the old and new pool ?

Sorry for this last asking,
Hope you're well as possible,

Romain

Marcin Haba

unread,
Oct 2, 2024, 12:26:57 PM10/2/24
to Romain Delmas, bacularis
On Wed, 2 Oct 2024 at 17:18, Romain Delmas <romain...@5group.fr> wrote:
>
> Hi Marcin,
>
> One last thing, to be sure, by changing pool, I'm changing retention policy, I'm also changing migration policy.
>
> If a volume is changed of pool, it takes the new migration policy.
>
> The MediaType that determines the physical location causes no problem, as you said, when restoring data, but in this particular case, as this affects the pool level (and I understand that the MediaType still stays identifiable as long as the elements remain in the bacula-dir/bacula-sd directives), what about the migration ? Will the volume information (MediaType) be sufficient for temporary local passage (the directory where the file transits before the s3 push) and final migration, even if informations differs between the old and new pool ?

Hi Romain,

Yes, you are right, for migration it has meaning. If you have migrate
job that have a source pool AAA and destination pool BBB, then it is
important to be sure that:

1) on volumes in Pool AAA we have all needed volumes with jobs to migrate
2) on pool BBB is set destination storage in the "Storage" pool directive

Once you are sure of the two points, rest should go easy. For volumes
with different media types in one source pool it isn't a problem
because Bacula is able to change reading device "on the fly" during
migrate job. So if you have jobs to migrate on volumes with media type
X, Y and Z, then Bacula can use first the Storage that reads X, then
it will use the storage that is able to read Y and at the end will be
used storage that reads Z. All this will happen in the one migrate
job.

> Sorry for this last asking,
> Hope you're well as possible,

Don't worry, Romain. This user group is exactly for this purpose - to
help users :-) It is great when this group lives.
Reply all
Reply to author
Forward
0 new messages