Running jobs in parallel to single local file path

[Bruno Marsal]

Hi, consider following setup:
* Storage daemon configured with archive device: /mnt/backup with media
type: file
* Storage daemon connected via 10Gbps Ethernet
* Many jobs. For example: Backup server1, Backup server2, . Backup server70
* Each server has its own volume pool: server1-fd-Pool, server2-fd-Pool, .
server70-df-Pool

This configuration worked great but the problem was the jobs are running consecutively and would run sometimes several days (in case of full backups). Seeking for an option to run the backup jobs in parallel I created
4 Device definitions on the storage daemon and 4 storage definitions on the
director:
On Storage daemon:
* Device { Name = Backup_01; Archive device = /mnt/backup; media type = file; [...]}
* Device { Name = Backup_02; Archive device = /mnt/backup; media type = file; [...]}
* Device { Name = Backup_03; Archive device = /mnt/backup; media type = file; [...]}
* Device { Name = Backup_04; Archive device = /mnt/backup; media type = file; [...]} On Director:
* Storage { Name = Storage_01; Device = Backup_01; [...]} ...
* Storage { Name = Storage_04; Device = Backup_04; [...]}

The pools were randomly configured to use either Storage_01, Storage_02,
Storage_03 and Storage_04 --> Now 4 jobs are running in parallel which is almost perfect.

The problem with this configuration is: Whenever a new server is being added to be backed up, we manually must choose a storage (01-04) which is all but perfect.
Is there a way to configure/modify the job scheduler in order it assigns one storage after the other to the scheduled jobs? Could this be done using plugins?
Is there some other nicer way to more or less automatically balance the jobs to the available storages?

Bruno

[Bruno Friedmann]

What you're looking for is "spooling", so you could have on storage and device, and all jobs will be spooled together
then for x reasons write done to the device.


--

Bruno Friedmann
Ioda-Net Sàrl www.ioda-net.ch

openSUSE Member & Board, fsfe fellowship
GPG KEY : D5C9B751C4653227
irc: tigerfoot

[Bruno Marsal]

Thank you for the answer. Will consider using a single pool for all servers/job.

I hoped there is a way to comfortably manage a pool per server but still have them on a single fileserver and be able to run backups in parallel. Seems there is no easy way to modify the scheduler to fit my needs. At least none I am aware of.

[Bruno Marsal]

Finally I came up with this:
Pool {
  Name = [...]
  @|"sh -c 'echo Storage = Storage_0$(shuf -i 1-4 -n 1)'"
}

This will randomly configure one of the 4 devices which is exactly what I looked for. Syntax is strange, but works very fine. This can be even improved by creating a file "echo 4 > PARALLELJOBS" and read this "variable" within the shuf command.

Hope you get the idea (having a pool per client and still run jobs in parallel).

Regards, Bruno Marsal
office +49-7053-9380604 | mobile +49-176-67393223 | Schlehenweg 1, 75387 Neubulach, Germany

--
You received this message because you are subscribed to a topic in the Google Groups "bareos-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bareos-users/iZhmf9IfE1c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bareos-users...@googlegroups.com.
To post to this group, send email to bareos...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert N]

Hi Bruno,

Happy New Year!

what version of Bareos do you use?

I try to use the same on 15.2.3 on Centos6:

Storage {

@|"sh -c 'idx=$(cat /tmp/bkp_idx 2>/dev/null || echo 0) ; echo -n Device = DiskDeviceIncr ; echo $((($idx % 5) + 1)) | tee /tmp/bkp_idx'"

}

but when I try to validate the config, it fails with following error, although the output generated by the script is correct. I tried replacing the @| line with :"Device = DiskDeviceIncr2" the exact output of script, and it works.


[root@muc1pro-backup-1 bareos]# bareos-dir -t -v
bareos-dir: ERROR TERMINATION at res.c:575
Config error: Could not find config Resource "Device" referenced on line 1 : Device = DiskDeviceIncr2


: line 1, col 24 of file |sh -c 'idx=$(cat /tmp/bkp_idx 2>/dev/null || echo 0) ; echo -n Device = DiskDeviceIncr ; echo $((($idx % 5) + 1)) | tee /tmp/bkp_idx'
Device = DiskDeviceIncr2

Thanks,
Robert

[Bruno Friedmann]

Hi Robert,

I wouldn't expect that the lexical parser is able to run the script.
It expect to have a mandatory Device parameter an equal sign (=) then a value
quoted or not and a end of line.

So Probably a line with
Device = "__some_kind_of_magic_good_luck_script"
and the correct @| \@| would work and make lexical parser happier.

Also not sure how the general daemon would react on dynamic device ....
Which is still for my brain not completely understable.

--

Bruno Friedmann
Ioda-Net Sàrl www.ioda-net.ch

Bareos Partner, openSUSE Member, fsfe fellowship

GPG KEY : D5C9B751C4653227
irc: tigerfoot

openSUSE Tumbleweed
Linux 4.9.0-2-default x86_64 GNU/Linux, nvidia: 375.26
Qt: 5.7.1, KDE Frameworks: 5.29.0, Plasma: 5.8.4, kmail2 5.4.0

[Robert N]

Hi Bruno,

thanks for your reply.
I've got it in the end working with Device = "| magic script" I think, but unfortunately doesn;t solve the problem. I think with @| bareos will execute it only once at startup, because it's a config include, I needed it executed every time a resource is being used. Nevertheless, the problem is always the same, when bareos has more than 1 device to write to, the jobs fail with the "No volume name given error", see here : https://groups.google.com/forum/?fromgroups=#!topic/bareos-users/Nbv-_fRWxaY.

I tried 3 diff scenarios:

1. define multiple devices in storage resource, as recommended in bareos doku.
2. script the device definition as described above.
3. use a virtual autochanger as described here: http://blog.bacula.org/whitepapers/CommunityDiskBackup.pdf

the problem is the volume selection I think, when more than 1 device is available in the storage.

any clues how to solve this?

thanks,
Robert

[Bruno Friedmann]

I'm using mono device for each storage location always since too long.
I've no autochanger and if I have to use one (when doing training) I'm using
mvhtl (available at bareos)

What I'm doing when I want to merge several backups jobs coming from several
backup clients is using spooling. And this for several years now, has always
works, and when the infrastructure permit it, the storage do all the
compression with autoxflate plugin.

Why, do I do this, because multiple device was initially implemented and made
for autoloaders with all the mechanics and so on....
;-)

[Robert N]

thanks.

what I'd like to achieve is, to run more jobs in parallel, doing 1 job/volume. not sure how could I achieve this with spooling and only one device in storage. talking about backup to disk.

[Bruno Friedmann]

> thanks.
>
> what I'd like to achieve is, to run more jobs in parallel, doing 1
> job/volume. not sure how could I achieve this with spooling and only one
> device in storage. talking about backup to disk.

I you want (the why is to determine, cause it can be a good idea, as a bad
one) to achieve one job per volume then you will need one storage per job.

Joba -> poolA -> storage A (with device type A) -> sd / device A

What is the motivation to have this one job per volume when using disk ?

[Robert N]

speed, if I use interleaved jobs on disks, copytotape takes very long...

--
You received this message because you are subscribed to a topic in the Google Groups "bareos-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bareos-users/iZhmf9IfE1c/unsubscribe.

To unsubscribe from this group and all its topics, send an email to bareos-users+unsubscribe@googlegroups.com.

[Bruno Friedmann]

Ok I'm pushing and try to keep it to mailing list :-)

On lundi, 9 janvier 2017 12.31:13 h CET Robert N wrote:
> let me see if I get this right. do you meand I should be using spooling for
> the backup to disk jobs, and copy jobs from disk volumes to tapes without
> spooling?
>

I was addressing only one type of backup, several jobs running concurrently
spooled, then merge in one device (disk based) (one or splitted by size
volumes).

Not playing enough with tapes, I can't say anything on the second part (copy/
migrate) to tapes.

But then you are reading one device and writing to another one, the slowest
making the io bandwidth.

It seems from what is expressed on the mailing list, that copy/migrate with
jobs are failing in a one after the other syndrom.
But I can't say more on that

> On Thu, Jan 5, 2017 at 9:42 AM, Bruno Friedmann <friedma...@gmail.com>
>
> wrote:

> > On mercredi, 4 janvier 2017 21.23:21 h CET you wrote:
> > > speed, if I use interleaved jobs on disks, copytotape takes very long...
> >

> > Which is why there's spooling to addresss that. with big chunk of spool
> > (I'm
> > using 128 to 512 GB) most of the job are spooled in one time and then
> > written
> > done to the media, with no interleave.

> >
> >


--

Bruno Friedmann
Ioda-Net Sàrl www.ioda-net.ch
Bareos Partner, openSUSE Member, fsfe fellowship
GPG KEY : D5C9B751C4653227
irc: tigerfoot

openSUSE Tumbleweed
Linux 4.9.0-2-default x86_64 GNU/Linux, nvidia: 375.26

Qt: 5.7.1, KDE Frameworks: 5.29.0, Plasma: 5.8.5, kmail2 5.4.0