Concurrent jobs and Consolidate Jobs

[Brock Palen]

We backup/spool to disk volumes and copy/migrate jobs to tape.  We have one tape drive.

If we have Concurrent jobs allowed so we can backup more than one host at a time to disk we run into issues with Consolidate jobs failing when jobs were migrated to the tape drive.   If we only allow 1 job at a time it works as no second job tries to grab the tape drive.  

The specific error is:

list joblog jobid=23422
 2020-02-13 21:56:23 myth-dir JobId 23422: Start Virtual Backup JobId 23422, Job=sch-hp-desktop-Users-Pictures.2020-02-13_21.56.19_07
 2020-02-13 21:56:23 myth-dir JobId 23422: Consolidating JobIds 22736,23413
 2020-02-13 21:56:23 myth-dir JobId 23422: Bootstrap records written to /var/lib/bareos/myth-dir.restore.2.bsr
 2020-02-13 21:56:23 myth-dir JobId 23422: Connected Storage daemon at myth..com:9103, encryption: PSK-AES256-CBC-SHA
 2020-02-13 21:56:23 myth-dir JobId 23422: Using Device "FileStorage" to read.
 2020-02-13 21:56:23 myth-dir JobId 23422: Created new Volume "AI-Consolidated-1886" in catalog.
 2020-02-13 21:56:23 myth-dir JobId 23422: Using Device "FileStorage3" to write.
 2020-02-13 21:56:23 myth-sd JobId 23422: stored/acquire.cc:159 Changing read device. Want Media Type="LTO5" have="File"
  device="FileStorage" (/mnt/bacula)
 2020-02-13 21:56:23 myth-sd JobId 23422: Releasing device "FileStorage" (/mnt/bacula).
 2020-02-13 21:56:23 myth-sd JobId 23422: Fatal error: stored/acquire.cc:219 No suitable device found to read Volume "DA1391L5"
 2020-02-13 21:56:23 myth-sd JobId 23422: Releasing device "Tand-LTO5" (/dev/nst1).
 2020-02-13 21:56:24 myth-dir JobId 23422: Error: Bareos myth-dir 19.2.5 (03Feb20):
  Build OS:               Linux-3.10.0-1062.9.1.el7.x86_64 ubuntu Ubuntu 16.04 LTS
<snip>
  SD Errors:              1
  SD termination status:  Fatal Error
  Accurate:               yes
  Bareos binary info:     bareos.org build: Get official binaries and vendor support on bareos.com
  Termination:            *** Backup Error ***


So when it swaps to the tape (which is busy with another Consolidate VIrtualFull this job fails.

The Tape Storage does have

  Maximum Concurrent Jobs = 1

Is there a way to make the job hold and wait for the drive to free up?

I can use

  Reschedule on Error = yes
  Reschedule Times = 5
  Reschedule Interval = 1800

But this causes very long pauses as many consolidation jobs are very short/fast, but can be long if it includes full.

Thoughts?

Brock Palen
1 (989) 277-6075
bro...@mlds-networks.com
www.mlds-networks.com
Websites, Linux, Hosting, Joomla, Consulting

[Brock Palen]

Any thoughts related to this? Be really great if could run more than one job at a time but to have the VirtualFull jobs created by consolidate to wait for drives to be available or only run when no other jobs are running etc.

Thanks

Brock Palen
1 (989) 277-6075
bro...@mlds-networks.com
www.mlds-networks.com
Websites, Linux, Hosting, Joomla, Consulting

[Urban Persson]

Hi, 

Do you run this consolidation job with higher priority number so the job will wait until all other backup jobs finished?

Normally backupjobs is Priority 10 and Consolidate is Priority 11.

Can that be the issue?

// Urban

[Brock Palen]

> On Feb 28, 2020, at 1:11 PM, Urban Persson <supe...@gmail.com> wrote:
>
> Hi,
>
> Do you run this consolidation job with higher priority number so the job will wait until all other backup jobs finished?
> Normally backupjobs is Priority 10 and Consolidate is Priority 11.

I don’t see how that would help. Multiple VirtualFull jobs that are started from the Consolidate job would all have the same priority and start running try to swap devices same error.

If you think it would help I can test though. If you think that though why do you think it would work?

Thanks!

> --
> You received this message because you are subscribed to the Google Groups "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/e582785a-6fb3-4d88-9d77-bcf2beedb783%40googlegroups.com.

[Brock Palen]

Very old thread but this was just tested on Bareos 21

If you get the error

No suitable device found to read Volume

When running VirutalFull or Consoidate jobs where you have jobs spread across devices (in our case disk and tape) when running concurrent jobs.

The mistake was setting the Storage {} in the DIrector Confix Max Conccurrent Jobs =1 for tape devices. These need to be set higher, but then in the tape Device{} in your SD config should be set to 1.

This will now cause multiple jobs waiting for that tape drive to block:

66159 Virtual mills-feldman-Users-no-photos.2022-09-14_13.47.22_04 is running
66160 Virtual archive-hp-z210-Users.2022-09-14_13.48.17_05 is waiting on Storage "T-LTO4”

Now that we figured it out it’s ‘obvious’ but it wasn’t earlier.

bareos-dir.conf

Storage {
Name = T-LTO4
Address = <snip>
Password = “<snip>"
Device = T-LTO4
Media Type = LTO4
Maximum Concurrent Jobs = 4
Auto Changer = no
}


bareos-sd.conf

Device {
Name = T-LTO4
Autochanger = no
Drive Index = 0
Media Type = LTO4
Archive Device = /dev/nst0
Device Type = Tape
Spool Directory = /mnt/spool/Q-LTO4
Maximum Job Spool Size = 80000000000
Maximum Spool Size = 160000000000
Drive Crypto Enabled = Yes
Query Crypto Status = yes

Maximum Concurrent Jobs = 1
}

note I have not tried increasing conccurrent on the tape drive because in our case this drive is molyst used for creating Archive jobs for offsite extra copies while using Always Incremental. So when that monthly set of jobs run I would rather serialize all this internal to server traffic to stream HDD’s and minimize random IO even with spooling.

Brock Palen

bro...@mlds-networks.com
www.mlds-networks.com
Websites, Linux, Hosting, Joomla, Consulting