Always Incremental: VirtualFull fails

[Marco Betschart]

Hi there,

I'm trying to get the Always Incremental Backup scheme running - but it fails at the VirtualFull consolidation. For whatever reason, Bareos thinks a consolidate volume is stored on the wrong device and hangs indefinetely with the followgin warning message:

ZH-lin10-sd JobId 2005: Warning: stored/acquire.cc:276 Read open device "ZH-lin10-HD-USB0-device" (/media/usb0) Volume "LTA-Consolidated-0069" failed: ERR=stored/dev.cc:731 Could not open: /media/usb0/LTA-Consolidated-0069, ERR=No such file or directory

The missing volume (LTA-Consolidated-0069) does exist -  but on the other device (ZH-lin10-HD-USB1-device). And the regular Incremental Backups run just fine. I don't have any clue what's wrong - and highly appreciate any pointers.

More details to the current configuration: I do have two devices attached to one storage daemon:

/etc/bareos/bareos-sd.d/device

Device {
  Name = ZH-lin10-HD-USB0-device

  Maximum Concurrent Jobs = 5
  Device Type = File
  Media Type = File
  Label Media = yes
  Random Access = yes
  Automatic Mount = yes
  Removable Media = yes
  Requires Mount = yes
  Always Open = no
  Archive Device = "/media/usb0"
  Mount Point = "/media/usb0"
  Mount Command = "/bin/mount /dev/sdc1"
  Unmount Command = "fusermount -u %m"
}

Device {
  Name = ZH-lin10-HD-USB1-device


  Maximum Concurrent Jobs = 5
  Device Type = File
  Media Type = File
  Label Media = yes
  Random Access = yes
  Automatic Mount = yes
  Removable Media = yes
  Requires Mount = yes
  Always Open = no
  Archive Device = "/media/usb1"
  Mount Point = "/media/usb1"
  Mount Command = "/bin/mount /dev/sdd1"
  Unmount Command = "fusermount -u %m"
}

According to this thread, the storage needs configured with "Device Reserve By Media Type = yes" (don't know what this exactly does, not found any further documentation of it):

/etc/bareos/bareos-sd.d/storage

Storage {
  Name = ZH-lin10-sd
  Maximum Concurrent Jobs = 20

  # https://bugs.bareos.org/view.php?id=740
  Device Reserve By Media Type = yes
}

And the jobs involved are defined as follows:

/etc/bareos/bareos-dir.d/jobdefs

JobDefs {
  Name = LinuxServerLTA-Incremental-jd
  Type = Backup

  #
  # IMPORTANT: Accurate = "no" is on purpose!
  # This disables the detection of deleted files,
  # which in turn keeps them in the consolidated
  # backup jobs.
  #
  Accurate = no

  # Enable Always Incremental Backup Scheme
  Always Incremental = yes

  # Sets the age where incrementals of this job
  # will be kept, older jobs will be consolidated.
  Always Incremental Job Retention = 7 days

  Pool = ZH-lin10-LTA-Incremental-pool
  Full Backup Pool = ZH-lin10-LTA-Consolidated-pool

  # The time specifies the maximum allowed time that a Full backup job may run.
  # counted from when the job starts in seconds (not necessarily the same as when the
  # job was scheduled).
  # set to half a day
  Full Max Runtime = 43000

  FileSet = LinuxServerLTA-fs
  Client = ZH-lin10-fd

  Storage = ZH-lin10-sd
  Schedule = WeeklyCycle-LinuxServerLTA-sched
  Messages = Standard
  Priority = 10
  Spool Attributes = yes
  Write Bootstrap = "/var/lib/bareos/%c.bsr"
}


JobDefs {
  Name = LinuxServerLTA-Consolidate-jd
  Type = Consolidate

  #
  # IMPORTANT: Accurate = "no" is on purpose!
  # This disables the detection of deleted files,
  # which in turn keeps them in the consolidated
  # backup jobs.
  #
  Accurate = no

  #Pool = ZH-lin10-LTA-Incremental-pool
  Pool = ZH-lin10-LTA-Consolidated-pool

  # The time specifies the maximum allowed time that a Full backup job may run.
  # counted from when the job starts in seconds (not necessarily the same as when the
  # job was scheduled).
  # set to half a day
  Full Max Runtime = 43000

  FileSet = LinuxServerLTA-fs
  Client = ZH-lin10-fd

  Storage = ZH-lin10-sd
  Schedule = WeeklyCycle-LinuxServerLTA-sched
  Messages = Standard
  Priority = 10
}

/etc/bareos/bareos-dir.d/job

Job {
  Name = ZH-LTA-Consolidate-job
  JobDefs = LinuxServerLTA-Consolidate-jd
}


Job {
  Name = ZH-LTA-Incremental-lin132-job
  JobDefs = LinuxServerLTA-Incremental-jd
  Client = ZH-lin132-fd
}

[Tomi Pozderec]

Hey Marco,

Please note that I'm not an expert and my advice should be taken as such. I'm still struggling with Bareos and my advice is likely to be wrong at times. I'll try my best.

As I understand it, you need at least 2 devices per `Archive Device` for consolidate job to be successful. One to read from the pool and another to write to it.

My experience also tells me a couple of things:

1. Bareos devices can only read/write into the same `Archive Device` path.

2. Bareos will use/choose/reserve devices with same `Archive Device` and `Media Type` for a specific job unless you set `Device Reserve By Media Type = yes`

This is what I think happens with your setup when you try to consolidate:

1. When consolidation starts, Bareos reserves device "ZH-lin10-HD-USB1-device"for writing.


2. If, at this point `Device Reserve By Media Type` is set to "no", your consolidation will break, because Bareos needs a device to read with, but there's no available devices with same `Archive Device` as `ZH-lin10-HD-USB1-device`. However:

3. If, at this point `Device Reserve By Media Type` is set to "yes", Bareos will ignore `Archive Device` when choosing a device for a job and will choose any free device with same `Media Type` for reading. This time Bareos chooses `ZH-lin10-HD-USB0-device` for reading. But this device is only aware of `Archive Device` (disk path) = `Archive Device = "/media/usb0"` That's why it's trying to find the specific volume at this path.

I hope this makes sense and that's it's not riddled with mistakes. Any corrections are welcome and appreciated!

Marco, let me know if you need additional help with your config/understanding and I'll try to help.

[Marco Betschart]

Hi Tomi,

Thank you so much for these insights! It makes sense - at least somehow ;)

Never thought of the Archive Device to be honest. It is set to the exact same path in the example I've found, where they state it works. Changes are good that you catched the culprit. I've read the documentation on this setting, but was not really able to wrap my head around what it's purpose exactly is and how it should be defined. However, if I change my device configuration as follows and restart bareos-sd, it does not complain - which is a good sign. Do you happen to know, if that's the correct way to do it (see config below)?

Unfortunately I messed up my Volumes while tinkering around with this yesterday, so it seems I have start backing up from scratch. That said, it'll take some time to figure out if everything is working correctly - so any pointers what might be wrong with the following config would be highly appreciated.

/etc/bareos/bareos-sd.d/device

Device {
  Name = ZH-lin10-HD-USB0-device
  ...
  Archive Device = "/media/usb0"
  Archive Device = "ZH-lin10-HD-USB1-device"
  ...
}


Device {
  Name = ZH-lin10-HD-USB1-device
  ...
  Archive Device = "/media/usb1"
  Archive Device = "ZH-lin10-HD-USB0-device"
  ...
}