Multiple Storage Device not moving to next device or recycling when device is full?
chris.fi...@ad-juster.com
I am currently having an issue when I run a Full job (50G vol size, 365d retention) the job does not switch to the next available device(s) or recycle an existing volume when the device(s) runs out of space, however it seems to work fine when I have a Incremental job (1G vol size, 30d retention).
Below I have listed my setup and configuration information that I feel is relevant, along with output from three jobs (2 of the jobs are a small job that fails in Full pool but works when Incremental pool, and the 3rd is the output from a DB backup that spans multiple Full volumes and caused the last disk to fill up over night, which was hoping it would roll back to Data1 or at least recycle the oldest on Data4 but it did not
Thank you for any assistance in this manner.
-Chris
##
## Setup
##
Bareos 17.2rc1/rc2
CentOS 7 server with 4 4TB JBOD disks for backup storage
Director and SD on same machine
Multiple FD clients
# Director storage/File.conf
Storage {
Name = File
Address = backup
Password = <password>
Device = FileStorage-Data1
Device = FileStorage-Data2
Device = FileStorage-Data3
Device = FileStorage-Data4
Media Type = File
Maximum Concurrent Jobs = 10
}
# Director pool/Full.conf
Pool {
Name = Full
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 365 days
Maximum Volume Bytes = 50G
Maximum Volumes = 200
Label Format = "Full-"
Recycle Oldest Volume = yes
Action On Purge = Truncate
Purge Oldest Volume = yes
}
# Director pool/Incremental.conf
Pool {
Name = Incremental
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 30 days
Maximum Volume Bytes = 1G
Maximum Volumes = 200
Label Format = "Incremental-"
Recycle Oldest Volume = yes
Action On Purge = Truncate
Purge Oldest Volume = yes
}
# Storage device/FileStorage-Data1.conf
Device {
Name = FileStorage-Data1
Media Type = File
Archive Device = /data1/bareos
LabelMedia = yes;
Random Access = yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
Description = "File device. A connecting Director must have the same Name and MediaType."
Maximum Concurrent Jobs = 10
}
# Storage device/FileStorage-Data2.conf
Device {
Name = FileStorage-Data2
Media Type = File
Archive Device = /data2/bareos
LabelMedia = yes;
Random Access = yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
Description = "File device. A connecting Director must have the same Name and MediaType."
Maximum Concurrent Jobs = 10
}
# Storage device/FileStorage-Data3, Data4
Same as above just Name and Archive Device updated accordingly
##
## Expected actions
##
# When all drives are not full
When FileStorage-Data1 fills up, the volume is created on FileStorage-Data2..4
# When all drives are full
When all drives are full, recycle the oldest volume which should be on FileStorage-Data1
## Workaround so far
When FileStorage-Data1 filled up, comment it out in Director storage/File.conf which allows it to be written to FileStorage-Data2, repeat till Data4.
This allowed me to kick the can down the road until last night when Data4 filled up and it did not recycle and got stuck.
I should note, that when Data1 first filled up, again only Full backups would not jump to Data2..4 but Incremental and Differential backups would.
##
## Output
##
# df
/dev/mapper/clc-data3 3.7T 3.6T 62G 99% /data3
/dev/mapper/cld-data4 3.7T 3.7T 747M 100% /data4
/dev/mapper/clb-data2 3.7T 3.7T 40K 100% /data2
/dev/mapper/cla-data1 3.0T 3.0T 1001M 100% /data1
#
# FAILS - Full job output (works as Incremental)
#
2017-11-27 14:00:03 bareos-dir JobId 702: Bareos bareos-dir 17.2.4 (21Sep17):
Build OS: x86_64-redhat-linux-gnu redhat Red Hat Enterprise Linux Server release 7.0 (Maipo)
JobId: 702
Job: Machine1-Wiki.2017-11-27_13.52.25_51
Backup Level: Full
Client: "Machine1-fd" 17.2.3 (14Aug17) x86_64-redhat-linux-gnu,redhat,Red Hat Enterprise Linux Server release 7.0 (Maipo),RHEL_7,x86_64
FileSet: "Machine1-Wiki" 2017-10-03 15:58:33
Pool: "Full" (From command line)
Catalog: "MyCatalog" (From Client resource)
Storage: "File" (From Job resource)
Scheduled time: 27-Nov-2017 13:52:25
Start time: 27-Nov-2017 13:52:28
End time: 27-Nov-2017 14:00:03
Elapsed time: 7 mins 35 secs
Priority: 10
FD Files Written: 0
SD Files Written: 0
FD Bytes Written: 0 (0 B)
SD Bytes Written: 0 (0 B)
Rate: 0.0 KB/s
Software Compression: None
VSS: no
Encryption: no
Accurate: no
Volume name(s):
Volume Session Id: 2
Volume Session Time: 1511818757
Last Volume Bytes: 0 (0 B)
Non-fatal FD errors: 1
SD Errors: 0
FD termination status: Error
SD termination status: Canceled
Termination: Backup Canceled
2017-11-27 14:00:03 bareos-dir JobId 702: Fatal error: No Job status returned from FD.
2017-11-27 14:00:03 bareos-sd JobId 702: Fatal error: Job 702 canceled.
2017-11-27 14:00:03 bareos-dir JobId 702: Fatal error: Network error with FD during Backup: ERR=Interrupted system call
2017-11-27 13:57:28 bareos-sd JobId 702: Warning: mount.c:250 Open device "FileStorage-Data1" (/data1/bareos) Volume "Full-0288" failed: ERR=dev.c:664 Could not open: /data1/bareos/Full-0288, ERR=No such file or directory
2017-11-27 13:52:28 bareos-sd JobId 702: Please mount append Volume "Full-0288" or label a new one for:
Job: Machine1-Wiki.2017-11-27_13.52.25_51
Storage: "FileStorage-Data1" (/data1/bareos)
Pool: Full
Media type: File
2017-11-27 13:52:28 bareos-sd JobId 702: Warning: mount.c:250 Open device "FileStorage-Data1" (/data1/bareos) Volume "Full-0288" failed: ERR=dev.c:664 Could not open: /data1/bareos/Full-0288, ERR=No space left on device
...
2017-11-27 13:52:28 bareos-sd JobId 702: Warning: label.c:376 Open device "FileStorage-Data1" (/data1/bareos) Volume "Full-0288" failed: ERR=dev.c:664 Could not open: /data1/bareos/Full-0288, ERR=No space left on device
2017-11-27 13:52:28 bareos-dir JobId 702: Using Device "FileStorage-Data1" to write.
2017-11-27 13:52:27 bareos-dir JobId 702: Start Backup JobId 702, Job=Machine1-Wiki.2017-11-27_13.52.25_51
#
# WORKS - Incremental job output (Same job)
#
2017-11-27 14:00:08 bareos-dir JobId 703: Bareos bareos-dir 17.2.4 (21Sep17):
Build OS: x86_64-redhat-linux-gnu redhat Red Hat Enterprise Linux Server release 7.0 (Maipo)
JobId: 703
Job: Machine1-Wiki.2017-11-27_13.59.58_02
Backup Level: Incremental, since=2017-11-27 07:48:39
Client: "Machine1-fd" 17.2.3 (14Aug17) x86_64-redhat-linux-gnu,redhat,Red Hat Enterprise Linux Server release 7.0 (Maipo),RHEL_7,x86_64
FileSet: "Machine1-Wiki" 2017-10-03 15:58:33
Pool: "Incremental" (From command line)
Catalog: "MyCatalog" (From Client resource)
Storage: "File" (From command line)
Scheduled time: 27-Nov-2017 13:59:58
Start time: 27-Nov-2017 14:00:04
End time: 27-Nov-2017 14:00:08
Elapsed time: 4 secs
Priority: 10
FD Files Written: 139
SD Files Written: 139
FD Bytes Written: 24,751,160 (24.75 MB)
SD Bytes Written: 24,770,990 (24.77 MB)
Rate: 6187.8 KB/s
Software Compression: 53.5 % (lzo)
VSS: no
Encryption: no
Accurate: no
Volume name(s): Incremental-0094
Volume Session Id: 3
Volume Session Time: 1511818757
Last Volume Bytes: 24,800,601 (24.80 MB)
Non-fatal FD errors: 0
SD Errors: 0
FD termination status: OK
SD termination status: OK
Termination: Backup OK
2017-11-27 14:00:08 bareos-dir JobId 703: sql_create.c:872 Insert of attributes batch table done
2017-11-27 14:00:07 bareos-sd JobId 703: Elapsed time=00:00:01, Transfer rate=24.77 M Bytes/second
2017-11-27 14:00:06 bareos-sd JobId 703: New volume "Incremental-0094" mounted on device "FileStorage-Data1" (/data1/bareos) at 27-Nov-2017 14:00.
2017-11-27 14:00:06 bareos-sd JobId 703: Recycled volume "Incremental-0094" on device "FileStorage-Data1" (/data1/bareos), all previous data lost.
2017-11-27 14:00:05 bareos-dir JobId 703: Recycled volume "Incremental-0094"
2017-11-27 14:00:05 bareos-dir JobId 703: All records pruned from Volume "Incremental-0094"; marking it "Purged"
2017-11-27 14:00:05 bareos-dir JobId 703: There are no more Jobs associated with Volume "Incremental-0094". Marking it purged.
2017-11-27 14:00:05 bareos-sd JobId 703: End of medium on Volume "Incremental-0093" Bytes=1,073,729,940 Blocks=2 at 27-Nov-2017 14:00.
2017-11-27 14:00:05 bareos-sd JobId 703: User defined maximum volume capacity 1,073,741,824 exceeded on device "FileStorage-Data1" (/data1/bareos).
2017-11-27 14:00:04 bareos-sd JobId 703: Ready to append to end of Volume "Incremental-0093" size=1073729940
2017-11-27 14:00:04 bareos-sd JobId 703: Volume "Incremental-0093" previously written, moving to end of data.
2017-11-27 14:00:04 bareos-dir JobId 703: Using Device "FileStorage-Data1" to write.
2017-11-27 14:00:04 bareos-dir JobId 703: Start Backup JobId 703, Job=Machine1-Wiki.2017-11-27_13.59.58_02
#
# FAILS Full job that filled Data4 device
#
2017-11-27 07:43:33 bareos-dir JobId 687: Bareos bareos-dir 17.2.3 (14Aug17):
Build OS: x86_64-redhat-linux-gnu redhat Red Hat Enterprise Linux Server release 7.0 (Maipo)
JobId: 687
Job: Machine2-MySQL.2017-11-26_21.05.00_43
Backup Level: Full
Client: "Machine2-fd" 17.2.3 (14Aug17) x86_64-redhat-linux-gnu,redhat,Red Hat Enterprise Linux Server release 7.0 (Maipo),RHEL_7,x86_64
FileSet: "MySQL-Data" 2017-10-04 15:39:20
Pool: "Full" (From Job FullPool override)
Catalog: "MyCatalog" (From Client resource)
Storage: "File" (From Job resource)
Scheduled time: 26-Nov-2017 21:05:00
Start time: 26-Nov-2017 21:05:03
End time: 27-Nov-2017 07:43:32
Elapsed time: 10 hours 38 mins 29 secs
Priority: 10
FD Files Written: 0
SD Files Written: 0
FD Bytes Written: 0 (0 B)
SD Bytes Written: 846,033 (846.0 KB)
Rate: 0.0 KB/s
Software Compression: None
VSS: no
Encryption: no
Accurate: no
Volume name(s): Full-0284|Full-0285|Full-0286
Volume Session Id: 564
Volume Session Time: 1507787926
Last Volume Bytes: 1,354,941 (1.354 MB)
Non-fatal FD errors: 1
SD Errors: 0
FD termination status: Error
SD termination status: Canceled
Termination: Backup Canceled
2017-11-27 04:43:14 bareos-sd JobId 687: Please mount append Volume "Full-0287" or label a new one for:
Job: Machine2-MySQL.2017-11-26_21.05.00_43
Storage: "FileStorage-Data4" (/data4/bareos)
Pool: Full
Media type: File
2017-11-27 04:38:14 bareos-sd JobId 687: Warning: mount.c:250 Open device "FileStorage-Data4" (/data4/bareos) Volume "Full-0287" failed: ERR=dev.c:663 Could not open: /data4/bareos/Full-0287, ERR=No such file or directory
...
2017-11-27 00:48:14 bareos-sd JobId 687: Warning: mount.c:250 Open device "FileStorage-Data4" (/data4/bareos) Volume "Full-0287" failed: ERR=dev.c:663 Could not open: /data4/bareos/Full-0287, ERR=No such file or directory
2017-11-27 00:43:14 bareos-sd JobId 687: Please mount append Volume "Full-0287" or label a new one for:
Job: Machine2-MySQL.2017-11-26_21.05.00_43
Storage: "FileStorage-Data4" (/data4/bareos)
Pool: Full
Media type: File
2017-11-27 00:38:14 bareos-sd JobId 687: Warning: mount.c:250 Open device "FileStorage-Data4" (/data4/bareos) Volume "Full-0287" failed: ERR=dev.c:663 Could not open: /data4/bareos/Full-0287, ERR=No such file or directory
...
2017-11-26 21:43:13 bareos-sd JobId 687: Warning: label.c:376 Open device "FileStorage-Data4" (/data4/bareos) Volume "Full-0287" failed: ERR=dev.c:663 Could not open: /data4/bareos/Full-0287, ERR=No space left on device
2017-11-26 21:43:13 bareos-dir JobId 687: All records pruned from Volume "Full-0287"; marking it "Purged"
2017-11-26 21:43:13 bareos-dir JobId 687: There are no more Jobs associated with Volume "Full-0287". Marking it purged.
2017-11-26 21:43:13 bareos-dir JobId 687: 1 File on Volume "Full-0287" purged from catalog.
2017-11-26 21:43:11 bareos-dir JobId 687: Purging oldest volume "Full-0287"
2017-11-26 21:43:11 bareos-sd JobId 687: End of medium on Volume "Full-0286" Bytes=1,354,941 Blocks=21 at 26-Nov-2017 21:43.
2017-11-26 21:43:11 bareos-sd JobId 687: End of Volume "Full-0286" at 0:1354940 on device "FileStorage-Data4" (/data4/bareos). Write of 64512 bytes got 21315.
2017-11-26 21:43:11 bareos-sd JobId 687: New volume "Full-0286" mounted on device "FileStorage-Data4" (/data4/bareos) at 26-Nov-2017 21:43.
2017-11-26 21:43:11 bareos-sd JobId 687: Wrote label to prelabeled Volume "Full-0286" on device "FileStorage-Data4" (/data4/bareos)
2017-11-26 21:43:10 bareos-sd JobId 687: Labeled new Volume "Full-0286" on device "FileStorage-Data4" (/data4/bareos).
2017-11-26 21:43:07 bareos-dir JobId 687: Recycled volume "Full-0286"
2017-11-26 21:43:07 bareos-dir JobId 687: All records pruned from Volume "Full-0286"; marking it "Purged"
2017-11-26 21:43:07 bareos-dir JobId 687: There are no more Jobs associated with Volume "Full-0286". Marking it purged.
2017-11-26 21:43:07 bareos-sd JobId 687: End of medium on Volume "Full-0285" Bytes=35,648,683,931 Blocks=552,590 at 26-Nov-2017 21:43.
2017-11-26 21:43:06 bareos-sd JobId 687: End of Volume "Full-0285" at 8:1288945562 on device "FileStorage-Data4" (/data4/bareos). Write of 64512 bytes got 61541.
2017-11-26 21:26:55 bareos-sd JobId 687: New volume "Full-0285" mounted on device "FileStorage-Data4" (/data4/bareos) at 26-Nov-2017 21:26.
2017-11-26 21:26:55 bareos-sd JobId 687: Wrote label to prelabeled Volume "Full-0285" on device "FileStorage-Data4" (/data4/bareos)
2017-11-26 21:26:55 bareos-sd JobId 687: Labeled new Volume "Full-0285" on device "FileStorage-Data4" (/data4/bareos).
2017-11-26 21:26:55 bareos-dir JobId 687: Recycled volume "Full-0285"
2017-11-26 21:26:54 bareos-dir JobId 687: All records pruned from Volume "Full-0285"; marking it "Purged"
2017-11-26 21:26:54 bareos-dir JobId 687: There are no more Jobs associated with Volume "Full-0285". Marking it purged.
2017-11-26 21:26:53 bareos-sd JobId 687: End of medium on Volume "Full-0284" Bytes=53,687,081,703 Blocks=832,204 at 26-Nov-2017 21:26.
2017-11-26 21:26:53 bareos-sd JobId 687: User defined maximum volume capacity 53,687,091,200 exceeded on device "FileStorage-Data4" (/data4/bareos).
2017-11-26 21:05:03 bareos-sd JobId 687: Ready to append to end of Volume "Full-0284" size=4857758442
2017-11-26 21:05:03 bareos-sd JobId 687: Volume "Full-0284" previously written, moving to end of data.
2017-11-26 21:05:03 Machine2-fd JobId 687: shell command: run ClientBeforeJob "/usr/local/bin/mysqladmin flush-tables"
2017-11-26 21:05:03 Machine2-fd JobId 687: ClientBeforeJob: Slave stopped
2017-11-26 21:05:03 Machine2-fd JobId 687: shell command: run ClientBeforeJob "/usr/local/bin/mysqladmin stop-slave"
2017-11-26 21:05:03 bareos-dir JobId 687: Using Device "FileStorage-Data4" to write.
2017-11-26 21:05:02 bareos-dir JobId 687: Start Backup JobId 687, Job=Machine2-MySQL.2017-11-26_21.05.00_43
chris.fi...@ad-juster.com
2017-11-27 23:30:11 bareos-sd JobId 714: Please mount append Volume "Full-0290" or label a new one for:
Job: BackupCatalog.2017-11-27_23.10.00_39
Storage: "FileStorage-Data1" (/data1/bareos)
Pool: Full
Media type: File
2017-11-27 23:30:11 bareos-sd JobId 714: Warning: mount.c:250 Open device "FileStorage-Data1" (/data1/bareos) Volume "Full-0290" failed: ERR=dev.c:664 Could not open: /data1/bareos/Full-0290, ERR=No space left on device
...
2017-11-27 23:30:10 bareos-sd JobId 714: Warning: label.c:376 Open device "FileStorage-Data1" (/data1/bareos) Volume "Full-0290" failed: ERR=dev.c:664 Could not open: /data1/bareos/Full-0290, ERR=No space left on device
2017-11-27 23:30:10 bareos-dir JobId 714: Recycled volume "Full-0290"
2017-11-27 23:30:10 bareos-dir JobId 714: All records pruned from Volume "Full-0290"; marking it "Purged"
2017-11-27 23:30:10 bareos-dir JobId 714: There are no more Jobs associated with Volume "Full-0290". Marking it purged.
2017-11-27 23:30:10 bareos-sd JobId 714: End of medium on Volume "Full-0289" Bytes=1,096,893 Blocks=17 at 27-Nov-2017 23:30.
2017-11-27 23:30:08 bareos-sd JobId 714: End of Volume "Full-0289" at 0:1096892 on device "FileStorage-Data1" (/data1/bareos). Write of 64512 bytes got 37699.
2017-11-27 23:30:08 bareos-sd JobId 714: New volume "Full-0289" mounted on device "FileStorage-Data1" (/data1/bareos) at 27-Nov-2017 23:30.
2017-11-27 23:30:07 bareos-sd JobId 714: Wrote label to prelabeled Volume "Full-0289" on device "FileStorage-Data1" (/data1/bareos)
2017-11-27 23:30:07 bareos-sd JobId 714: Labeled new Volume "Full-0289" on device "FileStorage-Data1" (/data1/bareos).
2017-11-27 23:30:02 bareos-dir JobId 714: Recycled volume "Full-0289"
2017-11-27 23:30:02 bareos-dir JobId 714: All records pruned from Volume "Full-0289"; marking it "Purged"
2017-11-27 23:30:01 bareos-dir JobId 714: There are no more Jobs associated with Volume "Full-0289". Marking it purged.
2017-11-27 23:30:01 bareos-sd JobId 714: End of medium on Volume "Full-0288" Bytes=285,336,758 Blocks=4,423 at 27-Nov-2017 23:30.
2017-11-27 23:30:01 bareos-sd JobId 714: End of Volume "Full-0288" at 0:285336757 on device "FileStorage-Data1" (/data1/bareos). Write of 64512 bytes got 43850.
2017-11-27 23:30:00 bareos-sd JobId 714: Wrote label to prelabeled Volume "Full-0288" on device "FileStorage-Data1" (/data1/bareos)
2017-11-27 23:30:00 bareos-sd JobId 714: Labeled new Volume "Full-0288" on device "FileStorage-Data1" (/data1/bareos).
2017-11-27 23:29:59 bareos-dir JobId 714: Using Device "FileStorage-Data1" to write.
2017-11-27 23:29:59 bareos-dir JobId 714: Start Backup JobId 714, Job=BackupCatalog.2017-11-27_23.10.00_39
2017-11-27 23:29:36 bareos-dir JobId 714: shell command: run BeforeJob "/usr/lib/bareos/scripts/make_catalog_backup.pl MyCatalog"
