Copy job between 2 storage daemons runs 3 jobs: 2 "copy jobs", and 1 "copy of a job"
283 views
Skip to first unread message
tim.b...@gmail.com
Apr 29, 2022, 8:51:26 AM4/29/22
to bareos-users
Hello,
There doesn't seem to be a lot of documentation for copy jobs, although I think they are extremely practical when having 2 storage daemons, and backing up remote clients to only use bandwidth once.
Problem: whenever I (manually, or scheduled) run a copy job, bareos actually runs 3 jobs:
1) one "copy job" named "myserver-sys-copy-file-to-tape" which only selects the job to be copied, and runs for not even a second
2) one "copy job" named "myserver-sys-copy-file-to-tape" as well, which seems to be actual copy job (it has the read, and write storage attributes)
3) one "copy of a job" named "myserver-sys-file" which is initially empty, but then fills with all the log-data of the initial job which was copied (also having the jobid of the file-backup which is copied to the tape)
Is this "as intended", or can I improve my config so I declutter my inbox/messages when doing a copy? I plan to have a couple of copy jobs which copies the last Full backup from file to tape. I could try one copy job for all remote clients (doing e.g. a SQL group by/distinct by client-name, and limiting the result to the amount of clients I want to backup), but I guess it would only save me one email/job per client (so for 5 clients, I would still have 11 emails).
It would be great if somebody can have a look at my config and point out what is wrong, so I only get (ideally) 1 email per copy job?
==== Here are the logs of the 3 jobs ====
1) copy job which seems to select the jobid to be copied:
*Director Running Jobs*
Running Jobs:
Console connected at 29-Apr-22 12:38
Console connected at 29-Apr-22 12:38
JobId Level Name Status
======================================================================
1084 Increme myserver-sys-copy-file-to-tape.2022-04-29_12.38.35_11 is waiting on max Job jobs
1085 Full myserver-sys-file.2022-04-29_12.38.35_12 is waiting execution
====
*Joblog*
Connecting to Director localhost:9101
Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
1000 OK: bareos-dir Version: 21.0.0 (21 December 2021)
bareos.org build binary
bareos.org binaries are UNSUPPORTED by bareos.com.
Get official binaries and vendor support on https://www.bareos.com
You are connected using the default console
Enter a period (.) to cancel a command.
list joblog jobid=1083
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
2022-04-29 12:38:35 bareos-dir JobId 1083: The following 1 JobId was chosen to be copied: 1069
2022-04-29 12:38:35 bareos-dir JobId 1083: Automatically selected Catalog: MyCatalog
2022-04-29 12:38:35 bareos-dir JobId 1083: Using Catalog "MyCatalog"
2022-04-29 12:38:35 bareos-dir JobId 1083: Job queued. JobId=1084
2022-04-29 12:38:35 bareos-dir JobId 1083: Copying JobId 1084 started.
2022-04-29 12:38:35 bareos-dir JobId 1083: Bareos bareos-dir 21.0.0 (21Dec21):
Build OS: Ubuntu 20.04.3 LTS
Current JobId: 1083
Current Job: myserver-sys-copy-file-to-tape.2022-04-29_12.38.33_06
Catalog: "MyCatalog" (From Default catalog)
Start time: 29-Apr-2022 12:38:35
End time: 29-Apr-2022 12:38:35
Elapsed time: 0 secs
Priority: 20
Bareos binary info: bareos.org build: Get official binaries and vendor support on bareos.com
Job triggered by: User
Termination: Copying OK
You have messages.
*Files backed up*
You have messages.
2) the actual copy job between the 2 storage daemons
*Director Running Jobs*
Running Jobs:
Console connected at 29-Apr-22 12:38
Console connected at 29-Apr-22 12:39
JobId Level Name Status
======================================================================
1085 Full myserver-sys-file.2022-04-29_12.38.35_12 has terminated
====
*Joblog*
Connecting to Director localhost:9101
Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
1000 OK: bareos-dir Version: 21.0.0 (21 December 2021)
bareos.org build binary
bareos.org binaries are UNSUPPORTED by bareos.com.
Get official binaries and vendor support on https://www.bareos.com
You are connected using the default console
Enter a period (.) to cancel a command.
list joblog jobid=1084
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
2022-04-29 12:38:35 bareos-dir JobId 1084: Copying using JobId=1069 Job=myserver-sys-file.2022-04-29_12.01.16_47
2022-04-29 12:38:35 bareos-dir JobId 1084: Bootstrap records written to /var/lib/bareos/bareos-dir.restore.9.bsr
2022-04-29 12:38:37 bareos-dir JobId 1084: Start Copying JobId 1084, Job=myserver-sys-copy-file-to-tape.2022-04-29_12.38.35_11
2022-04-29 12:38:37 bareos-dir JobId 1084: Connected Storage daemon at bareos.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
2022-04-29 12:38:37 bareos-dir JobId 1084: Using Device "bareos-file" to read.
2022-04-29 12:38:37 bareos-dir JobId 1085: Connected Storage daemon at eljadid.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
2022-04-29 12:38:37 bareos-dir JobId 1085: Using Device "Ultrium1760" to write.
2022-04-29 12:38:37 bareos-sd JobId 1084: Connected Storage daemon at eljadid.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
2022-04-29 12:38:37 bareos-sd JobId 1084: Ready to read from volume "Full-0154" on device "bareos-file" (/mnt/leah/var_lib_bareos_storage).
2022-04-29 12:38:37 eljadid-sd JobId 1085: Spooling data ...
2022-04-29 12:38:37 bareos-sd JobId 1084: Forward spacing Volume "Full-0154" to file:block 3:1118992026.
2022-04-29 12:38:57 bareos-sd JobId 1084: End of Volume at file 3 on device "bareos-file" (/mnt/leah/var_lib_bareos_storage), Volume "Full-0154"
2022-04-29 12:38:57 bareos-sd JobId 1084: End of all volumes.
2022-04-29 12:38:57 eljadid-sd JobId 1085: Committing spooled data to Volume "XM2724L3". Despooling 931,953,968 bytes ...
2022-04-29 12:39:17 eljadid-sd JobId 1085: Despooling elapsed time = 00:00:20, Transfer rate = 46.59 M Bytes/second
2022-04-29 12:39:17 eljadid-sd JobId 1085: Releasing device "Ultrium1760" (/dev/tape/by-id/scsi-HU19326DV6).
2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-109-generic] (local build)
2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert:
2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: === START OF READ SMART DATA SECTION ===
2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: TapeAlert Not Supported
2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: Error Counter logging not supported
2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert:
2022-04-29 12:39:19 eljadid-sd JobId 1085: Elapsed time=00:00:42, Transfer rate=22.07 M Bytes/second
2022-04-29 12:39:19 eljadid-sd JobId 1085: Sending spooled attrs to the Director. Despooling 24,595,188 bytes ...
2022-04-29 12:39:21 bareos-sd JobId 1084: Releasing device "bareos-file" (/mnt/leah/var_lib_bareos_storage).
2022-04-29 12:39:21 bareos-dir JobId 1085: Insert of attributes batch table with 74025 entries start
2022-04-29 12:39:23 bareos-dir JobId 1085: Insert of attributes batch table done
2022-04-29 12:39:23 bareos-dir JobId 1084: Bareos bareos-dir 21.0.0 (21Dec21):
Build OS: Ubuntu 20.04.3 LTS
Prev Backup JobId: 1069
Prev Backup Job: myserver-sys-file.2022-04-29_12.01.16_47
New Backup JobId: 1085
Current JobId: 1084
Current Job: myserver-sys-copy-file-to-tape.2022-04-29_12.38.35_11
Backup Level: Incremental
Client: myserver-fd
FileSet: "myserver-sys-file"
Read Pool: "Full" (From Job resource)
Read Storage: "bareos-file" (From Pool resource)
Write Pool: "tape-automated" (From Job's NextPool resource)
Write Storage: "eljadid_HP_1_8_G2" (From Storage from Job's NextPool resource)
Next Pool: "tape-automated" (From Job's NextPool resource)
Catalog: "MyCatalog" (From Default catalog)
Start time: 29-Apr-2022 12:38:37
End time: 29-Apr-2022 12:39:23
Elapsed time: 46 secs
Priority: 20
SD Files Written: 74,025
SD Bytes Written: 927,271,942 (927.2 MB)
Rate: 20158.1 KB/s
Volume name(s): XM2724L3
Volume Session Id: 10
Volume Session Time: 1651214033
Last Volume Bytes: 19,536,942,080 (19.53 GB)
SD Errors: 0
SD termination status: OK
Bareos binary info: bareos.org build: Get official binaries and vendor support on bareos.com
Job triggered by: User
Termination: Copying OK
You have messages.
*Files backed up*
You have messages.
3) the "copy of a job" with a new jobid (1085 in this case), but the logs of the job which is copied
*Director Running Jobs*
Running Jobs:
Console connected at 29-Apr-22 13:01
No Jobs running.
====
*Joblog*
Connecting to Director localhost:9101
Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
1000 OK: bareos-dir Version: 21.0.0 (21 December 2021)
bareos.org build binary
bareos.org binaries are UNSUPPORTED by bareos.com.
Get official binaries and vendor support on https://www.bareos.com
You are connected using the default console
Enter a period (.) to cancel a command.
list joblog jobid=1085
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
2022-04-29 12:01:18 bareos-dir JobId 1069: Start Backup JobId 1069, Job=myserver-sys-file.2022-04-29_12.01.16_47
2022-04-29 12:01:18 bareos-dir JobId 1069: Connected Storage daemon at bareos.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
2022-04-29 12:01:18 bareos-dir JobId 1069: Using Device "bareos-file" to write.
2022-04-29 12:01:18 bareos-dir JobId 1069: Probing client protocol... (result will be saved until config reload)
2022-04-29 12:01:22 bareos-dir JobId 1069: Connected Client: myserver-fd at sub.mydomain.org:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
2022-04-29 12:01:23 bareos-dir JobId 1069: Handshake: Immediate TLS 2022-04-29 12:01:23 bareos-dir JobId 1069: Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
2022-04-29 12:01:25 myserver-fd JobId 1069: shell command: run ClientBeforeJob "/usr/local/bin/bareos-runbefore.sh"
2022-04-29 12:01:25 myserver-fd JobId 1069: Extended attribute support is enabled
2022-04-29 12:01:25 myserver-fd JobId 1069: ACL support is enabled
2022-04-29 12:01:23 bareos-sd JobId 1069: Connected File Daemon at sub.mydomain.org:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
2022-04-29 12:01:25 bareos-sd JobId 1069: Volume "Full-0154" previously written, moving to end of data.
2022-04-29 12:01:25 bareos-sd JobId 1069: Ready to append to end of Volume "Full-0154" size=14003893914
2022-04-29 12:25:39 bareos-sd JobId 1069: Releasing device "bareos-file" (/mnt/leah/var_lib_bareos_storage).
2022-04-29 12:25:39 bareos-sd JobId 1069: Elapsed time=00:24:14, Transfer rate=637.7 K Bytes/second
2022-04-29 12:25:39 bareos-dir JobId 1069: Insert of attributes batch table with 74025 entries start
2022-04-29 12:25:41 bareos-dir JobId 1069: Insert of attributes batch table done
2022-04-29 12:25:41 bareos-dir JobId 1069: Bareos bareos-dir 21.0.0 (21Dec21):
Build OS: Ubuntu 20.04.3 LTS
JobId: 1069
Job: myserver-sys-file.2022-04-29_12.01.16_47
Backup Level: Full
Client: "myserver-fd" 21.0.0 (21Dec21) Debian GNU/Linux 11 (bullseye),debian
FileSet: "myserver-sys-file" 2022-04-28 17:29:32
Pool: "Full" (From command line)
Catalog: "MyCatalog" (From Client resource)
Storage: "bareos-file" (From Pool resource)
Scheduled time: 29-Apr-2022 12:01:16
Start time: 29-Apr-2022 12:01:25
End time: 29-Apr-2022 12:25:41
Elapsed time: 24 mins 16 secs
Priority: 10
FD Files Written: 74,025
SD Files Written: 74,025
FD Bytes Written: 881,441,457 (881.4 MB)
SD Bytes Written: 927,271,942 (927.2 MB)
Rate: 605.4 KB/s
Software Compression: 61.5 % (lz4)
VSS: no
Encryption: yes
Accurate: no
Volume name(s): Full-0154
Volume Session Id: 4
Volume Session Time: 1651214033
Last Volume Bytes: 14,936,345,934 (14.93 GB)
Non-fatal FD errors: 0
SD Errors: 0
FD termination status: OK
SD termination status: OK
Bareos binary info: bareos.org build: Get official binaries and vendor support on bareos.com
Job triggered by: User
Termination: Backup OK
2022-04-29 12:25:41 bareos-dir JobId 1069: shell command: run AfterJob "/bin/bash -c '/bin/echo "run myserver-sys-file-v yes" | bconsole >/dev/null'"
You have messages.
*Files backed up*
==== Here is my config ====
Job { #the job which backs up the remote client to file/disk
Name = "myserver-sys-file"
Type = Backup
Level = Incremental
Client = "myserver-fd"
FileSet = myserver-sys-file
Schedule = myserver-sys-file
Storage = bareos-file
Messages = Standard
Maximum Concurrent Jobs = 7
Pool = Incremental
Priority = 10
Write Bootstrap = "/var/lib/bareos/bootstrap/%c-%n.bsr"
Allow Mixed Priority = yes
#prevent duplicate jobs
Allow Duplicate Jobs = no
Cancel Lower Level Duplicates = yes
Cancel Queued Duplicates = yes
Full Backup Pool = Full # write Full Backups into "Full" Pool (#05)
Differential Backup Pool = Differential # write Diff Backups into "Differential" Pool (#08)
Incremental Backup Pool = Incremental # write Incr Backups into "Incremental" Pool (#11)
RunAfterJob = "/bin/bash -c '/bin/echo \"run myserver-sys-file-v yes\" | bconsole >/dev/null'"
RunScript {
Command = "/usr/local/bin/bareos-runbefore.sh"
RunsWhen = before
RunsOnClient = yes
}
}
Job { #the job which copies to tape
Name = myserver-sys-copy-file-to-tape
Type = Copy
Level = Full
Pool = Full
Selection Type = SQLQuery
Selection Pattern = "select jobid from job where name = 'myserver-sys-file' and level ='F' order by endtime DESC limit 1;"
Next Pool = tape-automated
Schedule = myserver-sys-copy-file-to-tape
Spool Data = Yes
Messages = Standard
Priority = 20
Write Bootstrap = "/var/lib/bareos/bootstrap/%c-%n.bsr"
Allow Duplicate Jobs = no
Cancel Lower Level Duplicates = yes
Cancel Queued Duplicates = yes
RunAfterJob = "/bin/bash -c '/bin/echo \"run myserver-sys-copy-file-to-tape-v yes\" | bconsole >/dev/null'"
}
Pool { #file/disk pool
Name = Full
Pool Type = Backup
Recycle = yes # Bareos can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
Volume Retention = 365 days # How long should the Full Backups be kept? (#06)
Job Retention = 365 days
File Retention = 365 days
Maximum Volume Bytes = 50G # Limit Volume size to something reasonable
Maximum Volumes = 200 # Limit number of Volumes in Pool
Label Format = "Full-" # Volumes will be labeled "Full-<volume-id>"
Storage = bareos-file
Next Pool = tape-automated
}
Pool { #tape pool
Name = tape-automated
Pool Type = Backup
Recycle = yes # Bareos can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
#Recycle Oldest Volume = yes
RecyclePool = Scratch
Maximum Volume Bytes = 0
Job Retention = 365 days
Volume Retention = 4 weeks
Volume Use Duration = 12 days
Cleaning Prefix = "CLN"
Catalog Files = yes #needed, this is default
Storage = tapeSD_HP_1_8_G2
}
Storage { #this is the file/disk storage daemon
Name = bareos-file
Address = "bareos.mydomain.org" # N.B. Use a fully qualified name here (do not use "localhost" here).
Password = "XXXX"
Device = bareos-file
Media Type = File
Maximum Concurrent Jobs = 7
#dashboard progress / statistics collection
Collect Statistics = yes
TLS Enable = yes
TLS Require = yes
TLS DH File = /etc/bareos/ssl/dh4096.pem
}
Storage { #this is the tape storage daemon on another server
Name = "tapeSD_HP_1_8_G2"
Address = "tapeSD.mydomain.org"
Lan Address = "tapeSD.directlink.mydomain.org"
Password = "XXXX"
Device = HP_1_8_G2
Autochanger = yes
MediaType = "LTO"
#dashboard progress / statistics collection
Collect Statistics = yes
TLS Enable = yes
TLS Require = yes
TLS DH File = /etc/bareos/ssl/dh4096.pem
}
There doesn't seem to be a lot of documentation for copy jobs, although I think they are extremely practical when having 2 storage daemons, and backing up remote clients to only use bandwidth once.
Problem: whenever I (manually, or scheduled) run a copy job, bareos actually runs 3 jobs:
1) one "copy job" named "myserver-sys-copy-file-to-tape" which only selects the job to be copied, and runs for not even a second
2) one "copy job" named "myserver-sys-copy-file-to-tape" as well, which seems to be actual copy job (it has the read, and write storage attributes)
3) one "copy of a job" named "myserver-sys-file" which is initially empty, but then fills with all the log-data of the initial job which was copied (also having the jobid of the file-backup which is copied to the tape)
Is this "as intended", or can I improve my config so I declutter my inbox/messages when doing a copy? I plan to have a couple of copy jobs which copies the last Full backup from file to tape. I could try one copy job for all remote clients (doing e.g. a SQL group by/distinct by client-name, and limiting the result to the amount of clients I want to backup), but I guess it would only save me one email/job per client (so for 5 clients, I would still have 11 emails).
It would be great if somebody can have a look at my config and point out what is wrong, so I only get (ideally) 1 email per copy job?
==== Here are the logs of the 3 jobs ====
1) copy job which seems to select the jobid to be copied:
*Director Running Jobs*
Running Jobs:
Console connected at 29-Apr-22 12:38
Console connected at 29-Apr-22 12:38
JobId Level Name Status
======================================================================
1084 Increme myserver-sys-copy-file-to-tape.2022-04-29_12.38.35_11 is waiting on max Job jobs
1085 Full myserver-sys-file.2022-04-29_12.38.35_12 is waiting execution
====
*Joblog*
Connecting to Director localhost:9101
Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
1000 OK: bareos-dir Version: 21.0.0 (21 December 2021)
bareos.org build binary
bareos.org binaries are UNSUPPORTED by bareos.com.
Get official binaries and vendor support on https://www.bareos.com
You are connected using the default console
Enter a period (.) to cancel a command.
list joblog jobid=1083
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
2022-04-29 12:38:35 bareos-dir JobId 1083: The following 1 JobId was chosen to be copied: 1069
2022-04-29 12:38:35 bareos-dir JobId 1083: Automatically selected Catalog: MyCatalog
2022-04-29 12:38:35 bareos-dir JobId 1083: Using Catalog "MyCatalog"
2022-04-29 12:38:35 bareos-dir JobId 1083: Job queued. JobId=1084
2022-04-29 12:38:35 bareos-dir JobId 1083: Copying JobId 1084 started.
2022-04-29 12:38:35 bareos-dir JobId 1083: Bareos bareos-dir 21.0.0 (21Dec21):
Build OS: Ubuntu 20.04.3 LTS
Current JobId: 1083
Current Job: myserver-sys-copy-file-to-tape.2022-04-29_12.38.33_06
Catalog: "MyCatalog" (From Default catalog)
Start time: 29-Apr-2022 12:38:35
End time: 29-Apr-2022 12:38:35
Elapsed time: 0 secs
Priority: 20
Bareos binary info: bareos.org build: Get official binaries and vendor support on bareos.com
Job triggered by: User
Termination: Copying OK
You have messages.
*Files backed up*
You have messages.
2) the actual copy job between the 2 storage daemons
*Director Running Jobs*
Running Jobs:
Console connected at 29-Apr-22 12:38
Console connected at 29-Apr-22 12:39
JobId Level Name Status
======================================================================
1085 Full myserver-sys-file.2022-04-29_12.38.35_12 has terminated
====
*Joblog*
Connecting to Director localhost:9101
Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
1000 OK: bareos-dir Version: 21.0.0 (21 December 2021)
bareos.org build binary
bareos.org binaries are UNSUPPORTED by bareos.com.
Get official binaries and vendor support on https://www.bareos.com
You are connected using the default console
Enter a period (.) to cancel a command.
list joblog jobid=1084
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
2022-04-29 12:38:35 bareos-dir JobId 1084: Copying using JobId=1069 Job=myserver-sys-file.2022-04-29_12.01.16_47
2022-04-29 12:38:35 bareos-dir JobId 1084: Bootstrap records written to /var/lib/bareos/bareos-dir.restore.9.bsr
2022-04-29 12:38:37 bareos-dir JobId 1084: Start Copying JobId 1084, Job=myserver-sys-copy-file-to-tape.2022-04-29_12.38.35_11
2022-04-29 12:38:37 bareos-dir JobId 1084: Connected Storage daemon at bareos.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
2022-04-29 12:38:37 bareos-dir JobId 1084: Using Device "bareos-file" to read.
2022-04-29 12:38:37 bareos-dir JobId 1085: Connected Storage daemon at eljadid.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
2022-04-29 12:38:37 bareos-dir JobId 1085: Using Device "Ultrium1760" to write.
2022-04-29 12:38:37 bareos-sd JobId 1084: Connected Storage daemon at eljadid.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
2022-04-29 12:38:37 bareos-sd JobId 1084: Ready to read from volume "Full-0154" on device "bareos-file" (/mnt/leah/var_lib_bareos_storage).
2022-04-29 12:38:37 eljadid-sd JobId 1085: Spooling data ...
2022-04-29 12:38:37 bareos-sd JobId 1084: Forward spacing Volume "Full-0154" to file:block 3:1118992026.
2022-04-29 12:38:57 bareos-sd JobId 1084: End of Volume at file 3 on device "bareos-file" (/mnt/leah/var_lib_bareos_storage), Volume "Full-0154"
2022-04-29 12:38:57 bareos-sd JobId 1084: End of all volumes.
2022-04-29 12:38:57 eljadid-sd JobId 1085: Committing spooled data to Volume "XM2724L3". Despooling 931,953,968 bytes ...
2022-04-29 12:39:17 eljadid-sd JobId 1085: Despooling elapsed time = 00:00:20, Transfer rate = 46.59 M Bytes/second
2022-04-29 12:39:17 eljadid-sd JobId 1085: Releasing device "Ultrium1760" (/dev/tape/by-id/scsi-HU19326DV6).
2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-109-generic] (local build)
2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert:
2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: === START OF READ SMART DATA SECTION ===
2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: TapeAlert Not Supported
2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert: Error Counter logging not supported
2022-04-29 12:39:19 eljadid-sd JobId 1085: Alert:
2022-04-29 12:39:19 eljadid-sd JobId 1085: Elapsed time=00:00:42, Transfer rate=22.07 M Bytes/second
2022-04-29 12:39:19 eljadid-sd JobId 1085: Sending spooled attrs to the Director. Despooling 24,595,188 bytes ...
2022-04-29 12:39:21 bareos-sd JobId 1084: Releasing device "bareos-file" (/mnt/leah/var_lib_bareos_storage).
2022-04-29 12:39:21 bareos-dir JobId 1085: Insert of attributes batch table with 74025 entries start
2022-04-29 12:39:23 bareos-dir JobId 1085: Insert of attributes batch table done
2022-04-29 12:39:23 bareos-dir JobId 1084: Bareos bareos-dir 21.0.0 (21Dec21):
Build OS: Ubuntu 20.04.3 LTS
Prev Backup JobId: 1069
Prev Backup Job: myserver-sys-file.2022-04-29_12.01.16_47
New Backup JobId: 1085
Current JobId: 1084
Current Job: myserver-sys-copy-file-to-tape.2022-04-29_12.38.35_11
Backup Level: Incremental
Client: myserver-fd
FileSet: "myserver-sys-file"
Read Pool: "Full" (From Job resource)
Read Storage: "bareos-file" (From Pool resource)
Write Pool: "tape-automated" (From Job's NextPool resource)
Write Storage: "eljadid_HP_1_8_G2" (From Storage from Job's NextPool resource)
Next Pool: "tape-automated" (From Job's NextPool resource)
Catalog: "MyCatalog" (From Default catalog)
Start time: 29-Apr-2022 12:38:37
End time: 29-Apr-2022 12:39:23
Elapsed time: 46 secs
Priority: 20
SD Files Written: 74,025
SD Bytes Written: 927,271,942 (927.2 MB)
Rate: 20158.1 KB/s
Volume name(s): XM2724L3
Volume Session Id: 10
Volume Session Time: 1651214033
Last Volume Bytes: 19,536,942,080 (19.53 GB)
SD Errors: 0
SD termination status: OK
Bareos binary info: bareos.org build: Get official binaries and vendor support on bareos.com
Job triggered by: User
Termination: Copying OK
You have messages.
*Files backed up*
You have messages.
3) the "copy of a job" with a new jobid (1085 in this case), but the logs of the job which is copied
*Director Running Jobs*
Running Jobs:
Console connected at 29-Apr-22 13:01
No Jobs running.
====
*Joblog*
Connecting to Director localhost:9101
Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
1000 OK: bareos-dir Version: 21.0.0 (21 December 2021)
bareos.org build binary
bareos.org binaries are UNSUPPORTED by bareos.com.
Get official binaries and vendor support on https://www.bareos.com
You are connected using the default console
Enter a period (.) to cancel a command.
list joblog jobid=1085
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
2022-04-29 12:01:18 bareos-dir JobId 1069: Start Backup JobId 1069, Job=myserver-sys-file.2022-04-29_12.01.16_47
2022-04-29 12:01:18 bareos-dir JobId 1069: Connected Storage daemon at bareos.ad.mydomain.org:9103, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
2022-04-29 12:01:18 bareos-dir JobId 1069: Using Device "bareos-file" to write.
2022-04-29 12:01:18 bareos-dir JobId 1069: Probing client protocol... (result will be saved until config reload)
2022-04-29 12:01:22 bareos-dir JobId 1069: Connected Client: myserver-fd at sub.mydomain.org:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
2022-04-29 12:01:23 bareos-dir JobId 1069: Handshake: Immediate TLS 2022-04-29 12:01:23 bareos-dir JobId 1069: Encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
2022-04-29 12:01:25 myserver-fd JobId 1069: shell command: run ClientBeforeJob "/usr/local/bin/bareos-runbefore.sh"
2022-04-29 12:01:25 myserver-fd JobId 1069: Extended attribute support is enabled
2022-04-29 12:01:25 myserver-fd JobId 1069: ACL support is enabled
2022-04-29 12:01:23 bareos-sd JobId 1069: Connected File Daemon at sub.mydomain.org:9102, encryption: TLS_CHACHA20_POLY1305_SHA256 TLSv1.3
2022-04-29 12:01:25 bareos-sd JobId 1069: Volume "Full-0154" previously written, moving to end of data.
2022-04-29 12:01:25 bareos-sd JobId 1069: Ready to append to end of Volume "Full-0154" size=14003893914
2022-04-29 12:25:39 bareos-sd JobId 1069: Releasing device "bareos-file" (/mnt/leah/var_lib_bareos_storage).
2022-04-29 12:25:39 bareos-sd JobId 1069: Elapsed time=00:24:14, Transfer rate=637.7 K Bytes/second
2022-04-29 12:25:39 bareos-dir JobId 1069: Insert of attributes batch table with 74025 entries start
2022-04-29 12:25:41 bareos-dir JobId 1069: Insert of attributes batch table done
2022-04-29 12:25:41 bareos-dir JobId 1069: Bareos bareos-dir 21.0.0 (21Dec21):
Build OS: Ubuntu 20.04.3 LTS
JobId: 1069
Job: myserver-sys-file.2022-04-29_12.01.16_47
Backup Level: Full
Client: "myserver-fd" 21.0.0 (21Dec21) Debian GNU/Linux 11 (bullseye),debian
FileSet: "myserver-sys-file" 2022-04-28 17:29:32
Pool: "Full" (From command line)
Catalog: "MyCatalog" (From Client resource)
Storage: "bareos-file" (From Pool resource)
Scheduled time: 29-Apr-2022 12:01:16
Start time: 29-Apr-2022 12:01:25
End time: 29-Apr-2022 12:25:41
Elapsed time: 24 mins 16 secs
Priority: 10
FD Files Written: 74,025
SD Files Written: 74,025
FD Bytes Written: 881,441,457 (881.4 MB)
SD Bytes Written: 927,271,942 (927.2 MB)
Rate: 605.4 KB/s
Software Compression: 61.5 % (lz4)
VSS: no
Encryption: yes
Accurate: no
Volume name(s): Full-0154
Volume Session Id: 4
Volume Session Time: 1651214033
Last Volume Bytes: 14,936,345,934 (14.93 GB)
Non-fatal FD errors: 0
SD Errors: 0
FD termination status: OK
SD termination status: OK
Bareos binary info: bareos.org build: Get official binaries and vendor support on bareos.com
Job triggered by: User
Termination: Backup OK
2022-04-29 12:25:41 bareos-dir JobId 1069: shell command: run AfterJob "/bin/bash -c '/bin/echo "run myserver-sys-file-v yes" | bconsole >/dev/null'"
You have messages.
*Files backed up*
==== Here is my config ====
Job { #the job which backs up the remote client to file/disk
Name = "myserver-sys-file"
Type = Backup
Level = Incremental
Client = "myserver-fd"
FileSet = myserver-sys-file
Schedule = myserver-sys-file
Storage = bareos-file
Messages = Standard
Maximum Concurrent Jobs = 7
Pool = Incremental
Priority = 10
Write Bootstrap = "/var/lib/bareos/bootstrap/%c-%n.bsr"
Allow Mixed Priority = yes
#prevent duplicate jobs
Allow Duplicate Jobs = no
Cancel Lower Level Duplicates = yes
Cancel Queued Duplicates = yes
Full Backup Pool = Full # write Full Backups into "Full" Pool (#05)
Differential Backup Pool = Differential # write Diff Backups into "Differential" Pool (#08)
Incremental Backup Pool = Incremental # write Incr Backups into "Incremental" Pool (#11)
RunAfterJob = "/bin/bash -c '/bin/echo \"run myserver-sys-file-v yes\" | bconsole >/dev/null'"
RunScript {
Command = "/usr/local/bin/bareos-runbefore.sh"
RunsWhen = before
RunsOnClient = yes
}
}
Job { #the job which copies to tape
Name = myserver-sys-copy-file-to-tape
Type = Copy
Level = Full
Pool = Full
Selection Type = SQLQuery
Selection Pattern = "select jobid from job where name = 'myserver-sys-file' and level ='F' order by endtime DESC limit 1;"
Next Pool = tape-automated
Schedule = myserver-sys-copy-file-to-tape
Spool Data = Yes
Messages = Standard
Priority = 20
Write Bootstrap = "/var/lib/bareos/bootstrap/%c-%n.bsr"
Allow Duplicate Jobs = no
Cancel Lower Level Duplicates = yes
Cancel Queued Duplicates = yes
RunAfterJob = "/bin/bash -c '/bin/echo \"run myserver-sys-copy-file-to-tape-v yes\" | bconsole >/dev/null'"
}
Pool { #file/disk pool
Name = Full
Pool Type = Backup
Recycle = yes # Bareos can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
Volume Retention = 365 days # How long should the Full Backups be kept? (#06)
Job Retention = 365 days
File Retention = 365 days
Maximum Volume Bytes = 50G # Limit Volume size to something reasonable
Maximum Volumes = 200 # Limit number of Volumes in Pool
Label Format = "Full-" # Volumes will be labeled "Full-<volume-id>"
Storage = bareos-file
Next Pool = tape-automated
}
Pool { #tape pool
Name = tape-automated
Pool Type = Backup
Recycle = yes # Bareos can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
#Recycle Oldest Volume = yes
RecyclePool = Scratch
Maximum Volume Bytes = 0
Job Retention = 365 days
Volume Retention = 4 weeks
Volume Use Duration = 12 days
Cleaning Prefix = "CLN"
Catalog Files = yes #needed, this is default
Storage = tapeSD_HP_1_8_G2
}
Storage { #this is the file/disk storage daemon
Name = bareos-file
Address = "bareos.mydomain.org" # N.B. Use a fully qualified name here (do not use "localhost" here).
Password = "XXXX"
Device = bareos-file
Media Type = File
Maximum Concurrent Jobs = 7
#dashboard progress / statistics collection
Collect Statistics = yes
TLS Enable = yes
TLS Require = yes
TLS DH File = /etc/bareos/ssl/dh4096.pem
}
Storage { #this is the tape storage daemon on another server
Name = "tapeSD_HP_1_8_G2"
Address = "tapeSD.mydomain.org"
Lan Address = "tapeSD.directlink.mydomain.org"
Password = "XXXX"
Device = HP_1_8_G2
Autochanger = yes
MediaType = "LTO"
#dashboard progress / statistics collection
Collect Statistics = yes
TLS Enable = yes
TLS Require = yes
TLS DH File = /etc/bareos/ssl/dh4096.pem
}
Brock Palen
Apr 29, 2022, 9:49:09 AM4/29/22
to tim.b...@gmail.com, bareos-users
That’s pretty much how it works, a copy jobs actually spawns other jobs that do the actual copy.
One thing about copy jobs that prevented me form using them the way I wanted (this may not be an issue for you as you have to SD’s online) is that Consolidation jobs for always incremental jobs works flaky with it, and has no provisioned for an offline copy.
I ended up using the archive job for my offline second copy.
Brock Palen
bro...@mlds-networks.com
www.mlds-networks.com
Websites, Linux, Hosting, Joomla, Consulting
> --
> You received this message because you are subscribed to the Google Groups "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/508d21ad-1d5e-449f-9753-9735dd97c03cn%40googlegroups.com.
One thing about copy jobs that prevented me form using them the way I wanted (this may not be an issue for you as you have to SD’s online) is that Consolidation jobs for always incremental jobs works flaky with it, and has no provisioned for an offline copy.
I ended up using the archive job for my offline second copy.
Brock Palen
bro...@mlds-networks.com
www.mlds-networks.com
Websites, Linux, Hosting, Joomla, Consulting
> You received this message because you are subscribed to the Google Groups "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/508d21ad-1d5e-449f-9753-9735dd97c03cn%40googlegroups.com.
tim.b...@gmail.com
May 2, 2022, 8:29:33 AM5/2/22
to bareos-users
Hi Brock,
thank you. If that might help somebody, instead of 2xN, I managed N+1 emails with the following query (which is also better in regard to failed backups, and making sure it copies the backups, not the copies, and assuming the latest full will always have the highest jobid):
...
Selection Type = SQLQuery
Selection Pattern = "select max(jobid), name from job where type = 'B' and level = 'F' and jobstatus IN ('T','W') and name = any(array['server1-sys-file','server2-sys-file']) group by name;"
Selection Pattern = "select max(jobid), name from job where type = 'B' and level = 'F' and jobstatus IN ('T','W') and name = any(array['server1-sys-file','server2-sys-file']) group by name;"
...
Reply all
Reply to author
Forward
0 new messages