massive runtime changes on only one job

[Silvio Schloeffel]

Hi,

I have a problem with one job which I can not understand.

We have a bareos server behind a firewall running in our dc.
This server connects to all the outside clients and backups the data
(client can not connect direct to the server).

It's running since ~ a year without major problem.

Additionally we have a vpn connection from an office to the fw gateway
and in the office we have some systems in the backup. Also running
without problem until 8th of july.

We have one job (not system because another job on this system is
running normal) with massive runtime changes.

The job is a always incremental and backups the home dirs.

On 7th of july and before it looked like this:

07-Jul 04:32 bareos-dir JobId 6278: Bareos bareos-dir
22.1.1~pre13.b49eea503 (27Jun23):
Build OS: Red Hat Enterprise Linux release 8.7 (Ootpa)
JobId: 6278
Job: bachserv17-home-backup.2023-07-07_03.00.00_47
Backup Level: Incremental, since=2023-07-06 03:00:03
Client: "bachserv17" 22.1.1~pre18.1c87f96b5 (28Jun23)
Red Hat Enterprise Linux Server release 7.9 (Maipo),redhat
FileSet: "Homedir" 2022-09-28 21:00:02
Pool: "AI-Incremental" (From Job resource)
Catalog: "MyCatalog" (From Client resource)
Storage: "File" (From Pool resource)
Scheduled time: 07-Jul-2023 03:00:00
Start time: 07-Jul-2023 03:00:03
End time: 07-Jul-2023 04:32:55
Elapsed time: 1 hour 32 mins 52 secs
Priority: 10
FD Files Written: 624
SD Files Written: 624
FD Bytes Written: 11,980,296,864 (11.98 GB)
SD Bytes Written: 11,980,414,768 (11.98 GB)
Rate: 2150.1 KB/s
Software Compression: 43.8 % (lz4)
VSS: no
Encryption: no
Accurate: yes
Volume name(s): AI-Incremental-0046
Volume Session Id: 209
Volume Session Time: 1687987409
Last Volume Bytes: 16,308,603,906 (16.30 GB)
Non-fatal FD errors: 0
SD Errors: 0
FD termination status: OK
SD termination status: OK
Bareos binary info: Bareos community build (UNSUPPORTED): Get
professional support from https://www.bareos.com
Job triggered by: Scheduler
Termination: Backup OK


So ~1.5h for the job over the vpn connection in the night was normal.

Since 8th of july i have this (every day).

08-Jul 08:09 bareos-dir JobId 6301: Bareos bareos-dir
22.1.1~pre13.b49eea503 (27Jun23):
Build OS: Red Hat Enterprise Linux release 8.7 (Ootpa)
JobId: 6301
Job: bachserv17-home-backup.2023-07-08_03.00.00_11
Backup Level: Incremental, since=2023-07-07 03:00:03
Client: "bachserv17" 22.1.1~pre18.1c87f96b5 (28Jun23)
Red Hat Enterprise Linux Server release 7.9 (Maipo),redhat
FileSet: "Homedir" 2022-09-28 21:00:02
Pool: "AI-Incremental" (From Job resource)
Catalog: "MyCatalog" (From Client resource)
Storage: "File" (From Pool resource)
Scheduled time: 08-Jul-2023 03:00:00
Start time: 08-Jul-2023 03:00:03
End time: 08-Jul-2023 08:09:08
Elapsed time: 5 hours 9 mins 5 secs
Priority: 10
FD Files Written: 489
SD Files Written: 489
FD Bytes Written: 10,235,370,879 (10.23 GB)
SD Bytes Written: 10,235,465,494 (10.23 GB)
Rate: 551.9 KB/s
Software Compression: 47.0 % (lz4)
VSS: no
Encryption: no
Accurate: yes
Volume name(s): AI-Incremental-0049
Volume Session Id: 231
Volume Session Time: 1687987409
Last Volume Bytes: 13,827,736,531 (13.82 GB)
Non-fatal FD errors: 0
SD Errors: 0
FD termination status: OK
SD termination status: OK
Bareos binary info: Bareos community build (UNSUPPORTED): Get
professional support from https://www.bareos.com
Job triggered by: Scheduler
Termination: Backup OK


As you can see the job now needs 5h and I do not know why.

I checked if we had some updates on client and server -> nothing.
Another job on this server shows not difference. Check the home storage
-> reading with maximum network speed. Checked the logs of both systems
-> nothing.

The vpn connection is over a 40Mbit/s upload link and is only used for
the jobs in the night -> also checked.


The homes have ~ 8Mio Files but this was not a problem untils this date.

Any ideas what I can check or do I eventually have to change some
parameters if I have a longer running system with bigger always inc jobs?

The job:

Job {
# required
Name = "bachserv17-home-backup"

# the default settings
JobDefs = "BackupHome"
Accurate = yes
Always Incremental = yes
Always Incremental Job Retention = 1 month
Always Incremental Keep Number = 28 days
Always Incremental Max Full Age = 2 months
Pool = AI-Incremental
Full Backup Pool = AI-Consolidated
# overwrite the client here
Client = "bachserv17"
}

JobDefs {
# name (required)
Name = "BackupHome"

# type can be backup/restore/verify
Type = Backup

# the default level bareos will try
# can also be Full/Differential(since last
full)/Incremental(since last incremental)
Level = Incremental

# the default client, to be overwritten by the job.conf
Client = bareos-fd

# what files to include/exclude
FileSet = "Homedir"

# the schedule we just created
Schedule = "Home-backup"

# where to store it
Storage = File

# the message reporting
Messages = Standard

# # the pool where to store it#
# Pool = Incremental

# the higher the value priority the lower it will be dropped in
the queue
# so for important jobs priority=1 will run first
Priority = 10

# the bootstrap file keeps a "log" of all the backups, and gets
rewritten every time a
# full backup is made, it can be used during recovery
Write Bootstrap = "/var/lib/bareos/%c.bsr"

# in case these value's get overwritten
# define where would be a good pool to write
# note that full backup will be used atleast once because no full
# backup will exist
Pool = AI-Incremental
Full Backup Pool = AI-Consolidated
}

The bareos server is a Xeon(R) Bronze 3204 with 64GB and the database is
on a ssd drive, so this should not be a problem.

Thanks for any idea

Best

Silvio

[Philipp Storz]

Hello,

you could compare how long the steps of the jobs take.

i.e.: pre job runscripts, sending accurate information, doing the real backup, doing the post backup
scripts.

Depending on the step that takes longer you can probably find out what is the problem.

best regards,

Philipp
Am 01.08.23 um 15:52 schrieb Silvio Schloeffel:

--
Mit freundlichen Grüßen

Philipp Storz philip...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221 63 06 93-92
http://www.bareos.com Fax: +49 221 63 06 93-10

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Geschäftsführer: Stephan Dühr, J. Steffens, P. Storz

[Philipp Storz]

Hello,

checking the joblog should show the consumed times.
The joblog has timestamps for each line so you should be able to estimate how long each step takes.

Am 01.08.23 um 16:33 schrieb Silvio Schloeffel:

> Thanks for the answer.
>
> I have not scripts in this job, it's a "simple backup this dir job".
> The only thing could be the calculation of "what to backup" and this should be done by the server.
>
> Do you know a way to see how long the server needs for this step?
> The clients have only the bareos-fd running, sd and dir are backup server only because most of the
> systems can not directly connect to the backup server.
>
> Best
>
> Silvio
>
> Am 01.08.23 um 16:25 schrieb Philipp Storz: