Weird issue when doing scheduled backups

[Valentin Dzhorov]

Hey guys! So, I am running Bareos experimentally in a test environment. I am backing up 10 clients on a remote storage server. The issue I am facing is that sometimes when doing scheduled backups (incremental or full), some of the backup jobs fail with the error message "Fatal error: Failed to authenticate Storage daemon.". However, when re-running the same job via WebUI, the job runs without any issue. One thing to mention is that I am backing up to disk and my setup is done according to the documentation for running concurrent backup jobs on a disk. I am running 5 concurrent backup jobs at the moment. Also, the authorization between client->dir and client->storage is done via a password. Any hints on what may be wrong are welcome! Here is the log from a failed backup job:

2020-01-10 00:00:06 client1 JobId 112: Fatal error: Authorization key rejected client1.

2020-01-10 00:00:06 client1 JobId 112: Fatal error: Failed to authenticate Storage daemon.

2020-01-10 00:00:06 bareos-dir JobId 112: Fatal error: Bad response to Storage command: wanted 2000 OK storage

, got 2902 Bad storage

2020-01-10 00:00:06 bareos-dir JobId 112: Error: Bareos bareos-dir 19.2.4~rc1 (19Dec19):

Build OS: Linux-5.3.14-200.fc30.x86_64 redhat CentOS Linux release 7.7.1908 (Core)

JobId: 112

Job: client1.2020-01-10_00.00.00_48

Backup Level: Incremental, since=2020-01-09 00:00:01

Client: "client1" 19.2.4~rc1 (19Dec19) Linux-5.3.14-200.fc30.x86_64,redhat,CentOS Linux release 7.7.1908 (Core)

FileSet: "LinuxAllwithSQL_EL" 2020-01-08 00:00:00

Pool: "client1" (From Job resource)

Catalog: "MyCatalog" (From Client resource)

Storage: "remote_storage_public_1" (From Pool resource)

Scheduled time: 10-Jan-2020 00:00:00

Start time: 10-Jan-2020 00:00:00

End time: 10-Jan-2020 00:00:06

Elapsed time: 6 secs

Priority: 20

FD Files Written: 0

SD Files Written: 0

FD Bytes Written: 0 (0 B)

SD Bytes Written: 0 (0 B)

Rate: 0.0 KB/s

Software Compression: None

VSS: no

Encryption: no

Accurate: no

Volume name(s):

Volume Session Id: 22

Volume Session Time: 1578498212

Last Volume Bytes: 0 (0 B)

Non-fatal FD errors: 2

SD Errors: 0

FD termination status: Fatal Error

SD termination status: Waiting on FD

Bareos binary info: pre-release version: Get official binaries and vendor support on bareos.com

Termination: *** Backup Error ***

2020-01-10 00:00:01 bareos-dir JobId 112: Using Device "FileStorage5" to write.

2020-01-10 00:00:01 bareos-dir JobId 112: Connected Client: client1 at 79.98.111.69:9102, encryption: None

2020-01-10 00:00:01 bareos-dir JobId 112: Handshake: Cleartext

2020-01-10 00:00:01 bareos-dir JobId 112: Encryption: None

2020-01-10 00:00:00 bareos-dir JobId 112: Start Backup JobId 112, Job=client1.2020-01-10_00.00.00_48

2020-01-10 00:00:00 bareos-dir JobId 112: Connected Storage daemon at 185.55.228.135:9103, encryption: None

[Anthony Vaccaro]

Hi Valentin,

My reply is a bit late, so you may have already solved this issue. However, I think this is caused by a mismatch of "maximum concurrent jobs" configuration between your director and storage daemon. Please make sure the director and storage daemon have the same value for this setting.

There's some more information here - https://docs.bareos.org/Appendix/Troubleshooting.html?highlight=concurrent .

Cheers, Anthony

--
You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/8a5aec1b-091f-4365-807f-e723c86bcd5c%40googlegroups.com.

[Ricardo Almeida]

Hi,

I am facing the same problem described by Valentin. Did you find the problem and solution?

Thanks, best regards.

To unsubscribe from this group and stop receiving emails from it, send an email to bareos...@googlegroups.com.

[Valentin Dzhorov]

Hey Ricardo,

I am sorry for the delayed response. I did find a solution to my issue, or so I think. At least I have not experienced any of the issue after changing the Maximum Connections in the storage resource on the storage server. The default is 42, I have increased it to 100 and no longer get that error. Here is the documentation for that var: https://docs.bareos.org/Configuration/StorageDaemon.html#config-Sd_Storage_MaximumConnections. Please let us know how it goes for you if you decide to have a go at increasing the value of this var.

[Ricardo Almeida]

Hi Valentin,

After trying many options, my mistake was the number of  "Maximum Concurrent Jobs" that I have configured different values between bareos-dir storage resource and bareos-sd storage resource. I followed the instructions in the comments of bareos-sd storage - (# It is usually best to make sure the value on the SD's Storage resource leaves a little room (if you set it to 40 on the director, try 50 on the SD).), - but I think that this comment is to set a greater "Maximum Concurrent Jobs" for the bareos-dir director resource and not for the bareos-dir storage resource. 

Now, everything is working as expected.

Thank you, Valentin. Best regards.