bareos-sd startup systemd dependency

[Silvio Schloeffel]

Hi,

not a showstopper but something that can make more work under certain
circumstances.

Our RHEL8 based backup system has the filestorage mounted over nfs and
today we had scheduled a UPS shutdown/restart test. Everything worked as
requested, except the bareos service.

The bareos-sd service was down.
[root@storage-serv ~]# systemctl status bareos-sd
● bareos-sd.service - Bareos Storage Daemon service
Loaded: loaded (/usr/lib/systemd/system/bareos-sd.service; enabled;
vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2023-08-25 06:19:41
UTC; 33min ago
Docs: man:bareos-sd(8)
Process: 1721 ExecStart=/usr/sbin/bareos-sd -f (code=exited,
status=1/FAILURE)
Main PID: 1721 (code=exited, status=1/FAILURE)

Aug 25 06:19:41 storage-serv.musicdna.com systemd[1]: Started Bareos
Storage Daemon service.
Aug 25 06:19:41 storage-serv.musicdna.com bareos-sd[1721]: bareos-sd
ERROR TERMINATION
Aug 25 06:19:41 storage-serv.musicdna.com bareos-sd[1721]:
stored/stored_conf.cc:524 Unable to stat path
'/mnt/backup-storage/backupdata/storage' for device FileStorage1:
ERR=Datei oder Verzeichnis nicht gefunden
Aug 25 06:19:41 storage-serv.musicdna.com bareos-sd[1721]: Consider
setting Device Type if device is not available when daemon starts.
Aug 25 06:19:41 storage-serv.musicdna.com systemd[1]: bareos-sd.service:
Main process exited, code=exited, status=1/FAILURE
Aug 25 06:19:41 storage-serv.musicdna.com systemd[1]: bareos-sd.service:
Failed with result 'exit-code'.

-> check if storage is mounted:

10.234.16.52:/backup 50T 6,1T 44T 13% /mnt/backup-storage

Manual restart:

[root@storage-serv ~]# systemctl start bareos-sd
[root@storage-serv ~]# systemctl status bareos-sd
● bareos-sd.service - Bareos Storage Daemon service
Loaded: loaded (/usr/lib/systemd/system/bareos-sd.service; enabled;
vendor preset: disabled)
Active: active (running) since Fri 2023-08-25 06:56:24 UTC; 4s ago
Docs: man:bareos-sd(8)
Main PID: 3497 (bareos-sd)
Tasks: 2 (limit: 409922)
Memory: 1.4M
CGroup: /system.slice/bareos-sd.service
└─3497 /usr/sbin/bareos-sd -f


-> works.

It looks like the bareos-sd daemon starts before all devices are
mounted. We mount the storage direct over the fstab, so it will mount as
early as possible but not before all network services are up and ready.

So eventually it would be a solution to start the bareos-sd service
later -> eventually a delay.

In your config you have:

# Restart on failure, wait 30 seconds
Restart=on-failure
RestartSec=30

if this works, it could solve the problem. But it looks like it does not
work in my configuration.


Best

Silvio


--
Silvio Schloeffel | CIO
silvio.s...@musicdna.com
Tel. +49 3677 2094 840

the evolution of music continues

MusicDNA GmbH
Schwanitzstrasse 6,
98693 Ilmenau, Germany
Phone: +49 3677 2094 840 | Fax: +49 322 298 222 00

Steuer-NR.: DE 156/114/04317

Handelsregisterzeichen: HRB Jena 517208,
Geschäftsführer: Sebastian Schmidt
www.musicdna.com

[Silvio Schloeffel]

[Bruno Friedmann]

We have detected some similar issues and a patch see PR1515 will improve even more the start of daemon, 

We have added local-fs.target to the After target (containing already remote-fs.target) to all daemons. Normally now systemd should wait until all local fs are up.

The failure setup has been also fixed in the meantime.