PID file /var/lib/bareos/bareos-dir.9101.pid not readable (yet?) after start: No such file or direct

[Helmut Pable]

I´m running Bareos Ver.17.2.4 on Uubuntu-16.04 LTS with database Mariadb since Feb 2018.

Now after creating a new client I tried

su bareos -s /bin/sh -c "bareos-dir -t" :

root@n8800:~# su bareos -s /bin/sh -c "bareos-dir -t"
bareos-dir: ERROR TERMINATION at parse_conf.c:260
Config error: expected an equals, got: {
: line 2, col 1 of file /etc/bareos/bareos-dir.d/jobdefs/filesvr02-pab-job.conf
{

a message concerning a job, which was successfully done a couple of times earlier.

A "systemctl status bareos-director" gave me the result contained in the attachment "Bareos director failed.txt".

What can I do to recreate the missing PID file /var/lib/bareos/bareos-dir.9101.pid ?

Any help is needed and appreciated!!

[Helmut Pable]

Bareos-Service is dead:
root@n8800:~# systemctl status bareos
bareos.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)

Here should be the missing file bareos-dir.9101.pid:
root@n8800:~# ls -al /var/lib/bareos/
insgesamt 316
drwxr-xr-x 4 bareos bareos 4096 Jun 25 14:19 .
drwxr-xr-x 46 root root 4096 Jun 9 13:43 ..
-rw-r----- 1 bareos bareos 87 Jun 22 01:00 bareos.17302.traceback
-rw-r----- 1 bareos bareos 1373 Jun 22 01:00 bareos-dir.17302.bactrace
-rw-r----- 1 bareos bareos 2112 Jun 25 11:31 bareos-dir.9101.state
-rw------- 1 bareos bareos 262300 Jun 25 11:31 bareos-dir.conmsg
-rw-r----- 1 root bareos 4 Jun 25 14:19 bareos-fd.9102.pid
-rw-r----- 1 root bareos 2112 Jun 25 14:18 bareos-fd.9102.state
-rw-r----- 1 bareos bareos 651 Jun 16 01:00 bareos-fd.bsr
-rw-r----- 1 bareos bareos 4 Jun 25 14:19 bareos-sd.9103.pid
-rw-r----- 1 bareos bareos 2112 Jun 25 14:18 bareos-sd.9103.state
drwxrwxr-x 3 root root 4096 Jun 9 13:27 dbconfig-common
-rw------- 1 bareos bareos 62 Jun 22 21:10 .my.cnf
drwxr-x--- 2 bareos bareos 4096 Aug 20 2017 storage

[Bruno Friedmann]

Your director can't run with a error in its configuration, which is exactly what indicate your bareos-sd -t command
Have a look at that file filesvr02-pad-job.conf

Compare it with the other client definition you have and you will find the obvious missing part.

I'm also a bit surprized you have the client definition in the jobdefs folder, wouldn't you have it located in jobs subdir instead ?

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Helmut Pable]

Thank you Bruno for your answer! I understand your brevity completely!
There is no error in the configuration, as exactly this job was running before many times and its resources were not touched, as I could proof.
To find out was going on I deleted some similar client jobs "xxx" configurations to test the situation and always got
"Config error: expected an equals, got: { : line 2, col 1 of file /etc/bareos/bareos-dir.d/jobdefs/xxx.conf"
for what job "xxx" ever, which all were running before smoothly.

All the client definitions are located as follows:
Director:/etc/bareos/bareos-dir.d/director/bareos-dir.conf
Client: /etc/bareos/bareos-dir.d/client/filesvr02-pab-fd.conf
Job: /etc/bareos/bareos-dir.d/job/filesvr02-pab-job.conf
JobDefs: /etc/bareos/bareos-dir.d/jobdefs/filesvr02-pab-job.conf
Pool: /etc/bareos/bareos-dir.d/pool/filesvr02-pab.conf
Fileset: /etc/bareos/bareos-dir.d/fileset/filesvr02-pab.conf
Schedule:/etc/bareos/bareos-dir.d/schedule/filesvr02-pab.conf
Device; /etc/bareos/bareos-sd.d/device/filesvr02-pab.sd.conf

I should point out, that all configs are created in the same way as described in OpenSource Conference Sept 25-26 2017 in Cologne by Dr. Stefan Vollmar - part "Adding clients by templates" (templates courtesy of J. Behrend, thanks!), https://osbconf.org/wp-content/uploads/2017/10/vollmar-obsconf-2017-15-p-g.pdf.

All this client-configs were tested and run without any error until the director died: I append here what is to find in /var/lib/bareos/bareos-dir.conmsg for the incriminated job "filesvr02-pab-job":

25-Jun 01:45 filesvr02-fd JobId 121: VSS Writer (BackupComplete): "System Writer", State: 0x1 (VSS_WS_STABLE)
25-Jun 01:45 filesvr02-fd JobId 121: VSS Writer (BackupComplete): "ASR Writer", State: 0x1 (VSS_WS_STABLE)
25-Jun 01:45 filesvr02-fd JobId 121: VSS Writer (BackupComplete): "Shadow Copy Optimization Writer", State: 0x1 (VSS_WS_STABLE)
25-Jun 01:45 filesvr02-fd JobId 121: VSS Writer (BackupComplete): "Registry Writer", State: 0x1 (VSS_WS_STABLE)
25-Jun 01:45 filesvr02-fd JobId 121: VSS Writer (BackupComplete): "DFS Replication service writer", State: 0x1 (VSS_WS_STABLE)
25-Jun 01:45 filesvr02-fd JobId 121: VSS Writer (BackupComplete): "IIS Metabase Writer", State: 0x1 (VSS_WS_STABLE)
25-Jun 01:45 filesvr02-fd JobId 121: VSS Writer (BackupComplete): "BITS Writer", State: 0x1 (VSS_WS_STABLE)
25-Jun 01:45 filesvr02-fd JobId 121: VSS Writer (BackupComplete): "IIS Config Writer", State: 0x1 (VSS_WS_STABLE)
25-Jun 01:45 filesvr02-fd JobId 121: VSS Writer (BackupComplete): "FRS Writer", State: 0x1 (VSS_WS_STABLE)
25-Jun 01:45 filesvr02-fd JobId 121: VSS Writer (BackupComplete): "WMI Writer", State: 0x1 (VSS_WS_STABLE)
25-Jun 01:45 filesvr02-fd JobId 121: VSS Writer (BackupComplete): "COM+ REGDB Writer", State: 0x1 (VSS_WS_STABLE)
25-Jun 01:45 filesvr02-fd JobId 121: VSS Writer (BackupComplete): "Dhcp Jet Writer", State: 0x1 (VSS_WS_STABLE)
25-Jun 01:45 filesvr02-fd JobId 121: VSS Writer (BackupComplete): "NTDS", State: 0x1 (VSS_WS_STABLE)
25-Jun 01:45 bareos-sd JobId 121: Elapsed time=00:15:02, Transfer rate=7.943 K Bytes/second
25-Jun 01:45 bareos-dir JobId 121: sql_create.c:872 Insert of attributes batch table done
25-Jun 01:45 bareos-dir JobId 121: Bareos bareos-dir 17.2.4 (21Sep17):
Build OS: i686-pc-linux-gnu ubuntu Ubuntu 16.04 LTS
JobId: 121
Job: filesvr02-pab-job.2018-06-25_01.05.00_16
Backup Level: Incremental, since=2018-06-24 01:08:00
Client: "filesvr02-pab-fd" 17.2.4 (21Sep17) xxxxxxxxxxxxxxxxxx
FileSet: "filesvr02-pab" 2018-06-24 01:05:00
Pool: "filesvr02-pab" (From Job resource)
Catalog: "MyCatalog" (From Client resource)
Storage: "File" (From Job resource)
Scheduled time: 25-Jun-2018 01:05:00
Start time: 25-Jun-2018 01:30:13
End time: 25-Jun-2018 01:45:46
Elapsed time: 15 mins 33 secs
Priority: 2
FD Files Written: 49
SD Files Written: 49
FD Bytes Written: 7,151,691 (7.151 MB)
SD Bytes Written: 7,165,057 (7.165 MB)
Rate: 7.7 KB/s
Software Compression: 16.1 % (gzip)
VSS: yes
Encryption: no
Accurate: no
Volume name(s): filesvr02-pab-0020
Volume Session Id: 29
Volume Session Time: 1529779051
Last Volume Bytes: 7,172,837 (7.172 MB)
Non-fatal FD errors: 2
SD Errors: 0
FD termination status: OK
SD termination status: OK
Termination: Backup OK -- with warnings

The warnings concerne a volume which was removed from the client machine without correcting the jobdef.

Now the director is dead I can´t work with bconsole:
root@n8800:~# bconsole
Connecting to Director localhost:9101

root@n8800:~# systemctl status bareos
bareos.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)

root@n8800:~#

What can I do to correct the situation? Is this only because file bareos-dir.9101.pid is missing? How the director could have crashed?
Can I restore somehow the last configuration?

[Bruno Friedmann]

Still on the road I resume what I've understood of your setup.

There's two main point you should concentrate on it.
- configuration parse error :
This should finish with no error, otherwise the standard systemd service will failed (there's a pre -t before running)
I've not the bandwith to check out the system of template cited in the pdf. So I'm only working on guessing ;-)

- When you run your bareos-dir -t you should also include the -u bareos -b bareos, so it run as the expected user forced (normally your su bareos should do that, but who know) Also check in /etc/bareos that every file are at least readable by bareos, same for all in /var/lib/bareos)

- The pid file not yet readable is just warning on systemd powered system, the check happen too quickly before the daemon start.
once the bareos daemon is started, a new call of systemctl start bareos-dir should show that everything is working well.

Here's there's something really strange on your system, systemctl return file not found, mean that it can't load /usr/lib/systemd/system/bareos-dir.service ? Are the service file still there ?

[Helmut Pable]

Thank you Bruno for your help and advice!
I checked configs of ALL clients - there is no config error:

root@n8800:~# su bareos - configuration parse error
root@n8800:~# ^C
root@n8800:~# su bareos -s /bin/sh -c "bareos-dir -t -u bareos -m"
a-\▒bareos-dir: ERROR TERMINATION at parse_conf.c:260

Config error: expected an equals, got: {
: line 2, col 1 of file /etc/bareos/bareos-dir.d/jobdefs/filesvr02-pab-job.conf
{

root@n8800:~#


But there is no directory "/usr/lib/systemd/system/". Iappend what I have found:

root@n8800:~# ls -al /usr/lib/systemd/system/
ls: Zugriff auf '/usr/lib/systemd/system/' nicht möglich: Datei oder Verzeichnis nicht gefunden
root@n8800:~# ls -al /usr/lib/systemd/
insgesamt 28
drwxr-xr-x 7 root root 4096 Aug 10 2017 .
drwxr-xr-x 49 root root 4096 Jun 9 13:30 ..
drwxr-xr-x 3 root root 4096 Aug 10 2017 boot
drwxr-xr-x 2 root root 4096 Jun 9 13:27 catalog
drwxr-xr-x 2 root root 4096 Aug 10 2017 network
drwxr-xr-x 2 root root 4096 Jun 9 13:27 user
drwxr-xr-x 2 root root 4096 Jun 9 13:27 user-generators
root@n8800:~#

root@n8800:~# ls -al /usr/lib/bareos/
insgesamt 1920
drwxr-xr-x 6 root root 4096 Jun 9 13:27 .
drwxr-xr-x 49 root root 4096 Jun 9 13:30 ..
drwxr-xr-x 2 root root 4096 Jun 9 13:27 backends
drwxr-xr-x 6 root root 4096 Jun 9 13:27 defaultconfigs
-rw-r--r-- 1 root root 504848 Dez 14 2017 libbareos-17.2.4.so
-rw-r--r-- 1 root root 9768 Dez 14 2017 libbareoscats-17.2.4.so
-rw-r--r-- 1 root root 105400 Dez 14 2017 libbareoscfg-17.2.4.so
-rw-r--r-- 1 root root 87988 Dez 14 2017 libbareosfind-17.2.4.so
-rw-r--r-- 1 root root 87808 Dez 14 2017 libbareoslmdb-17.2.4.so
-rw-r--r-- 1 root root 584864 Dez 14 2017 libbareosndmp-17.2.4.so
-rw-r--r-- 1 root root 350320 Dez 14 2017 libbareossd-17.2.4.so
-rw-r--r-- 1 root root 194804 Dez 14 2017 libbareossql-17.2.4.so
drwxr-xr-x 2 root root 4096 Jun 9 13:27 plugins
drwxr-xr-x 3 root root 4096 Jun 25 16:01 scripts
root@n8800:~#

root@n8800:~# ls -al /var/lib/bareos/

insgesamt 340
drwxr-xr-x 4 bareos bareos 4096 Jun 26 10:46 .

drwxr-xr-x 46 root root 4096 Jun 9 13:43 ..
-rw-r----- 1 bareos bareos 87 Jun 22 01:00 bareos.17302.traceback

-rw-r--r-- 1 root root 10240 Jun 26 09:33 bareos.capture.201806260933.tar
-rw-r--r-- 1 root root 10240 Jun 26 10:10 bareos.capture.201806261010.tar

-rw-r----- 1 bareos bareos 1373 Jun 22 01:00 bareos-dir.17302.bactrace
-rw-r----- 1 bareos bareos 2112 Jun 25 11:31 bareos-dir.9101.state
-rw------- 1 bareos bareos 262300 Jun 25 11:31 bareos-dir.conmsg

-rw-r----- 1 root bareos 4 Jun 26 08:43 bareos-fd.9102.pid
-rw-r----- 1 root bareos 2112 Jun 26 08:42 bareos-fd.9102.state

-rw-r----- 1 bareos bareos 651 Jun 16 01:00 bareos-fd.bsr

-rw-r----- 1 bareos bareos 4 Jun 26 08:43 bareos-sd.9103.pid
-rw-r----- 1 bareos bareos 2112 Jun 26 08:42 bareos-sd.9103.state

drwxrwxr-x 3 root root 4096 Jun 9 13:27 dbconfig-common
-rw------- 1 bareos bareos 62 Jun 22 21:10 .my.cnf
drwxr-x--- 2 bareos bareos 4096 Aug 20 2017 storage

root@n8800:~#

root@n8800:~# ls -al /etc/bareos/
insgesamt 160
drwxr-xr-x 6 root root 4096 Jun 25 13:56 .
drwxr-xr-x 95 root root 4096 Jun 9 13:45 ..
drwxr-x--- 14 bareos bareos 4096 Jun 9 13:27 bareos-dir.d
drwxr-x--- 3 bareos bareos 4096 Aug 19 2017 bareos-dir-export
drwxr-x--- 5 bareos bareos 4096 Aug 19 2017 bareos-fd.d
drwxr-x--- 6 bareos bareos 4096 Jun 9 13:27 bareos-sd.d
-rw-r----- 1 root bareos 236 Aug 19 2017 bconsole.conf
-rw-r--r-- 1 root root 230 Jun 9 13:41 bconsole.conf.dist
-rw-r--r-- 1 root root 297 Jun 21 22:13 exclude_linux
-rw-r--r-- 1 root root 1009 Jun 23 15:33 exclude_windows
-rw-r--r-- 1 root root 245 Jun 18 10:55 exclude_windows_dat
-r-------- 1 root root 522 Aug 19 2017 .rndpwd

[Helmut Pable]

Isn´t the message "Config error: expected an equals, got: { " itself false??
In all resourcefiles there must be an "{" on this place!! There cant be an "="!!!

[Jörg Steffens]

[...]

> Isn´t the message "Config error: expected an equals, got: { " itself false??
> In all resourcefiles there must be an "{" on this place!! There cant be an "="!!!

If bareos-dir -t indicates an error, bareos-dir will never start, as due
to an configuration error.

I've never seen this error, claiming an error on line 2.
What I can think of is that the error isn't in this configuration file,
but in the file parsed before. In principle. bareos merges all config
files together and just indicating you, at what position it find the
invalid syntax.
Just thinking: if you have an open { in the file before, without an
match closing }, it might leads to this kind of error.

regards,
Jörg
--
Jörg Steffens joerg.s...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221 630693-91
http://www.bareos.com Fax: +49 221 630693-10

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer:
S. Dühr, M. Außendorf, Jörg Steffens, P. Storz

[Helmut Pable]

Thanks Jörg!
I rechecked again all configurastion files, including these which I did NOT create and really, there was a closing bracket missing in /etc/bareos/bareos-dir.d/jobdefs/DefaultJob.conf, which I did not check before.

Thanks again Bruno for your dear support!

[Tam Angustia]

I kinda did this when I was setting up my first instance of bareos. I solved it after a few days and a lot of headaches. that "}" really messed me up. Good thing I use vim when editing.
Good to know you have found it as well!

Tam