Director failed to start

[Helmut Pable]

Hi,
my bareos installation worked well during the last 12 months without any change of config-files, operating on dedicated server under Ubuntu 16.04.5 LTS. Bareos Version: 18.2.5 (30 January 2019), Ver 15.1 Distrib 10.0.38-MariaDB, for debian-linux-gnu (i686).

On Nov.29th all jobs were finished successfully, but when I tried to look at the result of the backups,  I got "bareos-director.service: Failed with result 'signal'." (see the attached file)

I checked my database and it turned out to be OK (see the attached file also).

I did research as described in my last post but did not find the culprit. To my disappointment noone could lead me to a solution of the problem - thanks to all who tried to help!

What can I do now?

Please help

Helmut

[Chad W Seys]

Hi Helmut,
Try to increase the open file limit:

E.g.
# "/etc/systemd/system/bareos-director.service.d/override.conf"
[Service]
LimitNOFILE=10000

systemctl daemon-reload
systemctl restart bareos-director

Chad.

[Helmut Pable]

Hi Chad,

 thanks for your advice. This file does not exist already nor exists directory /etc/systemd/system/bareos-director.service.d

In dir /etc/systemd/system there is ony the file "bareos-dir.service", which  starts the bareos director daemon service (bareos-dir) ---> "will be installed as /lib/systemd/system/bareos-dir.service". These 2 files are identical.

Should I create /etc/systemd/system/bareos-director.service.d/override.conf ?

Helmut

[M-D-F]

Hello Helmut,

how did you get from your Unknown failure "singnal 11 segmentataion violation" to that rather concrete "resouce limit exceeded", if I may ask?

Did you start from scratch? 

One of my installations of bareos came to a grinding halt with "signal 11" as well...

Thanks and regards,

Matthias

Am Dienstag, 10. Dezember 2019 13:11:06 UTC+1 schrieb Helmut Pable:

[Chad W Seys]

Hi Helmut,
Yes, create this directory and the file.

Chad.

> --
> You received this message because you are subscribed to the Google
> Groups "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to bareos-users...@googlegroups.com
> <mailto:bareos-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/bareos-users/913f2a0c-8675-447b-b3cf-094e4157a3e4%40googlegroups.com
> <https://groups.google.com/d/msgid/bareos-users/913f2a0c-8675-447b-b3cf-094e4157a3e4%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Chad W Seys]

Hi Matthias,
Would you happen to be using mysql/mariadb? Would it happen to be
pegged at 100% CPU usage?

I had these symptoms only after backing up lots of files. No
configuration changes. Recently I switch to postgresql and am trying
again. So far no problems, but I'm still backing up. Once I'm
confident that the behavior is not coming

Postgresql is the recommended database on the setup, but possibly mysql
should be NOT mentioned as an alternative. Or at least some mention of
problems. Again, I'm not 100% confident that the problem won't come
back with postgresql, but I'm hopeful.

Chad.

> --
> You received this message because you are subscribed to the Google
> Groups "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to bareos-users...@googlegroups.com
> <mailto:bareos-users...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/bareos-users/4017c5b7-10ae-4fdc-9f09-7158577a5aa2%40googlegroups.com
> <https://groups.google.com/d/msgid/bareos-users/4017c5b7-10ae-4fdc-9f09-7158577a5aa2%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Chad W Seys]

Also, if the override file loads successfully you'll see it in this output:
# systemctl status bareos-director.service
● bareos-director.service - Bareos Director Daemon service
Loaded: loaded (/lib/systemd/system/bareos-director.service;
enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/bareos-director.service.d
└─override.conf


and
# systemctl show bareos-dir | grep LimitNOFILE
LimitNOFILE=10000
LimitNOFILESoft=10000


Chad.

[Helmut Pable]

Hi Chad,

I did as you told me, see the result:

root@n8800:~# mkdir /etc/systemd/system/bareos-director.service.d
root@n8800:~# nano /etc/systemd/system/bareos-director.service.d/override.conf

root@n8800:~# systemctl daemon-reload
root@n8800:~# systemctl restart bareos-director
Job for bareos-director.service failed because a fatal signal was delivered to the control process. See "systemctl status bareos-director.service" and "journalctl -xe" for details.
root@n8800:~#

root@n8800:~# systemctl status bareos-director.service
? bareos-director.service - Bareos Director Daemon service

   Loaded: loaded (/lib/systemd/system/bareos-director.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/bareos-director.service.d

           +-override.conf
   Active: failed (Result: signal) since Mit 2019-12-11 21:50:41 CET; 1min 32s ago
     Docs: man:bareos-dir(8)
  Process: 2241 ExecStartPre=/usr/sbin/bareos-dir -t -f (code=killed, signal=SEGV)

Any idea?

Helmut

[Chad W Seys]

Dang, possibly it is a different problem than what I experienced.
Does /var/log/bareos/bareos.log have any clues?

Chad.

> --
> You received this message because you are subscribed to the Google
> Groups "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to bareos-users...@googlegroups.com
> <mailto:bareos-users...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/bareos-users/b976bbd3-8642-4854-9e04-b3f90ad8808e%40googlegroups.com
> <https://groups.google.com/d/msgid/bareos-users/b976bbd3-8642-4854-9e04-b3f90ad8808e%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Andreas Rogge]

Hi Helmut,

please install the bareos debugging symbols (on fedora you can use
debuginfo-install) and gdb.
Afterwards the crashes should produce meaningful trace-files in
/var/lib/bareos/.

From there we can take a look what crashes and maybe why.

Best Regards,
Andreas
--
Andreas Rogge andrea...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221-630693-86
http://www.bareos.com

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: S. Dühr, M. Außendorf, J. Steffens, Philipp Storz

[Helmut Pable]

Hi Andreas,

thanks for your advice. What is the command to install the bareos debugging symbols on ubuntu:

root@n8800:~# apt-get install debuginfo
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
E: Paket debuginfo kann nicht gefunden werden.

Does exist a specific bareos-debug-package?

Helmut

[Helmut Pable]

Hi Chad,

there are no clues in /var/log/bareos/bareos.log, because I did what M-D-F described in an other post:

# /usr/sbin/bareos-dir -d 500 -u bareos -g bareos

which led also to the result: webui accessable and director started the first of in the meantime missed orders. This produced only messages concerning this job in /var/log/bareos/bareos.log.

Helmut

> an email to bareos...@googlegroups.com
> <mailto:bareos-users+unsub...@googlegroups.com>.

[Helmut Pable]

Hi Andreas,

I am unfamiliar to debug linux as I am a programming NET, but managed to install gdb and bareos debugging symbols, but now I need to start director under gdb. Can you please give the correct command to do so?

# gdb ... run ???? or what else ?

Thanx

Helmut

Am Mittwoch, 11. Dezember 2019 22:57:55 UTC+1 schrieb Andreas Rogge:

[Andreas Rogge]

Hi Helmut,

no need to do this manually.
Whenever a Bareos daemon crashes it will write a traceback using gdb
automatically. That traceback can be found in /var/lib/bareos/*.traceback

Best Regards,
Andreas

Am 12.12.19 um 10:49 schrieb Helmut Pable:

> Hi Andreas,
> I am unfamiliar to debug linux as I am a programming NET, but managed to
> install gdb and bareos debugging symbols, but now I need to start
> director under gdb. Can you please give the correct command to do so?
>
> # gdb ... run ???? or what else ?
>
> Thanx
> Helmut
>
> Am Mittwoch, 11. Dezember 2019 22:57:55 UTC+1 schrieb Andreas Rogge:
>
> Hi Helmut,
>
> please install the bareos debugging symbols (on fedora you can use
> debuginfo-install) and gdb.
> Afterwards the crashes should produce meaningful trace-files in
> /var/lib/bareos/.
>
> From there we can take a look what crashes and maybe why.
>
> Best Regards,
> Andreas
> --
>   Andreas Rogge                             andrea...@bareos.com

> <javascript:>

>   Bareos GmbH & Co. KG                      Phone: +49 221-630693-86
>   http://www.bareos.com
>
>   Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
>   Komplementär: Bareos Verwaltungs-GmbH
>   Geschäftsführer: S. Dühr, M. Außendorf, J. Steffens, Philipp Storz
>

> --
> You received this message because you are subscribed to the Google
> Groups "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to bareos-users...@googlegroups.com
> <mailto:bareos-users...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/bareos-users/c0f4e722-5bf2-4a24-b80d-258ae3d8e97f%40googlegroups.com
> <https://groups.google.com/d/msgid/bareos-users/c0f4e722-5bf2-4a24-b80d-258ae3d8e97f%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Helmut Pable]

Hi Andreas,

now I am confused a bit:

On my Ubuntu-Backup-Server (Ubuntu 16.04.6 LTS) I installed:

#:~# apt install gdb
... die neueste Version (7.11.1-0ubuntu1~16.5).
~# apt-get install bareos-dbg

... die neueste Version (18.2.5-147.2).

Then I did:

~# service bareos-director restart

Job for bareos-director.service failed because a fatal signal was delivered to the control process. See "systemctl status bareos-director.service" and "journalctl -xe" for details.

But in  /var/lib/bareos there were no files dated newer then date of director-restart (todays date):

# ls -l  /var/lib/bareos/*.traceback
-rw-r----- 1 bareos bareos 87 Mär  1  2019 /var/lib/bareos/bareos.1290.traceback
-rw-r----- 1 bareos bareos 87 Jul 18  2018 /var/lib/bareos/bareos.1302.traceback
-rw-r----- 1 bareos bareos 87 Jun 22  2018 /var/lib/bareos/bareos.17302.traceback
-rw-r----- 1 bareos bareos 87 Dez 12 00:06 /var/lib/bareos/bareos.2770.traceback
-rw-r----- 1 bareos bareos 87 Jul 10  2018 /var/lib/bareos/bareos.3022.traceback
-rw-r----- 1 bareos bareos 87 Nov  6  2018 /var/lib/bareos/bareos.931.traceback

(same result after a reboot).

Content of /var/lib/bareos/bareos.2770.traceback:

No debugger available on this platform,  please install one to get proper stack traces

Why?

Helmut

> an email to bareos...@googlegroups.com
> <mailto:bareos-users+unsub...@googlegroups.com>.

[Krisztian VASAS]

Hi,

i also have Ubuntu 16.04 LTS, the server is up2date. I also get the SEGV error, but only if i run "bareos-dir -t". After installing the bareos-dbg package i don't get the traceback file, but if i run the command with gdb i get this backtrace:

root@nas:/var/lib/bareos# gdb --args /usr/sbin/bareos-dir -t -f

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1

Copyright (C) 2016 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word"...

Reading symbols from /usr/sbin/bareos-dir...Reading symbols from /usr/lib/debug//usr/sbin/bareos-dir...(no debugging symbols found)...done.

(no debugging symbols found)...done.

(gdb) run

Starting program: /usr/sbin/bareos-dir -t -f

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.

0x00007ffff3bc4510 in ?? ()

(gdb) bt full

#0  0x00007ffff3bc4510 in ?? ()

No symbol table info available.

#1  0x00007ffff6a9d568 in ENGINE_remove (e=0x76c370) at eng_list.c:282

        to_return = 1

#2  0x00007ffff6a9d6b5 in engine_list_cleanup () at eng_list.c:92

        iterator = <optimized out>

#3  0x00007ffff6a9cc86 in engine_cleanup_cb_free (item=0x76c680) at eng_lib.c:198

No locals.

#4  0x00007ffff6aab921 in sk_pop_free (st=0x76c620, func=0x7ffff6a9cc80 <engine_cleanup_cb_free>) at stack.c:327

        i = 0

#5  0x00007ffff6a9cf8c in ENGINE_cleanup () at eng_lib.c:205

No locals.

#6  0x00007ffff6e471cb in CleanupCrypto() () from /usr/lib/bareos/libbareos.so.18

No symbol table info available.

#7  0x0000000000433ea7 in directordaemon::TerminateDird(int) ()

No symbol table info available.

#8  0x0000000000433b2b in main ()

No symbol table info available.

(gdb)

I hope this will help a bit. And this is a totally clean install (removed all bareos-related packages, cleaned all configs, removed the database). After installing the package i wanted to start, but i get the SEGV.

IroNiQ

[Helmut Pable]

Hi Andreas,

after having read the post of Krisztian Vasas (many thanks to him!), id tried to do the same as he did. The result is in the attached file - you certainly can see in it, that debugging this program is a book with seven seals for me.

What can I do now?

> an email to bareos...@googlegroups.com
> <mailto:bareos-users+unsub...@googlegroups.com>.

[Krisztian VASAS]

Some additional info: on a clean Ubuntu 18.04 this doesn't come. Only on 16.04. Something has happened there between mid of november and dec.2...

[Helmut Pable]

Hi Krisztian,

btw happy new year to all!

as I pointed out, my system was unchanged when this failure occurred.

Is anybody able to help me out debugging the cause of this failure ???

Many thanks in advance

Helmut