Groups keyboard shortcuts have been updated
Dismiss
See shortcuts
could not open catalog database
27 views
Skip to first unread message
Jon Schewe
Nov 27, 2024, 3:28:44 PM11/27/24
to bareos-users
In the last couple of days since upgrading bareos to the latest in the "current" repository I've found the director getting stuck talking to the backend database.
I'll execute "update slots" or "run" and then the console hangs and I get an error like this:
*run
Could not open catalog database "8^".
Could not open catalog database "8^".
The database name is some random set of characters. This kind of sounds like a memory bounds error.
Here's the version of bareos I'm running. Surprised that a "pre-release" is in "current" rather than "next" or "experimental".
*version
bareos-dir Version: 23.1.1~pre3.47aeb30a1 (21 November 2024) Red Hat Enterprise Linux release 9.5 (Plow) redhat Red Hat Enterprise Linux release 9.5 (Plow)
bareos-dir Version: 23.1.1~pre3.47aeb30a1 (21 November 2024) Red Hat Enterprise Linux release 9.5 (Plow) redhat Red Hat Enterprise Linux release 9.5 (Plow)
Watching /var/log/messages I see this when the job fails:
Nov 27 15:22:50 bareos bareos-dir[2021875]: Connect failure: ERR=error:0A0000C7:SSL routines::peer did not return a certificate
Nov 27 15:22:50 bareos bareos-dir[2021875]: lib/bnet.cc:125 TLS Negotiation failed.
Nov 27 15:22:50 bareos bareos-dir[2021875]: lib/bnet.cc:125 TLS Negotiation failed.
I do have all of my daemons setup to use TLS certificates and it's been working fine for a few months on this installation.
Disconnecting from bconsole and reconnecting and I see these errors:
27-Nov 15:18 bareos-dir JobId 0: Fatal error: cats/postgresql.cc:232 Unable to connect to PostgreSQL server. Database=8^ User=
Possible causes: SQL server not running; password incorrect; max_connections exceeded.
(connection pointer is NULL
)
27-Nov 15:22 bareos-dir JobId 0: Fatal error: cats/postgresql.cc:232 Unable to connect to PostgreSQL server. Database=8^ User=
Possible causes: SQL server not running; password incorrect; max_connections exceeded.
(connection pointer is NULL
)
Possible causes: SQL server not running; password incorrect; max_connections exceeded.
(connection pointer is NULL
)
27-Nov 15:22 bareos-dir JobId 0: Fatal error: cats/postgresql.cc:232 Unable to connect to PostgreSQL server. Database=8^ User=
Possible causes: SQL server not running; password incorrect; max_connections exceeded.
(connection pointer is NULL
)
The postgresql server is local and only used by bareos.
After restarting the console application things appear to be happier. I had left the console running for quite some time.
This morning when it happened bareos-dir crashed and gave me a traceback, shown below
Reading symbols from /usr/sbin/bareos-dir...
Reading symbols from .gnu_debugdata for /usr/sbin/bareos-dir...
(No debugging symbols found in .gnu_debugdata for /usr/sbin/bareos-dir)
Attaching to program: /usr/sbin/bareos-dir, process 1772827
[New LWP 1772830]
[New LWP 1772831]
[New LWP 1780188]
[New LWP 1799021]
[New LWP 1799033]
[New LWP 1803537]
[New LWP 1803538]
[New LWP 1803539]
[New LWP 1803540]
[New LWP 1803541]
[New LWP 1803542]
[New LWP 1803543]
[New LWP 1803544]
[New LWP 2018381]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install bareos-director-23.1.1~pre3.47aeb30a1-149.el9.x86_64
(gdb) 'my_name' has unknown type; cast it to its declared type
(gdb) 'exename' has unknown type; cast it to its declared type
(gdb) 'exepath' has unknown type; cast it to its declared type
(gdb) No symbol "catalog_db" in current context.
(gdb) 'kBareosVersionStrings' has unknown type; cast it to its declared type
(gdb) 'kBareosVersionStrings' has unknown type; cast it to its declared type
(gdb) 'host_name' has unknown type; cast it to its declared type
(gdb) Environment variable "TestName" not defined.
(gdb) #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x000055f9bacfb9fb in main ()
(gdb)
Thread 15 (Thread 0x7f06a2ffd640 (LWP 2018381) "bareos-dir"):
#0 0x00007f06e5301aff in poll () from /lib64/libc.so.6
#1 0x00007f06e596cdd9 in WaitForReadableFd(int, int, bool) () from /usr/lib64/bareos/libbareos.so.23
#2 0x00007f06e597c889 in TlsOpenSslPrivate::OpensslBsockReadwrite(BareosSocket*, char*, int, bool) () from /usr/lib64/bareos/libbareos.so.23
#3 0x00007f06e5902294 in BareosSocketTCP::recv() () from /usr/lib64/bareos/libbareos.so.23
#4 0x000055f9badaf070 in directordaemon::BgetDirmsg(BareosSocket*, bool) ()
#5 0x000055f9bad43f1f in msg_thread ()
#6 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#7 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 14 (Thread 0x7f06cbfff640 (LWP 1803544) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 13 (Thread 0x7f06e19f9640 (LWP 1803543) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 12 (Thread 0x7f06e29fb640 (LWP 1803542) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 11 (Thread 0x7f06e31fc640 (LWP 1803541) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 10 (Thread 0x7f06e09f7640 (LWP 1803540) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 9 (Thread 0x7f06c97fa640 (LWP 1803539) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 8 (Thread 0x7f06e11f8640 (LWP 1803538) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 7 (Thread 0x7f06caffd640 (LWP 1803537) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 6 (Thread 0x7f06ca7fc640 (LWP 1799033) "bareos-dir"):
#0 0x00007f06e52868ba in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1 0x00007f06e5289692 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6
#2 0x000055f9bad4a3e3 in directordaemon::WaitForStorageDaemonTermination(JobControlRecord*) ()
#3 0x000055f9bad9f2dd in directordaemon::DoNativeVbackup(JobControlRecord*) ()
#4 0x000055f9bad3ae7d in directordaemon::job_thread(void*) [clone .lto_priv.0] ()
#5 0x000055f9bad41e90 in jobq_server ()
#6 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#7 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 5 (Thread 0x7f06e21fa640 (LWP 1799021) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 4 (Thread 0x7f06e39fd640 (LWP 1780188) "bareos-dir"):
#0 0x00007f06e52d8b3f in wait4 () from /lib64/libc.so.6
#1 0x00007f06e597566b in SignalHandler () from /usr/lib64/bareos/libbareos.so.23
#2 <signal handler called>
#3 0x00007f06e5370c64 in __strnlen_evex () from /lib64/libc.so.6
#4 0x00007f06e58fdc52 in Bvsnprintf(char*, int, char const*, __va_list_tag*) () from /usr/lib64/bareos/libbareos.so.23
#5 0x00007f06e595beb3 in PoolMem::Bvsprintf(char const*, __va_list_tag*) () from /usr/lib64/bareos/libbareos.so.23
#6 0x000055f9bad77638 in directordaemon::UaContext::vSendMsg(int, char const*, char const*, __va_list_tag*) ()
#7 0x000055f9bad77e5a in directordaemon::UaContext::ErrorMsg(char const*, ...) ()
#8 0x000055f9bad6a6aa in directordaemon::OpenDb(directordaemon::UaContext*, bool) [clone .part.0] ()
#9 0x000055f9bad98846 in directordaemon::UpdateCmd(directordaemon::UaContext*, char const*) ()
#10 0x000055f9bad625df in directordaemon::Do_a_command(directordaemon::UaContext*) ()
#11 0x000055f9bad50ce2 in directordaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
#12 0x00007f06e59735a2 in WorkerThread(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>) () from /usr/lib64/bareos/libbareos.so.23
#13 0x00007f06e596c515 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition> > > >::_M_run() () from /usr/lib64/bareos/libbareos.so.23
#14 0x00007f06e56dbad4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#15 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#16 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 3 (Thread 0x7f06e41fe640 (LWP 1772831) "bareos-dir"):
#0 0x00007f06e5301aff in poll () from /lib64/libc.so.6
#1 0x00007f06e58fa598 in BnetThreadServerTcp(dlist<IPADDR>*, alist<s_sockfd*>*, ThreadList&, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, std::atomic<BnetServerState>*, std::function<void* (void*)>, std::function<void ()>) () from /usr/lib64/bareos/libbareos.so.23
#2 0x000055f9bad51fdf in connect_thread ()
#3 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#4 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 2 (Thread 0x7f06e49ff640 (LWP 1772830) "bareos-dir"):
#0 0x00007f06e52868ba in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1 0x00007f06e52893c4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6
#2 0x00007f06e59862c0 in watchdog_thread () from /usr/lib64/bareos/libbareos.so.23
#3 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#4 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 1 (Thread 0x7f06e4e69880 (LWP 1772827) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x000055f9bacfb9fb in main ()
(gdb) #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
(gdb) No symbol table info available.
(gdb) #1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
(gdb) No symbol table info available.
(gdb) #2 0x000055f9bacfb9fb in main ()
(gdb) No symbol table info available.
(gdb) No frame at level 3.
(gdb) No symbol table info available.
(gdb) No frame at level 4.
(gdb) No symbol table info available.
(gdb) No frame at level 5.
(gdb) No symbol table info available.
(gdb) No frame at level 6.
(gdb) No symbol table info available.
(gdb) No frame at level 7.
(gdb) No symbol table info available.
(gdb) Detaching from program: /usr/sbin/bareos-dir, process 1772827
[Inferior 1 (process 1772827) detached]
(gdb)
Reading symbols from .gnu_debugdata for /usr/sbin/bareos-dir...
(No debugging symbols found in .gnu_debugdata for /usr/sbin/bareos-dir)
Attaching to program: /usr/sbin/bareos-dir, process 1772827
[New LWP 1772830]
[New LWP 1772831]
[New LWP 1780188]
[New LWP 1799021]
[New LWP 1799033]
[New LWP 1803537]
[New LWP 1803538]
[New LWP 1803539]
[New LWP 1803540]
[New LWP 1803541]
[New LWP 1803542]
[New LWP 1803543]
[New LWP 1803544]
[New LWP 2018381]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install bareos-director-23.1.1~pre3.47aeb30a1-149.el9.x86_64
(gdb) 'my_name' has unknown type; cast it to its declared type
(gdb) 'exename' has unknown type; cast it to its declared type
(gdb) 'exepath' has unknown type; cast it to its declared type
(gdb) No symbol "catalog_db" in current context.
(gdb) 'kBareosVersionStrings' has unknown type; cast it to its declared type
(gdb) 'kBareosVersionStrings' has unknown type; cast it to its declared type
(gdb) 'host_name' has unknown type; cast it to its declared type
(gdb) Environment variable "TestName" not defined.
(gdb) #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x000055f9bacfb9fb in main ()
(gdb)
Thread 15 (Thread 0x7f06a2ffd640 (LWP 2018381) "bareos-dir"):
#0 0x00007f06e5301aff in poll () from /lib64/libc.so.6
#1 0x00007f06e596cdd9 in WaitForReadableFd(int, int, bool) () from /usr/lib64/bareos/libbareos.so.23
#2 0x00007f06e597c889 in TlsOpenSslPrivate::OpensslBsockReadwrite(BareosSocket*, char*, int, bool) () from /usr/lib64/bareos/libbareos.so.23
#3 0x00007f06e5902294 in BareosSocketTCP::recv() () from /usr/lib64/bareos/libbareos.so.23
#4 0x000055f9badaf070 in directordaemon::BgetDirmsg(BareosSocket*, bool) ()
#5 0x000055f9bad43f1f in msg_thread ()
#6 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#7 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 14 (Thread 0x7f06cbfff640 (LWP 1803544) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 13 (Thread 0x7f06e19f9640 (LWP 1803543) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 12 (Thread 0x7f06e29fb640 (LWP 1803542) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 11 (Thread 0x7f06e31fc640 (LWP 1803541) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 10 (Thread 0x7f06e09f7640 (LWP 1803540) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 9 (Thread 0x7f06c97fa640 (LWP 1803539) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 8 (Thread 0x7f06e11f8640 (LWP 1803538) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 7 (Thread 0x7f06caffd640 (LWP 1803537) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 6 (Thread 0x7f06ca7fc640 (LWP 1799033) "bareos-dir"):
#0 0x00007f06e52868ba in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1 0x00007f06e5289692 in pthread_cond_clockwait@GLIBC_2.30 () from /lib64/libc.so.6
#2 0x000055f9bad4a3e3 in directordaemon::WaitForStorageDaemonTermination(JobControlRecord*) ()
#3 0x000055f9bad9f2dd in directordaemon::DoNativeVbackup(JobControlRecord*) ()
#4 0x000055f9bad3ae7d in directordaemon::job_thread(void*) [clone .lto_priv.0] ()
#5 0x000055f9bad41e90 in jobq_server ()
#6 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#7 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 5 (Thread 0x7f06e21fa640 (LWP 1799021) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007f06e59045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055f9bad42803 in jobq_server ()
#4 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#5 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 4 (Thread 0x7f06e39fd640 (LWP 1780188) "bareos-dir"):
#0 0x00007f06e52d8b3f in wait4 () from /lib64/libc.so.6
#1 0x00007f06e597566b in SignalHandler () from /usr/lib64/bareos/libbareos.so.23
#2 <signal handler called>
#3 0x00007f06e5370c64 in __strnlen_evex () from /lib64/libc.so.6
#4 0x00007f06e58fdc52 in Bvsnprintf(char*, int, char const*, __va_list_tag*) () from /usr/lib64/bareos/libbareos.so.23
#5 0x00007f06e595beb3 in PoolMem::Bvsprintf(char const*, __va_list_tag*) () from /usr/lib64/bareos/libbareos.so.23
#6 0x000055f9bad77638 in directordaemon::UaContext::vSendMsg(int, char const*, char const*, __va_list_tag*) ()
#7 0x000055f9bad77e5a in directordaemon::UaContext::ErrorMsg(char const*, ...) ()
#8 0x000055f9bad6a6aa in directordaemon::OpenDb(directordaemon::UaContext*, bool) [clone .part.0] ()
#9 0x000055f9bad98846 in directordaemon::UpdateCmd(directordaemon::UaContext*, char const*) ()
#10 0x000055f9bad625df in directordaemon::Do_a_command(directordaemon::UaContext*) ()
#11 0x000055f9bad50ce2 in directordaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
#12 0x00007f06e59735a2 in WorkerThread(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>) () from /usr/lib64/bareos/libbareos.so.23
#13 0x00007f06e596c515 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition> > > >::_M_run() () from /usr/lib64/bareos/libbareos.so.23
#14 0x00007f06e56dbad4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#15 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#16 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 3 (Thread 0x7f06e41fe640 (LWP 1772831) "bareos-dir"):
#0 0x00007f06e5301aff in poll () from /lib64/libc.so.6
#1 0x00007f06e58fa598 in BnetThreadServerTcp(dlist<IPADDR>*, alist<s_sockfd*>*, ThreadList&, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, std::atomic<BnetServerState>*, std::function<void* (void*)>, std::function<void ()>) () from /usr/lib64/bareos/libbareos.so.23
#2 0x000055f9bad51fdf in connect_thread ()
#3 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#4 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 2 (Thread 0x7f06e49ff640 (LWP 1772830) "bareos-dir"):
#0 0x00007f06e52868ba in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1 0x00007f06e52893c4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6
#2 0x00007f06e59862c0 in watchdog_thread () from /usr/lib64/bareos/libbareos.so.23
#3 0x00007f06e5289d22 in start_thread () from /lib64/libc.so.6
#4 0x00007f06e530ed40 in clone3 () from /lib64/libc.so.6
Thread 1 (Thread 0x7f06e4e69880 (LWP 1772827) "bareos-dir"):
#0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x000055f9bacfb9fb in main ()
(gdb) #0 0x00007f06e52d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
(gdb) No symbol table info available.
(gdb) #1 0x00007f06e52d8d87 in nanosleep () from /lib64/libc.so.6
(gdb) No symbol table info available.
(gdb) #2 0x000055f9bacfb9fb in main ()
(gdb) No symbol table info available.
(gdb) No frame at level 3.
(gdb) No symbol table info available.
(gdb) No frame at level 4.
(gdb) No symbol table info available.
(gdb) No frame at level 5.
(gdb) No symbol table info available.
(gdb) No frame at level 6.
(gdb) No symbol table info available.
(gdb) No frame at level 7.
(gdb) No symbol table info available.
(gdb) Detaching from program: /usr/sbin/bareos-dir, process 1772827
[Inferior 1 (process 1772827) detached]
(gdb)
Jon Schewe
Nov 27, 2024, 5:04:45 PM11/27/24
to bareos-users
Just got another segementation fault. This time I had just started bconsole and queried the director for status.
In both cases I see that the thread in the signal handler is in the function __strnlen_evex
Reading symbols from /usr/sbin/bareos-dir...
Reading symbols from .gnu_debugdata for /usr/sbin/bareos-dir...
(No debugging symbols found in .gnu_debugdata for /usr/sbin/bareos-dir)
Reading symbols from .gnu_debugdata for /usr/sbin/bareos-dir...
(No debugging symbols found in .gnu_debugdata for /usr/sbin/bareos-dir)
Attaching to program: /usr/sbin/bareos-dir, process 2021875
[New LWP 2021881]
[New LWP 2021882]
[New LWP 2023997]
[New LWP 2024634]
[New LWP 2024637]
[New LWP 2024640]
[New LWP 2024645]
[New LWP 2028750]
[New LWP 2034391]
[New LWP 2035955]
[New LWP 2041800]
[New LWP 2043688]
[New LWP 2043940]
[New LWP 2043952]
[New LWP 2021881]
[New LWP 2021882]
[New LWP 2023997]
[New LWP 2024634]
[New LWP 2024637]
[New LWP 2024640]
[New LWP 2024645]
[New LWP 2028750]
[New LWP 2034391]
[New LWP 2035955]
[New LWP 2041800]
[New LWP 2043688]
[New LWP 2043940]
[New LWP 2043952]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install bareos-director-23.1.1~pre3.47aeb30a1-149.el9.x86_64
(gdb) 'my_name' has unknown type; cast it to its declared type
(gdb) 'exename' has unknown type; cast it to its declared type
(gdb) 'exepath' has unknown type; cast it to its declared type
(gdb) No symbol "catalog_db" in current context.
(gdb) 'kBareosVersionStrings' has unknown type; cast it to its declared type
(gdb) 'kBareosVersionStrings' has unknown type; cast it to its declared type
(gdb) 'host_name' has unknown type; cast it to its declared type
(gdb) Environment variable "TestName" not defined.
(gdb) #0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x000055964eede9fb in main ()
(gdb)
Thread 15 (Thread 0x7fa39a7fc640 (LWP 2043952) "bareos-dir"):
#0 0x00007fa3bb101aff in poll () from /lib64/libc.so.6
#1 0x00007fa3bb76cdd9 in WaitForReadableFd(int, int, bool) () from /usr/lib64/bareos/libbareos.so.23
#2 0x00007fa3bb77c889 in TlsOpenSslPrivate::OpensslBsockReadwrite(BareosSocket*, char*, int, bool) () from /usr/lib64/bareos/libbareos.so.23
#3 0x00007fa3bb702294 in BareosSocketTCP::recv() () from /usr/lib64/bareos/libbareos.so.23
#4 0x000055964ef92070 in directordaemon::BgetDirmsg(BareosSocket*, bool) ()
#5 0x000055964ef26f1f in msg_thread ()
#6 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#7 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 14 (Thread 0x7fa3a97fa640 (LWP 2043940) "bareos-dir"):
#0 0x00007fa3bb101aff in poll () from /lib64/libc.so.6
#1 0x00007fa3bb76cdd9 in WaitForReadableFd(int, int, bool) () from /usr/lib64/bareos/libbareos.so.23
#2 0x00007fa3bb77c889 in TlsOpenSslPrivate::OpensslBsockReadwrite(BareosSocket*, char*, int, bool) () from /usr/lib64/bareos/libbareos.so.23
#3 0x00007fa3bb702294 in BareosSocketTCP::recv() () from /usr/lib64/bareos/libbareos.so.23
#4 0x000055964ef92070 in directordaemon::BgetDirmsg(BareosSocket*, bool) ()
#5 0x000055964ef88f9e in directordaemon::WaitForJobTermination(JobControlRecord*, int) ()
#6 0x000055964ef1e174 in directordaemon::job_thread(void*) [clone .lto_priv.0] ()
#7 0x000055964ef24e90 in jobq_server ()
#8 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#9 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 13 (Thread 0x7fa3b97fd640 (LWP 2043688) "bareos-dir"):
#0 0x00007fa3bb0d8b3f in wait4 () from /lib64/libc.so.6
#1 0x00007fa3bb77566b in SignalHandler () from /usr/lib64/bareos/libbareos.so.23
#2 <signal handler called>
#3 0x00007fa3bb16e37c in __strlen_evex () from /lib64/libc.so.6
#4 0x000055964ef70009 in directordaemon::ListRunningJobs(directordaemon::UaContext*) ()
#5 0x000055964ef775ff in directordaemon::StatusCmd(directordaemon::UaContext*, char const*) ()
#6 0x000055964ef455df in directordaemon::Do_a_command(directordaemon::UaContext*) ()
#7 0x000055964ef33ce2 in directordaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
#8 0x00007fa3bb7735a2 in WorkerThread(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>) () from /usr/lib64/bareos/libbareos.so.23
#9 0x00007fa3bb76c515 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition> > > >::_M_run() () from /usr/lib64/bareos/libbareos.so.23
#10 0x00007fa3bb4dbad4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#11 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#12 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 12 (Thread 0x7fa3aaffd640 (LWP 2041800) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 11 (Thread 0x7fa39b7fe640 (LWP 2035955) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 10 (Thread 0x7fa39bfff640 (LWP 2034391) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 9 (Thread 0x7fa3abfff640 (LWP 2028750) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 8 (Thread 0x7fa3b8ffc640 (LWP 2024645) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 7 (Thread 0x7fa3ab7fe640 (LWP 2024640) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 6 (Thread 0x7fa3aa7fc640 (LWP 2024637) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 5 (Thread 0x7fa3a9ffb640 (LWP 2024634) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 4 (Thread 0x7fa3a8ff9640 (LWP 2023997) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 3 (Thread 0x7fa3b9ffe640 (LWP 2021882) "bareos-dir"):
#0 0x00007fa3bb101aff in poll () from /lib64/libc.so.6
#1 0x00007fa3bb6fa598 in BnetThreadServerTcp(dlist<IPADDR>*, alist<s_sockfd*>*, ThreadList&, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, std::atomic<BnetServerState>*, std::function<void* (void*)>, std::function<void ()>) () from /usr/lib64/bareos/libbareos.so.23
#2 0x000055964ef34fdf in connect_thread ()
#3 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#4 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 2 (Thread 0x7fa3ba7ff640 (LWP 2021881) "bareos-dir"):
#0 0x00007fa3bb0868ba in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1 0x00007fa3bb0893c4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6
#2 0x00007fa3bb7862c0 in watchdog_thread () from /usr/lib64/bareos/libbareos.so.23
#3 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#4 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 1 (Thread 0x7fa3bb208880 (LWP 2021875) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x000055964eede9fb in main ()
(gdb) #0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x000055964eede9fb in main ()
(gdb)
Thread 15 (Thread 0x7fa39a7fc640 (LWP 2043952) "bareos-dir"):
#0 0x00007fa3bb101aff in poll () from /lib64/libc.so.6
#1 0x00007fa3bb76cdd9 in WaitForReadableFd(int, int, bool) () from /usr/lib64/bareos/libbareos.so.23
#2 0x00007fa3bb77c889 in TlsOpenSslPrivate::OpensslBsockReadwrite(BareosSocket*, char*, int, bool) () from /usr/lib64/bareos/libbareos.so.23
#3 0x00007fa3bb702294 in BareosSocketTCP::recv() () from /usr/lib64/bareos/libbareos.so.23
#4 0x000055964ef92070 in directordaemon::BgetDirmsg(BareosSocket*, bool) ()
#5 0x000055964ef26f1f in msg_thread ()
#6 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#7 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 14 (Thread 0x7fa3a97fa640 (LWP 2043940) "bareos-dir"):
#0 0x00007fa3bb101aff in poll () from /lib64/libc.so.6
#1 0x00007fa3bb76cdd9 in WaitForReadableFd(int, int, bool) () from /usr/lib64/bareos/libbareos.so.23
#2 0x00007fa3bb77c889 in TlsOpenSslPrivate::OpensslBsockReadwrite(BareosSocket*, char*, int, bool) () from /usr/lib64/bareos/libbareos.so.23
#3 0x00007fa3bb702294 in BareosSocketTCP::recv() () from /usr/lib64/bareos/libbareos.so.23
#4 0x000055964ef92070 in directordaemon::BgetDirmsg(BareosSocket*, bool) ()
#5 0x000055964ef88f9e in directordaemon::WaitForJobTermination(JobControlRecord*, int) ()
#6 0x000055964ef1e174 in directordaemon::job_thread(void*) [clone .lto_priv.0] ()
#7 0x000055964ef24e90 in jobq_server ()
#8 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#9 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 13 (Thread 0x7fa3b97fd640 (LWP 2043688) "bareos-dir"):
#0 0x00007fa3bb0d8b3f in wait4 () from /lib64/libc.so.6
#1 0x00007fa3bb77566b in SignalHandler () from /usr/lib64/bareos/libbareos.so.23
#2 <signal handler called>
#3 0x00007fa3bb16e37c in __strlen_evex () from /lib64/libc.so.6
#4 0x000055964ef70009 in directordaemon::ListRunningJobs(directordaemon::UaContext*) ()
#5 0x000055964ef775ff in directordaemon::StatusCmd(directordaemon::UaContext*, char const*) ()
#6 0x000055964ef455df in directordaemon::Do_a_command(directordaemon::UaContext*) ()
#7 0x000055964ef33ce2 in directordaemon::HandleConnectionRequest(ConfigurationParser*, void*) ()
#8 0x00007fa3bb7735a2 in WorkerThread(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>) () from /usr/lib64/bareos/libbareos.so.23
#9 0x00007fa3bb76c515 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)> const&, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition>), std::shared_ptr<ThreadListContainer>, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, void*, std::shared_ptr<IsRunningCondition> > > >::_M_run() () from /usr/lib64/bareos/libbareos.so.23
#10 0x00007fa3bb4dbad4 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#11 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#12 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 12 (Thread 0x7fa3aaffd640 (LWP 2041800) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 11 (Thread 0x7fa39b7fe640 (LWP 2035955) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 10 (Thread 0x7fa39bfff640 (LWP 2034391) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 9 (Thread 0x7fa3abfff640 (LWP 2028750) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 8 (Thread 0x7fa3b8ffc640 (LWP 2024645) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 7 (Thread 0x7fa3ab7fe640 (LWP 2024640) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 6 (Thread 0x7fa3aa7fc640 (LWP 2024637) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 5 (Thread 0x7fa3a9ffb640 (LWP 2024634) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 4 (Thread 0x7fa3a8ff9640 (LWP 2023997) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x00007fa3bb7045fa in Bmicrosleep(int, int) () from /usr/lib64/bareos/libbareos.so.23
#3 0x000055964ef25803 in jobq_server ()
#4 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#5 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 3 (Thread 0x7fa3b9ffe640 (LWP 2021882) "bareos-dir"):
#0 0x00007fa3bb101aff in poll () from /lib64/libc.so.6
#1 0x00007fa3bb6fa598 in BnetThreadServerTcp(dlist<IPADDR>*, alist<s_sockfd*>*, ThreadList&, std::function<void* (ConfigurationParser*, void*)>, ConfigurationParser*, std::atomic<BnetServerState>*, std::function<void* (void*)>, std::function<void ()>) () from /usr/lib64/bareos/libbareos.so.23
#2 0x000055964ef34fdf in connect_thread ()
#3 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#4 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 2 (Thread 0x7fa3ba7ff640 (LWP 2021881) "bareos-dir"):
#0 0x00007fa3bb0868ba in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1 0x00007fa3bb0893c4 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libc.so.6
#2 0x00007fa3bb7862c0 in watchdog_thread () from /usr/lib64/bareos/libbareos.so.23
#3 0x00007fa3bb089d22 in start_thread () from /lib64/libc.so.6
#4 0x00007fa3bb10ed40 in clone3 () from /lib64/libc.so.6
Thread 1 (Thread 0x7fa3bb208880 (LWP 2021875) "bareos-dir"):
#0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
#2 0x000055964eede9fb in main ()
(gdb) #0 0x00007fa3bb0d4175 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
(gdb) No symbol table info available.
(gdb) #1 0x00007fa3bb0d8d87 in nanosleep () from /lib64/libc.so.6
(gdb) No symbol table info available.
(gdb) #2 0x000055964eede9fb in main ()
(gdb) No symbol table info available.
(gdb) No frame at level 3.
(gdb) No symbol table info available.
(gdb) No frame at level 4.
(gdb) No symbol table info available.
(gdb) No frame at level 5.
(gdb) No symbol table info available.
(gdb) No frame at level 6.
(gdb) No symbol table info available.
(gdb) No frame at level 7.
(gdb) No symbol table info available.
(gdb) Detaching from program: /usr/sbin/bareos-dir, process 2021875
[Inferior 1 (process 2021875) detached]
(gdb)
[Inferior 1 (process 2021875) detached]
(gdb)
Here is the bactrace that was dumped in /var/lib/bareos
Attempt to dump current JCRs. njcrs=5
threadid=0x00007fa3bb208880 killable=0 JobId=0 JobStatus=R jcr=0x55964f0cb0b0 name=*JobMonitor*.2024-11-27_07.40.49_01
UseCount=1
JobType=I JobLevel=
sched_time=27-Nov-2024 07:40 start_time=27-Nov-2024 07:40
end_time=31-Dec-1969 19:00 wait_time=31-Dec-1969 19:00
db=(nil) db_batch=(nil) batch_started=0
threadid=0x00007fa3bb208880 killable=0 JobId=972 JobStatus=p jcr=0x55964f0ce310 name=BackupCatalog.2024-11-27_12.10.00_30
UseCount=1
JobType=B JobLevel=F
sched_time=27-Nov-2024 12:10 start_time=27-Nov-2024 12:10
end_time=31-Dec-1969 19:00 wait_time=27-Nov-2024 16:39
db=0x7fa3ac057f40 db_batch=(nil) batch_started=0
BareosDb=0x7fa3ac057f40 db_name=bareos db_user=bareos connected=true
cmd="SELECT MediaId,VolumeName,VolJobs,VolFiles,VolBlocks,VolBytes,VolMounts,VolErrors,VolWrites,MaxVolBytes,VolCapacityBytes,MediaType,VolStatus,PoolId,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,Recycle,Slot,FirstWritten,LastWritten,InChanger,EndFile,EndBlock,LabelType,LabelDate,StorageId,Enabled,LocationId,RecycleCount,InitialWrite,ScratchPoolId,RecyclePoolId,VolReadTime,VolWriteTime,ActionOnPurge,EncryptionKey,MinBlocksize,MaxBlocksize FROM Media WHERE PoolId=10 AND MediaType='File' AND Enabled=1 AND VolStatus='Purged' AND Recycle=1 ORDER BY LastWritten ASC,MediaId LIMIT 1" changes=947
RWLOCK=0x7fa3ac057f48 w_active=0 w_wait=0
threadid=0x0000000000000000 killable=0 JobId=973 JobStatus=R jcr=0x7fa3ac219a70 name=ottertail.bbn.com-onsite.2024-11-27_15.29.56_33
UseCount=2
JobType=B JobLevel=F
sched_time=27-Nov-2024 15:29 start_time=27-Nov-2024 16:39
end_time=31-Dec-1969 19:00 wait_time=31-Dec-1969 19:00
db=0x7fa3ac057f40 db_batch=0x7fa39c01a480 batch_started=1
BareosDb=0x7fa3ac057f40 db_name=bareos db_user=bareos connected=true
cmd="SELECT MediaId,VolumeName,VolJobs,VolFiles,VolBlocks,VolBytes,VolMounts,VolErrors,VolWrites,MaxVolBytes,VolCapacityBytes,MediaType,VolStatus,PoolId,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,Recycle,Slot,FirstWritten,LastWritten,InChanger,EndFile,EndBlock,LabelType,LabelDate,StorageId,Enabled,LocationId,RecycleCount,InitialWrite,ScratchPoolId,RecyclePoolId,VolReadTime,VolWriteTime,ActionOnPurge,EncryptionKey,MinBlocksize,MaxBlocksize FROM Media WHERE PoolId=10 AND MediaType='File' AND Enabled=1 AND VolStatus='Purged' AND Recycle=1 ORDER BY LastWritten ASC,MediaId LIMIT 1" changes=947
RWLOCK=0x7fa3ac057f48 w_active=0 w_wait=0
threadid=0x00007fa3b97fd640 killable=0 JobId=0 JobStatus=C jcr=0x7fa3ac1fa210 name=-Console-.2024-11-27_16.31.02_34
UseCount=1
JobType=U JobLevel=F
sched_time=27-Nov-2024 16:31 start_time=27-Nov-2024 16:31
end_time=31-Dec-1969 19:00 wait_time=31-Dec-1969 19:00
db=(nil) db_batch=(nil) batch_started=0
threadid=0x00007fa39bfff640 killable=0 JobId=974 JobStatus=p jcr=0x7fa39403f9b0 name=BackupCatalog-offsite.2024-11-27_16.39.13_39
UseCount=2
JobType=B JobLevel=F
sched_time=27-Nov-2024 16:40 start_time=27-Nov-2024 16:39
end_time=31-Dec-1969 19:00 wait_time=27-Nov-2024 16:40
db=0x7fa3ac057f40 db_batch=(nil) batch_started=0
BareosDb=0x7fa3ac057f40 db_name=bareos db_user=bareos connected=true
cmd="SELECT MediaId,VolumeName,VolJobs,VolFiles,VolBlocks,VolBytes,VolMounts,VolErrors,VolWrites,MaxVolBytes,VolCapacityBytes,MediaType,VolStatus,PoolId,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,Recycle,Slot,FirstWritten,LastWritten,InChanger,EndFile,EndBlock,LabelType,LabelDate,StorageId,Enabled,LocationId,RecycleCount,InitialWrite,ScratchPoolId,RecyclePoolId,VolReadTime,VolWriteTime,ActionOnPurge,EncryptionKey,MinBlocksize,MaxBlocksize FROM Media WHERE PoolId=10 AND MediaType='File' AND Enabled=1 AND VolStatus='Purged' AND Recycle=1 ORDER BY LastWritten ASC,MediaId LIMIT 1" changes=947
RWLOCK=0x7fa3ac057f48 w_active=0 w_wait=0
dumping of jcrs finished. number of dumped = 5
threadid=0x00007fa3bb208880 killable=0 JobId=0 JobStatus=R jcr=0x55964f0cb0b0 name=*JobMonitor*.2024-11-27_07.40.49_01
UseCount=1
JobType=I JobLevel=
sched_time=27-Nov-2024 07:40 start_time=27-Nov-2024 07:40
end_time=31-Dec-1969 19:00 wait_time=31-Dec-1969 19:00
db=(nil) db_batch=(nil) batch_started=0
threadid=0x00007fa3bb208880 killable=0 JobId=972 JobStatus=p jcr=0x55964f0ce310 name=BackupCatalog.2024-11-27_12.10.00_30
UseCount=1
JobType=B JobLevel=F
sched_time=27-Nov-2024 12:10 start_time=27-Nov-2024 12:10
end_time=31-Dec-1969 19:00 wait_time=27-Nov-2024 16:39
db=0x7fa3ac057f40 db_batch=(nil) batch_started=0
BareosDb=0x7fa3ac057f40 db_name=bareos db_user=bareos connected=true
cmd="SELECT MediaId,VolumeName,VolJobs,VolFiles,VolBlocks,VolBytes,VolMounts,VolErrors,VolWrites,MaxVolBytes,VolCapacityBytes,MediaType,VolStatus,PoolId,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,Recycle,Slot,FirstWritten,LastWritten,InChanger,EndFile,EndBlock,LabelType,LabelDate,StorageId,Enabled,LocationId,RecycleCount,InitialWrite,ScratchPoolId,RecyclePoolId,VolReadTime,VolWriteTime,ActionOnPurge,EncryptionKey,MinBlocksize,MaxBlocksize FROM Media WHERE PoolId=10 AND MediaType='File' AND Enabled=1 AND VolStatus='Purged' AND Recycle=1 ORDER BY LastWritten ASC,MediaId LIMIT 1" changes=947
RWLOCK=0x7fa3ac057f48 w_active=0 w_wait=0
threadid=0x0000000000000000 killable=0 JobId=973 JobStatus=R jcr=0x7fa3ac219a70 name=ottertail.bbn.com-onsite.2024-11-27_15.29.56_33
UseCount=2
JobType=B JobLevel=F
sched_time=27-Nov-2024 15:29 start_time=27-Nov-2024 16:39
end_time=31-Dec-1969 19:00 wait_time=31-Dec-1969 19:00
db=0x7fa3ac057f40 db_batch=0x7fa39c01a480 batch_started=1
BareosDb=0x7fa3ac057f40 db_name=bareos db_user=bareos connected=true
cmd="SELECT MediaId,VolumeName,VolJobs,VolFiles,VolBlocks,VolBytes,VolMounts,VolErrors,VolWrites,MaxVolBytes,VolCapacityBytes,MediaType,VolStatus,PoolId,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,Recycle,Slot,FirstWritten,LastWritten,InChanger,EndFile,EndBlock,LabelType,LabelDate,StorageId,Enabled,LocationId,RecycleCount,InitialWrite,ScratchPoolId,RecyclePoolId,VolReadTime,VolWriteTime,ActionOnPurge,EncryptionKey,MinBlocksize,MaxBlocksize FROM Media WHERE PoolId=10 AND MediaType='File' AND Enabled=1 AND VolStatus='Purged' AND Recycle=1 ORDER BY LastWritten ASC,MediaId LIMIT 1" changes=947
RWLOCK=0x7fa3ac057f48 w_active=0 w_wait=0
threadid=0x00007fa3b97fd640 killable=0 JobId=0 JobStatus=C jcr=0x7fa3ac1fa210 name=-Console-.2024-11-27_16.31.02_34
UseCount=1
JobType=U JobLevel=F
sched_time=27-Nov-2024 16:31 start_time=27-Nov-2024 16:31
end_time=31-Dec-1969 19:00 wait_time=31-Dec-1969 19:00
db=(nil) db_batch=(nil) batch_started=0
threadid=0x00007fa39bfff640 killable=0 JobId=974 JobStatus=p jcr=0x7fa39403f9b0 name=BackupCatalog-offsite.2024-11-27_16.39.13_39
UseCount=2
JobType=B JobLevel=F
sched_time=27-Nov-2024 16:40 start_time=27-Nov-2024 16:39
end_time=31-Dec-1969 19:00 wait_time=27-Nov-2024 16:40
db=0x7fa3ac057f40 db_batch=(nil) batch_started=0
BareosDb=0x7fa3ac057f40 db_name=bareos db_user=bareos connected=true
cmd="SELECT MediaId,VolumeName,VolJobs,VolFiles,VolBlocks,VolBytes,VolMounts,VolErrors,VolWrites,MaxVolBytes,VolCapacityBytes,MediaType,VolStatus,PoolId,VolRetention,VolUseDuration,MaxVolJobs,MaxVolFiles,Recycle,Slot,FirstWritten,LastWritten,InChanger,EndFile,EndBlock,LabelType,LabelDate,StorageId,Enabled,LocationId,RecycleCount,InitialWrite,ScratchPoolId,RecyclePoolId,VolReadTime,VolWriteTime,ActionOnPurge,EncryptionKey,MinBlocksize,MaxBlocksize FROM Media WHERE PoolId=10 AND MediaType='File' AND Enabled=1 AND VolStatus='Purged' AND Recycle=1 ORDER BY LastWritten ASC,MediaId LIMIT 1" changes=947
RWLOCK=0x7fa3ac057f48 w_active=0 w_wait=0
dumping of jcrs finished. number of dumped = 5
Sebastian Sura
Nov 28, 2024, 4:53:46 AM11/28/24
to bareos...@googlegroups.com
Hi Jon,
thanks for the report. This looks very weird. Could you install our debug packages ? This way the trace backs will contain some addition information which might prove useful.
When you use the `--xc` option to dump your (director) config, does it look correct to you ?
Kind Regards
Sebastian Sura
Am 27.11.24 um 23:04 schrieb Jon
Schewe:
--
You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bareos-users/c3aaf621-123f-49c8-babb-790981bd7948n%40googlegroups.com.
-- Sebastian Sura sebasti...@bareos.com Bareos GmbH & Co. KG Phone: +49 221 630693-0 https://www.bareos.com Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646 Komplementär: Bareos Verwaltungs-GmbH Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz
Jon Schewe
Dec 3, 2024, 11:05:50 AM12/3/24
to bareos-users
Sebastian,
The director configuration appears correct. There are a lot of jobs, so it's hard to be sure they are all correct, but a scan of the jobs looks sane.
As far as the debug packages, do I only need the debug package for the director?
I believe the package I want is "bareos-director-debuginfo". I am using the community current EL9 repository.
Sebastian Sura
Dec 9, 2024, 3:48:17 AM12/9/24
to Jon Schewe, bareos-users
Hi Jon,
iirc you would also need to install install the `bareos-debuginfo` and `bareos-common-debuginfo` packages.
Kind Regards,
Sebastian Sura
Am 03.12.24 um 17:05 schrieb Jon
Schewe:
To view this discussion visit https://groups.google.com/d/msgid/bareos-users/57545a91-ed05-4202-b247-599f1abbc2d5n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages