tac_plus-ng issue. SIGSEGV

[Petr Issakov]

Hi Marc!

In log of tac_plus-ng service I notices periodically messages:

 GDB: backtrace start
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: GDB: running: "(printf 'bt
                                                       q
                                                       ';sleep 3)|gdb -n -q -p $CRASHPID 2>/dev/null"
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: Thank you.
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: the author, unless you have a prior permit for that.
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: Please do NOT mail bug reports to the private mail address of
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: and include the backtraces.
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]:     https://github.com/MarcJHuber/event-driven-servers/issues
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: a bug report via GitHub issues at
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: Reconfigure with --debug, recompile and reinstall. Then send
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: If the issue persists even with the most recent version:
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: still seeing the crash. After all, this bug may be already fixed.
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: please download, compile, install and check if you're
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]:     GIT info is unavailable. This build might be outdated. Please make sure to update to the latest>
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: If the version on that site doesn't equal
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]:     http://www.pro-bono-publico.de/projects/
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: you are running the most current version, downloadable from
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: Apologies ... this shouldn't have happened. Please verify that
Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: Catched SIGSEGV

I update my instance to latest commit, but errors still exists

Now I try to localize issue and compile latest commit with --debug option:

./configure --with-pam --with-pcre2 --debug tac_plus-ng

Which debug parameters I should use to catch error?

I affraid that debug all dramatically decrease system performance

[Marc Huber]

Hi Petr,

thanks for reporting this!

Yes, building with --debug is the right thing to do. That drops compiler
optimizations and adds more  debugging info.

If gdb is installed, the built-in "(printf...|gdb..." command should
then log a somewhat detailed backtrace. On Linux, a prior

    echo 0 > /proc/sys/kernel/yama/ptrace_scope

will be required to enable the debugger to attach to the crashed process.

you can use a different debugger, too, by specifying a custom debug
command. Default is

   debug-cmd = "(printf 'bt

q
';sleep 3)|gdb -n -q -p $CRASHPID 2>/dev/null"

Alternatively, you can easily run a debugger (gdb, lldb) via the CLI,
e.g. "lldb -p <pid>", then "cont" and wait until the SEGV occurs.

Thanks,

Marc

On 19.07.2024 09:45, Petr Issakov wrote:
> Hi Marc!
> In log of tac_plus-ng service I notices periodically messages:
>

> / GDB: backtrace start

> Jul 19 09:44:48 t2ru-tacacs-vm-01 tac_plus-ng[663270]: GDB: running:
> "(printf 'bt
>                                                        q
>                                                        ';sleep 3)|gdb
> -n -q -p $CRASHPID 2>/dev/null"

> /

[Petr Issakov]

Hi Marc!

I've installed gdb and lldb

I attached gdb and lldb logs

пятница, 19 июля 2024 г. в 17:55:02 UTC+6, Marc Huber:

[Marc Huber]

Hi Petr,

thanks, alas the backtraces weren't as detailed as I hoped, possibly due
to stripped binaries.

Nevertheless, I think I've found the issue -- could you please git pull
and retry? Commit bbd8248d32093ed48940a58c7b7134c598177cdd likely fixes
the problem.

Cheers,

Marc

On 22.07.2024 08:07, Petr Issakov wrote:
> Hi Marc!

[Petr Issakov]

Hi, Marc! it seems like the issue is resolved now!
Thank's a lot for fast resolving! 

понедельник, 22 июля 2024 г. в 19:32:47 UTC+6, Marc Huber:

[Petr Issakov]

Hi, Marc!

Due monitoring service working I noticed another segv.

May it will be usefull:

Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: running: "(printf 'bt
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: backtrace start
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Attaching to process 1004799
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Reading symbols from /usr/local/sbin/tac_plus-ng...(no debugging symbols found)...done.
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Reading symbols from /usr/local/lib64/libmavis.so...(no debugging symbols found)...done.
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Reading symbols from /../lib64/libcrypt.so.1...Reading symbols from .gnu_debugdata for /../lib64/libcrypt.so.1...(no debugging symbols found)...done.
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: (no debugging symbols found)...done.
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Reading symbols from /../lib64/libpcre2-8.so.0...Reading symbols from .gnu_debugdata for /../lib64/libpcre2-8.so.0...(no debugging symbols found)...done.
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: (no debugging symbols found)...done.
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Reading symbols from /../lib64/libc.so.6...(no debugging symbols found)...done.
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Reading symbols from /../lib64/libdl.so.2...(no debugging symbols found)...done.
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Reading symbols from /../lib64/libpthread.so.0...(no debugging symbols found)...done.
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: [Thread debugging using libthread_db enabled]
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Using host libthread_db library "/lib64/libthread_db.so.1".
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Reading symbols from /usr/local/lib64/mavis/libmavis_groups.so...(no debugging symbols found)...done.
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Reading symbols from /usr/local/lib64/mavis/libmavis_external.so...(no debugging symbols found)...done.
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: 0x00007fe883832585 in read () from /../lib64/libc.so.6
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-251.0.2.el8_10.2.x86_64 libxcrypt-4.1.1-6.el8.x86_64 pcre2-10.32-3.el8_6.x86_64
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: (gdb) #0  0x00007fe883832585 in read () from /../lib64/libc.so.6
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: #1  0x00007fe883796728 in __GI__IO_file_underflow () from /../lib64/libc.so.6
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: #2  0x00007fe883797886 in _IO_default_uflow () from /../lib64/libc.so.6
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: #3  0x00007fe88378b12a in _IO_getline_info () from /../lib64/libc.so.6
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: #4  0x00007fe88378a12f in fgets () from /../lib64/libc.so.6
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: #5  0x00007fe883fabb15 in catchsegv () from /usr/local/lib64/libmavis.so
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: #6  <signal handler called>
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: #7  0x00007fe883f9b899 in io_sched_exec () from /usr/local/lib64/libmavis.so
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: #8  0x00007fe883f9b9b0 in io_main () from /usr/local/lib64/libmavis.so
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: #9  0x000000000041aeb1 in main ()
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: (gdb) A debugging session is active.
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB:
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB:         Inferior 1 [process 1004799] will be detached.
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB:
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Quit anyway? (y or n) [answered Y; input not from terminal]
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: Detaching from program: /usr/local/sbin/tac_plus-ng, process 1004799
Jul 26 04:45:18 t2ru-tacacs-vm-02 tac_plus-ng[1004799]: GDB: [Inferior 1 (process 1004799) detached]

среда, 24 июля 2024 г. в 13:56:52 UTC+6, Petr Issakov:

[Marc Huber]

Hi Petr,

this is somewhat difficult, as your installation doesn't include debug
symbols (and I don't know why, I don't think I've added a strip (-s)
option to the install commands found in the various Makefiles).

I've pushed a potential fix, but I've never seen that issue before and
can't seem to reproduce it. Please continue watching for issues.

Thanks,

Marc