[Samba] samba crashing on start up INTERNAL ERROR: Signal 11

[Sonic via samba]

Samba has been running great but now I'm seeing a problem. Been
compiling from git 4-4-test, but even 4-4.stable (4.4.5) has the same
issue. I've tried to follow the troubleshooting guide and this is what
I come up with:

# nmbd -FS
nmbd version 4.4.6-GIT-1305ba6 started.
Copyright Andrew Tridgell and the Samba Team 1992-2016
STATUS=daemon 'nmbd' finished starting up and ready to serve connections
===============================================================
INTERNAL ERROR: Signal 11 in pid 11238 (4.4.6-GIT-1305ba6)
Please read the Trouble-Shooting section of the Samba HOWTO
===============================================================
PANIC (pid 11238): internal error
BACKTRACE: 13 stack frames:
#0 /usr/local/samba/lib/libsmbconf.so.0(log_stack_trace+0x1f) [0x7f3fe2250a81]
#1 /usr/local/samba/lib/libsmbconf.so.0(smb_panic_s3+0x6d) [0x7f3fe22508d2]
#2 /usr/local/samba/lib/libsamba-util.so.0(smb_panic+0x28) [0x7f3fe2cb7004]
#3 /usr/local/samba/lib/libsamba-util.so.0(+0x20ce2) [0x7f3fe2cb6ce2]
#4 /usr/local/samba/lib/libsamba-util.so.0(+0x20cf7) [0x7f3fe2cb6cf7]
#5 /lib64/libpthread.so.0(+0x10d90) [0x7f3fe3127d90]
#6 /usr/lib64/libtalloc.so.2(talloc_get_size+0x5) [0x7f3fe03163c5]
#7 /usr/local/samba/lib/libsmbconf.so.0(event_add_to_poll_args+0x72)
[0x7f3fe226bf81]
#8 nmbd(listen_for_packets+0x230) [0x56036fb9f2b1]
#9 nmbd(+0x9bdf) [0x56036fb88bdf]
#10 nmbd(main+0xe17) [0x56036fb89e04]
#11 /lib64/libc.so.6(__libc_start_main+0xf0) [0x7f3fdf74a720]
#12 nmbd(_start+0x29) [0x56036fb86e59]
smb_panic(): calling panic action [/bin/sleep 90000]

# smbd -FS
smbd version 4.4.6-GIT-1305ba6 started.
Copyright Andrew Tridgell and the Samba Team 1992-2016
===============================================================
INTERNAL ERROR: Signal 11 in pid 11198 (4.4.6-GIT-1305ba6)
Please read the Trouble-Shooting section of the Samba HOWTO
===============================================================
PANIC (pid 11198): internal error
===============================================================
INTERNAL ERROR: Signal 11 in pid 11199 (4.4.6-GIT-1305ba6)
Please read the Trouble-Shooting section of the Samba HOWTO
===============================================================
PANIC (pid 11199): internal error
BACKTRACE: 16 stack frames:
#0 /usr/local/samba/lib/libsmbconf.so.0(log_stack_trace+0x1f) [0x7efe57c8ca81]
#1 /usr/local/samba/lib/libsmbconf.so.0(smb_panic_s3+0x6d) [0x7efe57c8c8d2]
#2 /usr/local/samba/lib/libsamba-util.so.0(smb_panic+0x28) [0x7efe5a2bb004]
#3 /usr/local/samba/lib/libsamba-util.so.0(+0x20ce2) [0x7efe5a2bace2]
#4 /usr/local/samba/lib/libsamba-util.so.0(+0x20cf7) [0x7efe5a2bacf7]
#5 /lib64/libpthread.so.0(+0x10d90) [0x7efe5a526d90]
#6 /usr/lib64/libtevent.so.0(tevent_debug+0x5e) [0x7efe564f405e]
#7 /usr/lib64/libtevent.so.0(tevent_common_loop_timer_delay+0xba)
[0x7efe564f862a]
#8 /usr/local/samba/lib/libsmbconf.so.0(run_events_poll+0x76) [0x7efe57ca83a9]
#9 /usr/local/samba/lib/libsmbconf.so.0(+0x3b9f7) [0x7efe57ca89f7]
#10 /usr/lib64/libtevent.so.0(_tevent_loop_once+0x9d) [0x7efe564f3c1d]
#11 /usr/lib64/libtevent.so.0(tevent_req_poll+0x1f) [0x7efe564f4fbf]
#12 smbd(+0x75ea) [0x55e4354375ea]
#13 smbd(main+0xf5a) [0x55e43543a7c5]
#14 /lib64/libc.so.6(__libc_start_main+0xf0) [0x7efe56170720]
#15 smbd(_start+0x29) [0x55e435435b19]
smb_panic(): calling panic action [/bin/sleep 90000]
BACKTRACE: 6 stack frames:
#0 /usr/local/samba/lib/libsmbconf.so.0(log_stack_trace+0x1f) [0x7efe57c8ca81]
#1 /usr/local/samba/lib/libsmbconf.so.0(smb_panic_s3+0x6d) [0x7efe57c8c8d2]
#2 /usr/local/samba/lib/libsamba-util.so.0(smb_panic+0x28) [0x7efe5a2bb004]
#3 /usr/local/samba/lib/libsamba-util.so.0(+0x20ce2) [0x7efe5a2bace2]
#4 /usr/local/samba/lib/libsamba-util.so.0(+0x20cf7) [0x7efe5a2bacf7]
#5 /lib64/libpthread.so.0(+0x10d90) [0x7efe5a526d90]
smb_panic(): calling panic action [/bin/sleep 90000]

nmbd:
(gdb) bt
#0 0x00007f8fefbeaeea in waitpid () from /lib64/libc.so.6
#1 0x00007f8fefb7205b in ?? () from /lib64/libc.so.6
#2 0x00007f8ff265998b in smb_panic_s3 (why=0x7f8ff31106fd "internal error") at
../source3/lib/util.c:804
#3 0x00007f8ff30c0004 in smb_panic (why=0x7f8ff31106fd "internal error") at
../lib/util/fault.c:166
#4 0x00007f8ff30bfce2 in fault_report (sig=11) at ../lib/util/fault.c:83
#5 0x00007f8ff30bfcf7 in sig_fault (sig=11) at ../lib/util/fault.c:94
#6 <signal handler called>
#7 0x00007f8ff071f3c5 in talloc_get_size () from /usr/lib64/libtalloc.so.2
#8 0x00007f8ff2674f81 in event_add_to_poll_args (ev=0x56363bcbf2e0,
mem_ctx=0x0, pfds=0x56363ad170d8 <fds>, pnum_pfds=0x7ffe10d579a4,
ptimeout=0x7ffe10d579a8) at ../source3/lib/events.c:92
#9 0x000056363aaf22b1 in listen_for_packets (msg=0x56363bcbf410,
run_election=false) at ../source3/nmbd/nmbd_packets.c:1939
#10 0x000056363aadbbdf in process (msg=0x56363bcbf410) at
../source3/nmbd/nmbd.c:526
#11 0x000056363aadce04 in main (argc=2, argv=0x7ffe10d57d88) at
../source3/nmbd/nmbd.c:1100

smbd:
gdb) bt
#0 0x00007efe56207eea in waitpid () from /lib64/libc.so.6
#1 0x00007efe5618f05b in ?? () from /lib64/libc.so.6
#2 0x00007efe57c8c98b in smb_panic_s3 (why=0x7efe5a30b6fd "internal error") at
../source3/lib/util.c:804
#3 0x00007efe5a2bb004 in smb_panic (why=0x7efe5a30b6fd "internal error") at
../lib/util/fault.c:166
#4 0x00007efe5a2bace2 in fault_report (sig=11) at ../lib/util/fault.c:83
#5 0x00007efe5a2bacf7 in sig_fault (sig=11) at ../lib/util/fault.c:94
#6 <signal handler called>
#7 0x00007efe564f405e in tevent_debug () from /usr/lib64/libtevent.so.0
#8 0x00007efe564f862a in tevent_common_loop_timer_delay () from
/usr/lib64/libtevent.so.0
#9 0x00007efe57ca83a9 in run_events_poll (ev=0x55e43682c8e0, pollrtn=0,
pfds=0x0, num_pfds=0) at ../source3/lib/events.c:199
#10 0x00007efe57ca89f7 in s3_event_loop_once (ev=0x55e43682c8e0,
location=0x7efe564fa05f "../tevent_req.c:264") at ../source3/lib/events.c:303
#11 0x00007efe564f3c1d in _tevent_loop_once () from /usr/lib64/libtevent.so.0
#12 0x00007efe564f4fbf in tevent_req_poll () from /usr/lib64/libtevent.so.0
#13 0x000055e4354375ea in smbd_notifyd_init (msg=0x55e43682ca10,
interactive=false) at ../source3/smbd/server.c:411
#14 0x000055e43543a7c5 in main (argc=2, argv=0x7fff228aca28) at
../source3/smbd/server.c:1597

(gdb) bt
#0 0x00007efe56207eea in waitpid () from /lib64/libc.so.6
#1 0x00007efe5618f05b in ?? () from /lib64/libc.so.6
#2 0x00007efe57c8c98b in smb_panic_s3 (why=0x7efe5a30b6fd "internal error") at
../source3/lib/util.c:804
#3 0x00007efe5a2bb004 in smb_panic (why=0x7efe5a30b6fd "internal error") at
../lib/util/fault.c:166
#4 0x00007efe5a2bace2 in fault_report (sig=11) at ../lib/util/fault.c:83
#5 0x00007efe5a2bacf7 in sig_fault (sig=11) at ../lib/util/fault.c:94
#6 <signal handler called>
#7 0x0000000000000000 in ?? ()
#8 0x00007efe564f8640 in tevent_common_loop_timer_delay () from
/usr/lib64/libtevent.so.0
#9 0x00007efe57ca83a9 in run_events_poll (ev=0x55e43682c8e0, pollrtn=0,
pfds=0x0, num_pfds=0) at ../source3/lib/events.c:199
#10 0x00007efe57ca89f7 in s3_event_loop_once (ev=0x55e43682c8e0,
location=0x7efe564fa05f "../tevent_req.c:264") at ../source3/lib/events.c:303
#11 0x00007efe564f3c1d in _tevent_loop_once () from /usr/lib64/libtevent.so.0
#12 0x00007efe564f4fbf in tevent_req_poll () from /usr/lib64/libtevent.so.0
#13 0x000055e435437d71 in cleanupd_init (msg=0x55e43682ca10, interactive=false,
ppid=0x55e436831c70) at ../source3/smbd/server.c:515
#14 0x000055e43543a800 in main (argc=2, argv=0x7fff228aca28) at
../source3/smbd/server.c:1601

Where to go from here?

Thanks,

Chris

--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[Volker Lendecke via samba]

On Sun, Sep 04, 2016 at 02:39:16PM -0400, Sonic via samba wrote:
> Samba has been running great but now I'm seeing a problem. Been
> compiling from git 4-4-test, but even 4-4.stable (4.4.5) has the same
> issue. I've tried to follow the troubleshooting guide and this is what
> I come up with:

What platform are you on? Can you run that under valgrind?

Thanks,

Volker

[Sonic via samba]

On Mon, Sep 5, 2016 at 1:58 AM, Volker Lendecke <v...@samba.org> wrote:
> What platform are you on? Can you run that under valgrind?

Gentoo, using gcc-5.4.0.

initiate_name_query_packet: sending query for name GRIZZLY<1d>
(bcast=Yes) to IP 172.27.12.255
add_response_record: adding response record id:23039 to subnet
172.27.12.66. num_records:6
==10086== Invalid read of size 4
==10086== at 0x7C5C3C5: talloc_get_size (in /usr/lib64/libtalloc.so.2.1.8)
==10086== by 0x5D01040: event_add_to_poll_args (events.c:92)
==10086== by 0x12A8F0: listen_for_packets (nmbd_packets.c:1939)
==10086== by 0x11201E: process (nmbd.c:526)
==10086== by 0x113243: main (nmbd.c:1100)
==10086== Address 0xffffff9f is not stack'd, malloc'd or (recently) free'd
==10086==
===============================================================
INTERNAL ERROR: Signal 11 in pid 10086 (4.4.6-GIT-28d8230)

Please read the Trouble-Shooting section of the Samba HOWTO
===============================================================

PANIC (pid 10086): internal error
BACKTRACE: 13 stack frames:
#0 /usr/local/samba/lib/libsmbconf.so.0(log_stack_trace+0x1f) [0x5ce1da1]
#1 /usr/local/samba/lib/libsmbconf.so.0(smb_panic_s3+0x6d) [0x5ce1bf2]
#2 /usr/local/samba/lib/libsamba-util.so.0(smb_panic+0x28) [0x5279004]
#3 /usr/local/samba/lib/libsamba-util.so.0(+0x20ce2) [0x5278ce2]
#4 /usr/local/samba/lib/libsamba-util.so.0(+0x20cf7) [0x5278cf7]
#5 /lib64/libpthread.so.0(+0x10d90) [0x4e47d90]
#6 /usr/lib64/libtalloc.so.2(talloc_get_size+0x5) [0x7c5c3c5]
#7 /usr/local/samba/lib/libsmbconf.so.0(event_add_to_poll_args+0x72) [0x5d01041]
#8 /usr/local/samba/sbin/nmbd(listen_for_packets+0x230) [0x12a8f1]
#9 /usr/local/samba/sbin/nmbd(+0xa01f) [0x11201f]
#10 /usr/local/samba/sbin/nmbd(main+0xe17) [0x113244]
#11 /lib64/libc.so.6(__libc_start_main+0xf0) [0x86d3720]
#12 /usr/local/samba/sbin/nmbd(_start+0x29) [0x10fe59]
dumping core in /usr/local/samba/var/cores/nmbd
==10086==
==10086== Process terminating with default action of signal 6
(SIGABRT): dumping core
==10086== at 0x86E61D8: raise (raise.c:54)
==10086== by 0x86E7629: abort (abort.c:89)
==10086== by 0x5CF4DF8: dump_core (dumpcore.c:322)
==10086== by 0x5CE1D81: smb_panic_s3 (util.c:814)
==10086== by 0x5279003: smb_panic (fault.c:166)
==10086== by 0x5278CE1: fault_report (fault.c:83)
==10086== by 0x5278CF6: sig_fault (fault.c:94)
==10086== by 0x4E47D8F: ??? (in /lib64/libpthread-2.23.so)
==10086== by 0x7C5C3C4: talloc_get_size (in /usr/lib64/libtalloc.so.2.1.8)
==10086== by 0x5D01040: event_add_to_poll_args (events.c:92)
==10086== by 0x12A8F0: listen_for_packets (nmbd_packets.c:1939)
==10086== by 0x11201E: process (nmbd.c:526)
==10086==
==10086== HEAP SUMMARY:
==10086== in use at exit: 185,507 bytes in 960 blocks
==10086== total heap usage: 2,417 allocs, 1,457 frees, 366,952 bytes allocated
==10086==
==10086== LEAK SUMMARY:
==10086== definitely lost: 1,173 bytes in 1 blocks
==10086== indirectly lost: 0 bytes in 0 blocks
==10086== possibly lost: 74,956 bytes in 201 blocks
==10086== still reachable: 109,378 bytes in 758 blocks
==10086== suppressed: 0 bytes in 0 blocks
==10086== Rerun with --leak-check=full to see details of leaked memory
==10086==
==10086== For counts of detected and suppressed errors, rerun with: -v
==10086== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Aborted (core dumped)

[Sonic via samba]

Looks like tevent-0.9.30 is somehow the problem. After dropping back
to tevent-0.9.29 samba no longer crashes.

[Volker Lendecke via samba]

On Mon, Sep 05, 2016 at 08:58:38PM -0400, Sonic wrote:
> Looks like tevent-0.9.30 is somehow the problem. After dropping back
> to tevent-0.9.29 samba no longer crashes.

When you build, please have the correct header and -dev libs around
that you will later use.

Volker

[Sonic via samba]

On Tue, Sep 6, 2016 at 5:16 AM, Volker Lendecke <v...@samba.org> wrote:
> When you build, please have the correct header and -dev libs around
> that you will later use.

In most cases there are no separate dev packages in Gentoo, being a
source based distro.
When I run Samba (standalone server - smbd and nmbd) with
tevent-0.9.30 installed it coredumps (both nmbd and smbd) even if I
build Samba against tevent-0.9.30 (that's a make uninstall, make
clean, make distclean, configure, make, make install after updating
tevent and other packages that depend on it).

Chris

[Sonic via samba]

Also see:
https://bugs.gentoo.org/show_bug.cgi?id=592502

[Volker Lendecke via samba]

On Tue, Sep 06, 2016 at 10:55:28AM -0400, Sonic wrote:
> Also see:
> https://bugs.gentoo.org/show_bug.cgi?id=592502
>
>
> On Tue, Sep 6, 2016 at 9:28 AM, Sonic <sonic...@gmail.com> wrote:
> > On Tue, Sep 6, 2016 at 5:16 AM, Volker Lendecke <v...@samba.org> wrote:
> >> When you build, please have the correct header and -dev libs around
> >> that you will later use.
> >
> > In most cases there are no separate dev packages in Gentoo, being a
> > source based distro.
> > When I run Samba (standalone server - smbd and nmbd) with
> > tevent-0.9.30 installed it coredumps (both nmbd and smbd) even if I
> > build Samba against tevent-0.9.30 (that's a make uninstall, make
> > clean, make distclean, configure, make, make install after updating
> > tevent and other packages that depend on it).

Just dropping tevent-0.9.30 into an existing Samba installation
without recompiling Samba is invalid. Maybe we should document that
better, but Samba makes use of internal tevent structures that we give
no ABI guarantees for. See the comment in tevent_internal.h:

INTERNAL STRUCTS. THERE ARE NO API GUARANTEES.
External users should only ever have to include this header when
implementing new tevent backends.

Samba is one such user. Eventually this tight link will be removed,
but for the moment this is the case. With tevent-0.9.30 we have broken
the assumptions in tevent_internal.h by adding fields. Samba uses that
structure directly, so it needs to be compiled against the
tevent_internal.h of the tevent version that is later installed.

Volker

[Sonic via samba]

On Tue, Sep 6, 2016 at 1:24 PM, Volker Lendecke <v...@samba.org> wrote:
> Samba is one such user. Eventually this tight link will be removed,
> but for the moment this is the case. With tevent-0.9.30 we have broken
> the assumptions in tevent_internal.h by adding fields. Samba uses that
> structure directly, so it needs to be compiled against the
> tevent_internal.h of the tevent version that is later installed.

Again - I compiled Samba against tevent-0.9.30.
Did a "make uninstall" of Samba, uninstalled the working
tevent-0.9.29, installed tevent-0.9.30 and then rebuilt other packages
that depend on tevent (libverto, ldb, sssd). Then for Samba did a
"make clean" and "make distclean", followed by "configure", "make",
and "make install".
Result? Samba crashes - both nmbd and smbd dump core.

I've repeated the above steps many times, Samba fails with
tevent-0.9.30, and runs with tevent-0.9.29.

Chris

[Volker Lendecke via samba]

On Tue, Sep 06, 2016 at 01:46:11PM -0400, Sonic wrote:
> On Tue, Sep 6, 2016 at 1:24 PM, Volker Lendecke <v...@samba.org> wrote:
> > Samba is one such user. Eventually this tight link will be removed,
> > but for the moment this is the case. With tevent-0.9.30 we have broken
> > the assumptions in tevent_internal.h by adding fields. Samba uses that
> > structure directly, so it needs to be compiled against the
> > tevent_internal.h of the tevent version that is later installed.
>
> Again - I compiled Samba against tevent-0.9.30.
> Did a "make uninstall" of Samba, uninstalled the working
> tevent-0.9.29, installed tevent-0.9.30 and then rebuilt other packages
> that depend on tevent (libverto, ldb, sssd). Then for Samba did a
> "make clean" and "make distclean", followed by "configure", "make",
> and "make install".
> Result? Samba crashes - both nmbd and smbd dump core.
>
> I've repeated the above steps many times, Samba fails with
> tevent-0.9.30, and runs with tevent-0.9.29.

Does gentoo give any hints about what "tevent-0.9.30" exactly
contains? It should contain a tevent_internal.h file. This file should
contain something like "pthread_mutex_t scheduled_mutex". Can you
check that this exists when you compile Samba?

Volker

[Sonic via samba]

On Tue, Sep 6, 2016 at 2:05 PM, Volker Lendecke <v...@samba.org> wrote:
> Does gentoo give any hints about what "tevent-0.9.30" exactly
> contains? It should contain a tevent_internal.h file. This file should
> contain something like "pthread_mutex_t scheduled_mutex". Can you
> check that this exists when you compile Samba?

It's certainly in the tevent-0.9.30 tarball, however I don't see in
/usr/include after installing tevent-0.9.30 (assuming it's supposed to
be there), it only contains tevent.h. Manually adding 0.9.30's
tevent_internal.h and tevent_util.h to /usr/include does not help.

The Samba src contains ./lib/tevent/tevent_internal.h
./lib/tevent/tevent.h and ./lib/tevent/tevent_util.h, For one
./lib/tevent/tevent_internal.h is not equivalent to the 0.9.30
version. It seems that in building from src Samba is using it's
internal headers and not the system supplied ones, even though it
checks for and finds >=tevent-0.9.29 during conifigure. If I delete
the internal headers from git src then the compile fails with:
==========================
[1277/3352] Compiling source4/librpc/rpc/dcerpc_roh_channel_in.c
../source4/librpc/rpc/dcerpc_roh_channel_in.c:24:31: fatal error:
lib/tevent/tevent.h: No such file or directory
==========================
Even though tevent.h exists in /usr/include (and just in case, also a
a copy in /usr/include/tevent).

Chris

[Volker Lendecke via samba]

On Tue, Sep 06, 2016 at 02:52:20PM -0400, Sonic via samba wrote:
> On Tue, Sep 6, 2016 at 2:05 PM, Volker Lendecke <v...@samba.org> wrote:
> > Does gentoo give any hints about what "tevent-0.9.30" exactly
> > contains? It should contain a tevent_internal.h file. This file should
> > contain something like "pthread_mutex_t scheduled_mutex". Can you
> > check that this exists when you compile Samba?
>
> It's certainly in the tevent-0.9.30 tarball, however I don't see in
> /usr/include after installing tevent-0.9.30 (assuming it's supposed to
> be there), it only contains tevent.h. Manually adding 0.9.30's
> tevent_internal.h and tevent_util.h to /usr/include does not help.

This is a gentoo build issue, not a Samba issue. Please contact the
gentoo lists.

Volker