nginx + passenger = segv in _rtld_error on restart on FreeBSD 8.0?

Steven Hartland

unread,

Dec 8, 2009, 9:00:18 PM12/8/09

to freebsd...@freebsd.org, freebsd...@freebsd.org

I'm currently testing nginx + passenger on FreeBSD 8.0 and I'm seeing a strange
segv which seems to indicate a core library error in _rtld_error. Could this
be the case or is the stack just badly corrupted?

(gdb) bt
#0 0x00000008005577dc in _rtld_error () from /libexec/ld-elf.so.1
#1 0x0000000800557c3f in _rtld_error () from /libexec/ld-elf.so.1
#2 0x0000000800557d5e in _rtld_error () from /libexec/ld-elf.so.1
#3 0x000000080055851b in dladdr () from /libexec/ld-elf.so.1
#4 0x00000008005585f3 in dladdr () from /libexec/ld-elf.so.1
#5 0x000000080055576d in ?? () from /libexec/ld-elf.so.1
#6 0x0000000000000001 in ?? ()
#7 0x00000000004117f8 in boost::detail::sp_counted_impl_p<Passenger::Application::StandardSession>::dispose (this=0x800768980) at
sp_counted_impl.hpp:78
Previous frame inner to this frame (corrupt stack?)

Regards
Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postm...@multiplay.co.uk.

_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"

Kostik Belousov

unread,

Dec 9, 2009, 5:22:56 AM12/9/09

to Steven Hartland, freebsd...@freebsd.org, freebsd...@freebsd.org

On Wed, Dec 09, 2009 at 01:48:38AM -0000, Steven Hartland wrote:
> I'm currently testing nginx + passenger on FreeBSD 8.0 and I'm seeing a
> strange
> segv which seems to indicate a core library error in _rtld_error. Could this
> be the case or is the stack just badly corrupted?
>
> (gdb) bt
> #0 0x00000008005577dc in _rtld_error () from /libexec/ld-elf.so.1
> #1 0x0000000800557c3f in _rtld_error () from /libexec/ld-elf.so.1
> #2 0x0000000800557d5e in _rtld_error () from /libexec/ld-elf.so.1
> #3 0x000000080055851b in dladdr () from /libexec/ld-elf.so.1
> #4 0x00000008005585f3 in dladdr () from /libexec/ld-elf.so.1
> #5 0x000000080055576d in ?? () from /libexec/ld-elf.so.1
> #6 0x0000000000000001 in ?? ()
> #7 0x00000000004117f8 in
> boost::detail::sp_counted_impl_p<Passenger::Application::StandardSession>::dispose (this=0x800768980) at
> sp_counted_impl.hpp:78
> Previous frame inner to this frame (corrupt stack?)

You need to rebuild rtld with debugging information. Ideally, all shared
objects should have valid debug info.

At least, enter src/libexec/rtld-elf and do
make obj
make depend
make all install DEBUG_FLAGS=-g

Steven Hartland

unread,

Dec 9, 2009, 6:23:34 AM12/9/09

to Kostik Belousov, freebsd...@freebsd.org, freebsd...@freebsd.org

----- Original Message -----
From: "Kostik Belousov" <kost...@gmail.com>
To: "Steven Hartland" <kil...@multiplay.co.uk>
Cc: <freebsd...@freebsd.org>; <freebsd...@freebsd.org>
Sent: Wednesday, December 09, 2009 10:21 AM
Subject: Re: nginx + passenger = segv in _rtld_error on restart on FreeBSD8.0?

This is the trace once world had been recompiled with:-
CFLAGS=-pipe
WITH_CTF=1
DEBUG_FLAGS=-g

#0 0x0000000800c95eec in thr_kill () at thr_kill.S:3
#1 0x0000000800b22e9e in _thr_send_sig (thread=0x800f06600, sig=6) at /usr/src/lib/libthr/thread/thr_kern.c:92
#2 0x0000000800b1f878 in _raise (sig=6) at /usr/src/lib/libthr/thread/thr_sig.c:187
#3 0x0000000800d74003 in abort () at /usr/src/lib/libc/stdlib/abort.c:65
#4 0x000000000043b8a7 in Client::threadMain (this=0x800f9cf40) at ext/nginx/HelperServer.cpp:516
#5 0x0000000000411302 in boost::_mfi::mf0<void, Client>::operator() (this=0x7fffffa45ea8, p=0x800f9cf40) at
mem_fn_template.hpp:49
#6 0x0000000000411651 in boost::_bi::list1<boost::_bi::value<Client*> >::operator()<boost::_mfi::mf0<void, Client>,
boost::_bi::list0> (this=0x7fffffa45eb8, f=@0x7fffffa45ea8, a=@0x7fffffa45d7f) at bind.hpp:232
#7 0x0000000000411696 in boost::_bi::bind_t<void, boost::_mfi::mf0<void, Client>, boost::_bi::list1<boost::_bi::value<Client*> >
>::operator() (this=0x7fffffa45ea8) at bind_template.hpp:20
#8 0x00000000004116bd in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void,
Client>, boost::_bi::list1<boost::_bi::value<Client*> > >, void>::invoke (
function_obj_ptr=@0x7fffffa45ea8) at function_template.hpp:158
#9 0x000000000042e73a in boost::function0<void, std::allocator<void> >::operator() (this=0x7fffffa45ea0) at
function_template.hpp:825
#10 0x0000000000435760 in oxt::thread::thread_main (func=@0x7fffffa45ea0, data=@0x7fffffa45e90) at thread.hpp:107
#11 0x000000000041310e in boost::_bi::list2<boost::_bi::value<boost::function<void ()(), std::allocator<void> > >,
boost::_bi::value<boost::shared_ptr<oxt::thread::thread_data> > >::operator()<void (*)(boost::function<void ()(),
std::allocator<void> >, boost::shared_ptr<oxt::thread::thread_data>), boost::_bi::list0> (this=0x800f3ee80, f=@0x800f3ee78,
a=@0x7fffffa45f0f) at bind.hpp:289
#12 0x0000000000413196 in boost::_bi::bind_t<void, void (*)(boost::function<void ()(), std::allocator<void> >,
boost::shared_ptr<oxt::thread::thread_data>), boost::_bi::list2<boost::_bi::value<boost::function<void ()(), std::allocator<void>
> >, boost::_bi::value<boost::shared_ptr<oxt::thread::thread_data> > > >::operator() (this=0x800f3ee78) at bind_template.hpp:20
#13 0x00000000004131b9 in boost::thread::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()(),
std::allocator<void> >, boost::shared_ptr<oxt::thread::thread_data>), boost::_bi::list2<boost::_bi::value<boost::function<void
()(), std::allocator<void> > >, boost::_bi::value<boost::shared_ptr<oxt::thread::thread_data> > > > >::run (this=0x800f3ee00) at
thread.hpp:130
#14 0x0000000000443259 in thread_proxy (param=0x800f3ee00) at ext/boost/src/pthread/thread.cpp:127
#15 0x0000000800b1badd in thread_start (curthread=0x800f06600) at /usr/src/lib/libthr/thread/thr_create.c:288
#16 0x0000000000000000 in ?? ()
Cannot access memory at address 0x7fffffa46000
Current language: auto; currently asm

It seems that in the passenger client threads it calls closeStream which errors when
the socket close errors with ENOTCONN
virtual void closeStream() {
TRACE_POINT();
if (fd != -1) {
int ret = syscalls::close(fd);
fd = -1;
if (ret == -1) {
if (errno == EIO) {
throw SystemException("A write operation on the session stream failed",
errno);
} else {
throw SystemException("Cannot close the session stream",
errno);
}
}
}
}

This causes it to call abort on the the thread which then crashes the app with
the above stack trace, which seems really weird. Anyone got any ideas?

Regards
steve

Matt Reimer

unread,

May 7, 2010, 1:59:09 PM5/7/10

to Steven Hartland, freebsd...@freebsd.org, freebsd...@freebsd.org

On Tue, Dec 8, 2009 at 6:48 PM, Steven Hartland <kil...@multiplay.co.uk> wrote:
> I'm currently testing nginx + passenger on FreeBSD 8.0 and I'm seeing a
> strange
> segv which seems to indicate a core library error in _rtld_error. Could this
> be the case or is the stack just badly corrupted?
>
> (gdb) bt
> #0 �0x00000008005577dc in _rtld_error () from /libexec/ld-elf.so.1
> #1 �0x0000000800557c3f in _rtld_error () from /libexec/ld-elf.so.1
> #2 �0x0000000800557d5e in _rtld_error () from /libexec/ld-elf.so.1
> #3 �0x000000080055851b in dladdr () from /libexec/ld-elf.so.1
> #4 �0x00000008005585f3 in dladdr () from /libexec/ld-elf.so.1
> #5 �0x000000080055576d in ?? () from /libexec/ld-elf.so.1
> #6 �0x0000000000000001 in ?? ()
> #7 �0x00000000004117f8 in
> boost::detail::sp_counted_impl_p<Passenger::Application::StandardSession>::dispose
> (this=0x800768980) at sp_counted_impl.hpp:78
> Previous frame inner to this frame (corrupt stack?)
>
> � Regards
> � Steve

Steve,

Did you figure this out? We're seeing something very similar with
nginx + passenger + FreeBSD 8.0.

Matt

Matt Reimer

unread,

May 7, 2010, 5:33:24 PM5/7/10

to Steven Hartland, Kostik Belousov, freebsd...@freebsd.org, freebsd...@freebsd.org

Steve,

The patch for PR 144061 works for us.

http://lists.freebsd.org/pipermail/freebsd-hackers/2010-February/030741.html
http://www.freebsd.org/cgi/query-pr.cgi?pr=144061

Matt

Steven Hartland

unread,

May 8, 2010, 12:24:53 PM5/8/10

to Matt Reimer, freebsd...@freebsd.org, freebsd...@freebsd.org

As you may have guessed by my last reply no we didn't we had to revert to
apache + passenger, but seems you've found a fix anyway. Out of interest
what lead you to the close race condition PR as a potential fix?

----- Original Message -----
From: "Matt Reimer" <mattj...@gmail.com>
Steve,

Did you figure this out? We're seeing something very similar with
nginx + passenger + FreeBSD 8.0.

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postm...@multiplay.co.uk.

_______________________________________________