C++ std::thread: question about _M_start_thread and __gthread_active_p

[hw.cl...@gmail.com]

Hello,

I am having this nasty issue trying to start std::threads in AArch64, and I look for non-AArch64 advice regarding std::thread.

The thread that fails to start in particular is the reclaimer thread (from mempool.cc), which is of class reclaimer.

When _M_start_thread inside libstdc++ is called, the __gthread_active_p check evaluates to false in my case. This causes an exception to be thrown and hell breaks loose.

I probably have something wrong on my side, but I am looking for more general understanding of this issue.

In reclaimer::reclaimer we have:

reclaimer::reclaimer()
: _oom_blocked(), _thread(NULL)
{
// This cannot be a sched::thread because it may call into JNI functions,
// if the JVM balloon is registered as a shrinker. It expects the full
// pthread API to be functional, and for sched::threads it is not.
// std::thread is implemented ontop of pthreads, so it is fine
std::thread tmp([&] {
_thread = sched::thread::current();
_thread->set_name("reclaimer");
osv_reclaimer_thread = reinterpret_cast<unsigned char *>(_thread);
allow_emergency_alloc = true;
do {
_do_reclaim();
} while (true);
});
tmp.detach();
}

and in libstdc++-v3/src/c++11/thread.cc I see:

void
thread::_M_start_thread(__shared_base_type __b)
{
if (!__gthread_active_p())
#if __EXCEPTIONS
throw system_error(make_error_code(errc::operation_not_permitted),
"Enable multithreading to use std::thread");
#else
__throw_system_error(int(errc::operation_not_permitted));
#endif

__b->_M_this_ptr = __b;
int __e = __gthread_create(&_M_id._M_thread,
&execute_native_thread_routine, __b.get());
if (__e)
{
__b->_M_this_ptr.reset();
__throw_system_error(__e);
}
}

And in turn __gthread_active_p is:

static inline int
__gthread_active_p (void)
{
static void *const __gthread_active_ptr
= __extension__ (void *) &GTHR_ACTIVE_PROXY;
return __gthread_active_ptr != 0;
}

Where GTHR_ACTIVE_PROXY is:

#ifdef __GLIBC__
__gthrw2(__gthrw_(__pthread_key_create),
__pthread_key_create,
pthread_key_create)
# define GTHR_ACTIVE_PROXY __gthrw_(__pthread_key_create)
#elif defined (__BIONIC__)
# define GTHR_ACTIVE_PROXY __gthrw_(pthread_create)
#else
# define GTHR_ACTIVE_PROXY __gthrw_(pthread_cancel)
#endif

Does this sound familiar to anybody, who could share some pointers to me to cut some research time on this side? Who actually sets __gthread_active_ptr, and when? Or what does __gthrw_ mean?

Thanks,

Claudio

[Avi Kivity]

On 03/04/2014 05:47 PM, hw.cl...@gmail.com wrote:
> Hello,
>
> I am having this nasty issue trying to start std::threads in AArch64, and I look for non-AArch64 advice regarding std::thread.
>
> The thread that fails to start in particular is the reclaimer thread (from mempool.cc), which is of class reclaimer.
>
> When _M_start_thread inside libstdc++ is called, the __gthread_active_p check evaluates to false in my case. This causes an exception to be thrown and hell breaks loose.
>
> I probably have something wrong on my side, but I am looking for more general understanding of this issue.
>

<snip>

>
> and in libstdc++-v3/src/c++11/thread.cc I see:
>
> void
> thread::_M_start_thread(__shared_base_type __b)
> {
> if (!__gthread_active_p())
> #if __EXCEPTIONS
> throw system_error(make_error_code(errc::operation_not_permitted),
> "Enable multithreading to use std::thread");
> #else
> __throw_system_error(int(errc::operation_not_permitted));
> #endif

libstdc++ is trying to determine whether the application is linked with
libpthreads.so or not.

> __b->_M_this_ptr = __b;
> int __e = __gthread_create(&_M_id._M_thread,
> &execute_native_thread_routine, __b.get());
> if (__e)
> {
> __b->_M_this_ptr.reset();
> __throw_system_error(__e);
> }
> }
>
> And in turn __gthread_active_p is:
>
> static inline int
> __gthread_active_p (void)
> {
> static void *const __gthread_active_ptr
> = __extension__ (void *) &GTHR_ACTIVE_PROXY;
> return __gthread_active_ptr != 0;
> }
>
> Where GTHR_ACTIVE_PROXY is:
>
> #ifdef __GLIBC__
> __gthrw2(__gthrw_(__pthread_key_create),
> __pthread_key_create,
> pthread_key_create)
> # define GTHR_ACTIVE_PROXY __gthrw_(__pthread_key_create)
> #elif defined (__BIONIC__)
> # define GTHR_ACTIVE_PROXY __gthrw_(pthread_create)
> #else
> # define GTHR_ACTIVE_PROXY __gthrw_(pthread_cancel)
> #endif
>
>
> Does this sound familiar to anybody, who could share some pointers to me to cut some research time on this side? Who actually sets __gthread_active_ptr, and when? Or what does __gthrw_ mean?
>
>

See the comment above pthread_cancel() in libc/pthread.cc. Is pthread.o
linked with your image?

The way it normal works is that pthread_cancel is defined as a weak
symbol. If the application is linked with libpthread, the linker will
use pthread_cancel from that; otherwise it will set it to zero.

There could be two possible causes for the problem:
1. pthread.o is not linked in (or is stubbed out)
2. the macro tree evaluates GTHR_ACTIVE_PROXY to something else, other
than pthread_cancel, which we have not defined.

[Claudio Fontana]

Thank you for your answer, pthread.o is in, I'll check further but I
know a little bit what to look for now :)

Claudio

[Claudio Fontana]

I found out what appears to be the problem:

it has to do with the fact that the RELASZ in my resulting elf is
zero, while instead the RELA (.rela.dyn) contains quite some stuff,
and in particular it contains the data relocations for the GOT.

The GOT relocation for the object that points to __pthread_key_create
(the one to which GTHR_ACTIVE_PROXY resolves to) is thus not relocated
correctly at elf::get_init() time.

I am now digging into the aarch64 toolchain to try to find out why
RELASZ is not written to the correct value..

Ciao C