EPERM from pthread_mutex_unlock after fork using pthread_atfork()
Jason Roscoe
pthread_atfork() routine. The source code is attached.
Basically, I initialize a mutex, register the pthread_atfork() handlers,
and then call fork. In the prepare handler, I lock the mutex. In the
child and parent handlers, I try to unlock the mutex. In the parent
handler, it seems to work, however, when I try to unlock the mutex in
the child, I get an error (EPERM) that says the thread in the child does
not own the mutex.
By the way, I am not creating any other threads with pthread_create()
which I understand is the normal case.
My understanding of the way it should work is as follows. Please
forgive my terrible call stack depiction ...
parent
main()
fork()
atForkPrepare()
pthread_mutex_lock()
(logMutex is locked)
(child created, gets a copy of the locked logMutex, see child below)
atForkParent()
pthread_mutex_unlock(logMutex)
(logMutex is unlocked)
...
child
main()
fork()
atForkChild()
pthread_mutex_unlock(logMutex)
(here the pthread_mutex_unlock() returns EPERM)
...
This occurs on a current 2.6 Linux distribution (Debian 4.0) when the
attached source code is built with the following command:
% gcc -D_GNU_SOURCE -o thread thread.c -lpthread
(I defined _GNU_SOURCE to get PTHREAD_MUTEX_ERRORCHECK)
The output from running the program:
% ./thread
pid(3955)
main: set mutex type to error checking: : pid(3955)
atForkPrepare here pid(3955) logMutex(0x804a328)
atForkChild here pid(3956) logMutex(0x804a328)
atForkChild: unlock failed: EPERM: see man page pid(3956)
atForkParent here pid(3955) logMutex(0x804a328)
done waiting on child(3956)
%
Please help explain this one.
Thanks!
Jason
Jason Roscoe
pthread_atfork() routine. The source code is attached.
Basically, I initialize a mutex, register the pthread_atfork() handlers,
and then call fork. In the prepare handler, I lock the mutex. In the
child and parent handlers, I try to unlock the mutex. In the parent
handler, it seems to work, however, when I try to unlock the mutex in
the child, I get an error (EPERM) that says the thread in the child does
not own the mutex.
By the way, I am not creating any other threads with pthread_create()
which I understand is the normal case.
My understanding of the way it should work is as follows. Please
forgive my terrible call stack depiction ...
parent
main()
fork()
atForkPrepare()
pthread_mutex_lock(logMutex)
David Schwartz
> I have a simple test program that I wrote in order to understand the
> pthread_atfork() routine. The source code is attached.
>
> Basically, I initialize a mutex, register the pthread_atfork() handlers,
> and then call fork. In the prepare handler, I lock the mutex. In the
> child and parent handlers, I try to unlock the mutex. In the parent
> handler, it seems to work, however, when I try to unlock the mutex in
> the child, I get an error (EPERM) that says the thread in the child does
> not own the mutex.
> Please help explain this one.
What's to explain? A mutex can only be unlocked by the thread that
locked it. In the child, the thread that locked the mutex no longer
exists, so it cannot be unlocked. This seems like exactly what one
would expect.
DS
Jason Roscoe
A copy of the thread that locked the mutex does exist in the child process.
The mutex is not locked until the "atForkPrepare" handler. In this test
program, there is only one thread - the one that calls fork(). So, if
the fork() creates a copy of the parent (single thread in this case),
the child should get a copy of the mutex as well. The mutex should be
in a locked state after the "atForkPrepare" routine exits. So, it
should be locked in the child process and the parent process once the
fork() processing is complete. The child process has a copy of the
thread that called fork() which is the thread that locked the mutex.
Thanks for your help but it still needs explaining.
BTW, the behavior is different on AIX 4.3 and Solaris 8, i.e., I don't
get an error on the unlock in the "child" handler.
Chris Thomasson
news:S6WdnZw2IYqycCnb...@comcast.com...
> David Schwartz wrote:
>> On Aug 4, 6:56 am, Jason Roscoe <jason.ros...@gmail.com> wrote:
>>> I have a simple test program that I wrote in order to understand the
>>> pthread_atfork() routine. The source code is attached.
>>> Please help explain this one.
>>
>> What's to explain? A mutex can only be unlocked by the thread that
>> locked it. In the child, the thread that locked the mutex no longer
>> exists, so it cannot be unlocked. This seems like exactly what one
>> would expect.
[...]
> A copy of the thread that locked the mutex does exist in the child
> process.
[...]
I don't use fork all that often, however, I think the child thread should
have a different tid than the parent thread, so, its not really the same
exact thread that originally locked the mutex...
Jason Roscoe
I know - fork()ing is making this more complicated but that is how the
code is currently written. My test program shows that the tid as
returned by pthread_self() is the same in both the parent and the child
although I'm not sure if the following code is reasonable:
tid = pthread_self();
fprintf(stdout,"parent(%d) here tid(0x%lx)\n",getpid(),tid);
if(0 == (child = fork()))
{
/* child here */
fprintf(stdout,"child(%d) here old tid(0x%lx)\n",getpid(),tid);
tid = pthread_self();
fprintf(stdout,"child(%d) here new tid(0x%lx)\n",getpid(),tid);
sleep(1);
exit(EXIT_SUCCESS);
}
Here is the output:
parent(6691) here tid(0xb7e8b6c0)
child(6692) here old tid(0xb7e8b6c0)
child(6692) here new tid(0xb7e8b6c0)
Note the thread ids are the same.
Thanks for your help.
I'm still looking for some explanation for the original problem. Anyone?
Bert....@googlemail.com
pthread_self() return a pointer, and more precisly an address. But the
child have a new address space, other than that of the parent. So
these 'tid' are not the same, because there live in different address
spaces. You can get under linux a global uniq thread id with the
gettid(2) call, und than you will notice that these two 'threads' are
different.
But this is only a side note. I just read the SUSv3 page again, and
IMHO your test should succeed. I can imagine why this failed on linux.
In linux the owner of a mutex is specified by the tid from gettid()
and therefore this can not function under linux.
There is also no atfork test in the glibc source for this case. I can
realy imagine that this is a real bug?
Bert
Jason Roscoe
Thanks- that is what I was thinking regarding the opaque type.
>
> But this is only a side note. I just read the SUSv3 page again, and
> IMHO your test should succeed. I can imagine why this failed on linux.
> In linux the owner of a mutex is specified by the tid from gettid()
> and therefore this can not function under linux.
>
> There is also no atfork test in the glibc source for this case. I can
> realy imagine that this is a real bug?
>
> Bert
>
I've built and run the same test on multiple other (older) systems
(Slackware 8.1 kernel 2.4.31, RedHat 8 kernel 2.4.18-14, RedHat 7.2
kernel 2.4.7-10, AIX 4.3, Solaris 5.8 sparc, HP-UX 11) and the output is
what I expect based on the documentation - no EPERM return in the child
handler.
Do you know if there are some compiler flags or other libraries that I
may need to get the expected behavior on the newer Linux systems? I've
verified the problem on RedHat ES4 and Debian 4.0 both with stock 2.6
kernels. Or, if this is a bug, do you know who or how I can get it
verified/fixed? Thanks a lot for your humble opinion! =)
Jason
Bert....@googlemail.com
the same error.
And your successfull systems are all none linux 2.6.
> Do you know if there are some compiler flags or other libraries that I
> may need to get the expected behavior on the newer Linux systems? I've
> verified the problem on RedHat ES4 and Debian 4.0 both with stock 2.6
> kernels. Or, if this is a bug, do you know who or how I can get it
> verified/fixed? Thanks a lot for your humble opinion! =)
I will post this to the libc-alpha list, with a back reference.
http://sourceware.org/ml/libc-alpha/
bert
>
> Jason
David Schwartz
> I know - fork()ing is making this more complicated but that is how the
> code is currently written. My test program shows that the tid as
> returned by pthread_self() is the same in both the parent and the child
> although I'm not sure if the following code is reasonable:
IT SHOWS NO SUCH THING!
The only way to compare two pthread_t's for equality is what
pthread_equal. The result of calling 'pthread_equal' with one or more
invalid pthread_t's is undefined. The pthread_t for the thread that
existed in the other process is invalid, so there is nothing legal you
can do with it.
DS
David Schwartz
> tid = pthread_self();
> fprintf(stdout,"parent(%d) here tid(0x%lx)\n",getpid(),tid);
This code is broken. See my other post.
DS
Jason Roscoe
Perhaps you missed the clause about " ...although I'm not sure if the
following code is reasonable..." Thanks for clearing it up again, though.
Jason
David Schwartz
> Perhaps you missed the clause about " ...although I'm not sure if the
> following code is reasonable..." Thanks for clearing it up again, though.
Yeah, it's not. Mishandling a pthread_t is a common mistake. Linux
with NPTL has 'gettid' which has semantics closer to what you're
looking for.
DS
David Schwartz
> A copy of the thread that locked the mutex does exist in the child process.
I realized another way to wrap your brain around it. Imagine if it's a
process-shared mutex. If the copy could unlock it, then after the
'fork', there would be two threads that own the mutex. That obviously
can't be right.
DS
Chris Thomasson
news:1186370680.9...@w3g2000hsg.googlegroups.com...
Bingo! That is the end of the discussion.
:^)
Yes. The copied thread's tid and its sources may indeed be equal values,
however you have to keep in mind that the new thread is concurrently
executing along side its source.
Dave Butenhof
> I have a simple test program that I wrote in order to understand the
> pthread_atfork() routine. The source code is attached.
>
> Basically, I initialize a mutex, register the pthread_atfork() handlers,
> and then call fork. In the prepare handler, I lock the mutex. In the
> child and parent handlers, I try to unlock the mutex. In the parent
> handler, it seems to work, however, when I try to unlock the mutex in
> the child, I get an error (EPERM) that says the thread in the child does
> not own the mutex.
>
> By the way, I am not creating any other threads with pthread_create()
> which I understand is the normal case.
>
> My understanding of the way it should work is as follows. Please
> forgive my terrible call stack depiction ...
This is an unfortunately complicated issue. The complications stem
somewhat from the inherent nature of fork and exec (and signals) and
unavoidable collisions between these traditional UNIX artifacts and the
concept of threads. They stem partly from some compromises in the text,
in these same areas, necessary to get approval of the standard by some
voting parties who weren't willing to ensure some aspects of
consistency. But they also stem partly from bugs in the original POSIX
text, which the working group determined deliberately (due to the
inherent implementation conflicts) might be better off not solved.
The fact is that POSIX provided the pthread_atfork() mechanism to do
precisely what you want to do. That's the only reason it exists, and
about the only thing it's good for.
POSIX specifically, unambiguously states that you cannot do this.
POSIX places specific requirements about the child process ID; which of
course is essential because pid is a globally visible identifier. But
POSIX does not place any specific requirements on the child THREAD ID,
though it IMPLIES that it be the same because "A process shall be
created with a single thread. If a multi-threaded process calls fork( ),
the new process shall contain a replica of the calling thread and its
entire address space, possibly including the states of mutexes and other
resources."
It goes on to say "Consequently, to avoid errors, the child process may
only execute async-signal-safe operations until such time as one of the
exec functions is called."
And then follows with "Fork handlers may be established by means of the
pthread_atfork( ) function in order to maintain application invariants
across fork( ) calls." -- even though the former statement has
effectively outlawed anything useful one might wish to accomplish with
the atfork handlers. Particularly, of course, unlocking those mutexes
locked by the parent's prepare handler, since all mutex operations are
NOT async-signal-safe.
In other words... any code that does anything like this is "not strictly
conforming". The behavior on which you depend is ALLOWED by POSIX, but
not REQUIRED. If it works on some implementations, great; but you can't
count on it everywhere.
(Very early on, I tried to fix the "may only execute async-signal-safe
operations" bit, because it nullified the clear intent of the working
group in defining the pthread_atfork() mechanism. Instead the
interpretations committee identified additional problems, including the
fact that fork() is technically async-signal-safe whereas prepare atfork
handlers, which generally are used to lock mutexes, are NOT. So instead
of removing the restriction, the working group ended up ADDING the
restriction that you're responsible for not trying to use
non-async-signal-safe prepare handlers if fork might ever be called from
a signal handler in the process.)
Martin Vuille
And yet this is what the rationale for pthread_atfork has
to say on the matter (from IEEE Std. 1003.1-2004/SuSv3):
"The pthread_atfork() function provides multi-threaded libraries
with a means to protect themselves from innocent application
programs that call fork(), and it provides multi-threaded
application programs with a standard mechanism for protecting
themselves from fork() calls in a library routine or the
application itself.
"The expected usage is that the prepare handler acquires all mutex
locks and the other two fork handlers release them.
"For example, an application can supply a prepare routine that
acquires the necessary mutexes the library maintains and supply
child and parent routines that release those mutexes, thus ensuring
that the child gets a consistent snapshot of the state of the
library (and that no mutexes are left stranded). Alternatively,
some libraries might be able to supply just a child routine that
reinitializes the mutexes in the library and all associated states
to some known value (for example, what it was when the image was
originally executed).
"When fork() is called, only the calling thread is duplicated in
the child process. Synchronization variables remain in the same
state in the child as they were in the parent at the time fork()
was called. Thus, for example, mutex locks may be held by threads
that no longer exist in the child process, and any associated
states may be inconsistent. The parent process may avoid this by
explicit code that acquires and releases locks critical to the
child via pthread_atfork(). In addition, any critical threads need
to be recreated and reinitialized to the proper state in the child
(also via pthread_atfork()).
"A higher-level package may acquire locks on its own data
structures before invoking lower-level packages. Under this
scenario, the order specified for fork handler calls allows a
simple rule of initialization for avoiding package deadlock: a
package initializes all packages on which it depends before it
calls the pthread_atfork() function for itself."
This would seem to support the exact usage pattern that the
OP is trying to achieve.
MV
--
I do not want replies; please follow-up to the group.
Jason Roscoe
Thanks for your support and posting the documentation that I've been
using and trying to understand regarding pthread_atfork() and its
intended use. I'm not sure everyone understood that I was just
following (well, mostly) the documentation that came with my system...
JR
Jason Roscoe
Dave: I appreciate the lengthy explanation and now it all makes sense to
me. I've since rewritten the code so it does not require mutexes but
now I understand that I shouldn't use pthread_atfork() for its intended
use... =)
Thanks again!
Jason
Steve Watt
Jason Roscoe <jason....@gmail.com> wrote:
>David Schwartz wrote:
>> On Aug 4, 6:56 am, Jason Roscoe <jason.ros...@gmail.com> wrote:
>>> I have a simple test program that I wrote in order to understand the
>>> pthread_atfork() routine. The source code is attached.
>>>
>>> Basically, I initialize a mutex, register the pthread_atfork() handlers,
>>> and then call fork. In the prepare handler, I lock the mutex. In the
>>> child and parent handlers, I try to unlock the mutex. In the parent
>>> handler, it seems to work, however, when I try to unlock the mutex in
>>> the child, I get an error (EPERM) that says the thread in the child does
>>> not own the mutex.
>> [snip]
>>> Please help explain this one.
>>
>> What's to explain? A mutex can only be unlocked by the thread that
>> locked it. In the child, the thread that locked the mutex no longer
>> exists, so it cannot be unlocked. This seems like exactly what one
>> would expect.
>
As does a *copy* of the mutex. See another thread for discussion about the
usefulness (and legality) of mutex copies.
>The mutex is not locked until the "atForkPrepare" handler. In this test
>program, there is only one thread - the one that calls fork(). So, if
>the fork() creates a copy of the parent (single thread in this case),
>the child should get a copy of the mutex as well. The mutex should be
>in a locked state after the "atForkPrepare" routine exits. So, it
>should be locked in the child process and the parent process once the
>fork() processing is complete. The child process has a copy of the
>thread that called fork() which is the thread that locked the mutex.
>Thanks for your help but it still needs explaining.
Technically, it's not a mutex any more. If it wasn't in shared address
space, it's a copy of a mutex, which is a rather unpleasant thing to
be. If it was in shared address space (sorry, I didn't read your code
sample because your prose seemed pretty clear), then the child thread
wasn't an owner, and thus couldn't unlock it.
The best thing to do with private mutexes in child atfork handlers is to
initialize them.
>BTW, the behavior is different on AIX 4.3 and Solaris 8, i.e., I don't
>get an error on the unlock in the "child" handler.
David B. covered that one pretty well.
--
Steve Watt KD6GGD PP-ASEL-IA ICBM: 121W 56' 57.5" / 37N 20' 15.3"
Internet: steve @ Watt.COM Whois: SW32-ARIN
Free time? There's no such thing. It just comes in varying prices...
Dave Butenhof
> In article <S6WdnZw2IYqycCnb...@comcast.com>,
> Jason Roscoe <jason....@gmail.com> wrote:
>> David Schwartz wrote:
>>> On Aug 4, 6:56 am, Jason Roscoe <jason.ros...@gmail.com> wrote:
>>>> I have a simple test program that I wrote in order to understand the
>>>> pthread_atfork() routine. The source code is attached.
>>>>
>> program, there is only one thread - the one that calls fork(). So, if
>> the fork() creates a copy of the parent (single thread in this case),
>> the child should get a copy of the mutex as well. The mutex should be
>> in a locked state after the "atForkPrepare" routine exits. So, it
>> should be locked in the child process and the parent process once the
>> fork() processing is complete. The child process has a copy of the
>> thread that called fork() which is the thread that locked the mutex.
>> Thanks for your help but it still needs explaining.
>
> Technically, it's not a mutex any more. If it wasn't in shared address
> space, it's a copy of a mutex, which is a rather unpleasant thing to
> be. If it was in shared address space (sorry, I didn't read your code
> sample because your prose seemed pretty clear), then the child thread
> wasn't an owner, and thus couldn't unlock it.
Well, POSIX is squishy here, but definitely allows that (at the sole
discretion of the implementation) it MAY BE a mutex. "Possibly including
the states of mutexes", remember?
Sure, a portable APPLICATION cannot legally copy a pthread_mutex_t, but
only because it doesn't know what a mutex means "inside". The
implementation can, because it can copy what needs to be copied and
reconstruct or ignore what needs to be reconstructed or ignored. (After
all, it knows how to create a mutex from scratch, and can pick apart and
analyze every aspect of the old one... how could it NOT be able to
construct a duplicate with whatever semantic properties are considered
desirable?) For example, copying the lock state, marking it owned by the
child thread, and eliminating any waiters. If a mutex needs to work by
pointing to some special type of memory (as some exotic architectures
require), then the implementation can do that, too. But, more
importantly, no implementation is REQUIRED to do any of this.
So, yeah; a portable conforming application must assume that the data in
any pthread_mutex_t, within the child, is meaningless and unusable.
Not that it matters, because such an application (or at least its
developer) is aware that no pthread function can be called before wiping
the address space completely via exec*(), anyway. ;-)
> The best thing to do with private mutexes in child atfork handlers is to
> initialize them.
Except you can't do that either, because (1) the mutex may be locked,
depending on what the implementation actually does to these mutex
copies, and you can't initialize a locked mutex; and (2) because
pthread_mutex_init() isn't any more async-signal-safe than
pthread_mutex_unlock() and therefore also cannot be portably used in a
child atfork handler.
The fact is that atfork handlers were a necessary mechanism to ensure
that the C runtime can preserve enough state across the fork that the
child can actually back out into the application code and survive long
enough to call exec*(). (If it hadn't been standardized it would have
had to have been invented everywhere anyway.) ESPECIALLY where fork()
may be a pure syscall() (though it often isn't) and the C runtime data
and locks may be in a completely unpredictable state due to other
threads' activities at the time of the fork. The C runtime has a special
contract and relationship with the kernel and knows how far it can push
the rules -- it need not be (and in fact can't be) "portable POSIX code"
anyway.
For the application, the safe solution (the only conforming or portable
solution) is to forget about the enticing illusion of atfork handlers
entirely. Just exec*() in the child and start fresh. Of course it's
perfectly legitimate to exec*() your original argv[0] and restart the
same binary in the child, if that's what you want to do.
Markus Elfring
> fprintf(stdout,"parent(%d) here tid(0x%lx)\n",getpid(),tid);
I suggest to check such data type conversions for correctness ...
Regards,
Markus