Design Questions on Termination
rgerhards
I have a design question and I have to admit that I worked for most of
the past 10 or so years on Windows and now have come back to Linux for
a while.
I am working on a syslogd (http://www.rsyslog.com), inherited much of
code from the standard sysklogd package. That was single-threaded and
monolithic. In essence it accepts data from a number of fds (waited on
by select()), runs that through some logic and pushes it out to some
handlers. I've already changed that to use the same input logic but
then push the messages to a queue where a (so far single) worker
thread pulls them off the wire.
One of my next steps is to provide modular and loadable inputs. To do
so, each of them should run on its own thread. Now I have a question
in regard to termination.
Let's assume my module still uses select() (or a similar primitive) to
wait for a specific number of selectors. No problem in handling this.
The problem occurs, however, when it comes to termination. So far, my
select() is blocking and I would like to have it that way (if
possible) because I do no periodic wake-ups. But how can I terminate
the select() in case of a program termination?
In Win32 API, there is a WSACancelAsyncCall (or so) API, which makes
all *socket* operations return with a failure state. Is there anything
similar in Linux?
I know that a signal will cancel the select(), but I am hesitant to
rely on signals as I have read so often that pthreads and signals does
not go well together.
I think my problem is a quite basic one, one that often must be
solved. What's the standard / best solution for it under Linux?
Any feedback would be deeply appreciated.
Thanks,
Rainer
PS: I have also written a somewhat larger blurb about my problem at
http://rgerhards.blogspot.com/2007/12/design-problem.html
But I guess my posting here is a pretty good and up to the point wrap-
up.
Eric Sosman
> [...]
> Let's assume my module still uses select() (or a similar primitive) to
> wait for a specific number of selectors. No problem in handling this.
> The problem occurs, however, when it comes to termination. So far, my
> select() is blocking and I would like to have it that way (if
> possible) because I do no periodic wake-ups. But how can I terminate
> the select() in case of a program termination?
>
> In Win32 API, there is a WSACancelAsyncCall (or so) API, which makes
> all *socket* operations return with a failure state. Is there anything
> similar in Linux?
>
> I know that a signal will cancel the select(), but I am hesitant to
> rely on signals as I have read so often that pthreads and signals does
> not go well together.
>
> I think my problem is a quite basic one, one that often must be
> solved. What's the standard / best solution for it under Linux?
If you don't like signals, you could create a pipe and
include its reading end among the select() file descriptors.
When you want to awaken the select() "artificially" for any
reason, write() a byte to the pipe's other end.
--
Eric Sosman
eso...@ieee-dot-org.invalid
Chris Thomasson
news:ueCdnbNpmYICYsLa...@comcast.com...
> rgerhards wrote:
>> [...]
[...]
> If you don't like signals, you could create a pipe and
> include its reading end among the select() file descriptors.
> When you want to awaken the select() "artificially" for any
> reason, write() a byte to the pipe's other end.
I second that advise because it presents a workable solution indeed.
rgerhards
> > include its reading end among the select() file descriptors.
> > When you want to awaken the select() "artificially" for any
> > reason, write() a byte to the pipe's other end.
>
> I second that advise because it presents a workable solution indeed.
Thanks to everyone. Is there any noticable performance drawback from
this? I am asking because whenever I read about this approach, I also
read about performance bottlenecks. But that won't apply in my case -
after all, the pipe would only be written once, at program
termination.
Also, I'd just like to check if it is true that signals do no good
with pthreads (but the "pipe solution" sounds better to me...).
Thanks again,
Rainer
Eric Sosman
>>> If you don't like signals, you could create a pipe and
>>> include its reading end among the select() file descriptors.
>>> When you want to awaken the select() "artificially" for any
>>> reason, write() a byte to the pipe's other end.
>> I second that advise because it presents a workable solution indeed.
>
> Thanks to everyone. Is there any noticable performance drawback from
> this? I am asking because whenever I read about this approach, I also
> read about performance bottlenecks. But that won't apply in my case -
> after all, the pipe would only be written once, at program
> termination.
The performance impact is probably small: Just one more
file descriptor for select() or poll() or /dev/poll or epoll()
or whatever to keep track of, and all these are designed to
keep track of multiple FD's with reasonable efficiency. (If
you have more than a couple dozen FD's, though, select() may
not be the best choice.)
> Also, I'd just like to check if it is true that signals do no good
> with pthreads (but the "pipe solution" sounds better to me...).
There are two principal difficulties: (1) Most Pthreads
operations are off-limits to a signal handler, and (2) it can
be hard to control which of the program's many threads handles
a given signal.
Both problems can be addressed (in part, at least) by blocking
all the interesting signals in all threads except one, and by
using sigwait() in that special thread. Since only one thread
can receive the signals, you've "controlled" their dispatch. And
since the signal is processed synchronously in the thread's "normal"
code instead of asynchronously in a signal handler, the thread is
free to lock mutexes, launch additional threads, and do all the
other things a signal handler mustn't attempt.
It's not perfect. For example, if you're using a third-party
multi-threaded library whose innards you don't control, you may
have a hard time controlling the signal masks of that library's
threads. Even if the third-party library is single-threaded, you
may find that it expects to use signals in a way that doesn't mesh
with the block-them-all-almost-everywhere strategy. Still, if you
can exercise some control over these other components, you can
manage to mix signals and Pthreads.
Oil and water don't mix -- and pass me the salad dressing,
will you?
--
Eric Sosman
eso...@ieee-dot-org.invalid
Markus Elfring
> all the interesting signals in all threads except one, and by
> using sigwait() in that special thread. Since only one thread
> can receive the signals, you've "controlled" their dispatch. And
> since the signal is processed synchronously in the thread's "normal"
> code instead of asynchronously in a signal handler, the thread is
> free to lock mutexes, launch additional threads, and do all the
> other things a signal handler mustn't attempt.
I doubt that your conclusion is correct. I recommend to stick to
async-signal-safe functions.
http://opengroup.org/onlinepubs/009695399/functions/xsh_chap02_04.html#tag_02_04_03
Regards,
Markus
Scott Gifford
>> > If you don't like signals, you could create a pipe and
>> > include its reading end among the select() file descriptors.
>> > When you want to awaken the select() "artificially" for any
>> > reason, write() a byte to the pipe's other end.
>>
>> I second that advise because it presents a workable solution indeed.
>
> Thanks to everyone. Is there any noticable performance drawback from
> this? I am asking because whenever I read about this approach, I also
> read about performance bottlenecks. But that won't apply in my case -
> after all, the pipe would only be written once, at program
> termination.
Right, there should be basically no performance issues.
> Also, I'd just like to check if it is true that signals do no good
> with pthreads (but the "pipe solution" sounds better to me...).
If all pthreads except the one which should receive the signal ignore
it in their signal mask, you should be able to use it safely. If the
signal is sent to a specific thread with pthread_kill(), that should
also be safe.
This page reports problems with using select() and signals at all (and
suggests the above pipe trick as a solution), but the reported problem
might not apply to newer Unixes:
http://cr.yp.to/docs/selfpipe.html
----Scott.
Chris Friesen
Can you provide some basis for your doubts?
In the case above the signal (even if it is an async signal) is
processed synchronously, so the handler is not limited to the async-safe
functions.
From the rationale for sigwait():
"In summary, when it is necessary for code run in response to an
asynchronous signal to notify a thread, sigwait() should be used to
handle the signal. Alternatively, if the implementation provides
semaphores, they also can be used, either following sigwait() or from
within a signal handling routine previously registered with sigaction()."
Chris
Markus Elfring
Signal handler implementations must be async-signal-safe. There is no portable
way to loosen this POSIX standard requirement.
Regards,
Markus
Dave Butenhof
The point is that there IS no signal handler here. You have a thread
parked in a sigwait() loop, just as if it was calling read(), or
select(). When an asynchronous signal arrives, the thread is
synchronously UNBLOCKED from sigwait() and returns with an indication of
the signal. The thread then proceeds, synchronously, with whatever it
needs to do.
Or, to put this another way, sigwait() explicitly IS precisely a
"portable way to loosen this POSIX standard requirement".
You must however block the signal in all other threads in the process in
order for this to be effective -- otherwise it's possible that the
signal action will be taken in another thread even while the sigwaiter
is actually blocked. In a way, it's unfortunate that POSIX didn't
specify that a sigwaiter takes precedence (and some systems DO actually
implement such a precedence) -- however even with that, you still need
to block signals in case the signal should arrive while your sigwait()
thread is awake processing a previous signal.
The only practical way to accomplish the block in all threads (except in
an embedded application where you control all possible threads) is to
call sigprocmask() before the first thread can be created. (Which
generally requires that you control main(), if nothing else.)
Markus Elfring
> "portable way to loosen this POSIX standard requirement".
Thanks for your explanation.
The function "sigwait" has got different properties for async-signal-safety in
comparison to the function "sigaction".
http://opengroup.org/onlinepubs/009695399/functions/sigwait.html
http://opengroup.org/onlinepubs/009695399/functions/sigaction.html
Regards,
Markus
Markus Elfring
Do you distinguish the terms "signal handling routine" and "signal-catching
function" more precisely?
Regards,
Markus
rgerhards
> There are two principal difficulties: (1) Most Pthreads
> operations are off-limits to a signal handler, and (2) it can
> be hard to control which of the program's many threads handles
> a given signal.
Thanks for this clarification. Actually, it proved that I understood
something awfully wrong: I thought that when a signal occurs, all
select() operations would be unblocked in all threads. Obviously, this
is not the case.
The prime use case, I thought, of the signal was to break out of the
blocking operations. This (now I see) obviously is nonsene and so I do
not need signals at all for my implementation. Except, unfortunately,
SIGHUP, which I need to preserve for historical syslogd compatibility
- it is used to request a config reload request. I can block signals
for all threads but a housekeeping thread, so this is fine. Thanks a
lot for this advise (and those of the others), it gets me closer to
understanding all of my options.
But now let me ask another question - as I said, I'd like to know all
about my options (from an experience point of view that you guys
obviously have) before I settle down for a solution.
The self-pipe trick seems to play very nicely with select() and I
assume it does with poll(), epoll() etc. For sleeps, I can also use
select(), so it plays nicely with that too. What I am now looking for
is a generic approach on how to terminate blocking operations. Let's
for example say I'd implement tls secured connections via openssl. A
simple way to do that is by utilizing blocking openssl operations. But
this means my thread will block in the openssl read/write. So if I now
get a termination request, how do I gracefully shutdown that thread? I
know that I can simply cancel it, but that is not my understanding of
graceful termination... (e.g. I'd like to do connection cleanups,
whatever - the actual thing is not of relevance here).
What I can envision is to do the following: I have one thread (let's
call it Twork) that does all of the modules work, except for the
blocking operation. That thread has associated a condition variable
(CTwork). This condition variable is also known to a high-level thread
that does app-internal housekeeping - the one that receives the
shutdown request. Let's call that thread Thk. When thread Twork
initializes, it creates another thread (called Tio) that does nothing
but the blocking IO. Tio also has an associated condition variable
(CTio). Its pseudo-code is as follows:
Tio:
while(1) {
pthread_cond_wait(CTio)
select(params placed by Twork in shared memory)
pthread_cond_signal(CTwork)
}
It is more or less a wrapper for the select. The reason for it is to
make sure my module worker thread isn't blocked. Of course you can
replace the select() with any other blocking operation (e.g. openssl
calls).
The pseudo code for Twork is as follows:
Twork:
While(1) {
Do some work, eg. fill select params
pthread_cond_signal(CTio)
pthread_cond_wait(CTwork)
If(termination flag set) {
kill thread Tio
Do graceful shutdown
Exit thread
}
Do work
}
The point here is that Twork waits on CTio, which is signaled when the
blocking operation is finished. Now comes the termination criteria:
When housekeeping thread Thk receives the termination request, Thk
first sets the termination flag and then signals CTwork, too. Thus,
upon termination Twork awakes, detects it is requested to terminate
and then "cancels" the blocking request and finishes graceful.
I think that approach will work. However, it has the drawback of
additional complexity and I am also concerned about the overhead
involved. After all, I need to at least do two additional context
switches to activate Tio and come back after blocking call completion.
Also, the minimum number of threads per module doubles. I always need
at least two, because one is exclusively reserved for the blocking
operation. This all just to handle a quite exceptional situation...
How do you feel about this approach? Any feedback is highly
appreciated. And if I didn't express myself clearly enough, please let
me know.
Thanks again,
Rainer
Dave Butenhof
>> Or, to put this another way, sigwait() explicitly IS precisely a
>> "portable way to loosen this POSIX standard requirement".
>
> Thanks for your explanation.
> The function "sigwait" has got different properties for async-signal-safety in
> comparison to the function "sigaction".
No, the point is that "async-signal-safety" is irrelevant for use of
sigwait(); deliberately and by design.
Maybe a little background will help:
A signal handler is invoked at the kernel exit boundary by "faking" a
new call frame asynchronously on top of the user stack active at the
time the thread entered the kernel, using a special user-mode label (a
"trampoline") that builds enough stack context to call your handler.
When the application handler returns to the trampoline, it dodges back
into the kernel with a special syscall allowing the kernel to clean
things up and get back to business as usual.
It's this asynchronous pseudo-call that makes signal handlers dangerous;
the next call frame down is not necessarily at a call point. Stack
unwind descriptors may not work, and the kernel may have interrupted the
code in a nasty spot (it may have taken a signal, or a pagefault). In a
threaded program, the most common nastiness is that the thread owns a
mutex -- perhaps a [g]libc mutex that your signal handler might want to
lock. Even if it's recursive, this is seriously bad, because as there
was no explicit call point in most cases there's nothing the "mainline"
code can possibly do to ensure that critical data is consistent for the
signal handler. "Async-signal safety" means that operations are
completed atomically with respect to interrupts -- even pagefaults. It's
not easy, and in many cases is nearly impossible without horrendous
performance cost. Which is why POSIX requires it in very few cases.
sigwait() is completely different. There's no signal handler, no funky
trampoline code. It's exactly like a synchronous read() operation.
Thread calls into kernel and blocks. When the kernel detects that a
signal in the sigwait() mask has arrived, it unblocks the thread and
schedules it for return to user space. Just like completion of any
synchronous read(), or select(), or whatever.
There's no asynchronous context. No trampoline frame, no broken data
invariants unless there's an application bug -- let's say, no broken
data invariants that the application can't easily avoid by standard
correct programming techniques; just as for any other call it might make.
And when sigwait() returns, the kernel returns a value indicating the
signal number that occurred. So we have a normal thread, returning from
a normal call, with the number of a signal that the kernel or some other
process (or user) wanted it to know about. It can do anything it could
had it been a read(). It can lock mutexes (or hold a mutex across the
call but, just as with an indefinitely blocking read(), that's usually a
bad idea). There's nothing at all special about its context here... it's
just a plain old thread.
(Note that you could implement sigwait() on top of asynchronous signals.
For example, sigwait() might block the thread on a sem_wait(). The
signal action for the signals of interest could be set to a signal
handler that would use the async-signal safe sem_post() function to wake
the sigwaiter. Of course it's more complicated -- but with careful
application of simple lock-free coding mechanisms common in any SMP OS
implementation, it's not difficult. Again, the sigwait thread would
simply block synchronously and then return safely from the block once
the signal had arrived -- it wouldn't need to be aware of the interim
asynchrony.)
Scott Gifford
[...]
> The self-pipe trick seems to play very nicely with select() and I
> assume it does with poll(), epoll() etc. For sleeps, I can also use
> select(), so it plays nicely with that too. What I am now looking for
> is a generic approach on how to terminate blocking operations. Let's
> for example say I'd implement tls secured connections via openssl. A
> simple way to do that is by utilizing blocking openssl operations. But
> this means my thread will block in the openssl read/write. So if I now
> get a termination request, how do I gracefully shutdown that thread? I
> know that I can simply cancel it, but that is not my understanding of
> graceful termination... (e.g. I'd like to do connection cleanups,
> whatever - the actual thing is not of relevance here).
I solve this problem with signals sent with pthread_kill(). I
register an empty signal handler for SIGUSR1, and instruct sigaction()
not to automatically restart system calls. In worker threads I
unblock only that signal.Then to cancel a thread that is blocking, I
sent SIGUSR1 to that thread. The blocking call returns EINTR, and I
propogate that error back up to the thread's initial function, which
returns. Something like this:
/* In main() */
/* We ignore these, but setting a handler will allow them to
* interrupt syscalls
*/
sa.sa_handler = sighandle_ignore;
sigaction(SIGUSR1, &sa, NULL);
sigaction(SIGPIPE, &sa, NULL);
/* In worker threads */
/* Ignore these signals, so they will be caught by the main
* thread.
*/
sigset_t ignore_signals;
sigemptyset(&ignore_signals);
sigaddset(&ignore_signals, SIGINT);
sigaddset(&ignore_signals, SIGHUP);
sigaddset(&ignore_signals, SIGTERM);
sigaddset(&ignore_signals, SIGQUIT);
pthread_sigmask(SIG_SETMASK, &ignore_signals, NULL);
-----Scott.
Chris Thomasson
news:fjs8g2$r6t$1...@usenet01.boi.hp.com...
[...]
> (Note that you could implement sigwait() on top of asynchronous signals.
> For example, sigwait() might block the thread on a sem_wait(). The signal
> action for the signals of interest could be set to a signal handler that
> would use the async-signal safe sem_post() function to wake the sigwaiter.
> Of course it's more complicated -- but with careful application of simple
> lock-free coding mechanisms common in any SMP OS implementation, it's not
> difficult. Again, the sigwait thread would simply block synchronously and
> then return safely from the block once the signal had arrived -- it
> wouldn't need to be aware of the interim asynchrony.)
Right. IMO, this is one of the benefits of lock-free algorithms. One can
construct asynchronous-signal safe algorithms. Also, you can blend the
lock-free aspect with an eventcount algorithm that uses a semaphore as a
waitset, which means that you can add an asynchronous-signal safe
conditional blocking mechniasm to an existing lock-free algorithm. Lets see
if I can quickly sketch out some pseudo-code:
____________________________________________________________________
/* Semaphore-Based EventCount
This does not honor wait-epochs like algorithm that
is based on condition variables:
http://groups.google.com/group/comp.programming.threads/browse_frm/thread/aa8c62ad06dbb380
__________________________________________________________*/
#define WAITERS_BIT() 0x80000000
#define WAITERS_BITMASK() 0x7FFFFFFF
typedef int32_t atomicword32;
typedef struct evcount_s evcount;
typedef atomicword32 evkey;
struct evcount_s {
evkey count; /* init to 0 */
int waiters; /* init to zero */
sem_t wmtx; /* bin-sema; init to 1 */
sem_t wset; /* normal semaphore */
};
#define evsys_wmtxlock(mp_this) ( \
sem_wait(&mp_this->wmtx) \
)
#define evsys_wmtxunlock(mp_this) ( \
sem_post(&mp_this->wmtx) \
)
static evkey
evgetkey(evcount* const _this) {
evkey const key = ATOMIC_LOAD(&_this->count);
ATOMIC_BITSET(&_this->count, WAITERS_BIT());
membar #StoreLoad | #LoadLoad;
return key & WAITERS_BITMASK();
}
static void
evbroadcast(evcount* const _this) {
evkey key;
membar #StoreLoad | #StoreStore;
key = ATOMIC_LOAD(&_this->count);
if (key & WAITERS_BIT()) {
int waiters;
evsys_wmtxlock(_this);
while (! ATOMIC_CAS(
&_this->count,
key,
(key + 1) & WAITERS_BITMASK())) {
key = _this->_this->count;
}
waiters = _this->waiters;
_this->waiters = 0;
evsys_wmtxunlock(_this);
if (waiters) {
sem_post_multiple(&_this->wset, waiters);
}
}
}
static void
evwait(evcount* const _this, evkey const key) {
membar #LoadLoad;
evsys_wmtxlock(_this);
evkey const keycmp = ATOMIC_LOAD(&_this->count);
if (key == keycmp & & WAITERS_BITMASK()) {
sem_wait(&_this->wset);
}
evsys_wmtxunlock(_this);
}
____________________________________________________________________
Here is a usage example:
____________________________________________________________________
static evcount g_wset; /* = { static-init }; */
static lockfree_queue g_queue; /* = { static-init }; */
static lockfree_stack g_cache; /* = { static-init }; */
void consumer_sighandler(int sig) {
lockfree_node* node;
/* async-safe wait for a node */
while(! (node = lockfree_queue_trypop(&g_queue))) {
evkey const wkey = evwait(&g_wset);
if (node = lockfree_queue_trypop(&g_queue)) {
break;
}
evwait(&g_wset, wkey);
}
/* do whatever async-signal safe processing you need */
Process_Node(node);
/* return node to cache */
lockfree_stack_push(&g_cache, node);
}
void producer_sighandler(int sig) {
/* get node from cache */
lockfree_node* const node = lockfree_stack_trypop(&g_cache);
if (node) {
/* init the node state */
Init_Node(node);
/* add the node to the queue */
lockfree_queue_push(&g_queue, node);
/* issue a wake-up */
evbroadcast(&g_wset);
}
}
____________________________________________________________________
That scheme will give you 100% async-safe lock-free queue/pop with
conditional waits, thanks to the eventcount.
What do you think of that setup? Notice any potential problems?
;^0
Chris Thomasson
news:OvqdnZ-bx70oVPza...@comcast.com...
[...]
> Lets see if I can quickly sketch out some pseudo-code:
>
>
> ____________________________________________________________________
[...]
OOPS! Let me correct the following function:
> static void
> evwait(evcount* const _this, evkey const key) {
> membar #LoadLoad;
>
> evsys_wmtxlock(_this);
> evkey const keycmp = ATOMIC_LOAD(&_this->count);
> if (key == keycmp & & WAITERS_BITMASK()) {
> sem_wait(&_this->wset);
> }
> evsys_wmtxunlock(_this);
> }
static void
evwait(evcount* const _this, evkey const key) {
membar #LoadLoad;
evsys_wmtxlock(_this);
evkey const keycmp = ATOMIC_LOAD(&_this->count);
if (key == keycmp & & WAITERS_BITMASK()) {
++_this->waiters;
evsys_wmtxunlock(_this);
sem_wait(&_this->wset);
return;
}
evsys_wmtxunlock(_this);
}
> ____________________________________________________________________
[...]
I forgot to increment the evcount::waiters count and unlock the
evcount::wmtx mutex before waiting on the evcount::wset. Humm, I wonder if I
should code this up for real. I wonder how useful a full-blown async-signal
safe waitable lock-free collection would actually be...?
rgerhards
This sounds quite interesting. Are there any known drawbacks? For
example, I read that SIGUSR1 is used by some pthread implementatons
internally - not sure if that still is a concern.
Rainer
Markus Elfring
> signal number that occurred. So we have a normal thread, returning from
> a normal call, with the number of a signal that the kernel or some other
> process (or user) wanted it to know about. It can do anything it could
> had it been a read(). It can lock mutexes (or hold a mutex across the
> call but, just as with an indefinitely blocking read(), that's usually a
> bad idea). There's nothing at all special about its context here... it's
> just a plain old thread.
Thanks for your clarification.
The calling function will do something with the returned value. Is a specific
term available if this usual processing is not performed by a "signal handler"
from the operating system view?
Regards,
Markus
Dave Butenhof
No. It's just a thread that found a signal number. I suppose you could
say it's "synchronously performing actions related to receipt of a
signal", but even that may imply too much.
Scott Gifford
> On Dec 14, 12:11 am, Scott Gifford <sgiff...@suspectclass.com> wrote:
>> rgerhards <rgerha...@gmail.com> writes:
>>
>> [...]
>>
>> > The self-pipe trick seems to play very nicely with select() and I
>> > assume it does with poll(), epoll() etc. For sleeps, I can also use
>> > select(), so it plays nicely with that too. What I am now looking for
>> > is a generic approach on how to terminate blocking operations.
[...]
>> I solve this problem with signals sent with pthread_kill(). I
>> register an empty signal handler for SIGUSR1, and instruct sigaction()
>> not to automatically restart system calls. In worker threads I
>> unblock only that signal.Then to cancel a thread that is blocking, I
>> sent SIGUSR1 to that thread. The blocking call returns EINTR, and I
>> propogate that error back up to the thread's initial function, which
>> returns.
[...]
> This sounds quite interesting. Are there any known drawbacks? For
> example, I read that SIGUSR1 is used by some pthread implementatons
> internally - not sure if that still is a concern.
It seems intuitively correct and has worked for me so far, though I
have not read through the POSIX standards to see if it is guaranteed
to work on all platforms.
I know that LinuxThreads, the old Linux thread implementation, used
signals to interrupt itself, but SIGUSR1 is designed for applications
to ue internally, so I suspect lots of things would break if they just
used it without providing some kind of emulation to hide that.
At any rate, any signal wiill work. I handle SIGPIPE in the child
too, in case it is writing to a filehandled that is closed, so I
suppose you could just send SIGPIPE instead if SIGUSR1 is a problem.
Or just use SIGUSR2 for that matter.
Hope that helps,
---Scott.
Markus Elfring
> say it's "synchronously performing actions related to receipt of a
> signal", but even that may imply too much.
An user space function will be notified about the signal number. It should not
be called "signal handler" in this use case.
I was used to call functions with such behaviour as (message) handlers.
Regards,
Markus
Dave Butenhof
> rgerhards <rger...@gmail.com> writes:
>
>> This sounds quite interesting. Are there any known drawbacks? For
>> example, I read that SIGUSR1 is used by some pthread implementatons
>> internally - not sure if that still is a concern.
>
> It seems intuitively correct and has worked for me so far, though I
> have not read through the POSIX standards to see if it is guaranteed
> to work on all platforms.
>
> I know that LinuxThreads, the old Linux thread implementation, used
> signals to interrupt itself, but SIGUSR1 is designed for applications
> to ue internally, so I suspect lots of things would break if they just
> used it without providing some kind of emulation to hide that.
POSIX requires that the standard POSIX signals be available to
applications. In the strictest technical sense, an implementation that
consumes SIGUSR1 (or any other standard signal) is NOT "pthreads", or
"POSIX", in that use of the term implies conformance with the standard.
This was one of many conformance issues with Linuxthreads. (And I don't
mean that as criticism -- Xavier never claimed 100% POSIX conformance,
and what he was able to accomplish within the restrictions of the old
Linux kernel was more than a little astonishing. It just wasn't POSIX.)
There are implementations that need signals, or signal-like behavior,
for internal functions. That's perfectly OK, and the implementation can
provide an additional NON-standard signal number to fill that need; for
example, Sun's SIGLWP. But the standard signals must be available.
Then again -- there are viable and useful libraries that resemble
pthreads in useful ways that are not (and can't be) "POSIX", and making
allowance for glitches like this can enable application portability that
you'd lose otherwise. Linuxthreads uses SIGUSR1, sure; but what about
OpenVMS or the Win32 "pthread" library, which don't have real UNIX
signals at all?
It's best to avoid reliance on signals, if you can, because they're hard
to do on non-UNIX platforms. If you must use signals and tie yourself to
implementations that support them, try to be flexible in your choice of
signal rather than hardcoding something like SIGUSR1 throughout your
application. And if you can't even do that, just be alert for
implementations that do non-conforming things, which may present
problems for you.
Gil Hamilton
news:lybq8tm...@gfn.org:
> At any rate, any signal wiill work. I handle SIGPIPE in the child
> too, in case it is writing to a filehandled that is closed, so I
FYI:
This is tangential to the larger thread but... You do realize that you
don't need to handle SIGPIPE (i.e. you can ignore it and I mean ignore as
in SIG_IGN, not just install an empty handler function). If you attempt to
write to a broken pipe, write will return -1 with errno EPIPE anyway.
SIGPIPE is intended to kill "dumb" programs that might neglect to check the
return value from write (such as the huge majority of command line programs
that call printf and friends). So that if you type:
# program1 | program2 | ... | programN | less
and then quit out of 'less', the programs don't continue to run
(potentially forever). Instead, programN gets SIGPIPE when it next tries
to write to the pipe that 'less' used to be reading. Once it exits, the
programN-1 will get SIGPIPE on its next write. And so forth until every
program in the pipeline is dead.
GH
Scott Gifford
> Scott Gifford <sgif...@suspectclass.com> wrote in
> news:lybq8tm...@gfn.org:
>
>> At any rate, any signal wiill work. I handle SIGPIPE in the child
>> too, in case it is writing to a filehandled that is closed, so I
>
> FYI:
> This is tangential to the larger thread but... You do realize that you
> don't need to handle SIGPIPE (i.e. you can ignore it and I mean ignore as
> in SIG_IGN, not just install an empty handler function). If you attempt to
> write to a broken pipe, write will return -1 with errno EPIPE
> anyway.
Yes, that's right. I had already written the code to handle SIGUSR1
(which needs an empty handler to get the behavior I want, I believe)
and I just did the same thing with SIGPIPE. I will change it to
SIG_IGN, that is a better idea.
Thanks!
----Scott.
Markus Elfring
> all the interesting signals in all threads except one, and by
> using sigwait() in that special thread.
How does your wording fit to the following description?
http://opengroup.org/onlinepubs/009695399/functions/sigwait.html
"The signals defined by set shall have been blocked at the time of the call to
sigwait(); otherwise, the behavior is undefined."
I interpret the specification that the interesting signals should be blocked in
all threads (without exception). This setting would be a bit different if the
function "sigaction" will be used instead.
Regards,
Markus
Marcin ‘Qrczak’ Kowalczyk
Dnia 15-12-2007, So o godzinie 07:47 +0100, Markus Elfring pisze:
> "The signals defined by set shall have been blocked at the time of the
> call to sigwait(); otherwise, the behavior is undefined."
>
> I interpret the specification that the interesting signals should be
> blocked in all threads (without exception).
I interpret otherwise. The intention was definitely that they should be
blocked by the thread doing sigwait().
--
__("< Marcin Kowalczyk
\__/ qrc...@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/
Dave Butenhof
> Dnia 15-12-2007, So o godzinie 07:47 +0100, Markus Elfring pisze:
>
>> "The signals defined by set shall have been blocked at the time of the
>> call to sigwait(); otherwise, the behavior is undefined."
>>
>> I interpret the specification that the interesting signals should be
>> blocked in all threads (without exception).
>
> I interpret otherwise. The intention was definitely that they should be
> blocked by the thread doing sigwait().
These are part of the same issue. There's no defined precedence between
sigwait() and a signal action. So if both are "in scope", you don't know
which will happen. If a thread is blocked in sigwait() AND there's a
signal action, it's possible that the sigwait() won't be unblocked;
you'll get the defined signal action instead. (And of course the defined
signal action might be to terminate the process, which could possibly
interfere with the intended algorithm. ;-) )
The same holds if there are OTHER threads with the signal unblocked,
because the signal action might be triggered in one of them rather than
unblocking the sigwait()-er.
We toyed with the idea of requiring precedence for a sigwait() over a
signal action; there was some resistance (especially for the convenience
of pthread implementations without kernel support, which was much more
of an issue back then than it is now), but the main reason we didn't was
realization that it simply wouldn't help. A sigwait()-er will eventually
return, and do something with the information. There's nothing to
prevent the signal from recurring, unless it's blocked. And if there's
no thread blocked in sigwait(), it will trigger the signal action.
So in order to get reliable sigwait() behavior, you absolutely need to
have the signal blocked in all thread. It's not an arbitrary rule
imposed by sigwait(), but a simple declaration of inevitable fact.
Eric Sosman
>> Both problems can be addressed (in part, at least) by blocking
>> all the interesting signals in all threads except one, and by
>> using sigwait() in that special thread.
>
> How does your wording fit to the following description?
> http://opengroup.org/onlinepubs/009695399/functions/sigwait.html
> "The signals defined by set shall have been blocked at the time of the call to
> sigwait(); otherwise, the behavior is undefined."
My wording fits poorly, that's how it fits. ;-)
> I interpret the specification that the interesting signals should be blocked in
> all threads (without exception). This setting would be a bit different if the
> function "sigaction" will be used instead.
I think I was muddling the two together and creating a
sort of signal succotash. Thanks for the correction.
--
Eric Sosman
eso...@ieee-dot-org.invalid
Markus Elfring
> blocked by the thread doing sigwait().
Other documentation is more explicit than the POSIX specification about the
involved relationships.
A) HP-UX 11i Version 3 Reference
http://www.docs.hp.com/en/B2355-60130/sigwait.2.html
"APPLICATION USAGE
For a given signal number, the sigwait family of routines should not be used in
conjunction with sigaction() or any other functions which change signal action.
If they are used together, the results are undefined.
[...]
In order to ensure that the dedicated thread handles the signal, it is essential
that all threads, including the thread issuing the sigwait call, block the
signals of interest. Otherwise, the signal could be delivered to a thread other
than the dedicated signal handling thread. This could result in the default
action being carried out for the signal. It is important that the thread issuing
the sigwait call also block the signal. This will prevent signals from carrying
out the default signal action while the dedicated signal handling thread is
between calls to a sigwait function."
B) Solaris 10 Software Developer Collection >> Multithreaded Programming Guide
>> 5. Programming With the Solaris Software >> Extending Traditional Signals
http://docs.sun.com/app/docs/doc/816-5137/gen-ex-24?a=view
"Waiting for a Specified Signal
[...]
All signals identified by the set argument must be blocked on all threads,
including the calling thread. Otherwise, sigwait() might not work correctly."
C) IRIX 6.5 » Man Pages
http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?db=man&fname=/usr/share/catman/p_man/cat3thr/sigwait.z
"A thread must block the signals it waits for using sigprocmask() prior to
calling this function."
D) Article "Thread-Specific Data and Signal Handling in Multi-Threaded
Applications" by Martin McCarthy
http://www.linuxjournal.com/article/2121
"The main() function sets the signal mask to block all signals, so all threads
created after this point will have all signals blocked, including the
signal-handling thread. Strange as it may seem at first sight, this is exactly
what we want. The signal-handling thread expects signal information to be
provided by the sigwait() function, not directly by the operating system.
sigwait() will unmask the set of signals that are given to it, and then will
block until one of those signals occurs."
Regards,
Markus
Chris Thomasson
> news:fjs8g2$r6t$1...@usenet01.boi.hp.com...
> [...]
>> (Note that you could implement sigwait() on top of asynchronous signals.
>> For example, sigwait() might block the thread on a sem_wait(). The signal
>> action for the signals of interest could be set to a signal handler that
>> would use the async-signal safe sem_post() function to wake the
>> sigwaiter. Of course it's more complicated -- but with careful
>> application of simple lock-free coding mechanisms common in any SMP OS
>> implementation, it's not difficult. Again, the sigwait thread would
>> simply block synchronously and then return safely from the block once the
>> signal had arrived -- it wouldn't need to be aware of the interim
>> asynchrony.)
>
> Right. IMO, this is one of the benefits of lock-free algorithms. One can
> construct asynchronous-signal safe algorithms. Also, you can blend the
> lock-free aspect with an eventcount algorithm that uses a semaphore as a
> waitset, which means that you can add an asynchronous-signal safe
> conditional blocking mechniasm to an existing lock-free algorithm. Lets
> see if I can quickly sketch out some pseudo-code:
WHOOPS! Forget waiting on the event-count in a signal-handler! I can't
believe nobody caught the mistake I made here... See, I was using sem_wait
in a signal-handler == Dumb ass retarded mistake!!!
Anyway, why is sem_wait not async-signal safe? What happens if you use
sem_wait in a signal-handler? For some reason, I thought that you could do
something like:
void sighandler(int sig) {
int status = sem_wait(...);
if (! status) {
// consumed semaphore post
} else if (status == EINTR) {
// consumed signal
}
}
What happens when sem_post hits a slow-path and needs to wake a waiter?
Doesn't it need to take a lock on the semaphore waitset or something?
I need some sleep!
:^(...
Chris Thomasson
I need to make a clarification, and fix the example code I gave. I wrote
that code in a FOG that made me think that sem_wait was okay with being
calling in a signal-handler!
OUCH!
Anyway, you can use a producer/consumer relationship with signal-handlers if
you follow a simple rule:
The signal-handler can never be a consumer of something if it hits a
slow-path and has to wait!
For instance, the signal-handler has no choice but to fail and do nothing in
the following case because it failed to pop an item off the lock-free stack:
________________________________________________________________
static lockfree_stack g_stack;
void sighandler_consumer(int sig) {
lockfree_node* const node = g_stack.trypop();
if (! node) {
// there is nothing we can do! We have to fail.
return;
}
}
________________________________________________________________
Sorry about all that non-sense! Well, not to leave everybody high and dry
here, there is something that you can do with signals:
http://groups.google.com/group/comp.programming.threads/browse_frm/thread/9545d3e17806ccfe
you can use them for quiescent-states...
Chris Thomasson
Sorry for all the posts... But I have one more question...
Why isn't sem_trywait able to be called from a signal-handler?
Markus Elfring
Will a deadlock happen if the corresponding thread will be blocked?
Regards,
Markus
Dave Butenhof
If you BLOCK in a signal handler, in a sense you're blocking TWO
threads... the signal context that's actually executing, but also the
"mainline" thread whose stack the signal context has preempted.
Technically, a "trywait" isn't as bad, but there wasn't any real
purpose. We made sem_post() async-signal safe specifically to allow a
signal action routine to wake a thread. Nobody had any interest in
encouraging excessive use of async signal actions... and just about any
use is excessive. ;-)
rgerhards
many thanks for your excellent advise - really appreciated. I have now
been able to make up my mind. In the initial implementation, I will
actually use pthread_kill() for this type of synchronization. However,
I will create a wrapper class around almost all of the threading
calls, so changing to a different tool (like the self-pipe trick)
should be relatively painless. I know I can not do it with a zero
percent change to the user code, but it should be fairly
straightforward (and with a few smart preprocessor macros I may even
get close to 0% change outside of the wrapper... let's see).
I would once again thank all of you. The information presented here
was very in-depth, up to the point and obviously backed by experience.
I am glad I found this place and I am sure the information obtained
here will help implement an even better syslogd.
Rainer
Markus Elfring
> of the threading calls, so changing to a different tool
> (like the self-pipe trick) should be relatively painless.
Would you like to reuse any existing class library?
Regards,
Markus
rgerhards
I have just begun to implement using the signal() approach to prevent
blocking. However, I now thing that it probably is prone to a race
condition.
Let's just quickly wrap up:
I have one thread (Tw) that does
Tw:
in infinite loop
do some work
select()
if terminate flag set
pthread_exit()
do some more work
end loop
Tw has an signal handler handler that is just use to awake Tw if if is
in the select(). My main thread (Tm) termination code works as
follows:
Tm:
set Tw's terminate flag
pthread_kill(Tw, SIGUSR1)
pthread_join(Tw)
The race condition is that I may signal Tw while it is not at the
select. If so, the signal handler will execute, but it won't affect
the *future* select at all. Then, Tw runs into the select() and blocks
there. A (too-)obvious solution is to check the termination flag one
more time before we call select(). But then, what happens if Tw is
preempted after the check but before the select() is being fully
executed? Blocking again. Now I can put a mutex around these calls,
like in that way:
Tw:
in infinite loop
do some work
pthread_mutex_lock()
if terminate flag set
pthread_exit()
select()
pthread_mutex_unlock()
if terminate flag set
pthread_exit()
do some more work
end loop
But then I can't wait on the mutex. I could use a condition and signal
it before the select... But again, all of this boils down to be no
atomic operations.
So my conclusion is that I cannot use this approach in a reliable way.
Is this correct? What have I overlooked?
Feedback would much be appreciated.
Rainer
rgerhards
Oops, Markus, sorry. I overlooked this message. No, the code is quite
special and I tend to have as few dependencies as possible - except,
of course, when there is strong reasoning. My wrapper is tightly
integrated into the rsyslog logic and also has just a few lines.
Rainer
Marcin ‘Qrczak’ Kowalczyk
Dnia 20-12-2007, Cz o godzinie 00:37 -0800, rgerhards pisze:
> I have just begun to implement using the signal() approach to prevent
> blocking. However, I now thing that it probably is prone to a race
> condition.
See pselect.
Or poll / ppoll. It has a more convenient interface than select.
Or wake up with a pipe. It's easier to get right than with a signal.
Scott Gifford
[...]
> I have one thread (Tw) that does
>
> Tw:
> in infinite loop
> do some work
> select()
> if terminate flag set
> pthread_exit()
> do some more work
> end loop
>
> Tw has an signal handler handler that is just use to awake Tw if if is
> in the select(). My main thread (Tm) termination code works as
> follows:
>
> Tm:
> set Tw's terminate flag
> pthread_kill(Tw, SIGUSR1)
> pthread_join(Tw)
>
> The race condition is that I may signal Tw while it is not at the
> select. If so, the signal handler will execute, but it won't affect
> the *future* select at all. Then, Tw runs into the select() and blocks
> there. A (too-)obvious solution is to check the termination flag one
> more time before we call select(). But then, what happens if Tw is
> preempted after the check but before the select() is being fully
> executed? Blocking again.
You're absolutely right. In my case, Tm also closes the file
descriptor that Tw is select()ing on before sending the signal, so the
race is solved: If Tw it hasn't started select() then when it does it
will fail because of an invalid file descriptor; and if it is
currently in select() then it will return EINTR and then notice the
flag.
-----Scott.
Marcin ‘Qrczak’ Kowalczyk
> In my case, Tm also closes the file descriptor that Tw is select()ing
> on before sending the signal, so the race is solved:
But another race is introduced: if a different thread opens a file at
the same time, it might get the same descriptor.
Scott Gifford
> Dnia 20-12-2007, Cz o godzinie 13:10 -0500, Scott Gifford pisze:
>
>> In my case, Tm also closes the file descriptor that Tw is select()ing
>> on before sending the signal, so the race is solved:
>
> But another race is introduced: if a different thread opens a file at
> the same time, it might get the same descriptor.
Yes indeed. And that's quite likely, too, because of the way FDs are
assigned. Hrm, I'll have to think about that...
----Scott.
rgerhards
mmhhh... doesn't the kernel prevent the same fd to be reused as long
as it is in a select? That sounds somewhat scary... Anyway, as I don't
have this scenario, I've not thought more than a minute about it ;)
To go back to my original problem. I have changed my classes in a way
that everything has a separate cleanup function. So what I now do is
pthread_cancel() the select() (and any other blocking operations). So
far, I have not found any place where this causes grief, provided that
the cleanup function is called. I can envision, however, that there
may be some mutexs locked, in which case it may not be wise to cancel
the thread that holds them. If so, I think I can work-around that, but
I have yet to see this issue coming up. As far as I can see, it won't
materialize in what I need to do.
Rainer
rgerhards
> > I have just begun to implement using the signal() approach to prevent
> > blocking. However, I now thing that it probably is prone to a race
> > condition.
>
> See pselect.
>
> Or poll / ppoll. It has a more convenient interface than select.
>
> Or wake up with a pipe. It's easier to get right than with a signal.
I am using select() as a synonym for any blocking call. pselect() is
the obvious answer if you think about select. But it doesn't fit all
situations I expect. This is also what makes the self-pipe approach
unusable in my project - at least in some cases. As I have posted a
few minutes ago, it looks like I found my solution. The post was to
check if I was thinking correctly.
Thanks again for all the good advise here.
Rainer
Marcin ‘Qrczak’ Kowalczyk
> doesn't the kernel prevent the same fd to be reused as long
> as it is in a select?
What do you mean by "in a select"? The select call does not have to be
active at the moment of opening a new fd for the problem to occur.
I think that only waking up select with a pipe, or pthread_kill
+ pselect instead of select, are reasonably easy to get right.
The pipe is easier.
Marcin ‘Qrczak’ Kowalczyk
> I am using select() as a synonym for any blocking call. pselect() is
> the obvious answer if you think about select. But it doesn't fit all
> situations I expect.
For some other calls there is sigsafe library.
The project seems to be dead however. I sent him some bug reports 2 years
ago, and he later responded that he might release a fixed version, but
he hasn't done this.
rgerhards
wrote:
> Dnia 21-12-2007, Pt o godzinie 02:14 -0800, rgerhards pisze:
>
> > doesn't the kernel prevent the same fd to be reused as long
> > as it is in a select?
>
> What do you mean by "in a select"? The select call does not have to be
> active at the moment of opening a new fd for the problem to occur.
I think it is best to disregard that part of my reply. I am obviously
not 100% sure what was meant. If one thread works on a fd and the
other closes it, that calls for trouble. But anyhow ;)
>
> I think that only waking up select with a pipe, or pthread_kill
> + pselect instead of select, are reasonably easy to get right.
> The pipe is easier.
Yeah, that's true. But as I said pthread_cancel() and a decent cleanup
handler is even easier.
Rainer
Chris Thomasson
> Chris Thomasson wrote:
>> "Chris Thomasson" <cri...@comcast.net> wrote in message
>> news:OvqdnZ-bx70oVPza...@comcast.com...
>>> "Dave Butenhof" <david.b...@hp.com> wrote in message
>> [...]
>>
>> Sorry for all the posts... But I have one more question...
>>
>> Why isn't sem_trywait able to be called from a signal-handler?
>
> If you BLOCK in a signal handler, in a sense you're blocking TWO
> threads... the signal context that's actually executing, but also the
> "mainline" thread whose stack the signal context has preempted.
>
> Technically, a "trywait" isn't as bad, but there wasn't any real purpose.
;^)
Well, if you were forced to stick a wait function into the list of
async-signal-safe ones, IMO, a try-wait semantics would be a welcome
requirement...
> We made sem_post() async-signal safe specifically to allow a signal action
> routine to wake a thread. Nobody had any interest in encouraging excessive
> use of async signal actions... and just about any use is excessive. ;-)
Okay. However, IMHO, any "try" wait logic "should" be able to fit in within
the scope of async-signal-safety parameters...
Humm.
Dave Butenhof
True so far as it goes. But a try-wait may fail; and what do you do? You
either hack up some way to wait and try-wait again... or you go ahead as
if there had been no attempt to wait. Philosophically, the latter is
what you should simply ALWAYS do in a signal handler, keeping in mind
that the signal handler itself is always blocking an independent context
(even without threads) and that you're blocking IT as well as the signal
handler context.
And if you follow that model, then you're better off simply not even
doing the try-wait -- just presume that it might not have worked anyway,
do whatever you would have done had you tried and it failed, and get out
of there so the program can get back to normal operation. This path has
to work anyway, and now you only have to test the one path that MUST be
there. ;-)
Chris Thomasson
> Chris Thomasson wrote:
>> "Dave Butenhof" <david.b...@hp.com> wrote in message
>> news:fk4klr$dvt$1...@usenet01.boi.hp.com...
>>> Chris Thomasson wrote:
>>>> "Chris Thomasson" <cri...@comcast.net> wrote in message
>>>> news:OvqdnZ-bx70oVPza...@comcast.com...
>>>>> "Dave Butenhof" <david.b...@hp.com> wrote in message
>>>> [...]
>>>>
>>>> Sorry for all the posts... But I have one more question...
>>>>
>>>> Why isn't sem_trywait able to be called from a signal-handler?
>>>
>>> If you BLOCK in a signal handler, in a sense you're blocking TWO
>>> threads... the signal context that's actually executing, but also the
>>> "mainline" thread whose stack the signal context has preempted.
>>>
>>> Technically, a "trywait" isn't as bad, but there wasn't any real
>>> purpose.
>>> We made sem_post() async-signal safe specifically to allow a signal
>>> action routine to wake a thread. Nobody had any interest in encouraging
>>> excessive use of async signal actions... and just about any use is
>>> excessive. ;-)
>>
>> Okay. However, IMHO, any "try" wait logic "should" be able to fit in
>> within the scope of async-signal-safety parameters...
>
> True so far as it goes. But a try-wait may fail; and what do you do? You
> either hack up some way to wait and try-wait again... or you go ahead as
> if there had been no attempt to wait. Philosophically, the latter is what
> you should simply ALWAYS do in a signal handler, keeping in mind that the
> signal handler itself is always blocking an independent context (even
> without threads) and that you're blocking IT as well as the signal handler
> context.
Right. IMO, a signal-handler can never be a consumer of anything if it hits
a
slow-path and has to wait. You have to just return, or do something else in
the
case of a failed try-wait...
> And if you follow that model, then you're better off simply not even doing
> the try-wait -- just presume that it might not have worked anyway, do
> whatever you would have done had you tried and it failed, and get out of
> there so the program can get back to normal operation. This path has to
> work anyway, and now you only have to test the one path that MUST be
> there. ;-)
I advise avoiding signal-handlers as well. However, if you have to use a
signal-handler, its good to know that there are tools available which can
help out a great deal.
Scott Gifford
Here's a possible solution for sockets. Instead of calling close() on
the file descriptor, Tm calls shutdown(). That leaves the file
descriptor around so the same fd number won't be assigned to a
different connection, but it will be in a state where select() will
find it readable, read will return 0, and write will fail. The child
should notice one of those conditions, and can then exit its select
loop and close the socket safely.
----Scott.
David Schwartz
> Here's a possible solution for sockets. Instead of calling close() on
> the file descriptor, Tm calls shutdown(). That leaves the file
> descriptor around so the same fd number won't be assigned to a
> different connection, but it will be in a state where select() will
> find it readable, read will return 0, and write will fail. The child
> should notice one of those conditions, and can then exit its select
> loop and close the socket safely.
That's a common pattern and it definitely works, but I don't think it
will solve the race condition. If you call 'shutdown' without
synchronizing with the other thread, what happens in the following
case:
1) You are about to call 'shutdown', you have made that decision.
2) The thread returns from 'select' because the connection you were
going to shutdown just happened to close normally from the other side.
3) That thread goes through its set and finds the connection shut down
normally.
4) It calls 'close'.
5) Something happens that results in a new connection being made or
accepted, it gets the same file descriptor as the thread closed in
step 4.
6) We go back to the 'shutdown' thread, and it shuts down the new
connection made in step 5.
Oops.
So you have to synchronize with the thread that can call 'close' to
call 'shutdown' anyway. So how much does 'shutdown' help you? It will
get you out of 'select', but how do you know when it is safe to call
it?
DS
Scott Gifford
> On Dec 28, 9:45 am, Scott Gifford <sgiff...@suspectclass.com> wrote:
>
>> Here's a possible solution for sockets. Instead of calling close() on
>> the file descriptor, Tm calls shutdown(). That leaves the file
>> descriptor around so the same fd number won't be assigned to a
>> different connection, but it will be in a state where select() will
>> find it readable, read will return 0, and write will fail. The child
>> should notice one of those conditions, and can then exit its select
>> loop and close the socket safely.
>
> That's a common pattern and it definitely works, but I don't think
> it will solve the race condition. If you call 'shutdown' without
> synchronizing with the other thread, what happens in the following
> case:
[...]
> So you have to synchronize with the thread that can call 'close' to
> call 'shutdown' anyway. So how much does 'shutdown' help you? It
> will get you out of 'select', but how do you know when it is safe to
> call it?
You're right. Synchronizing the shutdown() and close() is
straightforward, though; trying to use synchronization around a
blocking select is much more difficult.
----Scott.
David Schwartz
> You're right. Synchronizing the shutdown() and close() is
> straightforward, though; trying to use synchronization around a
> blocking select is much more difficult.
>
> ----Scott.
That may be. For example, most of the code in the 'select' loop is
probably performance critical. But the code to close dead connections
is likely not called quite as often. So it may be easier to
synchronize with a call to 'close' than with a call to 'select'.
For example, you can do the following:
1) Set a flag that stops all connections from being closed.
2) Make sure the connection you want to shutdown still exists.
3) Call shutdown.
4) Clear the flag.
This allows the 'select' loop to operate without any synchronization.
A lock need only be held when a connection is added, removed, or shut
down.
In this case, the 'shutdown' function acts to ensure the 'select'
thread breaks out of 'select', the subsequent 'read' gets an error,
and the thread goes on to close down the connection.
You might think that since this requires synchronization in the
'close' area, the 'shutdown' might not help you -- why not just call
'close'? The difference is that a 'close' can race with any other
thread in the system. A 'shutdown' can only race with a thread that
has the right to close down that very connection.
DS
Scott Gifford
> On Dec 28, 5:47 pm, Scott Gifford <sgiff...@suspectclass.com> wrote:
>
>> You're right. Synchronizing the shutdown() and close() is
>> straightforward, though; trying to use synchronization around a
>> blocking select is much more difficult.
>>
>> ----Scott.
>
> That may be. For example, most of the code in the 'select' loop is
> probably performance critical. But the code to close dead connections
> is likely not called quite as often. So it may be easier to
> synchronize with a call to 'close' than with a call to 'select'.
It's not just a matter of performance, though, it's that it's that
select() will hold the lock while it blocks. Consider this pseudocode
(using the OP's Tw for worker thread and Tm for main thread):
Tw.mainloop:
lock(mutex);
select(socket for read);
unlock(mutex);
Tm.cancel:
lock(Tw.mutex);
close(Tw.socket);
unlock(Tw.mutex);
If Tw is blocked in select() and you wish to cancel it, Tm.cancel will
deadlock; it's waiting for the mutex to interrupt Tw, but Tw is
holding the mutex it needs.
I can't think of any way to safely synchronize around a blocking
select().
> For example, you can do the following:
>
> 1) Set a flag that stops all connections from being closed.
> 2) Make sure the connection you want to shutdown still exists.
> 3) Call shutdown.
> 4) Clear the flag.
I think it's possible to synchronize on something thread-specific,
rather than a global flag to stop all new connections. Maybe
something like this:
Tw.mainloop:
while(!error(socket) and !eof(socket)) {
select(socket);
/* ... */
}
lock(Tw.close_mutex);
close(socket);
socket = -1;
unlock(Tw.close_mutex);
Tm.cancel:
lock(Tw.close_mutex);
if (Tw.socket != -1)
shutdown(socket);
unlock(Tw.close_mutex);
[...]
> In this case, the 'shutdown' function acts to ensure the 'select'
> thread breaks out of 'select', the subsequent 'read' gets an error,
> and the thread goes on to close down the connection.
>
> You might think that since this requires synchronization in the
> 'close' area, the 'shutdown' might not help you -- why not just call
> 'close'? The difference is that a 'close' can race with any other
> thread in the system. A 'shutdown' can only race with a thread that
> has the right to close down that very connection.
Yes, exactly. That race on a reused new file descriptor is the error
I made in my suggestion (and some code I'm using, actually), which
Marcin very helpfully pointed out.
----Scott.
Chris Thomasson
news:lyd4sq8...@gfn.org...
FWIW, the synchronizing between a graceful shutdown and a final close on the
socket should resemble "something" like the following:
http://groups.google.com/group/alt.winsock.programming/msg/4ab835a6eaff3947
David Schwartz
wrote:
> It's not just a matter of performance, though, it's that it's that
> select() will hold the lock while it blocks. Consider this pseudocode
> (using the OP's Tw for worker thread and Tm for main thread):
>
> Tw.mainloop:
> lock(mutex);
> select(socket for read);
> unlock(mutex);
Nobody would ever do that under any circumstances. It's more like:
lock(mutex);
copy_fd_sets();
set_in_select_flag(1);
unlock(mutex);
> Tm.cancel:
> lock(Tw.mutex);
> close(Tw.socket);
> unlock(Tw.mutex);
>
> If Tw is blocked in select() and you wish to cancel it, Tm.cancel will
> deadlock; it's waiting for the mutex to interrupt Tw, but Tw is
> holding the mutex it needs.
Nobody would ever hold a mutex while they called a blocking function
in core I/O code unless they were a complete moron. (Or the lock
locked only a single object and there was no better way.)
> I can't think of any way to safely synchronize around a blocking
> select().
See my code above.
> > For example, you can do the following:
>
> > 1) Set a flag that stops all connections from being closed.
> > 2) Make sure the connection you want to shutdown still exists.
> > 3) Call shutdown.
> > 4) Clear the flag.
>
> I think it's possible to synchronize on something thread-specific,
> rather than a global flag to stop all new connections. Maybe
> something like this:
>
> Tw.mainloop:
> while(!error(socket) and !eof(socket)) {
> select(socket);
> /* ... */
> }
> lock(Tw.close_mutex);
> close(socket);
> socket = -1;
> unlock(Tw.close_mutex);
>
> Tm.cancel:
> lock(Tw.close_mutex);
> if (Tw.socket != -1)
> shutdown(socket);
> unlock(Tw.close_mutex);
That works.
> > In this case, the 'shutdown' function acts to ensure the 'select'
> > thread breaks out of 'select', the subsequent 'read' gets an error,
> > and the thread goes on to close down the connection.
> > You might think that since this requires synchronization in the
> > 'close' area, the 'shutdown' might not help you -- why not just call
> > 'close'? The difference is that a 'close' can race with any other
> > thread in the system. A 'shutdown' can only race with a thread that
> > has the right to close down that very connection.
> Yes, exactly. That race on a reused new file descriptor is the error
> I made in my suggestion (and some code I'm using, actually), which
> Marcin very helpfully pointed out.
Glad to hear you've got it sorted out.
DS