Using select(2) and pthread_cond_wait()/sem_wait() simultaneously
ge...@my-deja.com
pthreads. Threads will exchange information between them selfs and use
tcp to communicate with processes on other computers.
I expect to use select(2) to check for and wait for data on the tcp
sockets, and perhaps use a condition variable or a semaphore to check
for and wait for messages from other threads.
How can I combine these two operations, i.e. how can I wait both for
data on a tcp socket and messages from another thread?
Tips on how to solve this problem is much appreciated.
Thanks,
Geir Pedersen.
Sent via Deja.com http://www.deja.com/
Before you buy.
Ian Collins
>
> I am planning a Linux application that will use a small number of
> pthreads. Threads will exchange information between them selfs and use
> tcp to communicate with processes on other computers.
>
> I expect to use select(2) to check for and wait for data on the tcp
> sockets, and perhaps use a condition variable or a semaphore to check
> for and wait for messages from other threads.
>
> How can I combine these two operations, i.e. how can I wait both for
> data on a tcp socket and messages from another thread?
>
buffers and use messages to indicate that they are ready to be used.
Ian
Kaz Kylheku
>How can I combine these two operations, i.e. how can I wait both for
>data on a tcp socket and messages from another thread?
Use a thread which waits on the condition variable, and another thread which is
dedicated to the I/O multiplexing. That's one thing that threads are for; if
you can't multiplex waiting on two things, you can use two threads.
--
Any hyperlinks appearing in this article were inserted by the unscrupulous
operators of a Usenet-to-web gateway, without obtaining the proper permission
of the author, who does not endorse any of the linked-to products or services.
Dan Maas
> you can't multiplex waiting on two things, you can use two threads.
Make that "If the operating system's API is so brain-dead that you can't
wait on an arbitrary set of objects, you can kludge it with threads."
They're all brain-dead; pthread condition variables don't mix with
semaphores, message queues, or file descriptors; Win32 has a more unified
API but you still can't multiplex on FIFOs.**
This is a *fundamental flaw* in the UNIX API -- the original goal of making
I/O as stateless as possible (select/poll) has resulted in an interface that
is, in hindsight, inferior to the Windows-style model of building up the
waitable set as you go. (Contrast select/poll with WSAAsyncSelect; the
latter is more scalable and integrates seamlessly with other waitable
objects)
Cascading several threads to simulate true multiplexed waiting is an
inelegant hack for poorly-designed I/O API's. Every study that has looked
into this concluded that non-blocking state-machine I/O systems are more
scalable (though slightly harder to code) than those that use one thread per
I/O source. The difficulty of programming state-machine systems can be
conquered at user level with a library or interpreter that emulates
autonomous threads a la N-by-M threading. On real SMP systems, true
concurrency can be orthogonally exploited by running several non-blocking
engines in parallel.
Sorry for the rant; I don't mean any disrespect to Gier, Ian, or Kaz. I
simply think we've just brought to light one of the "dirty secrets" of how
people use threads =).
BTW, to answer Geir's original question... How about sticking with one I/O
transport for your whole system, namely sockets? My benchmarks indicate
that, on Linux, UNIX domain sockets are faster for IPC than you'll probably
ever need. My sockets go about 60% as fast as raw memcpy() -- that's
~120MB/sec -- and you can go to bed earlier since the kernel takes care of
all locking/synchronization for you =)
Dan
** IRIX's native threading API let you obtain a waitable file descriptor
from a condition variable, solving one of those problems...
Ian Collins
> > That's one thing that threads are for; if
> > you can't multiplex waiting on two things, you can use two threads.
>
> Make that "If the operating system's API is so brain-dead that you can't
> wait on an arbitrary set of objects, you can kludge it with threads."
> They're all brain-dead; pthread condition variables don't mix with
> semaphores, message queues, or file descriptors; Win32 has a more unified
> API but you still can't multiplex on FIFOs.**
>
> This is a *fundamental flaw* in the UNIX API -- the original goal of making
> I/O as stateless as possible (select/poll) has resulted in an interface that
> is, in hindsight, inferior to the Windows-style model of building up the
> waitable set as you go. (Contrast select/poll with WSAAsyncSelect; the
> latter is more scalable and integrates seamlessly with other waitable
> objects)
>
The problem with this approach is that when ever you add some new form of
waitable object, you have to re-jig existing kernel code to integrate the new
object.. Otherwise you end up with a partial solution with exceptions, FIFOs
on NT are a good example.
In the Unix world, features have evolved and been added over time, select/poll
preceded threads by many years (decades even), so it hardly surprising they
don't have very clean (ok, any) interactions.
The ideal OS would have a waitable object base that all waitable objects
derived from. May be one day we will see such a beast.
>
> Cascading several threads to simulate true multiplexed waiting is an
> inelegant hack for poorly-designed I/O API's. Every study that has looked
> into this concluded that non-blocking state-machine I/O systems are more
> scalable (though slightly harder to code) than those that use one thread per
> I/O source. The difficulty of programming state-machine systems can be
> conquered at user level with a library or interpreter that emulates
> autonomous threads a la N-by-M threading. On real SMP systems, true
> concurrency can be orthogonally exploited by running several non-blocking
> engines in parallel.
>
There is nothing wrong with the APIs, they just don't assume each other's
existence. Pthreads can exist under any OS, or even be the OS. With a tightly
coupled model, the OS and the libraries can not exist in isolation. This is a
design call, you can't say one is better than the other. NT does it one way,
so you get a nice friendly integrated environment - but only with one OS.
>
> Sorry for the rant; I don't mean any disrespect to Gier, Ian, or Kaz. I
> simply think we've just brought to light one of the "dirty secrets" of how
> people use threads =).
>
None taken, a good rant a day keep the ulcers away!
>
> BTW, to answer Geir's original question... How about sticking with one I/O
> transport for your whole system, namely sockets? My benchmarks indicate
> that, on Linux, UNIX domain sockets are faster for IPC than you'll probably
> ever need. My sockets go about 60% as fast as raw memcpy() -- that's
> ~120MB/sec -- and you can go to bed earlier since the kernel takes care of
> all locking/synchronization for you =)
>
I've tried this, and is does work rather well in abstracting the interface so
the objects/threads can exist on one or many hosts.
Ian
Dan Maas
> waitable object, you have to re-jig existing kernel code to integrate the
new
> object.. Otherwise you end up with a partial solution with exceptions,
FIFOs
> on NT are a good example.
True; although the "re-jigging" could be quite minor; I believe a simple
mechanism like the Linux's struct file_operations would work. (but then
again I'm talking about ripping up lots of kernel code already =)
I'd love to find a nice solution to the following problem... Say you have
two libraries that want to perform non-blocking I/O (eg a DNS resolver and a
CORBA ORB). How can a programmer easily use both together?
UNIX says: ask the library what file descriptors it cares about, then write
your own I/O loop and call back into the libraries as appropriate... This is
the best answer I've seen yet (Windows says: "Screw you, use threads" =),
although it will break if, say, one of the libraries also wants to wait on a
non-file-descriptor object. Also, adding another library means rewriting
your loop code.
I propose a solution more along the lines of POSIX RT signals (that lovely
new I/O model the Linux camp is raving about that bears a funny resemblance
to Windows events =). Have each library declare a set of objects on which it
wants to receive I/O completions, as well as a signal handler function. Then
your own code just calls dispatch_next_io_event(), which returns into the
appropriate signal handler. Since dispatching has been pushed down into the
kernel, the I/O loop is trivial, and new libraries can be added without
changing the core code. (The whole app remains single-threaded, avoiding the
illusion of concurrency that other models present =).
I'm not fully satisfied with the system I just described though, mainly
because it demands that your code conform to a non-concurrent, non-blocking
(state machine) I/O model. It seems to achieve a seamless interface by
sacrificing the flexibility of true multithreading. Say you want to write a
scalable web-server using the non-blocking DNS and CORBA libraries from
above... On SMP machines, you'd want to run several of these state machine
processes in parallel. Now what has to be done so that they can share one
common file cache? Add another library for I/O to the cache, complete with
the ability to block on semaphores for synchronization? Perhaps that would
work, but the whole design strikes me as a bit odd. How should
responsibilities be distributed among the threads? I dont' know. And
programming the system has degenerated from nice, procedural code into
various libraries of non-concurrent I/O handlers. (It feels like I'm groping
around for some deep principle about the nature of concurrency here, but I'm
too dumb to see it =)
> The ideal OS would have a waitable object base that all waitable objects
> derived from. May be one day we will see such a beast.
Mach - 1985! (grinning, ducking, running...)
Best regards,
Dan
Ian Collins
friendly GUI toolkit - Sun's XView.
> I'm not fully satisfied with the system I just described though, mainly
> because it demands that your code conform to a non-concurrent, non-blocking
> (state machine) I/O model. It seems to achieve a seamless interface by
> sacrificing the flexibility of true multithreading. Say you want to write a
> scalable web-server using the non-blocking DNS and CORBA libraries from
> above... On SMP machines, you'd want to run several of these state machine
> processes in parallel. Now what has to be done so that they can share one
> common file cache? Add another library for I/O to the cache, complete with
> the ability to block on semaphores for synchronization? Perhaps that would
> work, but the whole design strikes me as a bit odd. How should
> responsibilities be distributed among the threads? I dont' know. And
> programming the system has degenerated from nice, procedural code into
> various libraries of non-concurrent I/O handlers. (It feels like I'm groping
> around for some deep principle about the nature of concurrency here, but I'm
> too dumb to see it =)
>
Sounds like a step back in time to me. You'd be better of with one
thread per IO 'family' multiplexing into a sinle CV. Each family has
its own bit in a bit mask and the application uses these to determine
which family is ready. I used this to simulate the interrupt structure
of a communications processor ages ago and have found it to be a good
general purpose solution.
> > The ideal OS would have a waitable object base that all waitable objects
> > derived from. May be one day we will see such a beast.
>
> Mach - 1985! (grinning, ducking, running...)
>
Ouch!
Ian
Dan Maas
> thread per IO 'family' multiplexing into a sinle CV. Each family has
> its own bit in a bit mask and the application uses these to determine
> which family is ready. I used this to simulate the interrupt structure
> of a communications processor ages ago and have found it to be a good
> general purpose solution.
Interesting... It does sound reasonable to split into processes on
boundaries of functionality. I was never a big fan of shared-memory
multiprocessing anyway; I'm starting to prefer the idea of many independent
processes with small local working sets -- leave data transport and
synchronization up to the kernel.
(A microkernel? What's that? =)
Thanks for the comments,
Dan
Joerg Faschingbauer
> I am planning a Linux application that will use a small number of
> pthreads. Threads will exchange information between them selfs and use
> tcp to communicate with processes on other computers.
>
> I expect to use select(2) to check for and wait for data on the tcp
> sockets, and perhaps use a condition variable or a semaphore to check
> for and wait for messages from other threads.
>
> How can I combine these two operations, i.e. how can I wait both for
> data on a tcp socket and messages from another thread?
You cannot - at least not straightforwardly.
In every reasonable threads book there is an example of how to pass
data between threads, using a mutex and two condition variables
guarding a queue. Nothing to do with eventdrivenness (which is what
your select() loop is for - every return from select() resembles an
event).
Here's what I have.
You have thread A sending data to thread B (using one such queue), and
thread B wants to be notified of that. The only way to do this under
Unix (and I don't consider signals a way of doing it) is to use
select(). Thread B has to employ a socketpair(), waiting for events on
one side. The other side of the socket pair is open for every other
thread to send small messages of a defined format (writes of
reasonable small chunks of data atomic, so there's no need to
synchronize).
Thread A, after having written the data to the queue, writes a small
message to the receiving thread's socketpair client end. The message
contains things like the address or some identifier of the queue
object where the data have to be read (thread B might want to wait for
more than one queue).
Eventually, you would not want to write to the socket everytime you
write to the queue, but rather only if the queue was empty before.
The concept can be nicely abstracted, and extended to wait for
output-events (the third parameter to select(), meaning "the queue has
available room to be filled"). There are also some subtle issues
involved - if you like we can discuss in more detail.
Yours,
Joerg