Thread-per-socket vs select()
Rob Scullion
finding a definitive answer on Usenet or the web.
I'm wondering what the current wisdom would be on the design
philosophy for a single-process system under Windows NT which needs to
send/receive on 10 to 20 TCP/IP sockets at the same time.
The thread-per-socket approach seems closer to the traditional well
understood accept()/fork() method used in most Unix servers. The
drawback of this approach seems to be the overhead cost of thread
context switching can make this expensive resource-wise and thus cause
it to scale poorly.
The approach involving a single thread per processor select()ing on a
set of active sockets and servicing them as needed, has less overhead.
Unfortunately, if the processing of a given socket's data requires
some significant amount of work, this can cause delays in processing
other sockets. This situation can be worked around but doing so
creates a design which is a lot more complicated than the
thread-per-socket approach.
I'm hoping to keep the implementation as simple as possible, which to
me seems to be the thread-per-socket approach. Given that the proposed
application will only need to deal with a relatively small number of
sockets, it seems that the context-switching overhead shouldn't really
be an issue. I'm just wondering if there's anything beyond the
scalability of this approach that should be taken into consideration.
If there's a good discussion/reference someone could point me at, I'd
be much obliged.
Thanks in advance,
Rob
David M. Lloyd
> The approach involving a single thread per processor select()ing on a
> set of active sockets and servicing them as needed, has less overhead.
> Unfortunately, if the processing of a given socket's data requires some
> significant amount of work, this can cause delays in processing other
> sockets.
Set up a queue. One thread selects on all the fds you are interested in.
When a select returns that a fd is ready, remove the fd from the set and
throw an event into the queue that states, for instance, fd "x" is ready
for reading, and go back to selecting again. Then have a pool of threads
reading from the queue and processing requests. After the threads read or
write to the appropriate fd, re-add the fd to the set using a mutex or
something, and then signal the thread that is selecting to reload its set
and select again (this is somewhat of a simplification but you get the
idea).
This way you only need one thread selecting, but all threads can operate
on the results of it.
- D
Patrick TJ McPhee
Rob Scullion <googl...@ichiban.ichiban.org> wrote:
% The thread-per-socket approach seems closer to the traditional well
% understood accept()/fork() method used in most Unix servers. The
You could just create a new process for each new connection. The overhead
will be similar and you avoid problems with threads contending for
resources -- I would expect a multi-process server to outperform
a thread-per-connection server in most circumstances.
% The approach involving a single thread per processor select()ing on a
% set of active sockets and servicing them as needed, has less overhead.
You can have a single thread select()ing on all active sockets and have
it pass connections to a pool of so-called worker threads which do
whatever needs to be done. I understand you to suggest that each thread
would have a set of sockets associated with it, which doesn't seem
like a good idea.
% Unfortunately, if the processing of a given socket's data requires
% some significant amount of work, this can cause delays in processing
% other sockets.
If you have more than one worker thread, then the threads which aren't
processing that given socket can take up the slack. If adding more
threads will be helpful, then you can add more threads. This doesn't
have to be done by the software -- you can create tuning parameters
and adjust the number of threads until you get the best performance.
If you can break a single socket's data into pieces, you can get
more than one thread working on it simultaneously, so you can
improve the performance of single connections under low load as
well as scaling better.
% I'm hoping to keep the implementation as simple as possible, which to
% me seems to be the thread-per-socket approach.
The only really complicated part of the thread pool approach is
handing the socket back to the select()ing thread when you're done
with it. All the worker threads just sit on a queue waiting for
something to do.
--
Patrick TJ McPhee
East York Canada
pt...@interlog.com
David Schwartz
> Set up a queue. One thread selects on all the fds you are interested in.
> When a select returns that a fd is ready, remove the fd from the set and
> throw an event into the queue that states, for instance, fd "x" is ready
> for reading, and go back to selecting again. Then have a pool of threads
> reading from the queue and processing requests. After the threads read or
> write to the appropriate fd, re-add the fd to the set using a mutex or
> something, and then signal the thread that is selecting to reload its set
> and select again (this is somewhat of a simplification but you get the
> idea).
>
> This way you only need one thread selecting, but all threads can operate
> on the results of it.
Imagine you're handling 1,000 connections and you find work to do on
100 of them. You will wind up restarting the 'select' 100-ish times as
you add each one back to the set. In most cases, you're better off
reading all the data before restarting 'select'. (Not necessarily
processing it, just reading it.)
DS
Steve Watt
David Schwartz <dav...@webmaster.com> wrote:
>"David M. Lloyd" wrote:
>
[ ... ]
Note that he said to remove the fd from the set, which means that it
won't be selected for again "for a while" (until signalled ready again).
Additionally, if you process *all* of the notified fds in the loop
before calling select again (which is the only sensible thing to do),
you won't go through the loop anywhere near 100 times unless the
connections readied at exactly the right (wrong) times.
That model works pretty well for medium large values of file
descriptors, though I prefer the optimization that any thread can be
in the select, and will yank off all of the fds into the queue, start
working on the first one, and broadcast a cv to let the other threads
pick stuff off of the queue. First one who finds the queue empty goes
back into select.
--
Steve Watt KD6GGD PP-ASEL-IA ICBM: 121W 56' 57.8" / 37N 20' 14.9"
Internet: steve @ Watt.COM Whois: SW32
Free time? There's no such thing. It just comes in varying prices...
Frank Swarbrick
I don't have an answer, but I do have a related question.
In Java there doesn't seem to be any kind of 'select' function. Is
there any way to do what the OP is asking *other* than one thread per
socket? Actually, it seems like you'd need two threads per socket. One
to read from the socket and one to read from an external source
(keyboard or whatever) and do a send when there is data to be sent.
I only dabble in Java, but I am curious...
--
Frank Swarbrick -- now powered by SuSE Linux 7.2
"I'm very seldom naughty" --Willow Rosenberg, 'Buffy the Vampire Slayer'
Sean P. Burke
googl...@ichiban.ichiban.org (Rob Scullion) writes:
For 10 to 20 sockets, thread-per-socket will be the simplest
to implement, and the performance will be comparable to any
other approach. Don't create unecessary work for yourself.
If you get to a hundred sockets, you should look at the
work-queue/thread-pool solutions that other posters
have outlined.
-SEan
Juergen Kreileder
> In Java there doesn't seem to be any kind of 'select' function.
J2SE 1.4 has multiplexed I/O:
Take a look at http://java.sun.com/j2se/1.4/docs/guide/nio/index.html
and http://java.sun.com/j2se/1.4/docs/api/java/nio/channels/Selector.html
Juergen
--
Juergen Kreileder, Blackdown Java-Linux Team
http://www.blackdown.org/java-linux/java2-status/
Rob Scullion
I had been looking at the idea of worker threads working off of a
queue to handle the processing. While I think I understand the issues
involved, it is quite a bit more complicated than the
thread-per-socket approach. I guess my real question is this: Beyond
scalability, is there any advantage to this approach over the
conceptually simpler thread-per-socket design? My app will only be
using 10-20 sockets at a time. I'm thinking this might put it in a
realm where the performance gained using the select() method won't
justify the extra design/implementation/debugging time. That changes
if there's another factor beyond scalability of course.
Rob Scullion
> In article <faafd23c.0212...@posting.google.com>,
> Rob Scullion <googl...@ichiban.ichiban.org> wrote:
>
> % The thread-per-socket approach seems closer to the traditional well
> % understood accept()/fork() method used in most Unix servers. The
>
> You could just create a new process for each new connection. The overhead
> will be similar and you avoid problems with threads contending for
> resources -- I would expect a multi-process server to outperform
> a thread-per-connection server in most circumstances.
There are some communication issues unrelated to the sockets which
would make a multiple-process approach more cumbersome than a
multi-threaded approach. As far as performance goes, I'm thinking that
the difference should be slight considering the low number of sockets
I'll be dealing with.
>
> % The approach involving a single thread per processor select()ing on a
> % set of active sockets and servicing them as needed, has less overhead.
>
> You can have a single thread select()ing on all active sockets and have
> it pass connections to a pool of so-called worker threads which do
> whatever needs to be done. I understand you to suggest that each thread
> would have a set of sockets associated with it, which doesn't seem
> like a good idea.
>
I was thinking of allowing for a thread per processor if the machine
contained more than one. I'd try to distribute the open sockets over
the available select threads in an effort to balance the processing
load. I guess I could just as well assume that the other processors
would be given over to the worker threads. In the end, I just wanted
to take maximum advantage of the number of processors available.
> % Unfortunately, if the processing of a given socket's data requires
> % some significant amount of work, this can cause delays in processing
> % other sockets.
>
> If you have more than one worker thread, then the threads which aren't
> processing that given socket can take up the slack. If adding more
> threads will be helpful, then you can add more threads. This doesn't
> have to be done by the software -- you can create tuning parameters
> and adjust the number of threads until you get the best performance.
> If you can break a single socket's data into pieces, you can get
> more than one thread working on it simultaneously, so you can
> improve the performance of single connections under low load as
> well as scaling better.
>
> % I'm hoping to keep the implementation as simple as possible, which to
> % me seems to be the thread-per-socket approach.
>
> The only really complicated part of the thread pool approach is
> handing the socket back to the select()ing thread when you're done
> with it. All the worker threads just sit on a queue waiting for
> something to do.
True, but it is still more complicated. I'm not saying that I couldn't
do it, but the more complicated a design I implement now, the more
expensive it'll be over the product's lifecycle. I figure the
thread-per-socket approach is more easily understood, not only by the
initial implementers, but by the hypothetical programmer who hasn't
even been hired yet :). I guess I'm just trying to get a good feel for
whether or not a 10-20 socket app is going to be significantly better
using the select method. My gut feeling is no, but I'm worried that
there might be issues beyond scalability that I'm not taking into
account with my gut :).
Rob
Rob Scullion
I was kind of wondering at what order of magnitude the
thread-per-socket approach begins to really suffer compared to
select(). As long as it's not at the tens of sockets level, I think
we'll be OK. If we go this route, I'll be sure to make a design note
to the effect that increasing the # of threads to 100 or more should
trigger another look at the design philosophy. Can't see that we'll
ever need to get to that level, but I've learned to never assume :)
Rob
Ziv Caspi
wrote:
>Set up a queue. One thread selects on all the fds you are interested in.
>When a select returns that a fd is ready, remove the fd from the set and
>throw an event into the queue that states, for instance, fd "x" is ready
>for reading, and go back to selecting again. Then have a pool of threads
>reading from the queue and processing requests. After the threads read or
>write to the appropriate fd, re-add the fd to the set using a mutex or
>something, and then signal the thread that is selecting to reload its set
>and select again (this is somewhat of a simplification but you get the
>idea).
>
>This way you only need one thread selecting, but all threads can operate
>on the results of it.
An even better way would be to use IO completion ports. They provide
the functionality you describe, and they scale far better than
select(). Virtually any serious network server software written for
the NT "line" (NT, 2000, XP, .NET) works that way.
Ziv
Frank Swarbrick
>
> Frank Swarbrick <inf...@sprynet.com> writes:
>
> > In Java there doesn't seem to be any kind of 'select' function.
>
> J2SE 1.4 has multiplexed I/O:
>
> Take a look at http://java.sun.com/j2se/1.4/docs/guide/nio/index.html
> and http://java.sun.com/j2se/1.4/docs/api/java/nio/channels/Selector.html
Ah, I only have 1.3 right now, which is probably why I couldn't find it.
Thanks.
Rob Scullion
<snip>
Thanks for all the info folks. I think I've got a good idea of how to
proceed and what to watch out for as well as a better understanding of
the alternatives.
Thanks again!
Rob
Heretic
Horses for courses.. most socket applications are crying out for threads
as the IO is quite slow so having a thread per socket is a very neat way
of using a single processes resources.
Myself I have writen webapplications which run up a couple of hundred
threads on dedicated ports all blocking in accept. Very efficent. My
performance tests show that the conventional single thread doing a
accept then a pthread-create is considabley faster. Strangly the act of
creating and destorying threads appears to graduly slow over time.
Simething I've never got to the bottom off.
A pool of threads is good but if you are going to throttle your
application to n hundred connections (or less) why not just have the
threads suspended in accept() calls?
Regards
Heretic
Rob Scullion wrote:
>
>
>
> I had been looking at the idea of worker threads working off of a
> queue to handle the processing. While I think I understand the issues
> involved, it is quite a bit more complicated than the
> thread-per-socket approach. I guess my real question is this: Beyond
> scalability, is there any advantage to this approach over the
> conceptually simpler thread-per-socket design? My app will only be
> using 10-20 sockets at a time. I'm thinking this might put it in a
> realm where the performance gained using the select() method won't
> justify the extra design/implementation/debugging time. That changes
> if there's another factor beyond scalability of course.
--
If fifty million people say a foolish thing, it's still a foolish thing.
-- Bertrand Russell