pthreads vs. IPC: what to use when?
Christian Näger
when designing an application with multiple threads of execution, one
basically has two options: several processes with IPC, or threads.
In principle, you could fork() a new processes for every thread and
communicate via traditional IPC with no need for threads at all.
On the other hand, you could also build a single application with a
multitude of threads (which communicate via global variables and function
calls) and use only one single process at all.
However, most applications use a mixture of multi-processes and
multi-threads.
Now, in search of a good design, I wonder what (processes or threads) to
use when? What are the advantages of threads over processes and vice versa?
I work under Linux on StrongArm, so speed and memory are issues.
Threads are said to be created and destroyed faster, but Linux forks() are
also fast (and Linux has copy-on-write). Threads can communicate more
easily and faster, but Linux pipes and sockets are very fast (and in case
of sockets, very flexible) and there is also shared memory.
Please share your opinion!
Thanks, Chris
David Butenhof
> when designing an application with multiple threads of execution, one
> basically has two options: several processes with IPC, or threads.
>
> In principle, you could fork() a new processes for every thread and
> communicate via traditional IPC with no need for threads at all.
>
> On the other hand, you could also build a single application with a
> multitude of threads (which communicate via global variables and function
> calls) and use only one single process at all.
Of course; and you could write in C++ with no need for traditional C at
all. Or in traditional C with no need for C++ at all. We could phrase this
essential age-old balance in a multitude of ways. It really comes down to
one thing: when you design, you make choices. That's what design means!
> However, most applications use a mixture of multi-processes and
> multi-threads.
Some certainly do. I wouldn't say "most".
> Now, in search of a good design, I wonder what (processes or threads) to
> use when? What are the advantages of threads over processes and vice
> versa? I work under Linux on StrongArm, so speed and memory are issues.
Then do what's fastest -- except where it uses too much memory. And use
less memory, except when it's too slow. ;-)
> Threads are said to be created and destroyed faster, but Linux forks() are
> also fast (and Linux has copy-on-write). Threads can communicate more
> easily and faster, but Linux pipes and sockets are very fast (and in case
> of sockets, very flexible) and there is also shared memory.
On Linux, threads ARE processes, created using clone() instead of fork();
but fork() is nothing but a particular "mode" of clone(). While any two
processes can share memory, a set of threads exists within a single address
space. This may make the kernel context switch paths faster, depending on
how well the kernel you use is optimized for the full-sharing mode of
clone(). On some other platforms, where "threads" are inherently different
than "processes", you may see even greater benefits... or even less, of
course, depending on how well each is optimized. ;-)
The really basic differences between threads and processes are that
a) With threads, shared memory isn't an add-on; it's pervasive and
convenient. You just read and write variables like normal. This can be
fantastically advantageous for high-bandwidth/frequent communications, such
as fine grained parallelism. This is especially true when there's
relatively little synchronization required -- which complicates and slows
the execution. (And coding.)
b) With processes, each "entity" is an isolated security domain. Failures
in one process don't affect another directly, and the communication
channels (IPC) can be monitored and controlled to avoid contamination from
a failing entity. This can be powerful for servers that need to operate in
multiple security domains or need reliability. (Some have complained that
POSIX mutexes don't allow a locking thread to detect that the previous
owner terminated "abnormally"... that's because it wouldn't do any good to
know without isolation between the entities involved. With IPC mechanisms
it makes sense to know, because you can do something about it.)
Coarse grained parallelism, where security isn't critical, can be managed
either way -- for example, via MPI or OpenMP/threads. One or the other may
give some benefit in performance, scalability, or reliability, depending on
the details.
This is what you need to consider when designing an application. As much as
I like threads, I always experience a twinge when I hear someone say "I've
decided to design a threaded application". You should always set out to
design "an application"... and go on from there to determine what
technologies are most appropriate for that particular application based on
your needs.
/------------------[ David.B...@compaq.com ]------------------\
| Compaq Computer Corporation POSIX Thread Architect |
| My book: http://www.awl.com/cseng/titles/0-201-63392-2/ |
\-----[ http://home.earthlink.net/~anneart/family/dave.html ]-----/
Alex
You should generally fork processes when you don't need too many of
them, and you don't want them to communicate too much. forks are
expensive as each process gets a copy of the parent environment.
Sockets are NOT fast; they are reasonably fast for most applications
though. I believe that it widely accepted that IPC is slow.
Now threads on the other hand share the environment with the process
that creates them, therefore allowing for significantly faster
inter-thread communication. So they should theoretically be both
faster and consume less memory.
But of course this all boils down to design considerations, you should
see best which model or a combination suits your particular problem.
Alex
Christian Näger
> You should generally fork processes when you don't need too many of
> them, and you don't want them to communicate too much. forks are
> expensive as each process gets a copy of the parent environment.
> Sockets are NOT fast; they are reasonably fast for most applications
> though. I believe that it widely accepted that IPC is slow.
Well, I have seen statistics (sorry, can't find the URL) which show that
pipes under Linux are in fact quite fast (several times faster than under
windows). However, I don't know if this is "sufficient" or how sockets
compare to pipes (shouldn't make much difference, but dunno).
> Now threads on the other hand share the environment with the process
> that creates them, therefore allowing for significantly faster
> inter-thread communication. So they should theoretically be both
> faster and consume less memory.
But, I think this is not necessarily true for Linux. As David pointed out,
threads are implemented with processes. Processes also don't really get a
copy of the parent's environment, it only looks like that. In fact only
memory segments which are modified are duplicated (copy-on-write). So, I
doubt that threads are faster or consume less memory (under Linux!).
As a conclusion, it boils down, to the level of communication if it's
low-bandwidth or can be serialized easily use processes, otherwise use
threads (if you need frequent function calls between the different threads
or some manipulation of a shared complex data structure).
Additionaly, if your software components need to be able to fail
independently, use processes. In a multi-threaded app, if one thread
crashes, it brings down the whole application.
Chris
Christian Näger
> This is what you need to consider when designing an application. As much
> as I like threads, I always experience a twinge when I hear someone say
> "I've decided to design a threaded application". You should always set out
> to design "an application"... and go on from there to determine what
> technologies are most appropriate for that particular application based on
> your needs.
Thanks a lot for your answer!
Is there something that prevents the use of Linux threads on StrongArm
which does not have a floating point unit for example?
(I far as I can see, libpthread.so only adds an additional 106K, which
should be no problem)?
Chris
Alex
> Alex wrote:
>
> > You should generally fork processes when you don't need too many of
> > them, and you don't want them to communicate too much. forks are
> > expensive as each process gets a copy of the parent environment.
> > Sockets are NOT fast; they are reasonably fast for most applications
> > though. I believe that it widely accepted that IPC is slow.
>
> Well, I have seen statistics (sorry, can't find the URL) which show that
> pipes under Linux are in fact quite fast (several times faster than under
> windows). However, I don't know if this is "sufficient" or how sockets
> compare to pipes (shouldn't make much difference, but dunno).
I'm not sure about exact statistics, but imagine a task where all you need
access to a huge chunk of data, to which you only realistically need a
pointer. Would you really want to make an exact copy of it using IPC? (Of
course the producer process can do the active processing, but that's again a
design issue...). No matter how fast it is, if you dealing with a lot of
data this is overkill.
> > Now threads on the other hand share the environment with the process
> > that creates them, therefore allowing for significantly faster
> > inter-thread communication. So they should theoretically be both
> > faster and consume less memory.
>
> But, I think this is not necessarily true for Linux. As David pointed out,
> threads are implemented with processes. Processes also don't really get a
> copy of the parent's environment, it only looks like that. In fact only
> memory segments which are modified are duplicated (copy-on-write). So, I
> doubt that threads are faster or consume less memory (under Linux!).
I am not going to argue with David, if he says it is so, it must be so :)
But, even using copy-on-write, you are assuming that you are going to be
working mostly with static data, otherwise, still memory issues. But this
does bring up the point of actual vendor implementations, this is why I said
"theoretically". And still, this mostly depends on the task you are doing. I
think that it is safe to assume that threads are _at least_ as fast, and
most of the time faster than forked processes.
> As a conclusion, it boils down, to the level of communication if it's
> low-bandwidth or can be serialized easily use processes, otherwise use
> threads (if you need frequent function calls between the different threads
> or some manipulation of a shared complex data structure).
Yes.
> Additionaly, if your software components need to be able to fail
> independently, use processes. In a multi-threaded app, if one thread
> crashes, it brings down the whole application.
>
Correct. But should not be used a license to freely corrupt memory and
end process execution by dereferencing NULL :)
Alex