BSD vs Linux Threads
Jack J. Woehr
(if any) in threads under BSD vs Linux?
--
Jack J. Woehr # The Drug War is Race War
PO Box 51, Golden, CO 80402 # The Drug War is Class War.
jwo...@ibm.net j...@well.com # The Drug War is Civil War.
http://www.well.com/~jax/rcfb # Arrest the War on Drugs.
Chuck
> Can anyone discourse on difference in implementation
> (if any) in threads under BSD vs Linux?
vs. solaris vs. irix which are the two that I've seen that do it
best.
Clifton Royston
I was waiting to see more discussion on this, since I have no
first-hand experience, but as far as I know, *BSD currently implement
Posix-style threads only in user-land (application-space) whereas Linux
implements them in kernel space.
This is a recurrent issue for an app like Squid which has an option
to configure using threads for I/O (--async-io) and assumes they're in
kernel space. Helps performance somewhat on Linux, but if you use that
configure option on FreeBSD (for instance) you get slaughtered on CPU
with no performance advantage.
-- Clifton
--
Clifton Royston -- LavaNet Systems Architect -- clif...@lava.net
"An absolute monarch would be absolutely wise and good.
But no man is strong enough to have no interest.
Therefore the best king would be Pure Chance.
It is Pure Chance that rules the Universe;
therefore, and only therefore, life is good." - AC
Corey Brenner
> > best.
>
>I was waiting to see more discussion on this, since I have no
>first-hand experience, but as far as I know, *BSD currently
>implement Posix-style threads only in user-land (application-space)
>whereas Linux implements them in kernel space.
Right. BSD uses a userland thread library to schedule threads, while
Linux' threads are scheduled just like normal "processes" in the
kernel.
This means that, for BSD to properly handle threads, async I/O must
be used and handled by the thread library, otherwise when a single
thread blocks for I/O, the whole process blocks, and no other work
may be done until the I/O request is complete. But, properly
managing async I/O from multiple callers is kinda hard (at least to
do it efficiently).
Linux, on the other hand, with the kernel-thread model, assigns a
kernel "task" to each executing thread in a program, and simply
allows the threads to share the vm map, fd table, etc. among them-
selves (the clone() syscall under Linux handles this, taking diff-
erent flags for emulation of fork(), vfork(), etc.). Threaded
programs under Linux, then, can use regular blocking I/O, managed
by the kernel, to do what has to be emulated on other systems in
the thread libraries.
Solaris (used to?) uses a hybrid method whereby multiple
userland-scheduled threads are bound to a kernel-level "lightweight
process", of which there can be more than one designated a given
"normal process".
Basically, Solaris' threads are (were?) run on a two-tiered scheduler,
with the system kernel handling I/O for a group of threads running in
a process. I'm pretty sure the basic I/O primitives were wrapped in
their thread library (like they are in BSD) and made use of async
I/O, those signals being caught and handled by the thread scheduler
in userland to wake up threads blocked for I/O when a request was
done. Multiple paths of I/O, happening on multiple LWPs (scheduled
on possibly multiple CPUs, possibly completely concurrently) were
therefore possible. Not sure how big a difference (or penalty) multi-
threads makes on a single CPU system, though, in any of the models.
Irix probably follows a similar model to Solaris, but I don't know
for sure.
AFAIK, Digital has a decent implementation of pthreads, at least. One
of their guys, Dave Butenhof, is one of the main architects of the
standard. I think, though, that Digital's thread package operates
like that of BSD, completely in userland.
Just as a follow-on question, does anyone know the current status
of threads in Solaris? They had a good, solid thread library before
pthreads was standardized which included the ability to force another
thread to sleep (pthreads, AFAIK, includes no such functionality).
How difficult would it be to implement a mezzanine-scheduled thread
package for OpenBSD, given that there is not yet support for MP?
Would it even be worth doing on a single CPU?
What I've been wondering about is, having the main thread of a
program (the one active when main() is called) register an entry
point with the kernel, much like a signal handler. When that
process is being switched to by the system's scheduler, it will
make a check for that entry point (skip if NULL), and will enter
that code so that the userland thread library can perform any
cleanups, etc. that it needs to accomplish before the actual program
code is entered again.
This entry code would run in user mode, obviously, and would be
scheduled like normal by the kernel (except that, if the main thread
was preempted by the kernel while executing that code, there would
have to be some sort of flag set on the process and all its peers so
that none of them could be run until the thread scheduler/library
entry point had returned normally). Upon exit from that handler, the
process' normal code could be set in motion for the balance of that
thread's time quantum.
Children of the main thread (in a kernel-thread/user-thread hybrid
scheme) would not be able to register a handler, then. Such a system
might make possible some interesting GC systems in userland code, too.
--Corey
______________________________________________________
Get Your Private, Free Email at http://www.hotmail.com
Monty Brandenberg
> AFAIK, Digital has a decent implementation of pthreads, at least. One
> of their guys, Dave Butenhof, is one of the main architects of the
> standard. I think, though, that Digital's thread package operates
> like that of BSD, completely in userland.
Nope. Digital Unix (er, Tru64) was developed from Mach and has had
kernel based threads from day 1. Happy little processless kernel
threads are also available if you like, say, driver work threads,
etc. And no sproc hackery... Also, another userspace abstraction
called TIS is available though that may be deprecated these days.
--
Monty Brandenberg, Software Consultant MCB, Inc.
mcb...@world.std.com P.O. Box 426188
mcb...@ne.mediaone.net Cambridge, MA 02142-0021
617.864.6907
Trevor Schroeder
> Solaris (used to?) uses a hybrid method whereby multiple
> userland-scheduled threads are bound to a kernel-level "lightweight
> process", of which there can be more than one designated a given
> "normal process".
Which always seemed silly to me... Just FWIW, there are threads packages
(later versions of C threads?) that would do this with Mach.
> therefore possible. Not sure how big a difference (or penalty) multi-
> threads makes on a single CPU system, though, in any of the models.
Actually, multiple threads can often speed up execution, even on a single
CPU system. It's not that they're somehow magical, but the use of the
thread model allows you to essentially employ a state machine approach
without the headache associated with manually managing moving from state
to state.
> AFAIK, Digital has a decent implementation of pthreads, at least. One
> of their guys, Dave Butenhof, is one of the main architects of the
> standard. I think, though, that Digital's thread package operates
> like that of BSD, completely in userland.
I'm not so sure. I believe their earlier OSF/1 implementations did (as did
OpenVMS's prior to 7.0), but I'm given to understand that it tracked
Mach's development of C threads fairly closely and when C threads allowed
you to spawn multiple Mach threads, so did DU / OSF/1. But then again, I
heard this second hand in the first place, so I could certainly be wrong.
> Just as a follow-on question, does anyone know the current status
> of threads in Solaris? They had a good, solid thread library before
They've got nice threads support. POSIX. Some soft realtime. Never used
'em myself. ;)
> pthreads was standardized which included the ability to force another
> thread to sleep (pthreads, AFAIK, includes no such functionality).
You might be able to use pthread_kill() to do such a thing.
..........................................................................
: "I knew it was going to cost me my head and also my swivel chair, but :
: I thought: What the hell--better men than I have risked their heads :
: and their swivel chairs for truth and justice." -- James P. Cannon :
:........... http://www.zweknu.org/ for PGP key and more ................:
Joe Abley
> On Tue, Dec 21, 1999 at 02:32:43PM -0800, Chuck wrote:
> > Quoting Jack J. Woehr (jwo...@ibm.net):
> > > Can anyone discourse on difference in implementation
> > > (if any) in threads under BSD vs Linux?
> >
> > best.
>
> I was waiting to see more discussion on this, since I have no
> first-hand experience, but as far as I know, *BSD currently implement
> Posix-style threads only in user-land (application-space) whereas Linux
> implements them in kernel space.
In case you're interested, this is an area of active development in
FreeBSD (which is not to say it isn't active elsewhere). Walnut Creek
seem to have employed a developer specifically to implement a modern
threads implementation. Much discussion of the issues on the freebsd-arch
mailing list, which you should find archived somewhere at www.freebsd.org.
No running code yet, to my knowledge, but lots of good-sounding words :)
Joe
--
Ua lawa küpono ka hakahaka pä o këia pä malule
David Leonard
While ever openbsd does not support multiple processors, work will probably
be concentrated on the user-land thread implementation.
> This means that, for BSD to properly handle threads, async I/O must
> be used and handled by the thread library, otherwise when a single
> thread blocks for I/O, the whole process blocks, and no other work
> may be done until the I/O request is complete. But, properly
> managing async I/O from multiple callers is kinda hard (at least to
> do it efficiently).
I can't fully agree with this. Disk and network writes are typically
io enqueuings in the kernel. To me, the argument that you need kernel threads
to efficiently pump data out doesn't hold water. The user level scheduler
does an adequate job of starting the io - the kernel manages the io.
The user-land scheduler is very careful to ensure that none of the (wrapped)
syscalls block on i/o. If it does block or delay, then thats a problem of
the kernel not honouring O_NONBLOCK in the way that I expect.
Unfortunately, now I think about it, syscalls like fsync() will block
the other threads.
Where the user-level threads lose, for me, is in the syscall overhead of
calling gettimeofday() for each re-schedule, and in the signal overhead of the
scheduler's itimer. Otherwise I/O is handled well on a uniprocessor
user-level scheduler (but don't mention bpf).
BTW there is a directory called BENCH under libc_r. If you want to port it to
linux and compare it on the same machine, be my guest. Its mainly testing
primitives like mutex locking, but some i/o benchmarks would be interesting
to write - hmmm you'd need some way to distinguish the components of
the measurements - kernel i/o improvements and thread i/o improvements...
> What I've been wondering about is, having the main thread of a
> program (the one active when main() is called) register an entry
> point with the kernel, much like a signal handler. When that
> process is being switched to by the system's scheduler, it will
> make a check for that entry point (skip if NULL), and will enter
> that code so that the userland thread library can perform any
> cleanups, etc. that it needs to accomplish before the actual program
> code is entered again.
>
> This entry code would run in user mode, obviously, and would be
> scheduled like normal by the kernel (except that, if the main thread
> was preempted by the kernel while executing that code, there would
> have to be some sort of flag set on the process and all its peers so
> that none of them could be run until the thread scheduler/library
> entry point had returned normally). Upon exit from that handler, the
> process' normal code could be set in motion for the balance of that
> thread's time quantum.
>
> Children of the main thread (in a kernel-thread/user-thread hybrid
> scheme) would not be able to register a handler, then. Such a system
> might make possible some interesting GC systems in userland code, too.
hmm. this sounds a lot like what currently happens with existing signal
handling...
btw look at pthread_switch_add_np() in
src/lib/libc_r/uthread/uthread_switch_np.c. you can write your own
thread-space fiber scheduler (inside a user-space thread sceduler)!
d
--
David Leonard David....@csee.uq.edu.au
Dept of Comp. Sci. and Elec. Engg _ Room:78-624 Ph:+61 7 336 52447
The University of Queensland |+| http://www.csee.uq.edu.au/~leonard/
QLD 4072 AUSTRALIA ~` '~ E2A24DC6446E5779D7AFC41AA04E6401
Trevor Schroeder
> user-level scheduler (but don't mention bpf).
*smirk* Ask me about BPF and threads if you're really interested... :/ ;)
It's a veritable minefield trying to use both.