Tcl Threads
USCode
How does the Tcl implementation of Threads compare to other
implementations such as in Java or .NET?
Is Threading in Tcl as robust or maybe even more-so than those other 2?
From what I've read, Threads originally came from the development of
AOLServer, so I imagine it must be very effective (for lack of a better
word)?
Anyone have any experience with threading with Tcl/Java/.NET who can
compare/contrast their strengths/weaknesses?
Thanks
tom.rmadilo
As someone who used AOLserver for a decade before using Tcl threads, I
can say that I thought the same thing. I've even said it. Threads came
from AOLserver. But apparently, there are major differences. The
documentation is ambiguous for Tcl threads. AOLserver uses pthreads,
or whatever is the most efficient threading for the platform.
Another difference is that in AOLserver you don't really think about
threads, they are managed by the environment, but you can configure
many details, if you find the default configuration doesn't work for
you.
Of course, AOLserver is an application server, optimized as an HTTP
server, so it comes with baggage.
If you are writing a client/server, the AOLserver API is a good
choice. I have no experience in other areas.
USCode
> As someone who used AOLserver for a decade before using Tcl threads, I
> can say that I thought the same thing. I've even said it. Threads came
> from AOLserver. But apparently, there are major differences. The
> documentation is ambiguous for Tcl threads. AOLserver uses pthreads,
> or whatever is the most efficient threading for the platform.
>
> Another difference is that in AOLserver you don't really think about
> threads, they are managed by the environment, but you can configure
> many details, if you find the default configuration doesn't work for
> you.
>
> Of course, AOLserver is an application server, optimized as an HTTP
> server, so it comes with baggage.
>
> If you are writing a client/server, the AOLserver API is a good
> choice. I have no experience in other areas.
>
I was just curious as to how Tcl Threads compares to other
implementations such as Java and .NET.
Alexandre Ferrieux
> Thanks for the info. Tom, helps put things into perspective.
> I was just curious as to how Tcl Threads compares to other
Maybe one salient aspect: Tcl threads resemble little "subprocesses",
in that they host independent Tcl interpreters, which have non risk of
accidental concurrent access to data (no shared vars, so you won't
need a mutex, and [thread send] everywhere if you need to
synchronize). So in a sense, Tcl threads are "higher level" than Java
ones (no idea about .NET).
-Alex
USCode
> Maybe one salient aspect: Tcl threads resemble little "subprocesses",
> in that they host independent Tcl interpreters, which have non risk of
> accidental concurrent access to data (no shared vars, so you won't
> need a mutex, and [thread send] everywhere if you need to
> synchronize). So in a sense, Tcl threads are "higher level" than Java
> ones (no idea about .NET).
>
> -Alex
http://tcl.cvs.sourceforge.net/*checkout*/tcl/thread/doc/html/thread.html
and it lists mutex commands.
Does that mean hosting an independent Tcl interpreter in the thread is
optional?
Or are there other uses for mutexes with an threaded independent Tcl
interpreter?
Thanks!
Alexandre Ferrieux
> Alexandre Ferrieux wrote:
> > Maybe one salient aspect: Tcl threads resemble little "subprocesses",
> > in that they host independent Tcl interpreters, which have non risk of
> > accidental concurrent access to data (no shared vars, so you won't
> > need a mutex, and [thread send] everywhere if you need to
> > synchronize). So in a sense, Tcl threads are "higher level" than Java
> > ones (no idea about .NET).
>
> > -Alex
>
> and it lists mutex commands.
What I meant was that you don't need a mutex for the most frequent
shared resource, which is plain memory (vars in Tcl), since the
"address spaces" of Tcl threads are as separate as can be (though they
share the same underlying OS-level address space of course). But you
may still need a mutex for other resources (like a file), like you
would do between process with a system-wide C-level mutex.
But just like system-wide mutexes are available in C but seldom used,
you can do an awful lot of things with Tcl threads without needing to
call [mutex]. For example, a simple method is to dedicate a resource
to one thread, which listens to stimuli from others saying [thread
send] and serializes access to the resource.
Not using [mutex] explicitly also lessens the risk of deadlock, though
of course it doesn't bring it to zero.
Indeed, a deadlock can be produced in all cases by introducing a cycle
in a dependency graph; the graph is just slightly (!) harder to "read"
from mutex ordering than from "threadA-waits-for-threadB"
relationships.
-Alex
USCode
> What I meant was that you don't need a mutex for the most frequent
> shared resource, which is plain memory (vars in Tcl), since the
> "address spaces" of Tcl threads are as separate as can be (though they
> share the same underlying OS-level address space of course). But you
> may still need a mutex for other resources (like a file), like you
> would do between process with a system-wide C-level mutex.
>
> But just like system-wide mutexes are available in C but seldom used,
> you can do an awful lot of things with Tcl threads without needing to
> call [mutex]. For example, a simple method is to dedicate a resource
> to one thread, which listens to stimuli from others saying [thread
> send] and serializes access to the resource.
>
> Not using [mutex] explicitly also lessens the risk of deadlock, though
> of course it doesn't bring it to zero.
> Indeed, a deadlock can be produced in all cases by introducing a cycle
> in a dependency graph; the graph is just slightly (!) harder to "read"
> from mutex ordering than from "threadA-waits-for-threadB"
> relationships.
>
> -Alex
>
Thanks Alex!
tom.rmadilo
wrote:
> What I meant was that you don't need a mutex for the most frequent
> shared resource, which is plain memory (vars in Tcl), since the
> "address spaces" of Tcl threads are as separate as can be (though they
> share the same underlying OS-level address space of course). But you
> may still need a mutex for other resources (like a file), like you
> would do between process with a system-wide C-level mutex.
>
> But just like system-wide mutexes are available in C but seldom used,
> you can do an awful lot of things with Tcl threads without needing to
> call [mutex]. For example, a simple method is to dedicate a resource
> to one thread, which listens to stimuli from others saying [thread
> send] and serializes access to the resource.
>
> Not using [mutex] explicitly also lessens the risk of deadlock, though
> of course it doesn't bring it to zero.
> Indeed, a deadlock can be produced in all cases by introducing a cycle
> in a dependency graph; the graph is just slightly (!) harder to "read"
> from mutex ordering than from "threadA-waits-for-threadB"
> relationships.
This sounds nearly identical to AOLserver, the real unknown is how
different Tcl Threads is from AOLserver threads. AOLserver pretty much
follows the Steven's IPC and Networking APIs. Read those books and you
will understand the AOLserver API. But AOLserver adds some additional
APIs, such as nsv (shared arrays). These are variables which don't
need to be initialized like a traditional mutex, similar to Tcl vars
which initialize on first use. Tcl Threads has a similar tsv, but I
haven't used them, so I can't say how similar they are. They are used
to share data between threads. The concept of nsv/tsv is that you can
group mutexes, so you don't have to allocate them for every case. That
is a huge simplification for shared data, it is something like an
access pool, and you probably don't have this in Java or .NET.
Basically this recognizes the fact that there are different types of
exclusive access, and also that access in general is a shared system,
not just the resources, but the system is shared. The nsv/tsv
variables implement a shared 'system' to share resources, very
different from the basic mutex type locking.
It would be helpful to have more information on the Tcl Threads
implementation. My experience with performance testing indicates that
at least threads are assigned to all processors on the system, but I
have no experience with tsv's or other uses of Tcl Threads. My general
impression is that they work well, they just lack documentation.
Alexandre Ferrieux
> On Feb 14, 2:02 pm, Alexandre Ferrieux <alexandre.ferri...@gmail.com>
> wrote:
>
>
>
> > What I meant was that you don't need a mutex for the most frequent
> > shared resource, which is plain memory (vars in Tcl), since the
> > "address spaces" of Tcl threads are as separate as can be (though they
> > share the same underlying OS-level address space of course). But you
> > may still need a mutex for other resources (like a file), like you
> > would do between process with a system-wide C-level mutex.
>
> > But just like system-wide mutexes are available in C but seldom used,
> > you can do an awful lot of things with Tcl threads without needing to
> > call [mutex]. For example, a simple method is to dedicate a resource
> > to one thread, which listens to stimuli from others saying [thread
> > send] and serializes access to the resource.
>
> > Not using [mutex] explicitly also lessens the risk of deadlock, though
> > of course it doesn't bring it to zero.
> > Indeed, a deadlock can be produced in all cases by introducing a cycle
> > in a dependency graph; the graph is just slightly (!) harder to "read"
> > from mutex ordering than from "threadA-waits-for-threadB"
> > relationships.
>
> This sounds nearly identical to AOLserver, the real unknown is how
> different Tcl Threads is from AOLserver threads.
No. The OP asked about the comparison with other generic frameworks,
and only mentioned AOLserver as an acknowledge of performance. I have
tried to characterize Tcl threads ex nihilo and in comparison to those
frameworks, not in comparison to whatever you're fond of.
> It would be helpful to have more information on the Tcl Threads
> implementation.
Read The Fine Source :^)
> My experience with performance testing indicates that
> at least threads are assigned to all processors on the system,
That's a property of the underlying OS threads, of course.
In case you didn't know, Tcl doesn't come with its own syscalls :)
-Alex
Donal K. Fellows
> In case you didn't know, Tcl doesn't come with its own syscalls :)
QOTW?
Donal.
tom.rmadilo
wrote:
> > This sounds nearly identical to AOLserver, the real unknown is how
> > different Tcl Threads is from AOLserver threads.
>
> No. The OP asked about the comparison with other generic frameworks,
> and only mentioned AOLserver as an acknowledge of performance. I have
> tried to characterize Tcl threads ex nihilo and in comparison to those
> frameworks, not in comparison to whatever you're fond of.
Well, you failed. There isn't much of a framework in Tcl Threads, you
have to do it all yourself. AOLserver has an extensive thread/interp
management framework, that simply doesn't exists in Tcl Threads. But
both of these implementations are built on the operating system
'syscalls', which I already said. But where exactly was your
comparison? The only comparison I see is that you say Tcl Threads are
"higher level than Java ones". In contrast, I pointed out that Tcl
Threads, is based on AOLserver code, which follows Steven's IPC and
Networking books. If you read those books, which have nothing to do
with AOLserver, you will understand the thread/mutex IPC concepts in
Tcl Threads. Obviously you haven't read these books either.
But talking about threads as if they are there to prevent or lessen
the use of mutexes, etc. is very misleading. Threads simplify coding
in many ways, but the existence of threads pretty much requires the
use of mutex/condition vars, etc., which is obvious because the major
use of threads is to perform similar tasks in parallel. AOLserver has
nsv's and Tcl Threads has tsv's, which hide the use of mutexes, but
they still exist. In fact, these APIs exist not only to share data,
but to share other mutex/condition variables so that you can perform
otherwise non-atomic operations on shared resources. Btw, condition
variables can use a timeout, which should avoid the deadlock problem
that you are talking about. Deadlocks are rare, however, less rare is
for a resource to become unavailable because the lock hasn't been
released. Timeouts at least allow for a thread to continue to signal
an error.
Jeff Hobbs
> Well, you failed. There isn't much of a framework in Tcl Threads, you
> have to do it all yourself. AOLserver has an extensive thread/interp
> management framework, that simply doesn't exists in Tcl Threads.
Are you comparing to Tcl core threads, or the Thread extension? From
your later points, it seems the latter, but that extension does
actually provide a sufficient framework to start from. AOLServer's is
more server oriented, but actually it leaves a lot to be desired,
mostly inherited from legacy stuff.
> But talking about threads as if they are there to prevent or lessen
> the use of mutexes, etc. is very misleading. Threads simplify coding
> in many ways, but the existence of threads pretty much requires the
> use of mutex/condition vars, etc., which is obvious because the major
> use of threads is to perform similar tasks in parallel. AOLserver has
OK, on this I will strongly disagree, but maybe just on
clarification. The existence and usage of threads in Tcl does not
require mutex/cvars in most cases. The abstractions are, like so many
other in Tcl, a good way to provide the facility of concurrent threads
without forcing painful details on the coder. Underlying the Tcl
threads, yes, mutex/cvars are in rampant use because they are
required. I just want to separate _Tcl_ threads from raw threads
programming.
Jeff
USCode
> Are you comparing to Tcl core threads, or the Thread extension? From
> your later points, it seems the latter, but that extension does
> actually provide a sufficient framework to start from. AOLServer's is
> more server oriented, but actually it leaves a lot to be desired,
> mostly inherited from legacy stuff.
>
Jeff, are you saying AOLServer's threads leave a lot to be desired or
the Threads extension leaves a lot to be desired?
Is the Tcl Threads extension a complete and solid implementation?
tom.rmadilo
> On Feb 15, 9:09 am, "tom.rmadilo" <tom.rmad...@gmail.com> wrote:
>
> > Well, you failed. There isn't much of a framework in Tcl Threads, you
> > have to do it all yourself. AOLserver has an extensive thread/interp
> > management framework, that simply doesn't exists in Tcl Threads.
>
> Are you comparing to Tcl core threads, or the Thread extension? From
> your later points, it seems the latter, but that extension does
> actually provide a sufficient framework to start from. AOLServer's is
> more server oriented, but actually it leaves a lot to be desired,
> mostly inherited from legacy stuff.
>
The Tcl Thread extension provides a great API for building a
framework, it just doesn't do anything, obviously, it is just an API.
But I'm not sure what you find lacking in AOLserver. It starts with
the basic OS level API and abstracts these to NS_* C API. Maybe you
are talking about how threads are created/initialized/managed. This
obviously places limits on how things are done, that is the framework
that I talk about. Obviously frameworks restrict what you can do, but
also free you from having to do most things manually.
The Tcl Threads extension appears to make simple communication between
parent/child threads very easy. This doesn't even happen in AOLserver.
The issue in AOLserver is for worker threads to obtain/use/share
access to other threads which access external resources. Usually this
is done without using the concept of obtaining a mutex, but by
obtaining a handle.
If this type of communication is possible in Tcl Threads without using
a mutex, that would be very cool.
Jeff Hobbs
> Jeff Hobbs wrote:
> > Are you comparing to Tcl core threads, or the Thread extension? From
> > your later points, it seems the latter, but that extension does
> > actually provide a sufficient framework to start from. AOLServer's is
> > more server oriented, but actually it leaves a lot to be desired,
> > mostly inherited from legacy stuff.
>
> My original post was in regards to the Threads extension.
> Jeff, are you saying AOLServer's threads leave a lot to be desired or
> the Threads extension leaves a lot to be desired?
While the threads support in Tcl is derived largely from early
AOLserver, it should be noted that AOLserver was written coincident
with Tcl 7.x to start. It took a while for AOLserver to make use of
better APIs, but structurally it still has architectural approaches
that made sense in Tcl 7.x days, but would be done differently today.
It could be a lot more cleanly and efficiently integrated into Tcl.
This is not to say that AOLserver isn't a well-designed system, just
that it has warts that reflect the original design era. I've had
these discussions over time with the core developers - they are aware
of it. However, there isn't a driving need to update. It work, and
is quite a capable system in itself regardless.
> Is the Tcl Threads extension a complete and solid implementation?
Yes.
Jeff
tom.rmadilo
> > Is the Tcl Threads extension a complete and solid implementation?
>
> Yes.
I agree, I just wish there were examples covering more complex uses
and explaining more about what goes on with the different types of
memory. The model is also somewhat unique, making it hard to compare
to other threading/shared memory models.
Donal K. Fellows
> Well, you failed. There isn't much of a framework in Tcl Threads, you
> have to do it all yourself.
The Thread extension comes with a thread pool engine (check out the
tpool manual page). It's up to you to use it, but it's certainly
sufficient for many tasks.
> The only comparison I see is that you say Tcl Threads are
> "higher level than Java ones".
The normal way of using the Thread extension involves message passing
instead of shared state. It's easier to make message passing work (in
many ways, it's very much like working with other processes via sockets
or a pipe) and it is known to scale better. With shared memory threading
you have to worry about concurrent access and locking, and these issues
are the bane of many programmers' lives.
Yes, there are ways of doing more "traditional threading" things with
the Thread extension, but you don't usually need to bother.
> Threads simplify coding
> in many ways, but the existence of threads pretty much requires the
> use of mutex/condition vars, etc., which is obvious because the major
> use of threads is to perform similar tasks in parallel.
Actually no. Message passing has a different theoretical basis. OK, the
implementation might use some short-term locks on the message queues
when implemented over the other sort of thread system (e.g. over Posix
threads), but that's definitely an implementation detail.
Donal.
tom.rmadilo
Okay, I'm starting to get what you guys are talking about. Maybe
correct me, where I'm wrong, but, given that Jeff said that an
Apartment model is used, and that we have message passing, this must
be the single-thread per apartment model.
In this case you do have a manager which handles the interaction/
scheduling of threads. Since there is a manager, it is easy to hide
the boring details of mutex/locking. If this is the case, then there
isn't much relationship with AOLserver, or Java. I think .NET can
support this model, plus the multi-threaded apartment.
This would also explain the slight performance advantage for AOLserver
for a real world service, maybe 50% higher. But! It only took me a few
hours to figure out how to write a threaded application using Tcl
Threads, as opposed to the many man-years that went into developing
AOLserver.
tom.rmadilo
> > in many ways, but the existence of threads pretty much requires the
> > use of mutex/condition vars, etc., which is obvious because the major
> > use of threads is to perform similar tasks in parallel.
>
> Actually no. Message passing has a different theoretical basis. OK, the
> implementation might use some short-term locks on the message queues
> when implemented over the other sort of thread system (e.g. over Posix
> threads), but that's definitely an implementation detail.
Sorry Donal, I didn't mean to imply that the normal developer would be
using the mutex/condition vars. Instead, someone implements a feature,
like a queue of threads or database handles, and protects this list
with a mutex/condition var. That is how it works in AOLserver, where
the usual application developer never deals with locks of any kind,
heck, they don't even deal with threads, basically like there are
threads in Tcl core right now, but it isn't something you think about.
But if locks are an implementation detail, so must be threads, so I'm
not sure I understand the distinction here. Maybe concurrency and
shared resources are the goal, everything else is details. That is why
I referred to IPC. But if you look into IPC, message queues are
distinct from mutex/condition protected queues, and it would be very
cool if that was the implementation in Tcl.
Neil Madden
> On Feb 15, 4:40 pm, "Donal K. Fellows"
>>> Threads simplify coding
>>> in many ways, but the existence of threads pretty much requires the
>>> use of mutex/condition vars, etc., which is obvious because the major
>>> use of threads is to perform similar tasks in parallel.
>> Actually no. Message passing has a different theoretical basis. OK, the
>> implementation might use some short-term locks on the message queues
>> when implemented over the other sort of thread system (e.g. over Posix
>> threads), but that's definitely an implementation detail.
>
> Okay, I'm starting to get what you guys are talking about. Maybe
> correct me, where I'm wrong, but, given that Jeff said that an
> Apartment model is used, and that we have message passing, this must
> be the single-thread per apartment model.
>
> In this case you do have a manager which handles the interaction/
> scheduling of threads. Since there is a manager, it is easy to hide
> the boring details of mutex/locking. If this is the case, then there
> isn't much relationship with AOLserver, or Java. I think .NET can
> support this model, plus the multi-threaded apartment.
Yes, Tcl's model is single-thread per apartment by default. Each thread
has one or more interpreters that are restricted to that thread - an
interpreter (and all the state it contains) is never shared between
threads. When you need to communicate between threads you do so by
sending a message. This message is picked up by the receiving thread
which executes it, possibly returning a result (if using synchronous
thread::send).
The Tcl threading model is extremely well thought out. It is essentially
the event loop model extended to multiple threads. You still have the
nice property that a piece of code executing in a thread is atomic by
default. This eliminates need for explicit locking in your code. (It's
also good from a security point of view:
http://www.erights.org/elib/concurrency/event-loop.html)
There is a great paper called "The Problem with Threads" by Edward Lee
[1] that nicely describes the problem with shared-state concurrency.
Basically, the default threading model of languages like Java and C is
to introduce rampant non-determinism into the code and then try to prune
it back using locks. This can result in very efficient code, but it is a
huge amount of effort to get right. Tcl's approach on the other hand
allows you to go from purely deterministic code and then gradually
introduce more non-determinism/concurrency as it is required. You can
nicely move along the following scale in Tcl:
1. Purely declarative single threaded code (i.e., functional programming).
2. Single-threaded plus mutable state.
3. Event loop (still single threaded, deterministic execution of event
handlers, but possible non-deterministic execution order for these
handlers).
4. Message-passing threaded. Essentially only slightly more complicated
than an event loop (multiple event loops).
5. Shared state concurrency (via the tsv mechanism).
This is really a superb situation to be in. The shared state mechanisms
of the Thread package are also pretty good in allowing you to build more
complex abstractions on top: e.g. this explicit message queue:
http://wiki.tcl.tk/17652
[1] http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf
-- Neil
Alexandre Ferrieux
>
> Okay, I'm starting to get what you guys are talking about.
Then do us a favour, and think twice before insulting those trying to
enlighten you with pearls like "Obviously you haven't read these books
either.".
-Alex
tom.rmadilo
wrote:
When did you become an 'us'? You haven't enlightened me at all, and
none of the above comments even get to the reality of Tcl Threads.
Mostly there is a bunch of bs above, your comments being the least
helpful, so don't flatter yourself. I don't need to think twice before
rejecting your useless comments.
And what is with the two week delay to respond?
Alexandre Ferrieux
Oh, delighted :-) I'll know what to do with your questions, then.
> And what is with the two week delay to respond?
In addition to being repeatedly insulted in public, do I also need to
give details about by network access ?
-Alex
Donal K. Fellows
> When did you become an 'us'? You haven't enlightened me at all, and
> none of the above comments even get to the reality of Tcl Threads.
Do chill out. As Neil said, Tcl's threading support gives you almost
everything on the threading spectrum from wholly single-threaded to
almost total shared memory parallelism. It doesn't go the whole way,
mostly because the fully shared memory model is a real PITA to work
with in practice (see evidence from lots of other programming
languages) despite scaling fairly poorly.
The last point I make above is actually interesting in itself, as it
depends on the fact that getting coherent shared memory is very
complex once you can no longer co-locate CPUs. Indeed, it is this sort
of thing that makes supercomputers so expensive relative to clusters.
In cluster HPC, you explicitly give up on shared memory and use
message passing to synchronize (MPI is a good example of this, but the
map-reduce algorithm beloved of Google is actually another message
passing algorithm; different use of messages though) and it's proved
to be the case that clusters can scale much much larger. Once you get
over 60k machines in your cluster, you have other problems too (e.g.
non-scalability of scheduling algorithms) but most of us won't be
getting to that scale for a while yet. :-)
> And what is with the two week delay to respond?
Sometimes life gets in the way. Sometimes we're under the impression
that all's been said that needs to be said. YMMV.
Donal.
tom.rmadilo
On Mar 2, 1:45 am, Alexandre Ferrieux <alexandre.ferri...@gmail.com>
wrote:
> In addition to being repeatedly insulted in public, do I also need to
> give details about by network access ?
Alexandre,
The insulting was started by you in this thread when you decided that
you needed to read the mind of the 'OP', and that I should in essence
shut up and not talk about 'whatever [I'm] fond of':
> No. The OP asked about the comparison with other generic frameworks,
> and only mentioned AOLserver as an acknowledge of performance. I have
> tried to characterize Tcl threads ex nihilo and in comparison to those
> frameworks, not in comparison to whatever you're fond of.
Then you responded to something else I said with an example of exactly
what I was getting at: there is no external documentation or
interesting examples of Tcl Threads. I said:
> > It would be helpful to have more information on the Tcl Threads
> > implementation.
And you helpful response was:
> Read The Fine Source :^)
Why didn't you just ask the 'OP' to do the same? It might have helped
more than anything you said, which was unilluminating.
Then I gave on additional observation, that Tcl isn't using 'green
threads', but real OS threads:
> > My experience with performance testing indicates that
> > at least threads are assigned to all processors on the system,
To which you had to illuminate everyone about how much you know about
what Tcl doesn't have, and asking me if I knew it or not (this isn't
insulting, maybe just flippant):
> That's a property of the underlying OS threads, of course.
> In case you didn't know, Tcl doesn't come with its own syscalls :)
All of these comments, _you_ made before I did any insulting of your
fragile psyche.
For the record, I did take you up on your offer to read the source,
and what I found is that none of the descriptions above correctly
characterize how Tcl Threads works.
For instance, there is no distinct message queue, or messages being
passed. There are three things being used together:
1. Threads with mutex/condition vars and shared memory,
2. Tcl event queues,
3. Tcl events.
The 'messaging' is really event queue management. The queue is neither
lifo nor fifo, events can be inserted anywhere, and also deleted. It
is an interesting idea, but it isn't really a queue. So maybe event
list would be a better term. However, when I think of messaging, I
usually associate this term with a separation of the sender and
receiver. In the Tcl Threads implementation it is quite the opposite.
Every sender becomes a queue manager of the receiver's queue.
Donal,
You could also chill. The reference provided by Neil are about as far
off the mark of how Tcl Threads works as could be. It is a good
description of what is wrong with Java, and a good description of the
difficulty of using threads correctly, but these apply to any system
which uses threads. Obviously Tcl Threads uses threads, mutex/
condition vars and shared memory, so how this paper Neil references
somehow supports Tcl's thread system is beyond me. Simply hiding the
details doesn't remove any of the difficulty of getting it right.
Every system hides the details, they have to be hidden for obvious
reasons, the main one being that you have to be able to find the name/
reference of shared stuff in order to use it.
Donal K. Fellows
> Donal,
>
> You could also chill.
You're the primary source of non-chill-ness round here right now, though
Alex isn't too great at smoothing his ruffled feathers either.
Please *everyone*, stick to discussions of facts or at least informed
technical opinion. Ad hominem attacks are not permitted in this group,
not even for being Steve Ballmer or Richard Stallman. :-)
> The reference provided by Neil are about as far
> off the mark of how Tcl Threads works as could be. It is a good
> description of what is wrong with Java, and a good description of the
> difficulty of using threads correctly, but these apply to any system
> which uses threads.
Anything that uses a threading model based on shared memory and locks.
That's not the only model of parallelism though; the other model is
message passing, and that's known to scale up to the size of the
internet. In fact, that *is* the internet! It's a massive massive
parallel processing system without shared memory, without locks, and
with message passing as its fundamental primitive.
> Obviously Tcl Threads uses threads, mutex/
> condition vars and shared memory, so how this paper Neil references
> somehow supports Tcl's thread system is beyond me. Simply hiding the
> details doesn't remove any of the difficulty of getting it right.
That really depends. A key factor is that Tcl hides locks, and does so
in such a way that the implementation doesn't hold multiple locks at
once (except for circumstances where we can prove that the locks form a
DAG, and where the inner locks are not exposed to the script level).
Since locks aren't held, you can't deadlock. Since you're using message
passing, you have to write your code to work that way. Since you've
already bitten that bullet and avoided the Trap Of Shared Memory
Parallelism, going to the super-scalable message passing model is not a
big deal.
In other words, Tcl explicitly blocks you from doing something that is a
Bad Idea, while making something that is a Good Idea much easier to
achieve (it's easier to do message passing well in Tcl than in any other
language I know that doesn't make the message passing act itself part of
the fundamental linguistic fabric).
> Every system hides the details, they have to be hidden for obvious
> reasons, the main one being that you have to be able to find the name/
> reference of shared stuff in order to use it.
I currently fail to see what that's got to do with it. Not all
concealments are the same. For example, Tcl also hides the fine details
of working with multiplexed event sources (i.e. select()/poll()). While
in theory this makes Tcl less efficient, in practice it's a good thing
since it means we have our elegant [fileevent] system instead of the
horrible mess that other languages provide. Similarly, there's a
potential source of inefficiency from using our high level view of
threads that a low level shared memory system could avoid, but it turns
out that for real code the situation is different. Message passing works
better. Indeed, full production threaded code actually works by using a
database-backed message passing engine to distribute messages to
processing units that are typically managed in pools. I've seen that
before somewhere. :-)
Donal.
Darren New
> tom.rmadilo wrote:
>> You could also chill.
>
> You're the primary source of non-chill-ness round here right now,
It should be noted that it is possible to be both right *and* obnoxious
about it at the same time. It's also possible to tell someone they're
wrong without implying personal inadequacies.
--
Darren New / San Diego, CA, USA (PST)
"That's pretty. Where's that?"
"It's the Age of Channelwood."
"We should go there on vacation some time."
Donal K. Fellows
> It should be noted that it is possible to be both right *and* obnoxious
> about it at the same time. It's also possible to tell someone they're
> wrong without implying personal inadequacies.
This is Arguments. Abuse is down the hall on your right.
Donal.
George Peter Staplin
Donal, you don't own this newsgroup, any more than tom, or the other
people.
We are all equals here.
Donal K. Fellows
> Donal K. Fellows wrote:
> > This is Arguments. Abuse is down the hall on your right.
>
> people.
>
> We are all equals here.
I was referring (badly) to http://www.mindspring.com/~mfpatton/sketch.htm
Donal.
Pete
"Donal K. Fellows" <donal.k...@man.ac.uk> wrote in message
news:f05bf484-e0dd-409a...@f47g2000hsd.googlegroups.com...
Donal, you referred to it extremely well. Thank you for the link and thanks
for the very useful input you make to this group. I'm sure most readers
would agree.
Regards,
Pete
USCode
>
> Donal, you referred to it extremely well. Thank you for the link and thanks
> for the very useful input you make to this group. I'm sure most readers
> would agree.
>
> Regards,
> Pete
>
Pete's but nowhere as eloquently.
Thanks Pete and thanks Donal!
Darren New
I know that's supposed to be funny. I wasn't talking about Donal, nor
did I mean to abuse anyone. :-)
Neil Madden
[...]
> You could also chill. The reference provided by Neil are about as far
> off the mark of how Tcl Threads works as could be. It is a good
> description of what is wrong with Java, and a good description of the
> difficulty of using threads correctly, but these apply to any system
> which uses threads.
Could you clarify these remarks? If I am labouring under a
misapprehension of Tcl's threading model, I'd welcome being put straight
on the matter.
Regarding your last point, this is entirely wrong. The problems of
threads come only when mutable state is shared among threads. So systems
which use threads without mutable state (or in which the effects of such
mutation are effectively isolated) do not suffer from the problems
described in the paper.
> Obviously Tcl Threads uses threads, mutex/
> condition vars and shared memory, so how this paper Neil references
> somehow supports Tcl's thread system is beyond me.
Tcl's threads use shared memory and mutexes *in the implementation*.
This detail is effectively hidden from Tcl scripts running in a thread,
unless they explicitly choose to expose these details using the tsv
mechanisms.
> Simply hiding the
> details doesn't remove any of the difficulty of getting it right.
> Every system hides the details, they have to be hidden for obvious
> reasons, the main one being that you have to be able to find the name/
> reference of shared stuff in order to use it.
This is simply incorrect.
Neil Madden
[...]
>
> Donal,
>
> You could also chill. The reference provided by Neil are about as far
> off the mark of how Tcl Threads works as could be. It is a good
> description of what is wrong with Java, and a good description of the
> difficulty of using threads correctly, but these apply to any system
> which uses threads.
This isn't the case. Threads by themselves are not the problem; it is
the combination of threads and shared mutable state that cause the
problem. Systems that avoid one or the other avoid these problems. In
particular multi-threaded systems which do not share state avoid these
problems. This is Tcl's threading model: each thread has a separate
interpreter, and variables are not by default shared across threads (or
indeed, interps).
> Obviously Tcl Threads uses threads, mutex/
> condition vars and shared memory, so how this paper Neil references
> somehow supports Tcl's thread system is beyond me.
Tcl threads use shared memory and mutexes etc as part of the
implementation, not part of the model exposed to Tcl scripts. A script
can use explicit shared memory via the tsv mechanism, but by default no
state is shared. If you have a counterexample of this, I would be
interested to see it.
> Simply hiding the
> details doesn't remove any of the difficulty of getting it right.
I disagree. The user of Tcl's threading abstractions can completely
ignore all issues of shared memory unless they choose to introduce them.
Again, if you have a counterexample, I would be very interested to see it.
> Every system hides the details, they have to be hidden for obvious
> reasons, the main one being that you have to be able to find the name/
> reference of shared stuff in order to use it.
I don't understand what is meant here.
-- Neil
Joe English
[ drifting off-topic ]
Neil Madden wrote:
> Tcl threads use shared memory and mutexes etc as part of the
> implementation, not part of the model exposed to Tcl scripts. A script
> can use explicit shared memory via the tsv mechanism, but by default no
> state is shared. If you have a counterexample of this, I would be
> interested to see it.
The ::env array, current working directory, and
set of VFS mounts. If an interp in one thread
writes to ::env, or calls [cd ...] or [vfs::mount ...],
all interps in all threads are affected.
I'm hoping this can be changed for Tcl 9; it would
greatly improve Tcl's thread-safety.
--JE
Neil Madden
Good examples. None of these are things I use much, myself, except ::env
and that read-only. I'm surprised VFS mounts are shared. Of course, the
entire filesystem is one huge piece of shared state -- shared not only
between threads, but between processes, and potentially between
different machines. There's not really much Tcl could do about that though.
-- Neil
Alexandre Ferrieux
Neither about "cd", at least under Unix and until the implementation
abstracts away from the chdir() syscall, since the current-directory
is a process-wide notion shared among threads like open file
descriptors (can be observed under Linux by looking at /proc/$pid/cwd
for several threads doing a different chdir()).
-Alex
Neil Madden
That'd be nice. To my knowledge, I've only once used [cd] in recent
memory, and that was to work around DLL dependency issues on Windows
when loading libs out of a starkit (lib + dependency DLLs in starkit,
had to copy all to temp and [cd] there before would load correctly).
-- Neil
Alexandre Ferrieux
> Alexandre Ferrieux wrote:
> > Neither about "cd", at least under Unix and until the implementation
> > abstracts away from the chdir() syscall, since the current-directory
> > is a process-wide notion shared among threads like open file
> > descriptors (can be observed under Linux by looking at /proc/$pid/cwd
> > for several threads doing a different chdir()).
>
> That'd be nice. To my knowledge, I've only once used [cd] in recent
> memory, and that was to work around DLL dependency issues on Windows
> when loading libs out of a starkit (lib + dependency DLLs in starkit,
> had to copy all to temp and [cd] there before would load correctly).
Oh, but I'm just now realizing that abstraction is not enough: even if
the Tcl command [cd] is made per-thread by storing only a userland
variable and delaying the actual call to chdir(), it remains that the
chdir must be finalized just before OS operations depending on the
cwd, like [load] [open |] [exec]. That could be done, but a lock
should be used and kept during the length of the primitive, which may
be long for [exec]...
An alternative would be to define (in Tcl) and advertise a new
[thread_safe_cd] taking a script argument and keeping the lock during
the execution of the script:
thread_safe_cd $dir {exec foo bar baz}
-Alex
Donald G Porter
> The ::env array, current working directory, and
> set of VFS mounts.
Not quite the same thing, but the [exit] command brings down
an entire app, not just the thread it evaluates in.
--
| Don Porter Mathematical and Computational Sciences Division |
| donald...@nist.gov Information Technology Laboratory |
| http://math.nist.gov/~DPorter/ NIST |
|______________________________________________________________________|
Donald G Porter
> Oh, but I'm just now realizing that abstraction is not enough: even if
> the Tcl command [cd] is made per-thread by storing only a userland
> variable and delaying the actual call to chdir(), it remains that the
> chdir must be finalized just before OS operations depending on the
> cwd, like [load] [open |] [exec].
I think the simpler solution would be to have all of Tcl's system calls
that pass filenames always pass absolute filenames, so that the current
directory has no effect. Tcl's own sense of the "current directory" for
doing its own resolution of relative filenames could be contained
per-thread, or per-interp.
The catch there is probably the impact on extensions, having to train all
of them to always call the Tcl_FS* routines and not make calls to the
syscalls themselves. Still this impact is already in place for those
extensions which want to interact with virtual filesystems.
Alexandre Ferrieux
> Alexandre Ferrieux wrote:
> > Oh, but I'm just now realizing that abstraction is not enough: even if
> > the Tcl command [cd] is made per-thread by storing only a userland
> > variable and delaying the actual call to chdir(), it remains that the
> > chdir must be finalized just before OS operations depending on the
> > cwd, like [load] [open |] [exec].
>
> I think the simpler solution would be to have all of Tcl's system calls
> that pass filenames always pass absolute filenames, so that the current
> directory has no effect.
This is not possible in the general case, e.g. when [exec] spawns a
script which interprets some of its argument as relative paths (you
cannot know in a generic way which args are such paths and which are
other kinds of strings /usr/bin/unless /usr/bin/you /usr/bin/want to
prepare for trouble ;-)
-Alex
Jeff Hobbs
Actually vfs mount points are per-thread since 2006, when I had to
replumb tclvfs for various thread-safety issues. That means they are
still shared amongst all interps for a single thread though.
Jeff
Donald G Porter
Joe English wrote:
>>> set of VFS mounts. If an interp in one thread
>>> writes to ::env, or calls [cd ...] or [vfs::mount ...],
>>> all interps in all threads are affected.
Jeff Hobbs wrote:
> Actually vfs mount points are per-thread since 2006, when I had to
> replumb tclvfs for various thread-safety issues. That means they are
> still shared amongst all interps for a single thread though.
Note that Jeff's work only applies to the tclvfs package and those
virtual filesystems built on top of it. The core Tcl_Fileystem
interface continues to manage a process-global set of mountpoints.
Donald G Porter
> Note that Jeff's work only applies to the tclvfs package and those
> virtual filesystems built on top of it. The core Tcl_Fileystem
> interface continues to manage a process-global set of mountpoints.
That's not quite correct either. There is no process-global list
of mountpoints. Rather the invalidation of cached file name resolutions
when any change to the set of mountpoints happens is done by
Tcl_FSMountsChanged() and that routine acts process-globally.
Darren New
> chdir must be finalized just before OS operations depending on the
> cwd, like [load] [open |] [exec].
exec() and open() both do forks (at least under UNIX), so that's a
no-brainer. Fork, cd to where Tcl "thinks" is the current directory,
*then* exec() the appropriate program. The lock for a load() would
probably be pretty quick if it's coming off a local disk.
On the other hand, this is almost a bit silly. As has been pointed out,
there will always be shared state of *some* sort, even between separate
processes let alone separate threads. The trick is like with GC: make
the 99% use case the easy one, and let people spend more time figuring
out finalizers, out-of-memory conditions, and so on.
Alexandre Ferrieux
> Alexandre Ferrieux wrote:
> > chdir must be finalized just before OS operations depending on the
> > cwd, like [load] [open |] [exec].
>
> exec() and open() both do forks (at least under UNIX), so that's a
> no-brainer. Fork, cd to where Tcl "thinks" is the current directory,
> *then* exec() the appropriate program. The lock for a load() would
> probably be pretty quick if it's coming off a local disk.
Hey, you're right :-) Very nice. TIP ?
> On the other hand, this is almost a bit silly. As has been pointed out,
> there will always be shared state of *some* sort, even between separate
> processes let alone separate threads. The trick is like with GC: make
> the 99% use case the easy one, and let people spend more time figuring
> out finalizers, out-of-memory conditions, and so on.
I admit that hunting for shared state is not a tremendously exciting
goal in itself; it was only a pretext to make [cd] appear on our (and
more usefully your) radar screen. Now that we have detected the
problem, and that you have found the solution, it would be frustrating
to leave [cd] thread-unsafe for long !
-Alex
tom.rmadilo
wrote:
> I admit that hunting for shared state is not a tremendously exciting
> goal in itself; it was only a pretext to make [cd] appear on our (and
> more usefully your) radar screen. Now that we have detected the
> problem, and that you have found the solution, it would be frustrating
> to leave [cd] thread-unsafe for long !
[cd] is by definition thread-unsafe, always has been, always will be.
Jeff Hobbs
Er, yes and no. It can be made thread-safe, even interp-safe (in that
each interpreter has its own cwd). The question is whether it is
worth the effort to do so.
Jeff
Donal K. Fellows
> I admit that hunting for shared state is not a tremendously exciting
> goal in itself; it was only a pretext to make [cd] appear on our (and
> more usefully your) radar screen. Now that we have detected the
> problem, and that you have found the solution, it would be frustrating
> to leave [cd] thread-unsafe for long !
The fix is to get rid of [cd] entirely. :-)
I say that because I have always made scripts behave more like what
users expected by getting rid of calls to [cd]. The only time it is at
all useful is when you're using Tcl interactively.
Donal (yes, I'm being deliberately provocative).
tom.rmadilo
<donal.k.fell...@manchester.ac.uk> wrote:
> The fix is to get rid of [cd] entirely. :-)
>
> I say that because I have always made scripts behave more like what
> users expected by getting rid of calls to [cd]. The only time it is at
> all useful is when you're using Tcl interactively.
>
> Donal (yes, I'm being deliberately provocative).
I agree with this entirely. In scripts, and I mean Tcl shell scripts,
you need the [cd] command, or at least it is valuable to make it
easier to port the OS shell scripts to Tcl.
But the concept of the 'current working directory', or [pwd] in Tcl,
has a very specific meaning which is shared by most if not all
languages. The meaning is that it is process wide. Apparently on
Windows each thread can have an independent pwd. Considering Tcl needs
to run on many platforms, this exception isn't very useful. Also,
since Tcl can have multiple interps per thread, the result is that
allowing more than one pwd per process would require different scripts/
code depending on the environment.
Bryan Oakley
I'm with you, Donal. I can't remember the last time I used cd in a program.
peter....@gmail.com
<donal.k.fell...@manchester.ac.uk> wrote:
> The fix is to get rid of [cd] entirely. :-)
>
> I say that because I have always made scripts behave more like what
> users expected by getting rid of calls to [cd]. The only time it is at
> all useful is when you're using Tcl interactively.
>
> Donal (yes, I'm being deliberately provocative).
The only usage of cd I haven't been able to get rid of is this:
set old [pwd]
cd $dir
exec ...
cd $old
That is when the external depends on the current directory to do the
right thing.
If you can show me a way to do that I'd be very happy since I know
that will break down horribly if I ever go multithread.
If exec had a -pwd flag that takes care of that case MT safe that
would be great.
/Peter
Alexandre Ferrieux
Illuminating.
-Alex
Alexandre Ferrieux
A quick solution is, in line with Darren's idea but in Tcl:
exec sh -c "cd $dir;exec yourcommand"
exec cmd /c "cd $dir;exec yourcommand"
which effectively allows to get rid of [cd] as a Tcl command...
-Alex
Pete
"Jeff Hobbs" <jeff....@gmail.com> wrote in message
news:fca97857-d18b-450e...@e6g2000prf.googlegroups.com...
The working directory is, by definition, the working directory of a process,
therefore it cannot be made thread-safe. The program needs to be
redesigned, not the Tcl interpreter.
Pete
Alexandre Ferrieux
> "Jeff Hobbs" <jeff.ho...@gmail.com> wrote in message
The OS-level working directory, yes. A scripting language is allowed,
even encouraged, to offer abstraction above the OS level. Tcl already
does in numerous places (like Threads themselves, if you followed this
thread ;-). The only issue is the effort needed to reach that goal, as
Jeff pointed out. It seems [open] [load] [exec] are reasonable; but
the case of extensions is tough, it may be the real show stopper.
-Alex
Darren New
> The OS-level working directory, yes. A scripting language is allowed,
> even encouraged, to offer abstraction above the OS level.
Yet, some people actually count on the os-level semantics of [cd]
working. For example, I can't unmount a file system a process has cd'ed
into, which is a good thing. It would suck if all of a sudden it wasn't
unusual for [fileutil::find] to start throwing error halfway through the
iteration because someone ejected the CD, not realizing you're using it.
Just as an example.
I honestly think a better way to go would be to have a different name
for this functionality, like [file prefix] or some such, and leave [cd]
unsafe as it is. Just my 2 cents.
tom.rmadilo
> tom.rmadilo wrote:
> > Obviously Tcl Threads uses threads, mutex/
> > condition vars and shared memory, so how this paper Neil references
> > somehow supports Tcl's thread system is beyond me.
>
> This detail is effectively hidden from Tcl scripts running in a thread,
> unless they explicitly choose to expose these details using the tsv
> mechanisms.
Neil,
You asked for some clarification to my comments.
First, your 5 step list is very helpful, I think it explains very well
the progression of development in Tcl. (Although no examples of 2 come
to mind.)
What your list points out is that in Tcl you can start with a one
thread script and essentially avoid knowledge of the Tcl event loop.
The next stage (or 3) is concurrency. Concurrency is the main concept,
not the implementation. Step 3 implements concurrency with the Tcl
event loop. In my language, the event loop is the first primitive. I
call it a primitive because you can't divide it up any further. The
detailed implementation of the event loop is unimportant. As long as
it has the same behavior, you could change the internals.
Another primitive is an interp. An interp can contain code, data and
state which is considered independent of other interps. Obviously
there is a manager, which is another implementation detail, that
allows limited communication between interps. I'm not interested right
now with multiple interps in a thread, just in the fact that they
represent independent state (they seem very object-like to me).
The next step 4, requires threads. This is a new framework for
achieving concurrency. Here is where there seems to be a disagreement
with terminology. The paper you refer to "The Problem with Threads" is
a great reference, but mostly to the problem with Java threads.
Basically, Java didn't offer much in the way of synchronization
methods. The keyword synchronize appears to be the equivalent of a
critical section, which is one of the worst forms of synchronization,
but is absolutely critical in some situations. Critical sections allow
recursive entry into a routine by the same thread. For instance, in
AOLserver there is one critical section, used to maintain the 'master
lock', which is used to initialize mutex/condition variables, and
handle a few other recursive, application level tasks.
But Java uses 'critical sections" everywhere, and they are used by
programmers as an after-thought. The "Threads" paper correctly
identifies the horrors of this type of programming. But Java requires
this because an object can be called by any thread! If this is you
only way of synchronizing threads, it is easy to see why the author
thinks that threads are a bad model for achieving concurrency. This
doesn't apply to Tcl Thread, or C pthreads.
The problem with Java is that the real primitives were hidden behind
the synchronization keyword. Lately, Java 5 has added new concurrency
primitives and larger utilities:
http://java.sun.com/j2se/1.5.0/docs/guide/concurrency/overview.html
Most of these new thing exist in AOLserver, some of them in Tcl
Threads.
1. Task Scheduling Framework:
ns_schedule*
ns_job (warmed up thread queues)
ns_task (includes socket queue)
thread pools
2. Concurrency Collections:
nsv_*
::tsv::* (more facilities than nsv)
3. Atomic variables:
nsv_*
::tsv::* (more facilities than nsv)
4. Synchronizers:
mutex/condition vars
semaphores (built on mutex/condition)
cs lock (built on mutex/condition)
5. Locks:
built using mutex/timed condition (wait)
6. Nanosecond-granularity timing:
I'm not sure what the condition wait granularity is in pthreads.
All of this is what I think of as a concurrency system using threads.
Obviously you don't want your users to have to deal with the details
of every level, even within the internal implementation you build up
from the mutex/condition primitives to create something easier to
use.
It seems to me that Java tried to avoid using mutex/condition vars, so
they had threads without the other primitives. Eventually it was
discovered that this concept didn't work very well and the mutex/
condition primitives were added in. These concepts are hand and glove,
although the goal is always to hide details by creating easily reused
utilities.
Neil Madden
> On Mar 5, 11:44 am, Neil Madden <n...@cs.nott.ac.uk> wrote:
>> tom.rmadilo wrote:
>>> Obviously Tcl Threads uses threads, mutex/
>>> condition vars and shared memory, so how this paper Neil references
>>> somehow supports Tcl's thread system is beyond me.
>> Tcl's threads use shared memory and mutexes *in the implementation*.
>> This detail is effectively hidden from Tcl scripts running in a thread,
>> unless they explicitly choose to expose these details using the tsv
>> mechanisms.
>
> Neil,
>
> You asked for some clarification to my comments.
>
> First, your 5 step list is very helpful, I think it explains very well
> the progression of development in Tcl. (Although no examples of 2 come
> to mind.)
(2) is "single-threaded plus mutable state" -- this is the default mode
of operation for Tcl. [set], [incr] etc mutate state. You can write code
without them (i.e. option 1), but that is rare in Tcl (I don't usually
go that far myself, for instance).
>
> What your list points out is that in Tcl you can start with a one
> thread script and essentially avoid knowledge of the Tcl event loop.
>
> The next stage (or 3) is concurrency. Concurrency is the main concept,
> not the implementation. Step 3 implements concurrency with the Tcl
> event loop. In my language, the event loop is the first primitive. I
> call it a primitive because you can't divide it up any further. The
> detailed implementation of the event loop is unimportant. As long as
> it has the same behavior, you could change the internals.
Agreed.
>
> Another primitive is an interp. An interp can contain code, data and
> state which is considered independent of other interps. Obviously
> there is a manager, which is another implementation detail, that
> allows limited communication between interps. I'm not interested right
> now with multiple interps in a thread, just in the fact that they
> represent independent state (they seem very object-like to me).
I'm not sure what you mean by "manager" here. Interps can communicate
directly via command aliases, without need for a manager. You are right
that interps are object like.
>
> The next step 4, requires threads. This is a new framework for
> achieving concurrency. Here is where there seems to be a disagreement
> with terminology. The paper you refer to "The Problem with Threads" is
> a great reference, but mostly to the problem with Java threads.
It applies equally to C pthreads, and if I understand you correctly,
also to AOLserver's threading model (I have no experience with
AOLserver, so I am going on the details you hint at).
> Basically, Java didn't offer much in the way of synchronization
> methods. The keyword synchronize appears to be the equivalent of a
> critical section, which is one of the worst forms of synchronization,
> but is absolutely critical in some situations. Critical sections allow
> recursive entry into a routine by the same thread. For instance, in
> AOLserver there is one critical section, used to maintain the 'master
> lock', which is used to initialize mutex/condition variables, and
> handle a few other recursive, application level tasks.
Java's built-in synchronisation primitives are intended to be an
implementation of monitors. Each object has an implicit reentrant lock
(mutex) and entering a synchronized block/method causes the thread to
block (in a wait queue) until the lock is available, achieving mutual
exclusion between multiple threads accessing methods of the same object
(monitor). Each synchronized method therefore acts as a critical
section, but there is more to the model than this. Threads can also use
the methods wait() and notify()/notifyAll() to achieve condition
synchronisation. There is a paper by Per Brinch Hansen [1] that
discusses the problems with Java's approach compared to the original
monitors proposal.
> But Java uses 'critical sections" everywhere, and they are used by
> programmers as an after-thought. The "Threads" paper correctly
> identifies the horrors of this type of programming. But Java requires
> this because an object can be called by any thread! If this is you
> only way of synchronizing threads, it is easy to see why the author
> thinks that threads are a bad model for achieving concurrency. This
> doesn't apply to Tcl Thread, or C pthreads.
This applies identically to C pthreads! A C function can also be called
by any thread (if it is visible) and C variables can be altered by any
thread. You are correct that this doesn't apply to Tcl threads though
(in the default case). However, it does apply if you use the tsv
facilities (which is why you should be careful when using those facilities).
>
> The problem with Java is that the real primitives were hidden behind
> the synchronization keyword. Lately, Java 5 has added new concurrency
> primitives and larger utilities:
>
> http://java.sun.com/j2se/1.5.0/docs/guide/concurrency/overview.html
These new concurrency classes provide some more flexibility to Java, but
they are primarily motivated by performance. They don't address the
fundamental difficulties of Java's concurrency model. Tcl threads do not
have these problems to begin with, so by default, threaded Tcl code is safe.
>
> Most of these new thing exist in AOLserver, some of them in Tcl
> Threads.
>
[...]
>
> All of this is what I think of as a concurrency system using threads.
The mechanisms you describe are mostly low-level pthreads-style
concurrency primitives. You can write efficient code with them, but they
do not constitute a safe concurrency model.
As stated by Per Brinch Hansen in the paper referenced:
"Today we have three major communication paradigms:
monitors, remote procedures, and message passing. Any one of them would
have been a vast improvement over Java's insecure variant of shared
classes."
They are also vast improvements over pthreads. Pthreads is not a
concurrency model, it is merely a bag of useful concurrency utilities
with which you can attempt to reign in the rampant nondeterminism you
have unleashed upon your code (shutting the barn door after the horse
has bolted, in many cases). Tcl avoids this nondeterminism in the first
place, and allows you to selectively introduce it in just the spots you
want.
> Obviously you don't want your users to have to deal with the details
> of every level, even within the internal implementation you build up
> from the mutex/condition primitives to create something easier to
> use.
>
> It seems to me that Java tried to avoid using mutex/condition vars, so
> they had threads without the other primitives. Eventually it was
> discovered that this concept didn't work very well and the mutex/
> condition primitives were added in. These concepts are hand and glove,
> although the goal is always to hide details by creating easily reused
> utilities.
They are not hand and glove! There are alternative models that are much
cleaner. In addition to the alternative mechanisms described above,
there are also process calculi (CSP, CCS, pi-calculus, join calculus,
etc) that offer clean and simple models. Java adding more low-level
primitives has not solved any of the problems with its concurrency model
(you cannot fix a broken model by simply adding more primitives).
Refs:
[1] Per Brinch Hansen, "Java's Insecure Parallelism", SIGPLAN Notices
34(4), 1999. http://brinch-hansen.net/papers/1999b.pdf
-- Neil
Donal K. Fellows
> I honestly think a better way to go would be to have a different name
> for this functionality, like [file prefix] or some such, and leave [cd]
> unsafe as it is. Just my 2 cents.
My experience is that it's easier to write your code to work with
absolute filenames and never use [cd] (as long as there isn't some
horrible low-level library or non-Tcl program that needs it to work
right). Changing the working directory makes it harder to present a
consistent user experience, as it's not always practical to sanitize all
filenames before the [cd]. Luckily, Tcl makes it really easy to work
with absolute filenames; [file normalize] is good!
Sometimes you really need [cd]. Not that often though. (If it was only
ever needed for subprocesses - for me the main case - we could do more
elegant stuff. But that's not the only use.)
Donal.
Donal K. Fellows
> They are not hand and glove! There are alternative models that are much
> cleaner. In addition to the alternative mechanisms described above,
> there are also process calculi (CSP, CCS, pi-calculus, join calculus,
> etc) that offer clean and simple models.
FWIW, process calculi tend to orient towards message passing as a
fundamental model. (Or at least CSP, CCS and π-calculus do; I don't know
join calculus, and the paper that Wikipedia references is a little bit
too hard-core for reading when not at work... :-)) But there are many
types of message passing, with the main differences relating to the
sorts of message payloads and the degree of queueing.
Curiously, π-calculus also works as a model of OO.
Donal.