History of Active Messages
Robert Myers
bit of interesting work about active messages. I'm hoping that there
are people here who can help me to understand the history.
From the easiest to find citations, there was a flurry of optimistic
activity in the early 90's, including proposals with specialized
network processors, and then everything just seems to have died.
There is some recent work, but, by comparison with the 90's,
practically nothing is happening.
Some questions:
1. Did someone discover something fundamental that made people abandon
the idea, or did processors just become more powerful so rapidly that
people stopped thinking about innovative parallelism? Or was it the
DoD's directive to dumb it down (COTS only)?
2. How much can be learned from the Thinking Machines experience? Did
that failure play a significant role in getting us where we are now?
Do the Thinking Machines boxes teach anything about active messages in
general? If so, is it fundamental or just timing, politics,
execution, and, of course, money?
3. How much of a role, if any, did the collapse of the telecom bubble
play? Intel lost interest (since regained, I gather) in network
processors, which do business in much the way I imagine (accept
packets, do something with them, send them on) when the telecom market
collapsed. Is it just an accident that active messages no longer had
a future at just about the same time as network processors didn't look
as hot as they once did? Again, is it fundamental, or just a side-
effect of markets and timing?
Pardon me, please, if the questions seem obtuse or overly-broad.
Perhaps someone can sharpen them or ask better questions. Lots of
ideas just died on the vine. Is that because all the action was in
the x86 horse-race?
Robert.
Andy "Krazy" Glew
> A brief perusal of the ACM Digital Library and Google yields a fair
> bit of interesting work about active messages. I'm hoping that there
> are people here who can help me to understand the history.
>
> From the easiest to find citations, there was a flurry of optimistic
> activity in the early 90's, including proposals with specialized
> network processors, and then everything just seems to have died.
> There is some recent work, but, by comparison with the 90's,
> practically nothing is happening.
There's another consideration: the term "active messages" is a nice way
of describing the trends in much earlier work, from the 80s and earlier.
Many of us use active messages as a generic term, and are not
necesssarily referring specifically to Thorstein von Eicken's work when
we do so. (In some ways, that is the ultimate flattery: you have coined
a term so good that it becomes generic.)
For example, you already saw me confusing Actors with active messages.
Yes, there is an important difference, but in many ways I think of
active messages as the implementation model for some varieties of
Actors. Or, if you will, of Actors as the theoretical model for active
messages.
Now, basic Actors are just message passing. But Actors can spawn other
Actors.
The Actors model per se does not really comprehend physical locality.
But if you want to implement Actors in anything other than dance hall
shared memory, you end up sending packets that implement Actors from one
node to another where it should execute, and so on. Et voila - active
messages. Active messaging is the load balancing mechanism for an
Sctors implementation.
(With the caveat that most active messaging schemes assume that the
message handler is shortlived, and not necessarily a process that will
persist a long time.)
I was surprised that active messages seemed mainly to be a 90s sort of
thing, since I mainly played in this space prior to 1987, at Gould. But
I guess that when I saw the term "active messages" succintly describe
the sort of stuff I discuss above, I retroactively renamed it. Or,
rather, I was willing to accept someone (Berndt?) saying "That's not
Actors, that's active messages," even though my ideas were friom an
earlier decade.
> 1. Did someone discover something fundamental that made people abandon
> the idea, or did processors just become more powerful so rapidly that
> people stopped thinking about innovative parallelism? Or was it the
> DoD's directive to dumb it down (COTS only)?
Like I said, I played in this space mainly in the late 80s. The era of
the Japanese 5th Generation Project. New programming models to take
over the world. They didn't. Instead, the attack of the Killer Micros
happened.
However, as someone who was peripherally involved in such models of
innovative parallelism - at Gould we considered them, but discarded most
- let me say that one of the biggest problems was that the approaches
did not scale, when hardware support was provided. I don't mean scale
in terms of amount of parallelism, I mean that if, e.g., you were
providing a simple processing element in what we would no call a memory
controller or a router to execute the active message in a manner
supposedly more efficient that the big main processors could do well,
they tended to have too many arbitrary limits. Like "the active message
code can only access 128 bytes of local memory". "The active message
code be no more than 128 instructions long."
If you wanted something larger than these bounds, well, you could still
use the active messaging model (ok, so that wasn't the term we used at
the time. Or was it? darned if I can remember). But the message handler
had to run on the real CPU, not the assist CPU.
So much for the idea of adding stuff like this as an accelerator for
particular operations. It failed for the usual problem of accelerators:
heterogeneity is a pain, Especially if one of the heterogenous
processing elements is heavily constrained.
Burton Smith went a different way, and made the main CPU capable of
executing the message handler efficiently. But there were so many other
things in Burton's systems, that it's not clear why they failed. There
were many, many, good ideas, but it's not clear that all were good
ideas. I suspect that there would have been some fallout even with the
Attack of the Killer Micros.
But the Attack of the Killer Micros redirected everything. They were so
cheap you could build highly parallel systems; but they weren't
interested in adding features that earlier parallel computing work found
useful, because parallel computing wasn't their main market. Until
recently.
> 2. How much can be learned from the Thinking Machines experience? Did
> that failure play a significant role in getting us where we are now?
> Do the Thinking Machines boxes teach anything about active messages in
> general? If so, is it fundamental or just timing, politics,
> execution, and, of course, money?
>
> 3. How much of a role, if any, did the collapse of the telecom bubble
> play? Intel lost interest (since regained, I gather) in network
> processors, which do business in much the way I imagine (accept
> packets, do something with them, send them on) when the telecom market
> collapsed. Is it just an accident that active messages no longer had
> a future at just about the same time as network processors didn't look
> as hot as they once did? Again, is it fundamental, or just a side-
> effect of markets and timing?
Like I said, I really wasn't paying attention at this time.
I heard rumours that the old ideas were being applied, but never really
saw much evidence.
My guess is that it was not cause and effect. The NSPs were the great
white hope for a market for active messaging ideas, but people figured
out how to do advanced tewlecom without needed dramatically new
programming models. Or, maybe, as M. Paysan says, they were doing it.
In Forth. On general purpose parallel machines made out of Killer Micros.
===
I suspect that the problem with so many of these interesting ideas for
parallelism was that
a) people wanted to do hardware and software together
b) software only sounded interesting, but was slow on conventional
processors. If we just got this little bit of hardware support (like
lightweight user level interrupts) all would be well.
c) but in the meantime, people figured out how to do the problems that
needed solving on conventional machines, some parallel, some not, using
conventional programming models.
I contend that in order for something like this to succeed it must first
succeed in a purely software implementation on conventional machines.
Once the footr is in the door, we can optimize. Otherwise, it requires
a visionary leap of faith: and so often the visionaries are wrong.
Mayan Moudgill
> From the easiest to find citations, there was a flurry of optimistic
> activity in the early 90's, including proposals with specialized
> network processors, and then everything just seems to have died.
> There is some recent work, but, by comparison with the 90's,
> practically nothing is happening.
>
Depends on what you mean by active messages. However, here are a few
hypotheses:
1. Most machines went from distributed to shared memory, so that market
went away.
2. A lot of the interconnect became hard wired, so there wasn't much
room to play around (TCP/IP offload engines, hypertransport).
3. The generic software stacks/message handlers became more highly
optimized and integrated into the OS, so there was less of a gain.
4. User-level interrupt handler models became cleaner, so the overhead
of handling the message in userland wasn't so bad.
5. The security problem - do you really want to execute arbitrary code?
Robert Myers
> 5. The security problem - do you really want to execute arbitrary code?
Well, of course not. The question is, can you provide safeguards that
are safe enough? The answer to that question has to depend on details
of the implementation.
Every discussion that I've seen here that deals with concurrent
accesses to memory leaves me with conclusion that it simply can't be
done safely, or, if it could be done safely by some chosen few, the
vast majority of programmers will never even understand the possible
complications, much less know how to maneuver safely around them.
The problem of preventing execution of malicious code may be just as
hard or harder, but I can't think of two hard problems at once, and
I'm really attracted to making the problem of concurrent access to
memory just go away. If the system is set up so that all necessary
pieces (code and data) rendezvous automatically, another hard problem
is solved in addition to the concurrency problem.
I'll accept, for the time being, that the security problem may not be
solvable, but leave that question for later.
Robert.
Bernd Paysan
> I'll accept, for the time being, that the security problem may not be
> solvable, but leave that question for later.
As long as we are talking about shared memory vs. active messages, the
security implications aren't important. If you are allowed to directly
access all memory of another process, there is no security. It's not
even possible to get security by bolting it on afterwards.
The good think about active messages is that while security is still
bolted on afterwards, it's at least possible. You can inspect the code
you'll execute, and/or let it run with the right privilege levels and
address maps so that it can't do harm.
If I was to implement something like the Internet based on AMs, the
security model will be as follows:
* Addresses to be written to by remote hosts have to be mapped
explicitly into the AM address space. No other addresses can be written
to. Addresses to be read from (i.e. "copy me xxx:nnn") also have to be
cleared explicitly
* Instruction set is limited, uses a small set of data types where
semantics are strictly controlled (this probably means that hashes,
lists, and trees can't be composed out of uncontrolled pointers)
* Instruction set extensions (e.g. for HPC something like BLAS, for a
remote display, something like OpenGL) are controlled by the receiver
* Stored procedures use the same virtual machine model
--
Bernd Paysan
"If you want it done right, you have to do it yourself"
http://www.jwdt.com/~paysan/