Another communication channel I used between applications is mapped memory +
system events. This gives the fastest communication I could achieve between
2 applications.

(edif time_sim
  (edifVersion 2 0 0)
  (edifLevel 0)
  (keywordMap (keywordLevel 0))
  (status
    (written
      (timestamp 2000 5 24 12 0 29)
      (program "Xilinx NGD2EDIF" (version "C.18"))
      (comment "Command line: -w -v fndtn testus.nga time_sim.edn ")))
  (external SIMPRIMS
    (edifLevel 0)
    (technology (numberDefinition
      (scale 1 (E 1 -12) (unit TIME))))
    (cell x_and3
      (cellType GENERIC)
      (view view_1
        (viewType NETLIST)
        (interface
          (port IN0
            (direction INPUT))
          (port IN1
            (direction INPUT))
          (port IN2
            (direction INPUT))
          (port OUT
            (direction OUTPUT))
....

Looks like HW guys don't need XML though I'm sure some brilliant SW guy will
explain some day that these file should be written in XML to be used by
C#.NET...
When this arrives we could start to have to reboot, several time per day,
our watches, radio receivers, phones, lifts, flash lights,...

  TCP and IPv4 are _very_ good protocols.  (I don't know enough about
  IPv6 to rate it.)  The telecommunications protocols used on the
  gigabit trunks and on the long-haul high-speed interconnects are
  amazing feats of science.  Of the really, _really_ large protocol
  suites, I find ITU's Signalling System #7, the Digital Subscriber
  Signalling System, and the Integrated Digital Services Network
  protocols as well-designed as they are complex.  The Telecom people
  have probably done some of the best work there is in protocol
  design.  Just take a look at how they started out with very low
  speeds, like 300 bps, but over time managed to squeeze 56kbps
  through the feeble phone lines that were not upgraded, then got DSL
  to work across that same old copper wire.  Impresses me, anyway.

  FTP and SMTP are well-designed and fairly small protocols, too.
  (Small does not mean quick-and-dirty: If you spend less than 3
  months implementing a server or client for either of them, however,
  you're doing it wrong, as almost every implementation that is not
  regarded as a huge monstrosity are.)

  As an example of a very good way of separating abstractions and
  representation, ASN.1 and its encoding rules are worth a study, such
  as what is done with SNMP.

  Finally, if you don't know X.25, you don't know packet exchange, but
  it's _real_ old, now, and probably doesn't bring anything directly
  applicable to people who think in streams implemented on top of
  record-based protocols, except what _not_ to do with streams and
  when it really does make sense to use records.

  _Very_ few of the recent application-level Internet protocols are
  worth the paper they aren't published on.  The sheer lack of design
  behind the protocol that is _actually_ at work in the MIME cruft is
  astonishing, but trying to track that braindamaged piece of shit
  from its inception to its current specification level through the
  numerous RFCs will tell you what a few over-sized egos who had no
  clue what they were doing can do to a conceptually _very_ simple
  thing: Wrap various objects in a textual representation.

  On the other hand, the Network Time Protocol was done by people who
  really cared and also who managed to keep the dolts out of the way.

CORBA is _not_ anything like the best distributed object protocol that
one could define - probably a legacy of the C++ mentality of most of
the original vendors involved.  However, it _does_ work rather well
for many scenarios - most any that someone not in the protocol tool
business would care about.

The nice thing about using CORBA to do this is that it at least
eliminates all the marshalling and unmarshalling of data items
(primitive and structured) in a platform independent way.  Unless your
business is really line protocols or something similar, rolling your
own here with sockets is just a distracting waste of resources.

/Jon

  I disagree with this assessment.  It is the same argument that can
  be made for something overly complex and badly implemented, like
  SGML,  It has also been touted as solving problems that people would
  not be able to solve cheaper by themselves.  That was just plain
  false.  Similar problems crop up with other committee products that
  have tried to solve overly broad and thus non-existing problems
  instead of trying to solve real problems of real complexity.

  How much effort does it take a good system designer to come up with
  something better than XML for any _particular_ purpose?  If he
  grasps the good and useful qualities of SGML, a simplified syntax,
  slightly general tools, etc, can be written in a fairly short amount
  of time, and it costs less to develop, maintain, and use than a
  full-fledged SGML system does, because, importantly, SGML's full
  generality has surpassed the point where additional features bring
  nothing but more costs than benefits to the equation..  A particular
  markup language (or something similar) is also easier to learn for
  someone who has _not_ spent much too much time studying the original
  -- like one one who is only moderately familiar with the original.

| CORBA is _not_ anything like the best distributed object protocol
| that one could define - probably a legacy of the C++ mentality of
| most of the original vendors involved.  However, it _does_ work
| rather well for many scenarios - most any that someone not in the
| protocol tool business would care about.

  Precisely, but having designed a few protocols that have been in
  wide use for over a decade each, just how much effort do you think
  it is to design new ones?  The reason protocol design is hard is
  that people are told not to do it, and thus do not investigate the
  issues involved.  This also leads to incredibly bad protocols in
  use, because the users are told they are too hard to understand.

| This has not been true since at least CORBA 2.0.

  I'm "happy" to hear that.

| The nice thing about using CORBA to do this is that it at least
| eliminates all the marshalling and unmarshalling of data items
| (primitive and structured) in a platform independent way.

  You know, this is _not_ a hard problem.  It is a hard problem in C++
  and the like, but that is because those languages have lost every
  trace of the information the compiler had when building the code.
  That design mistakes has cost the world _billions_ of dollars.  If
  you don't make that design mistake, you don't have to worry.  I
  know, this sounds "too good to be true", but the inability of any
  small C++ program to read C++ source code and do useful things with
  it is the reason for all these weird data-oriented languages.

| Unless your business is really line protocols or something similar,
| rolling your own here with sockets is just a distracting waste of
| resources.

  Well, I have tried both approaches.  Have you?

| Only if you don't make use of threads in making the calls in the
| first place.  If each call is a thread, the "waiting for the
| synchronous reply" is irrelevant.  Most ORBs are multithreaded and
| can easily handle this.

  Right.  Not my experience.  Of course, everything gets better in the
  mythical "future", so do stay with CORBA if you have other things to
  do while waiting for the synchronous request for more features sees
  a response from the CORBA people.  (Sorry, a tad rambling here.)

| Well we have done it at 2000 miles and the results were basically
| "instantaneous" (well under a second).  Using straight X was death.
| Using HTTP was (of course) worse than death.

  How long time do you think you would have spent designing a protocol
  that would have been just as good?

Reality doesn't change...

Network programming isn't easy, and using CORBA in a slightly
different way the designers immagined will get you into a world of
pain. But it all appears so easy in the examples people are lured into
a false sense of security...

I think CORBA should be seen as "just" a RPC call round ASN.1, and
people should use its 'oo'-ness as sparingly as possible. Just have a
Server object replying to simple (non-object) commands with simple
(structure) replies. Stick to this and you'll be safe. Ignore the
C++/java people why cry you don't know OOP and want to make everything
an object. Remember that a RPC is inherently slow, so limit the amount
of calls needed to do a typical operation, also (unless you use
advanced stuff, not implemented in the lisp servers AFAIK) objects in
CORBA as horribly costly: they take about 1K of space on the wire and
most deadly of all *the never, ever die*. An object in CORBA, once
created and given out *has* to remain alive for the duration of the
lisp process.

Groetjes, Peter

  cb> Are there any non-C++ implementations of AMI? All the
  cb> documentation I've seen on AMI has been quite tightly tied to
  cb> C++...

not that I know of. Probably future releases of Orbix2000 for Java
will include AMI.

a CORBA method invocation does not necessarily involve network
communication: if the client and server are collocated (reside in the
same address space), the ORB can use a simple method call.

  pve> objects in CORBA as horribly costly: they take about 1K of
  pve> space on the wire and most deadly of all *the never, ever die*.
  pve> An object in CORBA, once created and given out *has* to remain
  pve> alive for the duration of the lisp process.

I haven't used the commercial CL ORB implementations, so I don't know
what they are capable of. However, I don't see anything in the CORBA
specifications which requires an object to stay alive indefinitely.

You publish the availability of a service through an object reference.
On the host+port specified by that object reference, the object
adapter is responsible for dispatching incoming invocations to the
appropriate servant (a programming language entity which implements
the service). In CL, a servant is probably represented by an instance
of a CLOS class.

The object adapter maintains a mapping between object identifiers (the
server-specific component of an object reference) and servants. When
publishing a service, the only thing which has to remain alive
server-side is its object identifier (a key in a hashtable, say). You
aren't obliged immediately to incarnate a servant: the object adapter
can do so on-demand. Once a servant has serviced a request the object
adapter can decide to kill it (called "etherealization"), to be
restarted when a new request comes in. You can also have one servant
which services requests for multiple objects.

See:
http://www.franz.com/support/documentation/6.0/orblink/doc/orblink-id...
http://www.franz.com/support/documentation/6.0/orblink/doc/standards.htm

Groetjes, Peter

For those that want to learn about protocols break open the OSI spec on
the reference model. Anybody remember the ISO spec number? I don't have
it handy anymore.

When I think what are good protocols I think they should have state
models that are totally defined.  Modern protocols seem to be lacking
proper definitions of their state machines and are more ambiguous in
their specification.  X.25 is very well defined with state tables
unambiguously placed in the specification.  Also I have found a good
feature of a protocol is that is does not try to do too much.  This
means that its finite state machine does not exceed 15 states with about
15 events.  Do not ask me to justify these arbitrary numbers.  I have
just found from experience that if the state machine exceeds these
limits there is probably more than one state machine that can be layered
into a protocol stack.

I am still divided on the issue of creating protocols between
cooperative machines/processes or going for a distributed OS view where
everything is one large memory space. I thought that is what CORBA is
offering, one large memory space.  There is room for both of them, a
distributed OS at some lower level must implement protocols.  Does a
programmer have to see that?

pve> objects in CORBA as horribly costly: they take about 1K of
pve> space on the wire

You're talking about object references, which can actually be much smaller
than 1 k (IOR strings don't tell you that much); it depends a bit on the ORB
vendor. The minimum it must contain (for IIOP) is protocol name +version,
hostname, portnumber, adaptor name and object identifier (the last two are
mangled into the object key, in a vendor-specific fashion). These components
would be necessary for any object reference that is supposed to work across a
network. I believe I read that an estimate of the lower bound on the data in
of an object reference is below 100 bytes (sorry, no 'references').

pve> and most deadly of all *the never, ever die*.
pve> An object in CORBA, once created and given out *has* to remain
pve> alive

Same is true for telephone numbers and URLs ... Somehow, the real world isn't
quite aware of this :-)

pve> for the duration of the lisp process.

Peter> In at least one cl-ORB you can force the deletion of the object (this
Peter> was added on request)

This sounds very suspicious to me. If an object should be deletable by a
client, then it's interface should bloody well have a delete() or destroy()
method or some such (and many do).

Peter> and further operations on the object result in
Peter> an exception.

exactly.

Peter> Note that the published CORBA mapping claims that Lisp objects do not
Peter> need the destroy method (sorry don't know the exact name)

'the destroy method' ? There is no such thing. It depends on the IDL, which
has nothing to do with languages.

Peter> as Lisp
Peter> doesn't need it...

??? If an IDL has interface Car { ...; destroy(); }

then this has nothing to do with Lisp garbage collection: it's simply a
method to have clients delete a particular Car object.

(yes)

This so-called DefaultServant approach is part of the POA, and therefore must
be implemented in _all_ ORBs that CORBA 2.1 compliant (were now at version
2.3a, btw). If not, they are not compliant. Incidentally, the DefaultServant
can be simulated by using a ServantLocator approach, where the preinvoke()
method always returns the same servant. It prolly is slightly more costly
though.

Peter> IMHO if you have a need for a large number of objects in your design
Peter> it's likely you're doing something wrong.

Why? It turns out that using objects in our field (bioinformatics) is very
natural and does make things easier. Since we're talking biggish databases
(1.5 million biological sequences, with many more sub-objects) here, we
inevitably have large numbers of objects. This has nothing to do with the
design: using 1.5 million little pieces of text (which is actually were we
come from) is no option.

When you're talking about large number of objects (all of them being equally
important), you _must_ have persistency behind it. Given this, CORBA
makes it easy to -- completely transparently to clients -- instantiate
some 'working set' of all those objects on demand (which, BTW, no other
distributed (object?) technology comes close to provding).

  ASN.1 is not a protocol design.  It's a data description language.
  The full name is Abstract Syntax Notation #1.  It does no more than
  define structures that are named and identified by mutual agreement.
  To use ASN.1, you have to choose some encoding rules, and there are
  unfortunately too many of them, but the Basic and Distinguished
  Encoding Rules are actually OK.  (C programmers hate them, because
  they make life so much harder in an untyped language like theirs,
  but if you know what kinds of objects you get, as in dynamic types,
  you can easily deal with these things.)

| They haven't solved any of the real problems but they added heaps of
| complexity in the process.

  I can't agree with your assessment.  Would you like to explain what
  you consider the real problems they have not solved and what the
  heaps of complexity added have been?

| Unfortunately, it's caught on in some part of the IETF world, so I'm
| considering writing some tools in CL to make it livable.

  I'm surprised that you haven't, already.  ASN.1 is about typed data.
  It does not live well in a "typed variable" approach to programming.

  What does this mean?  It makes absolutely no sense at it stands.

| The whole conceptual framework of protocols I think is best
| described by the OSI reference model.

  Not at all.  Only some frameworks are describable in ISORMOSI,
  specifically those that actually used it while being described.

  Please note that the full name is the _ISO_ Reference Model for Open
  Systems Interconnection.  It's their model.  It is _not_ a model
  that explains or can describe all Open Systems Interconnection.

  See The Elements of Networking Style: And Other Essays &
  Animadversions on the Art of Intercomputer Networking by
  M. A. Padlipsky for a full coverage of this.  ISBN 0-595-08879-1.

| TCP and IPv4 only live within the context of the data link and
| physical layer protocols that carry them.

  This is false.

| All the issues of reliability, delivery and such need to be looked
| at as protocol stacks.

  If you use the ISORMOSI on the Internet, you will end up with very,
  very weird attitudes towards a very fine technology, sort of like
  the people who believe that now that we have 8-bit byte-adressable
  machines, the concept "byte" is conflatable with "octet", and then
  their heads explode when other concepts of "byte" present themselves
  to them.

| For those that want to learn about protocols break open the OSI spec
| on the reference model.  Anybody remember the ISO spec number?  I
| don't have it handy anymore.

  I would recommend against ISORMOSI until you understand whence it
  came, that is: the history of protocol design preceding it, both
  within the CCITT/ITU and the ISO camps and without either.  There
  are very different cultures at work, here.  That's a contributory
  reason that TCP/IP won.  "If ISORMOSI really was so great, how come
  nobody talks their protocols?" actually applies.

| The major problem with the Telecom protocols I have found has been
| their defintions of their protocol data units which place a premimum
| on reducing the number of bits in network traffic.

  You _really_ want this, but perhaps you need to work with these guys
  to realize that 1% more protocol overhead means 1% more money to buy
  the same bandwidth, or slower deployment due to higher costs, etc.

| This makes the PDUs hard to parse.

  That does not follow _at_all_.

| A great example of this, if you can get the spec is IS-634, the
| control protocol between Cellular Base Stations and Management
| systems.   Whoever dreamed that up has never programmed a computer
| before.

  I'm making an unsubstantiated guess here, much like you do, and
  would counter that he probably does protocol implementation in
  dedicated chips.  So would you if you dealt that these things.

| Also the telecom protocols (at least the cellular ones) are tending
| to define the protocol state machines in SDL which allows protocol
| designers to leave gaps of what happens when error conditions occur.

  I have worked a little with SDL, but that is years ago, but most of
  what I have worked with in the Telecom world has been described with
  SDL (Z.100) or 'variations" like Estelle.

| ASN.1, what can I say, another language to learn.  Liking Lisp, I
| think a Lisp external syntax should be used for a platform
| independent encoding of PDUs.  It takes more bytes but it transports
| well (and I can see it clearly on a data scope).

  Notice that ASN.1's encoding rules are actually parsable with the
  same ease as (most) Lisp objects (that is, excluding symbols, which
  are a pain insofar as they overlap syntactically with numbers).

  I see a very clean mapping between ASN.1 and Lisp, qua languages to
  describe and encode data.

| I am still divided on the issue of creating protocols between
| cooperative machines/processes or going for a distributed OS view
| where everything is one large memory space.  I thought that is what
| CORBA is offering, one large memory space.  There is room for both
| of them, a distributed OS at some lower level must implement
| protocols.  Does a programmer have to see that?

  If you do the global memory thing, at least figure out the need for
  cache coherence and propagation protocols.  Nobody I talked to about
  CORBA have had the faintest clue what that would entail for CORBA,
  so I gave up talking to these guys.  (This was not in a Lisp
  context, BTW.)

  About half of what I have done over the years has had to do with
  ensuring that lots of computers very far apart agree to what the
  world looks like at exactly the same time.  "One large memory space"
  sort of covers it, since the idea is that through intelligent
  protocols and very careful programming, lots of computers actually
  get the same values without having to ask anyone _else_ for them.
  There are several interesting products that offer some of this
  functionality for databases, messages servers, etc, these days.

  I'm not sure a programmer needs to see everything, but being able to
  ensure that you have the information you need in time requires work,
  and I fail to see how anyone but programmers should do that.  Like
  TCP and IP do not live quietly down in the basement, but actually
  extend their services and _presence_ all the way up to the
  "application layer" (and you _really_ want this, by the way), there
  is no way you can avoid knowing about "the underworld" relative to
  your actual interests.  If you think you can avoid it, you are only
  saying that you are willing to accept some set of default behaviors,
  but if you don't know what they are, you are also likely to be wrong.

  The belief in cleanly separated layers is almost a religion.  It has
  no basis in reality or experience, but there is nothing that can
  make someone who believes in it stop believing in it.  There's
  always an answer for why it's a good idea despite all the hairy
  stuff they have to deal with.  Just like the whole ISORMOSI thing.
  And almost like the belief in structured programming and object
  orientation where the only time anyone can be trusted to use them
  well is when they know when _not_ to use them.

  Make that a more general assertion: If you think you have an answer
  for everything, the answer is probably not right for anything.  If
  you have an answer for some things, the answer is probably right for
  those things.

TIA O:-)

  Well, besides the hammer, nail/thumb analogy, you might decide not
  to "know" where your objects are.  This could lead to all _kinds_ of
  interesting performance pessimizations, just one of which is CORBA.

And the old copper wires aren't probably the problem, anyway.  They
should not be much worse than your ordinary 10Base-T twisted pair
ethernet wire.  The problems are probably more with the filters along
the way that have been optimized for base-band speech signals.

Everyone going off and reinventing this stuff on their own, _even if
they know they can do a better job of it_, when it is not the primary
work they are involved in, and when the available offerings will not
negatively impact this primary work is just counter productive.

This in no way suggests that it is not appropriate or sensible or even
"a good thing" for someone to focus on defining and producing better
such offerings and making them generally available.

Now, an order of magnitude better (depending on how that gain is
distribted), that would be a different story, but the effort would go
up dramatically and you are then basically in the distributed object
protocol _business_ (or at least you _should_ be).

/Jon

Humble old AppleTalk has been quietly driving up to 2 km of phone wire
(using the Farallon "PhoneNet" adaptors) for nearly 15 years now at
230,400 bps.  That's the same speed as a lot of people's budget-price
ADSL runs at, acheived using a handful of simple passive components
(transformer, resistor, capacitor) plugged into a serial port.  Now of
course ADSL can do better than that -- more like 2 Mbps - 6 Mbps -- if
the telco wants you to (or you pay them enough), but it *has* had a few
more years of development and those Nokia M10's aren't cheap at all.