One project I worked on chose XML as the basis for the data format because

1. It is human readable and editable.
2. It is language and platform independent, more or less.
3. It is simple.
4. Vendor and tool support is good.

XML does not solve a lot of problems.

It does not define a content model. Until recently there was no standard
method to include binary data. Version control. So we added solutions to those
to our in house standard.

It has worked well. To me it has evident advantages that binary formats lack
i.e. ease of debugging. You can look at a message and see if it is OK which is
not possible e.g. with ASN.1. Free/cheap friendly editors exist.

If someone claimed XML makes data and applications independent, they were
lying. Unfortunately that it all too common in IT. We have found though that a
message passing architecture does help keep applications from becoming
pathologically coupled. However the determined programmer can still make it
happen if he wants to.

However I don't think anything known to date will solve the problem of making
programs data independent.

To do that would require creating a universal knowledge schema from which all
message types could be subclassed or otherwise derived.

  Do you prefer XML to Common Lisp?  Have you ever implemented anything that
  talks ASN.1 in the native language compared to implementing something based
  on DOM?

  Knocking ASN.1 is common and accepted in some circles.  I have yet to meet
  one person who has been critical of ASN.1 who has any idea what it is like.
  It has this property in common with Common Lisp -- it is usually attacked
  only by people who have no idea what the language is like.  It is an amazing
  property of people who think they are so smart they do not need to know
  something before they make pronouncement about it that they do not even
  understand that their credibility is blown to bits.  It is probably part of
  the exaltation of illiteracy in the part of society that works with IT.

  By the way, the SGML Document Interchange Format (ISO 9069) uses ASN.1 to
  ship SGML documents around.  I wrote an implementation of SDIF in three days.
  Test runs showed that a major CALS application consumed approximately 40% of
  the character count of the SGML file, and with the then commonly available
  tools to parse and process SGML documents and ASN.1 processors, the SDIF data
  stream took around 1/200th as much CPU time and about 75% of the memory to
  reconstruct the identical in-memory version of the document.  This experiment
  was among the many data points that led me to conclude that SGML is insane
  and that those who think it is rational to require parsing of character data
  at each and every application interface are literally retarded and willfully
  blind.  Also, an SDIF data stream can only represent a validated document and
  the kinds of errors you get when parsing ASN.1 are unforgiving.  There is no
  doubt in my mind that if SDIF had won over the insanely verbose text format,
  even things like HTML would have been moderately sane.  Not to mention the
  fact that images could have been carried in the same data stream.  The world
  would have been a better place if SDIF had won over HTML, and if the nutjobs
  who "invented" XML had been moderately in touch with reality, they would have
  realized the insanity of requiring the verbose end-tags and the stupid syntax.
  XML-RPC and SOAP and the like could have been fairly inexpensive things.
  But, alas, people prefer buggy text formats that they can approximate rather
  than precise binary formats that follow general rules that are make them as
  easy to use as text formats.  Rationality is not part of the SGML philosophy,
  however, and SDIF was mainly an effort to keep the ODA and ODIF folks at bay
  and was a purely political stunt, not intended to be implemented.  When I
  went ahead and did it, I was not exactly applauded for the effort.  The fact
  that it was /vastly/ more efficient in all respects than the stupid character
  syntax was /most/ unwelcome by the community.

  So, what is your actual experience with ASN.1?

  Look, you are repeating this as if it is an argument.  It is like saying that
  Common Lisp does not define an application and shows a lack of insight.

| You can look at a message and see if it is OK which is not possible e.g.
| with ASN.1.

  This is just plain wrong.

| Free/cheap friendly editors exist.

  No difference from ASN.1 here.

| Searching the sgml bib for your name produces 0 hits...

  Why make such a fool of yourself annoying people on purpose?  What is /wrong/
  with you?  http://www.oasis-open.org/cover/biblio.html

| My main beef with XML is that it is very oriented to 'text'.

  Wrong on this count, too.  Do you have /any/ clue what you are talking about?

| Thus the lack of support for binary data, etc.  That just reflects its
| origins.  What do 'text guys' care about binary data, or compaction for that
| matter?

  Enough to make very good to excellent support for it, given the framework.
  XML's /origins/ were SGML and SGML has NOTATION declarations to communicate
  the meaning of non-SGML data held in external entities.  Most XML nuts do not
  understand the entity structure in SGML so reinvent it badly, but generally
  fail so miserably that they should feel ashamed.

  You clearly are clueless and are not at all ashamed to demonstrate it.

| However I don't think anything known to date will solve the problem of making
| programs data independent.  To do that would require creating a universal
| knowledge schema from which all message types could be subclassed or
| otherwise derived.

  *SIGH*  This does not even merit comment.  You are in a /Lisp/ newsgroup, for
  crying out loud!  Why are you posting here?  Did a google search for XML and
  just had to chip in or something?

A related point, it seems to me, is that reinvention is more-or-less
inevitable if the existing art is so bad that coping with it is too
hard.  So one escape from the wheel of reinvention is to try and come
up with systems that are well designed in the first place.  This is
obvious of course, but it explains quite a lot.

Two concrete examples.

1. Anyone who's read my previous article(s?) in this thread knows I
   think redoing emacs in scheme *or CL*[1] is insane.  Well, the
   underlying reason for that is that I think the existing system is
   not too bad.  Of course Emacs Lisp is far from perfect - I think
   it's probably the worst Lisp in common use - but it's not *bad*,`
   and the `API' to emacs is pretty usable.  And there isn't some
   enormous wall you have to climb before doing any programming of
   emacs as far as I can see - people start by setting a few variables
   and end up writing GNUS or something. So we have an acceptably good
   system already - the desire to reinvent it indicates something bad.

2. Anyone who has read the drivel I write for a bit longer knows my
   feelings about XML.  Every time I have to do something for which
   XML might seem suitable, I just invent something else, largely so
   that I don't have to cope with the ideas behind XML - other
   people's ideas.  Am I falling into the trap Erik describes?  Well,
   yes, I am, but I think this is OK.  XML and its encrustations seem
   to me to be classic examples of the existing art being just too bad
   to deal with.  XML itself is essentially too simple to do anything
   useful.  On top of it there is now a huge mass of stuff, the great
   majority of which is not well designed, which probably also fails
   to solve my problems, or does it in a way which means that simply
   dealing with the XML standards becomes about 90% of the code I need
   to write - bloating the effort involved in solving the problem by a
   factor of 10 over a reinvented solution.

So I think that in order to avoid endless reinvention it's *critical*
to use systems which are good enough[2] in the first place.  There are
rather few of those around in IT, because they are *very* hard to
produce.  The C/Unix pair is possibly one (but I think probably not),
Emacs/elisp is, I think, one.  Common Lisp is the only system that I
am pretty sure is one.

--tim

Footnotes:
[1]  That may not have been apparent to people I guess.

  This can be a good reason for re-invention, but then the smart re-inventor
  knows the past and what he needs to be better than.  If re-invention only
  means that one treads the same old path all over again, it is very bad.  And
  of course knowing the past is going to be hard.  That is why psychologists
  have to study so many completely bogus psychological theories -- left to
  their own devices, past psychologists have invented all these wacky ideas,
  and future psychological inventors please take note: they were discarded.

| So one escape from the wheel of reinvention is to try and come up with
| systems that are well designed in the first place.

  The force of genious lies behind a significant shift in perspective more
  than in new ways to perform old tasks.  We humans are pretty annoyingly
  limited when it comes to getting stuck in a well-known pattern and those who
  are not, tend to be more wacky than genius.

| 1. Anyone who's read my previous article(s?) in this thread knows I
|    think redoing emacs in scheme *or CL*[1] is insane.

  I once started down that path, but having spent a couple hundred hours on a
  Common Lisp-based Emacs, decided that it was going to take a couple thousand
  hours before I had a system I would use myself and would be able to attract
  others to join in building.  I still wonder what would have happened had I
  not gotten fed up, but the better system would also be harder to maintain
  than to build.  Emacs moves on, and if I were to build another Emacs, would
  have to duplicate a /lot/ of effort.  It would, in brief, be what I would do
  for the next few years.

| 2. Anyone who has read the drivel I write for a bit longer knows my feelings
|    about XML.  Every time I have to do something for which XML might seem
|    suitable, I just invent something else, largely so that I don't have to
|    cope with the ideas behind XML - other people's ideas.

  What can I say?  I wasted 6 years of my life on SGML and related technologies
  only to find that when I wanted to translate my experience and knowledge and
  significant grasp of this technology into a book that would teach what I had
  found to others, I had to look real hard at all the braindamaged things that
  I had been willing to sweep under the carpet and found, to my horror, that
  SGML, once understood, could not possibly be worse implemented than SGML
  itself -- if you get the very valuable core ideas behind SGML, SGML was self-
  defeating because it had to cater to a huge number of idiotic requests and
  requirements that took away from its ability to express what it wanted to
  express, and it became such a horribly application-dependent language that
  you would have to retarded to write anything worse.  I could not publish this
  in a book that would be a conceptual introduction to SGML and abandoned the
  whole language and my investment in time.  Once every two weeks or so, I get
  mail from somebody else who have discovered the same thing after actually
  having tried to use SGML for something other than a stupid markup language.

|   XML and its encrustations seem to me to be classic examples of the existing
|   art being just too bad to deal with.

  I takes considerable time to understand this, however.  If you do /not/ grasp
  the core ideas behind SGML and XML, you will probabl invent something worse,
  such as an ad-hoc binary format, and since most people would rather die than
  think (in fact they do), XML is better than what a braindead ignorant fool
  would invent from scratch.

| So I think that in order to avoid endless reinvention it's *critical* to use
| systems which are good enough[2] in the first place.  There are rather few of
| those around in IT, because they are *very* hard to produce.  The C/Unix pair
| is possibly one (but I think probably not), Emacs/elisp is, I think, one.
| Common Lisp is the only system that I am pretty sure is one.

  I think Common Lisp is the only fundamental IT system that is better than
  just Good Enough.  I am, however, dismayed at what every single vendor does
  with the language when it comes to presenting Common Lisp to "modern" stock
  computers.  For instance, the lack of the full range of hardware integers
  really, really bugs me.  I got a suggestion in the mail to look at the 8 new
  xmm<n> (128-bit) and mm<n> (64-bit) registers in the IA32 platform, and found
  that after spending a lot of time on programming specifically for these,
  there is absolutely no reason to stick to 32-bit integers on IA32.  The too
  few legacy registers should be used for interaction with memory, and the mmx
  registers should be used for integer arithmetic.  Not doing this is not using
  the Intel platform for what it's worth.  However, this is not done in any of
  the Common Lisps available for the Intel platform, although it would give the
  compiler writer 8 64-bit integer registers and 8 128-bit numeric registers
  that could both be split into smaller registers if need be, and this would
  solve so many of the register starvation issues on that platform.  This is
  the kind of thing that bugs the hell out of me.  Then there is the Unix
  integration, which is abysmal and forces people to use other languages to do
  anything resembling systems programming.  Dsspite all I know about Unix and
  Common Lisp, I do not feel that I write Common Lisp programs /under/ Unix.
  It is as if the Common Lisp programs run on a separate machine somewhere.
  And given that "feel", why not exploit the work in virtual machines and JIT
  compilers and related efforts in the Java and even the .NET community and let
  Common Lisp code become interchangeable and portable between all the
  implementations so that there could be a component market and you could
  actually develop on one platform and deploy on any number of others.  These
  are the developments in a significant part of the software market these days,
  and it bugs the hell out of me that I have to choose whether to continue to
  use Common Lisp if I want to be a part of this.  The effort to make a JVM run
  inside Common Lisp, for instance, is a really good idea.  Now we need a CLVM
  that actually supports CLOS.  Notice that these are not "user land" things,
  they require vendor support and just as much "develop a new system to use the
  new ideas" as any other revolution, but at least the programming language
  does not need to change, and we should be able to leverage off of all the
  existing Common Lisp skills, the standard behind it, textbooks, courses, and
  the long experience with the language to get things right.  However, it takes
  people who actually /like/ Common Lisp to do this and who are professional
  enough to set aside their personal issues when implementing the standard
  faithfully so others can trust that the specification will be sufficient and
  they can concentrate on their application and not have to waste time on
  subtle incompatibilities.  This would be slightly more interesting than a
  Common Lisp Emacs, but I actually think a Common Lisp Emacs would be so much
  easier to build once such a system was in place.

(X)Emacs would improve if it adopted and incorporated CL as I mentioned
previously because

 1) CL is a friendly, cleaner language to develop apps and extensions
    in. (my own opinion, of course)
 2) CL is well prepared for handling such a huge application, with so
    many sub-units, sub-programs, features, and developers
 3) CL is an ANSI-standardized language with a strong history and
    excellent community.
 4) CL would allow for a lot of simplification and performance gain
    for the existing (X)Emacs distros.  One would be the elimination of
    (require 'cl), which (as you can see) many developers enjoy using.

It is important to note, on that last item, that (require 'cl) is (or at
least used to be) supported by GNU Emacs *only* in situations where
there were compile-time effects (i.e. macros).  XEmacs allowed packages
to (require 'cl).  GNU Emacs always seemed (to me) to shy away from CL.
I don't know too many details here, but I'm not terribly amazed that the
same community wanted Guile over CL.  I've worked with a few Scheme guys
to dissociate them from the "Lisp" community.  It's not just the
language definition...it's the mindset of the developers.

Of course, it would be ludicrous to have to go through wads of elisp
code, convirting it to something else.  That was never an option.  The
idea here was to provide additional programming interfaces into the
emacsen.  I've seen some amazing feats in elisp of emulating
CL...everything from the 'cl package to a reasonable CL reader and an
implementation of CLOS.  But these are all memory hogs, slow as hell,
and still lacking (i.e. not up to spec).  A full CL is, in my opinion,
the right answer to the Emacs issue.  Guile is something that would make
(X)Emacs *worse*.  I think that is what you just don't get.  So if you
think that I am unnecessarily remorseful, it is because of the
infectation of the Scheme/Guile community dealing with Lisp-related
issues.  Keep diluting the already broad notion of what "Lisp" is with
Scheme Guile, and you just make the problem worse.

It seems like this is one of those cases.  In that case, good night and
sleep well.

Scheme is a degenerate -- a mutation, if you will.  There's enough
similarity to notice one of its ancestors, but more to show its
divergence (or deviance, depending on how you see it).

It's just as in biology: most mutations are bad, though some are good.
Scheme is a bad one.  But like a cancer, it grows, feeding mostly on the
vast pool of beginners who think it's easy and elegant.  Unfortunately,
it has grown into this newsgroup, as well as into applications some of
us care about.

Dorai can knock me all [s]he wants.  But it won't change the fact that
Scheme is viewed as a perversion, lacking some of the best features of
Lisp-family languages, and adding little useful.

I'm very interested in learning what these "core ideas" are, since I
suspect I am missing something significant (and/or obvious!), and
would appreciate a pointer to any available literature (including
any of your articles archived at <URL:http://www.oasis-open.org/> or
<URL:ftp://ftp.ifi.uio.no/pub/SGML> or elsewhere). Even just a set of
"bullet items" (one-liners) would be appreciated as hints for what to
go look for.

I assume you're *not* talking about trivial stuff like "<tag>foo</tag>"
syntax [which to me is just verbose way of externally representing
(serializing) trees (though not DAGs or more complex structures),
and which S-exprs do a much better job of (especially if you allow
"#n=" & "#n#" for the more complex graphs)], but about one or more
deeper issues that you've hinted at several times but that (not having
studied SGML per se) I don't have the context to follow up on without
more explicit pointers.

Thanks in advance,

-Rob

However, Emacs Lisp is slow, and also ugly in several respects.  This
became apparent when things like web browsers were attempted in elisp.
To cope with the problem, and to make programming more sophisticated
applications inside Emacs that would not require diving into C caused
serious discussion about how to make Emacs more easily programmable,
execute extension code more efficiently, and yet provide a clean
interface to legacy Emacs Lisp code.

There were many proposals.  Two were CL and Guile, and I'd say that
within each of these are pros and cons.  For me, the pros for the CL
camp were obvious:

 o excellent CL compilers exist in the open source community
 o CL is powerful and general enough to handle processing Emacs Lisp
 o CL is more in the spirit of the original Emacs Lisp language
 o CL is an ANSI-standardized language, and already had an impact
   on literally thousands of Emacs Lisp programs, meaning that by
   using CL, you are not turning your backs on the vast pool of
   existing elisp contributors.  You are giving them *more* rather than
   something completely different.

I can go on for many pages about all of the features of CL that I think
are useful, and would make adding to Emacs a joy, but some of the main
reasons are the few above.

If you wanted to start an Emacs-like editor from scratch, without
understanding the history, the elisp developer community, and the elisp
language features and existing codebase, then yeah...sure...do it in
Scheme or whatever language you want for that matter (several people
have already).  But we were talking about Emacs, and that's much more
than just an application at this point.

  Stallman wants to replace Elisp with Guile because Elisp has some
  problems that might not be there had he taken more care the first time
  around.

I agree with this fully.  But then again, I don't think that RMS really
knew what a phenomenon Emacs would become to care enough to do things
the right way.  I get a strong feeling that RMS has some hacker
tendencies.  That, coupled with 20/20 hindsight of what Emacs Lisp
should have been, can sometimes hurt his image a bit.  It's the part
about RMS supporting Guile as the evolution of Emacs Lisp that gets me.

Manual-inches, incidentally, is the best known unit.  Some people have
suggested things like manual-words but this is not enough as it
ignores typographical issues - manuals with too few words per inch are
typically very hard to use as the density of information is so low.
Unfortunately even manual-inches has some problems as it encourages
very small type sizes and long lines...

As an example, pointing to a URL with a broken search engine does not help.
Google finds some text by EN, but it is mostly more of the same type of
material we are unfortunately already so familiar with.

I don't think anyone has anything to gain from continuing this discussion.

Anyway, XML verus ASN.1 or SGML is not really relevant to lisp, except by way
of analogy. In some ways it is similar to Lisp versus C as in the "Worse is
Better" paper.

  Geez, you really are retarded.  Such rank incompetence at finding
  information is simply not possible in this day and age.  Your laziness and
  dishonesty speaks volume about your character.

  Since you need to be speoon-fed and need assistance finding your own butt,
  try <http://www.oasis-open.org/cover/bib-mn.html>, a link found on the very
  same page you incompetently could not read.  If you know how to avail
  yourself of the search function in the browser, use it.

  Jesus Christ, the idiots that storm the Net with their opinions these days.

| I don't think anyone has anything to gain from continuing this discussion.

  The people who hire you need might gain from it.

As an individual, I am just trying to deal with
reality too, and am observing that since you made an
explicit laundry-list of goodies (which you should be
thanked for), that you will find that you can deal with
Guile-Emacs if it happens.  

Software is not always about what we want exactly how
we want it -- there is a large social component
to it.  I am trying to find good points that I can deal
with.  Yes, I think Guile-Emacs will be "good enough",
to use another poster's phrase.  It will be dealable
with.  

Do I personally absolutely want Guile-Emacs?  No.  Will
I use it if it came about?  I'll probably have to, if I
want to keep in touch with the rest of the (not just
Lisp-) world.  Is it really worth my while to
spend extraordinary amounts of negative energy putting
it down?  I'd say not.

You say you've looked at my inter-language code.
That should tell you something at least about my
pragmatism in these matters.  

  To someone familiar with (Common) Lisp, it is much easier to grasp, and may
  indeed appear obvious or trivial.  These concepts are central:

· declarative elements with contents (not commands).
· hierarchical element structure.
· element structure validation (to a content model).
· element structure normalization (default values).
· element identity and referencing.
· declared entities with uniform references.
· declared notations.
· (abstract) public (entity and notation) identifiers
· a syntactic layer between entity and application.
· an access layer between file systems (networked, etc) and entity

  Less central, but still important with respect to the processing model:

· separate specification of position-in-hierarchy-dependent information.
· possible specification of parent, child and left and right sibling relations.
· dynamic inheritance of processing information.
· complete mapping between document and processing element hierarchies.
· marked sections with conditional inclusion.

  Things that the SGML community think are important, but which only detract
  from the real issues and confuse people tremendously:

· attributes to elements, notations, entities.
· attribute types
· character entities.
· parameter entities.
· marked sections with special syntax for contents.
· elements with special syntax for contents.
· tag minimization and omission.
· the actual syntax.

  Neither list should be deemed exhaustive.

| I assume you're *not* talking about trivial stuff like "<tag>foo</tag>"
| syntax [which to me is just verbose way of externally representing
| (serializing) trees (though not DAGs or more complex structures), and which
| S-exprs do a much better job of (especially if you allow "#n=" & "#n#" for
| the more complex graphs)]

  The rank incompetence of SGML and XML aficionados has led people to believe
  that the syntax cannot represent anything more complex than simple trees.
  This is just as wrong as claiming that Common Lisp cannot express circular
  lists and other forms of reused objects to preserve identity throughout the
  printed form of a structured.  Using ID and IDREF is exactly analogous to
  using #n= and #n#.  (And just as they have special syntax in Common Lisp,
  they should have special syntax in an attribute-free SGML.  It is the only
  thing that /needs/ attributes in the SGML syntax.)

  I tend to measure things by the weight and thickness of the paper and the
  spaciousness of the typography of the books you find on the market about
  something.  I have seen books on Visual Basic with the same page count as
  books on C but with 3 times the shelf space.  I have seen books on HTML with
  same amount of contents as books on Ada with 9 times the volume.  I have come
  to believe that large print, thick and heavy paper, and wide margins and
  oversize leading is indicative of the expected intelligence of the reader.
  If the reader is expected to be unable to concentrate or experiences mental
  fatigue just by looking at a page of text without oceans of whitespace, the
  material is probably geared towards people whose reading skills plateaued
  before they entered high school.  Compare children's books and books on Web
  Duhsign or other X-in-21-days books.  If the reading level of a specification
  is below college level, chances are the people behind it are morons and the
  result morose.

  If typography and reading level are comparable, manual-inches is probably a
  good measure, but a children's specification for something may be thinner
  than a solid work of engineering that it would actually take less time to
  grasp because it is so hard to sink to the level of children who need to be
  told things over and over and usually do not remeber subtle differences from
  repetition to repetion like reasonably smart people do.  (At least I find
  that I cannot read material written by people who are too stupid.  This was,
  incidentally, how I first began to understand that sports in the newsmedia is
  /intended/ to keep people in a semi-comatose, non-thinking state of mind
  where cheering for some idiot gang of testosterone bombs could be regarded as
  recreationally rewarding.)

--
Erik Naggum, Oslo, Norway

--tim