What I think I need now is what I needed at a comparable point in my Java journey -- a simple, on-line, indexed, HTML version of all the functions
described.

Something very much like Javadoc does.

I don't really want another book (I have a couple more, but nowadays, books are awkward and I flit around several computers and between home and
work).  As much as I admire Knuth, I don't want stuff in tex format (which I don't yet have any way of handling, especially a whole directory's
worth).

If it isn't HTML, it does have to be in a format that isn't Unix-specific and one which will work on a variety of platforms readily without me
learning another damn tool.

In short, I need a place to go to "find out" what I don't know about COMMON Lisp, function-by-function as I need it.

http://www.xanalys.com/software_tools/reference/HyperSpec/

http://www.xanalys.com/software_tools/reference/HyperSpec/FrontMatter...

Looks like it will serve very well.  Is there anything better (he asks greedily)?

This will give you a complete reference of the whole
standard-specified functionality of your implementation.  For
everything else, you'll have to take a look at the documentation that
comes with your implementation.

If you use Emacs, you will probably also want to use one of the Emacs
<-> HyperSpec interfaces, which will look up the HyperSpec entry to
the symbol-at-point at the touch of a button.  A couple of them come
with the ILISP distribution.

LispWorks on Windows/Linux already interfaces to the HyperSpec and
Xanalys' vendor documentation in an automated way.

For other implementations/editors/environments YMMV.

Regs, Pierre.

In other words:  Those tools are great for appeasing management that
insists on stacks of documentation (instead of focusing on quality
documentation), and/or for those that are somehow psychologically
incapable of reading source code, but other than that they are an
incredible waste of space.

A useful Lisp programming environment is going to beat any of those
systems hands-down:

- Most of the stuff those tools produce are implementation details,
  which are only relevant when you are changing/working with that
  unit's implementation, i.e. when you have the source-code those
  documents were generated from in front of you anyway.  No need to
  look at the same things in HTML/DOC/PDF at all.

- Those things which define interfaces, i.e. stuff you might want to
  reference while working on other files, are accessible much more
  directly in the Lisp environment, via symbol-completion/apropos,
  describe, doc-strings, argument lists, and if really necessary
  commands like edit-definition, find-callers, etc.

- In fact most Lisp environments offers much better cross-referencing
  tools than the simple-minded cross-linking most of the automatic
  documentation generation tools provide.

So, IMHO, put all of the low-level stuff into comments, doc-strings
and the source code, but let them stay there.  And then write some
high-level design documents, _BY HAND_, because those are documents
that will actually get read, because they answer questions which can
not be gotten more quickly from the source-code, and because they were
written by a human being, and are therefore at least half-way readable
and well-structured (in theory ;).

Regs, Pierre.

Kent M Pitman <pit...@world.std.com> writes:

Cheers

* Marco Antoniotti <marc...@cs.nyu.edu>

(macrolet
  ((dm (real-name arg1 ...)
     `(defmacro ,real-name (,arg1 ...)
       ... )))

  (dm boiler-plate1 ... ))

because it didn't actually define/macroexpand the code it analyzed
(which would have made it interesting when it was used to generate its
own documentation, I suppose).  I remember hacking it to handle the few
odd top-level cases like the above that I occasionally do.

The links for it at www.alu.org have gone stale, and I can't locate the
author.  I have my copy, and I don't see any licensing text in it, so I
suppose I could put it up for FTP, but I would feel better about doing
so if the author said it was OK to do so.  Anyone know how to contact
him?

I'm not familiar with javadoc, but perhaps Mark K's user_man package
is what you are looking for.  (See the CMU AI archives).  It extracts,
I think, the docstrings of functions and produces TeX output for the
function and docstring.

  I do _not_ propose that the URL should be (in) the documentation string.
  What I really wanted was a way to link the _real_ documentation to the
  symbol and documentation type through that function.  I think it would be
  ideal if that function launched the browser in a friendly way, and if it
  returned nil to the caller, that would not be a problem.

  Since documentation strings are routinely discarded by most CL systems,
  the best way to do this would be to use an interface that could take a
  package name, a symbol name, and the documentation type.  This would be a
  lot easier to implement on top of the generic function documentation than
  interfacing with magic values of the documentation string.

  The more standardized something is, the more one must expect people to
  know it.  A brief documentation string would not suffice for symbols in
  the common-lisp package, because anything that could be written briefly
  would already be known to the reader.  The need for access to the full-
  blown documentation in a nicely formatted and consistent form arises from
  the fact that people will look for different minute details and should be
  able to end their search quickly because they are in known territory.

  Writing good documentation is hard, because intelligent people bore
  easily and just put down documentation that explains the obvious, yet do
  not use documentation that fails to be readable.  What should be in the
  documentation is stuff that people cannot figure out for themselves
  faster than reading it.  That is why I love specifications to be online
  and why I think hypertext in specifications is such a wonderful use of
  the technology.  This is why I wrote hyperspec.el.  But now I want the
  same level of documentation for vendor-supplied pacakges that come with
  tons of high quality documentation, although more of a tutorial kind than
  specifications, since it is written after the fact, rather than before.

  The reason I think literate programming is so stupid is the same reason I
  found SGML's promise of "one source, multiple publications" to be so
  hollow and ignorant.  A piece of text has an audience, a context, and a
  set of expectations associated with it.  Until we can program computers
  to deal with this, _people_ need to write text for different audiences,
  different contexts, and different expectations.  When I look at code, I
  expect the documentation there to help me understand it.  When I read the
  documentation, I hope to understand the system design.  If the system
  design expectation is attempted filled by giving me a mere dump of the
  code documentation sans code, I have reason to accuse people of fraud.
  When I read specifications, I expect it to become the background noise of
  my own expectations when I deal with a conforming system.  (This is why
  lack of conformance is so much more annoying than bugs in stuff that does
  nto purport to conform to a specification; I have to become _conscious_
  of my expectation that the specification is actually implemented, and
  that means having to test for it and not being able to believe it.  This
  is like asking people if they understand every word you use, instead of
  assuming that you agree to speaking the same language.)

  Also, please note that a lot more than functions have documentation.
  That often tends to be forgotten in systems that primarly have functions
  as exportable interface concepts.  Classes, slots, types, packages,
  variables, and a lot more stuff have documentation in Common Lisp.  It is
  sometimes hard to understand how people who invented these things could
  be so futuristic that they would foresee such needs.  (It is likewise sad
  to see vendors completely ignoring the specification and reinventing the
  wheel badly when the "need" suddenly arises and half-assed stuff like the
  "web" suddenly appears.)

I don't read c.l.l very often these days but I got this forwarded.

I did this as a student job for some university project with tons of
source files with very strange loaders. And read or some functions using
it will be unhappy if they find a symbol 'somepackage:myfun before you
have an export definition for this. I was not happy with the solution
myself and I had no knowlegde of how to write a decent parser so it's a
hack. If anybody wants to do it right, here's the solution (just code
it):

1) use defsystem to define modules, load them
2) let the documentation tool go through the defsystem definitions and
scan the files with read/macroexpand/etc (the files are all loaded and
let's say well-defined)

You can find it at (how to point ALU to this?):

http://home.arcor-online.de/johannes.beck/lispdocu/

And if there's additional code I would collect them and publish them
here. Maybe I find the time on a rainy evening and even integrate
changes.

And well, there's no licence at all in this, so feel free to do what you
want with it.

I'm very capable of reading source code in a great many languages; I hope to include LISP in that set.  But, if I had to routinely look at nothing but
source code, my job (performance analysis) would be entirely impractical (at least if practical is defined as getting more than one thing done a
year).  My job is aptly defined in many cases as finding the right needle in the right haystack.  Sometimes, that requries code reading, but I need
something more than the root directory of a multi-million line of code OS to know where to start.  Good design doc is often a godsend.  We do seem to
agree on that.

I do take your point that automatic documentation generators (from code) have limits.  You'll note I said "good".  A forklift full of detail comments
is, I presume you agree, nonesense.

But, I've been pretty happy with Javadoc.  The tool is no more capable than the several I've used before and doubless earned the ire of us both.  It
must have something to do with the culture -- most of it is a pretty good predictor of how to use the widget, which is what I normally need.  And yet,
in theory, it is no better than any of the other "embed this in source code stuff" I've seen before.

Given that I've done a fair share of maintenance programming of other
foreign and own systems in my live, I think I have a fairly solid
grasp of what I've found to be useful (i.e. quality) documentation,
and what not.  And I have never found automatically generated
documentation to be any substitute for quality documentation.  What
automatically generated documentation is, is a statically restricted
and automatically cross-referenced view onto the source code.  This
doesn't constitute documentation, it is just a poor substitute for
interactive development systems that let me get at exactly the same
information (and often more of it) in a much more automated,
integrated and selective way.

As I (perhaps with less than the required clarity) stated in my
original posting, I see at least three-four different kinds of
information I might be interested in, when working on/with a system:

A) High- and medium-level requirements, analysis, design
   documentation.  This is completely independent of source code
   layout or information.  It should be written in a well-structured,
   prose style, possibly with relevant (pseudo-)formal notation thrown
   in for good measure (if you believe in formalisms at that level).
   See Erik Naggum's posting <3204468580853...@naggum.net> for more on
   this topic.

   Automatic documentation generators don't have a place at this level
   at all, IMNSHO, but sadly I've often found that their use in
   projects lowered the effort put into this kind of documentation,
   because it is easier to just dump all your knowledge into
   unstructured comments, than take the time to create a structured
   and polished presentation.

B) Public interface documentation, which includes signatures and short
   functional descriptions of said entities.  This is where automatic
   documentation generators shine, _in languages which don't offer
   environments that give you advanced interactive introspection and
   documentation capabilites, which will provide you with all of the
   information e.g. javadoc/doxygen give you, and some more, but in a
   much more directly useful way[1].  Again, take a look at any
   half-decent Lisp environment, and you will find all of this
   available at the touch of a button/key.  

   For example, in my working environment, C-c d will give me the
   doc-string for any entity in the system, C-c i will describe it,
   C-c I will inspect it, C-c a will give me the argument list for any
   function (it will also automatically flash the arg-list in the
   minibuffer while I type), C-c h will give me the HyperSpec entry
   for any standard-mandated entity, M-. will take me to the
   definition(s) of a given entity, C-c ^ will take me to the callers
   of a given function, etc. There's also tons of per-project
   customized stuff, so that I can get at specifications for certain
   functions, etc. with a simple command, etc.

B+) For high-quality libraries or semi-autonomous subsystems, there
    should be low-level API documentation that includes the
    information in B, but should include relevant information from A,
    and tie the whole API together in a well structured, presentation
    that combines tutorial and reference, specification and design
    in one.  The ("library part" of the) ANSI CL standard is an
    example of such high-quality documentation, IMHO, and I've only
    seen a couple of other projects that achieved that level of
    documentation.  Ideally, all units/subsystems of any project
    should achieve such a high level of quality of presentation, but
    realistically, this is very expensive, and very seldomly achieved
    in my experience.  It is though one of the necessary ingredients
    for succesful inter-project or even inter-team reuse, IMHO (which
    might explain the relatively low levels of reuse achieved by most
    companies).

C) Source-code unit-level implementation documentation.  Since you
   should by definition only need that kind of information when you
   are working at the unit in question (if that's not the case, then
   you already have even more urgent problems than documentation),
   the source for the unit should already be at your fingertips (in
   your editor).  So everything that javadoc-style systems will have
   generated from such a file will already be in plain view then.  No
   need for a separate, static, non-up-to-date view of the same,
   then.

   In combination with A and B (and B+), this should be pretty much
   all you need to work with the implementation.  And again, any
   half-way decent environment will give you even better access than
   javadoc-style documentation, by e.g. giving you interactive and
   batch regexp-search capabilities across whole projects,
   advanced cross-referencing, etc.

So while I think that javadoc-style ADGs might be an improvement in
languages that provide impoverished environments and language support
for introspection and documentation, I fail to see the advantage they
would bring to CL.  And I see the dangers that they inherently create,
since by incentivizing keeping documentation and source-code together,
they dis-incentivize the proper maintenance of separate documentation
(which is already suffering from enough pressures that it is often
sorely neglected), in my experience.

CL environments are guilty of that to some degree, too, since there is
no standard for integrating stand-alone documentation with the
environment, so that e.g. accessing CMU CL documentation is more
painful than e.g. accessing the HyperSpec, and that's a tiny bit more
painful than asking the system about some arglist or doc-string.
Dito for maintenance:  It's simpler to adjust the doc-string of a
function (or rewrite a comment), when something changes, than
adjusting the out-of-line documentation.  That's where management
processes and project-guidelines must step in, to reverse the
inevitable trend...

In any case,  I hope my current posting has cleared up any
mis-impressions of my position.

Regs, Pierre.

What do you think of gcl's approach where the doc-strings are really
held in an info file that describe and friends look through?  Granted,
the current info file basically looks something like what an ADG would
generate, it at least has the potential of being something more,
perhaps even like the Hyperspec.