Common Defsystem
David B. Lamkins
A few days ago I suggested that folks interested in putting a "common"
defsystem into MCL start by reading the lisp-standards/defsystem archive at
rochester.edu. I've been doing just that, and thought it might be
helpful if I summarized what I've found so far.
The archive contains source for three copyrighted, but (apparently)
redistributable defsystems:
- Kantrowitz
- MITRE/PR1ME (2 versions)
- UC, dcmartin et al
The archive also contains docs, but no code, for:
- Harlequin
- Franz
- Lucid
The archive also contains a very detailed defsystem proposal submitted by
Pittman et al to the ANSI committee; this was rejected twice "on
administrative grounds (too late in the process)".
Finally, there are quite a few discussion threads containing suggestions
for what a defsystem should do or be. I found two prevalent themes: "Blue
Sky" (i.e.: nobody has ever implemented feature X before, but I bet it
would be really useful) and "MFD" (My Favorite Defsystem.)
...
Next, I looked on the MCL CD-ROM, where I found still more defsystem code:
- Rand/HP
- Rand/PR1ME
- Felshin&Malone/MIT
- straz (Steve Strassman?)
- Bengtson
- Olivier
- Cartier
- Kantrowitz (multiple versions)
The MCL CD-ROM also contains graphical user interfaces for the Cartier,
Kantrowitz, and Rand defsystems.
The defsystems most commonly used by the contribs on the MCL CD-ROM are, in
no particular order: Kantrowitz (various versions) and Rand (various)
Lightweight, roll-your-own defsystems are also quite popular for many
systems found on the CD-ROM; these typically perform the algorithm: "given
an ordered list of files, compile and load if source is newer, otherwise
just load."
...
I'm not ready to offer a recommendation, because I'd like to be able to
first summarize the key features and differences among the various
candidates. However, I do have a strong preference toward adoption or
adaptation of existing code -- as opposed to writing new code to match a
spec or user documentation -- because (a) it's less work, and (b) the
existing code has probably seen enough field use to shake out some of the
design and implementation bugs.
Chris Reedy
At 01:00 AM 5/30/98 -0400, David B. Lamkins wrote:
>From: "David B. Lamkins" <dlam...@teleport.com>
>Date: Thu, 28 May 1998 23:07:32 -0700
>Subject: Common Defsystem
>
>A few days ago I suggested that folks interested in putting a "common"
>defsystem into MCL start by reading the lisp-standards/defsystem archive at
>rochester.edu. I've been doing just that, and thought it might be
>helpful if I summarized what I've found so far.
I reviewed the archive myself.
>[big snip - fine review of archive and MCL CD-ROM contents]
>I'm not ready to offer a recommendation, because I'd like to be able to
>first summarize the key features and differences among the various
>candidates. However, I do have a strong preference toward adoption or
>adaptation of existing code -- as opposed to writing new code to match a
>spec or user documentation -- because (a) it's less work, and (b) the
>existing code has probably seen enough field use to shake out some of the
>design and implementation bugs.
In general I agree with this, however, in this case I have very mixed
emotions. One minor quibble is that most of the available source versions do
not use CLOS and need to be updated to use the logical directory facility in
CLtL2 (this includes the Kantrowitz defsystem, which appears to be the most
popular).
My big concern is that all of the defsystems are based on a "file" notion
similar to the "make" facility typically used for C/C++. Unfortunately
(IMHO), this doesn't fit well with Lisp's notion of being able to
dynamically modify my environment while I am developing. This problem is
compounded by the fact that none of the defsystems in the archive provide a
mechanism for computing the dependencies among files. (One, LUCID's,
provides an analysis mechanism that could be used to manually determine the
dependencies).
I know what I would like: An IDE where I only have to list the files that
make up the system and the IDE would take care of all the rest of the
problems, including computing dependencies and updating the current image as
those files are changed. I can get this for C/C++ but not for Lisp.
This leaves me with a dilemma, work with one of the existing defsystems with
a GUI interface which seems to force me into a C/C++ style of development,
or ...? I would be interested in knowing what other people are doing. In
particular:
Do people make use of a defsystem during the heat of development, or only
for packaging code at intermediate points?
How do people deal with the potentially complex dependencies within Lisp? (I
would be interested in knowing whether anyone has ever tried to document
them all.)
Are there generally accepted good styles of packaging a Lisp system into
files (e.g. how do I package macro definition and usage, what about CLOS)?
And, in general, how do I keep my sanity when developing a large Lisp system?
Apologies for the length of the message, Chris
This is an informal message and not an official Mitretek Systems position.
Dr. Christopher L. Reedy, Mail Stop Z551
Mitretek Systems, 7525 Colshire Drive, McLean, VA 22102-7400
Email: cre...@mitretek.org Phone: (703) 610-1615 FAX: (703) 610-1603
Sunil Mishra
In message <2.2.32.19980601...@pacific.mitretek.org>, Chris Reedy w
rites:
>My big concern is that all of the defsystems are based on a "file" notion
>similar to the "make" facility typically used for C/C++. Unfortunately
>(IMHO), this doesn't fit well with Lisp's notion of being able to
>dynamically modify my environment while I am developing. This problem is
>compounded by the fact that none of the defsystems in the archive provide a
>mechanism for computing the dependencies among files. (One, LUCID's,
>provides an analysis mechanism that could be used to manually determine the
>dependencies).
I think this idea will also work better if one decides to incorporate the
defsystem into a larger version control system. I personally think that
smalltalk has done this part nicely, though I would prefer it if all my
code didn't end up in the same base image but could be maintained as an
external file/library or some such.
>I know what I would like: An IDE where I only have to list the files that
>make up the system and the IDE would take care of all the rest of the
>problems, including computing dependencies and updating the current image as
>those files are changed. I can get this for C/C++ but not for Lisp.
As you stated earlier, the file notion is not necessarily the best way to
go. We ought to be able to do better. (Though files are definitely nice for
moving code around.)
>This leaves me with a dilemma, work with one of the existing defsystems with
>a GUI interface which seems to force me into a C/C++ style of development,
>or ...? I would be interested in knowing what other people are doing. In
>particular:
>
>Do people make use of a defsystem during the heat of development, or only
>for packaging code at intermediate points?
I use mine at all points, though none of my programs have gotten past the
100K point (yet).
>How do people deal with the potentially complex dependencies within Lisp? (I
>would be interested in knowing whether anyone has ever tried to document
>them all.)
I have developed a routine for myself for organizing my programs that takes
care of all of my needs. Basically, anything involving mutual dependencies
ends up in a single file, and it is generally possible to give a coherent
description of what the content of the file does. Otherwise different
components of the program end up in different files.
If there are pieces of the code that can do something useful independent of
one another, then they end up in different systems. Makes for easier reuse
too. The largest such collection I have put together has gone up to ~250K.
>Are there generally accepted good styles of packaging a Lisp system into
>files (e.g. how do I package macro definition and usage, what about CLOS)?
Largely idiosyncratic for me, though I have picked up along the way some
invaluable advice.
>And, in general, how do I keep my sanity when developing a large Lisp
>system?
:-) I'll share when I get there.
Sunil
David B. Lamkins
At 3:53 PM -0400 6/1/98, Chris Reedy wrote:
>At 01:00 AM 5/30/98 -0400, David B. Lamkins wrote:
>>From: "David B. Lamkins" <dlam...@teleport.com>
>>Date: Thu, 28 May 1998 23:07:32 -0700
>>Subject: Common Defsystem
>>
>>A few days ago I suggested that folks interested in putting a "common"
>>defsystem into MCL start by reading the lisp-standards/defsystem archive at
>>rochester.edu. I've been doing just that, and thought it might be
>>helpful if I summarized what I've found so far.
>
>I reviewed the archive myself.
>
>>[big snip - fine review of archive and MCL CD-ROM contents]
>
>>I'm not ready to offer a recommendation, because I'd like to be able to
>>first summarize the key features and differences among the various
>>candidates. However, I do have a strong preference toward adoption or
>>adaptation of existing code -- as opposed to writing new code to match a
>>spec or user documentation -- because (a) it's less work, and (b) the
>>existing code has probably seen enough field use to shake out some of the
>>design and implementation bugs.
>
>In general I agree with this, however, in this case I have very mixed
>emotions. One minor quibble is that most of the available source versions do
>not use CLOS and need to be updated to use the logical directory facility in
>CLtL2 (this includes the Kantrowitz defsystem, which appears to be the most
>popular).
>
Actually, Yads <http://www.igortech.com> is based upon CLOS and supports
logical pathnames. I have yet to try it, but from the documentation it
looks like it's designed to be "plug compatible" with Kantrowitz's
defsystem.
I've hacked a version of Kantrowitz's defsystem to handle logical pathnames
and (for backward compatibility) MCL's older logical directory syntax.
Some of the changes are included the defsystem that Adam Alpern uses with
his Project Manager <file://MCL 4.2 CD:User Contributed
Code:Defsystem:Project Manager:defsystem:defsystem.lisp> -- unfortunately
the latest batch of changes (circa 1996) did not make it onto the CD. I'll
try to post a copy to my FTP site, soon.
My concern about the Kantrowitz defsystem is its size. If the Igor Tech
re-implementation does the same thing in 1/5 the number of LOC, well, I
know which one I'd rather maintain... (To be fair, the Kantrowitz version
contains a _lot_ of backward compatibility code for pre-ANSI CLs.)
>My big concern is that all of the defsystems are based on a "file" notion
>similar to the "make" facility typically used for C/C++. Unfortunately
>(IMHO), this doesn't fit well with Lisp's notion of being able to
>dynamically modify my environment while I am developing. This problem is
>compounded by the fact that none of the defsystems in the archive provide a
>mechanism for computing the dependencies among files. (One, LUCID's,
>provides an analysis mechanism that could be used to manually determine the
>dependencies).
>
Yes, this was a subject of debate in the archives. Personally, I'd like to
see the actual defsystem code implement a simple file dependency mechanism
(MCL is, after all, still a file-based Lisp) and have a separate package
grovel over a set of files to generate or update the defsystem description.
>I know what I would like: An IDE where I only have to list the files that
>make up the system and the IDE would take care of all the rest of the
>problems, including computing dependencies and updating the current image as
>those files are changed. I can get this for C/C++ but not for Lisp.
C/C++ has a conventional structure by which a "make-make" can dtermine all
of the dependencies: recursively list all of the include files required by
a source file. (Pascal systems which support UCSD-style units do a similar
thing with their "uses" clauses.) This works for UCSD units by design.
For C/C++ it works because of the long history of "conventional usage"
(e.g. include the header even if the compiler's default function prototype
would work) and because macros have file scope.
In Lisp, defmacro has global scope and indefinite extent; a macro defined
in one source file may be used (deliberately or not) in another. A
dependency generator might have to use a code walker to capture such
fine-grained dependencies for an arbitrary collection of Lisp files. And
what about load-time or compile-load dependencies? How would a walker
automatically determine these? I'm not saying that it's impossible, just
that it's not as simple as it first seems...
But, yes, the _interface_ to a defsystem should have hooks to support a
dependency generator.
>
>This leaves me with a dilemma, work with one of the existing defsystems with
>a GUI interface which seems to force me into a C/C++ style of development,
>or ...? I would be interested in knowing what other people are doing. In
>particular:
>
>Do people make use of a defsystem during the heat of development, or only
>for packaging code at intermediate points?
>
I use defsystem only for system-level projects. I tend to ignore it (more
from laziness than for a good, defensible reason) for projects that use
just one or two source files.
>How do people deal with the potentially complex dependencies within Lisp? (I
>would be interested in knowing whether anyone has ever tried to document
>them all.)
>
I'm not aware of any such list. Because things like defmacro and eval-when
can move the compile/evaluate/load boundaries, the (non-obvious)
dependencies are probably going to be a function of the program, rather
than the language.
>Are there generally accepted good styles of packaging a Lisp system into
>files (e.g. how do I package macro definition and usage, what about CLOS)?
>
Peter Norvig and Kent Pitman published a paper, "Tutorial on Good Lisp
Programming Style" <http://www.norvig.com/luv-slides.ps> at the LUV
conference in 1993.
>And, in general, how do I keep my sanity when developing a large Lisp system?
>
Take advantage of the environment. Build and test incrementally. Check
the Lisp and AI archives for existing code that does what you need. Be
careful when you try to go beyond the parts of Lisp that you understand.
Put aside what you know about C++ -- much of it will steer you in the wrong
direction when working with Lisp. (Norvig wrote an interesting article
about how many "Classic" patterns are irrelevant in Lisp
<http://www.norvig.com/design-patterns/index.htm>.) And take heart -- a
Lisp system will be about 1/6th the number of lines of code of a comparable
C++ system.
kr
At 15:53 6/1/98, Chris Reedy wrote:
>My big concern is that all of the defsystems are based on a "file" notion
>similar to the "make" facility typically used for C/C++. Unfortunately
>(IMHO), this doesn't fit well with Lisp's notion of being able to
>dynamically modify my environment while I am developing. This problem is
>compounded by the fact that none of the defsystems in the archive provide a
>mechanism for computing the dependencies among files. (One, LUCID's,
>provides an analysis mechanism that could be used to manually determine the
>dependencies).
why not adopt a common defsystem in a 2-phase process ? first settle on a suitable and wide-spread system that simplifies the more old-fashioned "make"-style "file" notions, and then enhance this with a more sophisticated system whenever a volunteer is willing to put in the effort. these two phases presumably can be made to be rather orthogonal.
IMHO, the most important thing right now is to settle on a wide-spread standard, and to make sure that fancy add-ons will be backwards compatible with respect to software distributions that only use a limited set of defsystem services.
to me it appears that the kantrowitz system is fairly wide-spread, and so this is my personal preference to start with.
Greetings
Markus Krummenacker
Alan Ruttenberg
I'll throw in a small suggestion, taken from my own defsystem, and an offer. Whereas the usual defsystem lists files by name, in mine I also allow wildcarded file names. For example,
(make-instance 'file-set
:define-logical-host? t
:name "customizationsPPC"
:files
'("finder-interact"
";code;PPC Friendly;*"
";code;PPC Only;*"))
which means: first load the file set "finder-interact". Then load all the files in (directory "customizationsPPC;code;PPC Friendly;*), and then all the files in (directory "customizationsPPC;code;PPC Only;*). That makes it real easy to maintain systems with frequently added files, but without interfile dependencies.
The logical host customizationsPPC is created to have its root in the same directory as the file set definition file.
I have a number of facilities in my system that I would be glad to port to whatever defsystem syntax that is adopted. Among them are:
- directory list search for file set definition file, as well as PBCatSearch (fast find file on one or all volumes)
- A menu of defined systems with hierarchical submenus corresponding to the directory structure in the system. That makes it easy to edit or load any file in a loaded system.
- Speedy routines which can check for and load any files changed since the last load of the system. I've used them to load changed files in a 300+ file system at lisp startup without onerous delays.
- A couple of tools to find differences in and help merge two copies of a defsystem, useful when there is more than one person developing with a copied source tree.
-Alan
Chris Reedy
At 09:48 PM 6/1/98 -0700, David B. Lamkins wrote:
>At 3:53 PM -0400 6/1/98, Chris Reedy wrote:
>>Do people make use of a defsystem during the heat of development, or only
>>for packaging code at intermediate points?
>
>I use defsystem only for system-level projects. I tend to ignore it (more
>from laziness than for a good, defensible reason) for projects that use
>just one or two source files.
I tend to find myself putting everything in one large file, even for things
that should reasonably be in four or five smaller files. I don't like this,
but I haven't found a good way to get started with a small project and then
build it up as I add features.
>>And, in general, how do I keep my sanity when developing a large Lisp system?
>>
>
[snip - a number of good suggestions, and thanks for the references to
Norvig's articles.]
One of the kinds of issues I was specifically interested in is: during
debugging I like to update my image by evaluating the defining form in the
FRED buffer. It seems to me that when I do this I'm simply taking advantage
of the capabilities of Lisp. This works fine when the defining form is
(defun ...). However, if it's (defmethod ...) and I've modified the types of
the parameters, I've actually created a new method and the old one is still
there, potentially giving erroneous results. On a couple of occasions, I've
spent some time trying to debug something that was correct, except for the
presence of an out-of-date method. (And, in case you're wondering, I tried
to check, more than once, that the old methods had been removed.)
My question is: How often do people normally reload a fresh version of their
system? Are there any events (such as the one above) when you want to do
this as a matter of standard practice?
Rainer Joswig
At 11:40 Uhr -0400 04.06.1998, Chris Reedy wrote:
>there, potentially giving erroneous results. On a couple of occasions, I've
>spent some time trying to debug something that was correct, except for the
>presence of an out-of-date method. (And, in case you're wondering, I tried
>to check, more than once, that the old methods had been removed.)
inspect the symbol (c-x c-i), inspect the generic function, select
the method, choose "Remove method" from the inspector's
Commands menu
Rainer Joswig, Lavielle EDV Systemberatung GmbH & Co, Lotharstrasse 2b, D22041
Hamburg, Tel: +49 40 658088, Fax: +49 40 65808-202,
Email: jos...@lavielle.com , WWW: http://www.lavielle.com/~joswig/
Doug Currie
At 11:40 AM -0400 6/4/98, Chris Reedy wrote:
>[...]
>My question is: How often do people normally reload a fresh version of their
>system? Are there any events (such as the one above) when you want to do
>this as a matter of standard practice?
We maintain an application of ~450 files and ~125,000 lines of code using a
defsystem based loosely on DFS (def-file-set), originally by Jim Salem at
Thinking Machines Corporation 10/86. We (actually Duncan Smith) completely
rewrote DFS in 1992 for MCL adding some important features and FRED
bindings. Since then we have used DFS continuously and maintained it across
all the MCL releases.
With DFS we organize the files into file-sets, ~40 in our application, as a
means of separating the work among a few developers while maintaining
dependencies among the files within a set and among the sets. DFS also
manages packages, forward definitions (to control compiler warnings),
imports and exports, etc.
In this environment we "never" evaluate forms in file buffers. It's just
too easy and quick to invoke an incremental make after completing an edit,
and avoids all the potential (e.g., macro) dependency issues. This doesn't
address the extraneous method problem you mentioned. For this we do
complete rebuilds frequently, which is very fast because it is automated,
and because only necessary compiles and loads are performed.
DFS handles dependencies in
- package creation (including a two phase package definition)
- symbol creation & export (in either phase)
- forward definition of functions
- load order
- compile order
- restricted external interfaces between file sets
- dependencies on the base (MCL & DFS) system with a recompile epoch
- transitive dependencies
(e.g., on macro files than expand into more macros in a third file)
DFS also provides
- compiled definition files
- external wrapper hooks to generate a complete application
from MCL image to save-application (except resources, we do that manually)
- tree shaking and unshaking
to eliminate and restore development features to the application
- integrated FRED tools and key bindings
- a DFS logical host (each file set is a directory in this host)
for searching or mapping over the files in the file set or all sets
- more
The definition files are pretty simple, ranging from a simple list of files
(in load order), to very elaborate specifications. Six years of use here
have proven DFS highly stable and useful.
DFS doesn't provide
- documentation
which is why I put off jumping into this thread until now. Yet if the
features above are interesting to intrepid MCL developers, especially if
you are considering creating/documenting/publishing a defsystem, Flavors
would be willing to donate the source to DFS.
e
Chris Reedy
At 12:50 PM 6/4/98 -0400, Doug Currie wrote:
>At 11:40 AM -0400 6/4/98, Chris Reedy wrote:
>>[...]
>>My question is: How often do people normally reload a fresh version of their
>>system? Are there any events (such as the one above) when you want to do
>>this as a matter of standard practice?
>
>We maintain an application of ~450 files and ~125,000 lines of code using a
>defsystem based loosely on DFS (def-file-set), originally by Jim Salem at
>Thinking Machines Corporation 10/86. We (actually Duncan Smith) completely
>rewrote DFS in 1992 for MCL adding some important features and FRED
>bindings. Since then we have used DFS continuously and maintained it across
>all the MCL releases.
>
>
>DFS doesn't provide
>- documentation
>
>which is why I put off jumping into this thread until now. Yet if the
>features above are interesting to intrepid MCL developers, especially if
>you are considering creating/documenting/publishing a defsystem, Flavors
>would be willing to donate the source to DFS.
>
in obtaining a copy of the source, even without documentation.
Thanks, Chris
Robert Futrelle
>
> At 12:50 PM 6/4/98 -0400, Doug Currie wrote:
> >At 11:40 AM -0400 6/4/98, Chris Reedy wrote:
> >>[...]
> The above features sound very interesting. I would be personnally interested
> in obtaining a copy of the source, even without documentation.
>
> Thanks, Chris
So would I!
-- Bob Futrelle
_______________________________________________________________
Prof. Robert P. Futrelle | Biological Knowledge Laboratory
Office: (617)-373-4239 | College of Computer Science 161CN
Fax: (617)-373-5121 | Northeastern University
futr...@ccs.neu.edu | 360 Huntington Ave.
| Boston, MA 02115
http://www.ccs.neu.edu/home/futrelle/
_______________________________________________________________
David B. Lamkins
At 6:19 PM +0200 6/4/98, Rainer Joswig wrote:
>At 11:40 Uhr -0400 04.06.1998, Chris Reedy wrote:
>
>>there, potentially giving erroneous results. On a couple of occasions, I've
>>spent some time trying to debug something that was correct, except for the
>>presence of an out-of-date method. (And, in case you're wondering, I tried
>>to check, more than once, that the old methods had been removed.)
>
>inspect the symbol (c-x c-i), inspect the generic function, select
>the method, choose "Remove method" from the inspector's
>Commands menu
There's also a nice contribution (undefine.lisp) that lets you undefine
(and optionally delete) the form under the cursor in a FRED window.