Anyway, what I'm really hoping is that someone can give me a pointer
to the right thing to read to wrap my head around these issues. (For
one, I don't even know if this is a language issue that I should be
reading about in the HyperSpec or an implementation issue.)

-Peter

    [...]
    PS> if i have
    PS> functions foo and bar and foo calls bar, I can redefine bar and a call
    PS> to foo will reflect the new behavior of bar. (Though I have an idea
    PS> that if I've compiled foo, as opposed to just eval'ing the definiton,
    PS> that might not be true.)

That would still be true (except in the case when BAR is
declared/proclaimed inline, but it is likely that you don't have to
concern yourself with that, at least for now).  By the way,
evaluating a DEFUN form might automatically compile.

    PS> But if bar was a macro, then I'd have to
    PS> re-eval the definition of foo if I wanted it to use a new definition
    PS> of bar, right?

Yes.  More precisely, you have to redefine FOO each time you change
the definition of a macro it uses.  You can redefine FOO by:

* re-evaluating the DEFUN form; or

* reloading the file where FOO is defined; or

* recompiling and then reloading the file where FOO is defined.

All these are equally good as far as having FOO pick up the new
macro definition is concerned.

    PS> And then there's compile-file and load. For the moment
    PS> I tend to kill my Lisp process every so often and recompile and reload
    PS> everything to make sure I've got a consistent image.

It should be enough to reload the files, if you have lost track of
which individual definitions need to be redone.  While developing,
especially if things change very often, you may want to load the
source files, and compile the files later when things are more
stable.  Loading the source files does not necessarily mean you
cannot run compiled code: invoking COMPILE by hand is tedious, but
if you find the switch that tells the implementation to
automatically compile on DEFUN, it will do it for you.  (This one,
of course, is an implementation-specific feature; not all of them
would have it.)

    PS> Anyway, what I'm really hoping is that someone can give me a pointer
    PS> to the right thing to read to wrap my head around these issues. (For
    PS> one, I don't even know if this is a language issue that I should be
    PS> reading about in the HyperSpec or an implementation issue.)

Start at www.lisp.org.

---Vassil.

Now, if you later change your mind about what BAR does, you need to
recompile and reload all files in which BAR is used, because those
functions never make any calls to BAR; the compiler does that, _once_,
at compilation time.

Functions are different; if BAR is also a function, then, at _runtime_,
the function object associated with the symbol BAR is retrieved when FOO
is called, so recompiling BAR is sufficient.
[Recompiling BAR or simply re-evaluating it are precisely the same module
 runtime efficiency.  But I always just hit C-c C-c in ilisp, so I always
 run compile code, since the evaluator doesn't issue warnings.]

Finally, if you declared a function INLINE, then for the purposes of
dependecies, it behaves as a macro, as FOO will (or, more properly, may)
have been compiled to simply include the body of FOO into the body of BAR.

Lastly, the way to not worry about such things is to use a build system
like DEFSYSTEM, and to think hard about what goes where.  You teach
the build system the compile and load time dependencies of your application,
and, when you've hacked willy-nilly, as you say, you just ask the build
system to recompile/reload your application, and it should be smart enough
to only recompile/reload the necessary files.

Even with CLOS the instances change when you change their class definitions:

* (defclass animal ()
    ((name :accessor name :initarg :name :initform nil)
     (age :accessor age :initarg :age :initform nil)))

#<STANDARD-CLASS ANIMAL {48038355}>
* (defclass bird (animal)
    ((can-fly-p :accessor can-fly-p :initarg :can-fly-p :initform t)))

#<STANDARD-CLASS BIRD {4808C5FD}>
* (defvar *zebra* (make-instance 'animal :name "Zebby" :age 4))

*ZEBRA*
* (defvar *parrot* (make-instance 'bird :name "Polly" :age 2))

*PARROT*
* (describe '*zebra*)

*ZEBRA* is an internal symbol in the COMMON-LISP-USER package.
It is a special variable; its value is #<ANIMAL {480AE755}>.

   #<ANIMAL {480AE755}> is an instance of class #<Standard-Class ANIMAL
                                                  {4803161D}>:
    The following slots have :INSTANCE allocation:
    AGE     4
    NAME    "Zebby"
*

;;; redefine animal

* (defclass animal ()
    ((name :accessor name :initarg :name :initform nil)
     (age :accessor age :initarg :age :initform nil)
     (owner :accessor owner :initarg :owner :initform "me")))

#<STANDARD-CLASS ANIMAL {48038355}>
(describe '*parrot*)

*PARROT* is an internal symbol in the COMMON-LISP-USER package.
It is a special variable; its value is #<BIRD {480B6495}>.

   #<BIRD {480B6495}> is an instance of class #<Standard-Class BIRD {48085D4D}>:
    The following slots have :INSTANCE allocation:
    OWNER        "me"
    AGE          2
    NAME         "Polly"
    CAN-FLY-P    T
*

If it is an issue somewhere, try the HyperSpec.  My (admittedly cursory) search
hasn't revealed anything.

But if you want something really quick and dirty then just create
a file/function that contains code to compile and then load each of
your files in turn and in the right order.  Instead of killing your
lisp just execute this function.

Barring some seriously screwed up interdependencies among your files
this will work just fine.  

Anyway, based on the responses, it seems like a "proper" (for
some--probably over-engineered--definition of "proper") Lisp build
system would keep track of dependencies at a function/macro level
(since that's the smallest unit you can compile). Do any of the
systems out there actually work that way. That is, you could imagine
keeping a big DAG of all the functions and macros (and, I guess,
defvars, defclasses, etc) and when you say (make) it would check which
ones had changed and recompile them in the proper order. And given the
code is data, data is code, thing it seems a lot easier to maintain
that information automatically than it is in languages like C or Java
where you don't necessarily have access to a parser for the language.

But that's probably crazy. I guess I'll go read about defsystem and
see if I can find that paper Kent Pitman wrote about what he wants
from a build system. Thanks to everyone who responded.

-Peter

Summary:  Macros and Defstructs will cause problems, others won't

Function calls work with redefinition in both interpreted and compiled
code.  This is one of the nice features of Lisp that allows hot patching
of running applications, as well as providing various debugging and code
development advantages.

Changing the definition of CLOS classes will also work on the fly
without requiring changes to the using code -- even if you add or delete
slots from the class.  (Unless of course the code happens to use slots
you decide to delete...)

The items that require re-evaluation or re-compilation of functions are
if you change a macro (since that is expanded when the function is
compiled) or if you change the structure of a defstruct (since the
accessor functions can be open-coded).

One more systematic approach to this (if you project gets big enough to
be in multiple files) is to use one of the DEFSYSTEM utilities that
manage the dependencies among source files.  Put macros and defstructs
in their own files, and mark the dependencies.

  It may help to understand how slot access works in CLOS compared to
  inferior object models.  If you try to access a slot with `slot-value´,
  and it has been deleted, the system invokes an error handler that may
  still return a useful value.  If you read a slot with an accessor generic
  function, that function can still exist and do useful things, and it
  might just be able to know enough about the caller to write a log entry
  that a maintenance programmer can use to update or correct it.  These are
  features that are not available in the simple-minded object systems that
  are used in "static" languages.

| One more systematic approach to this (if you project gets big enough to
| be in multiple files) is to use one of the DEFSYSTEM utilities that
| manage the dependencies among source files.  Put macros and defstructs in
| their own files, and mark the dependencies.

  Reasonably complete cross-referencing function are available in the real
  Common Lisp environments that can tell you which functions need to be
  recompiled if you change a macro (and indirectly a defstruct).  These
  features are enormously helpful in locating both the source file and the
  functions and macros calling and called from a function or macro.  They
  can also help you build a patch file of the functions that have changed
  after a macro has been changed for compilation as a patch.

  Depending on how you store your source files, extracting individual
  functions for recompilation may be a problem or no problem at all.

--
Erik Naggum, Oslo, Norway

(Assuming your implementation is even kind of conforming...
  [144]> ;;;Evaluating defclass basic-block
  WARNING:
  DEFCLASS: Class BASIC-BLOCK is being redefined, instances are obsolete
  #<STANDARD-CLASS IR:BASIC-BLOCK>
Whee! I get to restart CLISP! *sigh*)

  Since nobody has even mentioned this yet, I would like you to consider
  another option: Do not compile your code in the first place.  The great
  value of running interpreted is that macros you change will affect their
  uses immediately.  If you intend to do a lot of macro programming, you
  will find that compiling macros and functions using macros is a serious
  drag on programmer performance.  While testing and developing your code
  interactively, chances are pretty good that you will be far slower than
  the interpreted environment of your Common Lisp system and that any time
  you might save in executing compiled code is more than offset by the
  time you waste with your what-to-compile-when-problem.

  Programming in Common Lisp is all about not worrying about needless
  things, and if you need to worry about when a macro is being used because
  the caller has been compiled before you changed it, you should not ask
  what to recompile, but what to compile in the first place.

--
Erik Naggum, Oslo, Norway

-Peter

  You would have to find out if SBCL expands macros too early even when you
  want to run interpreted.  In my view, that is a serious disservice to its
  programmers, when the value of actually working with interpreted code is
  so obvious when you actually need it.  Performance is the key, but Common
  Lisp has generally been willing to consider programmer performance as a
  higher value than code performance in some stages of development.

| [Goes to browse through the HyperSpec].

  :)

| Okay, I see "3.2.2.3 Semantic Constraints" talking about the things that
| you can count on being the *same* between compiled and interpreted
| evaluation; is there somewhere else that talks about required
| *differences*.  Are there any?

  I expect to be able to change a macro definition and its interpreted
  users will reflect the chang while its compiled users would not.

(defmacro macro-expansion-test ()
  (print "I'm being expanded!")
  `(print "I'm the expansion!"))

(defun expanded-when ()
  (macro-expansion-test))

  Evaluate these and see what `(expanded-when)´ produces when you evaluate
  it twice in a row.  If you get the first `print´ while defining it, your
  environment is very eager.  If you get the first `print´ only once, it
  does the expansion on the first evaluation.  I expect it on both.  Then
  `(compile 'expanded-when)´ and it should print "I'm being expanded!" and
  calling it should only produce "I'm the expansion".

--
Erik Naggum, Oslo, Norway

* (defmacro macro-expansion-test () (print "I'm being expanded!") `(print "I'm the expansion!"))
MACRO-EXPANSION-TEST
* (defun expanded-when () (macro-expansion-test))
"I'm being expanded!"
EXPANDED-WHEN
* (expanded-when)
"I'm the expansion!"
"I'm the expansion!"
* (expanded-when)
"I'm the expansion!"
"I'm the expansion!"
* (compile 'expanded-when)
EXPANDED-WHEN
NIL
NIL
* (expanded-when)
"I'm the expansion!"
"I'm the expansion!"
*

-Peter

  Yes.  That premature line is telling.

  If SBCL is a reasonable Common Lisp environment, it should have a way to
  disable the premature macroexpansion.  If not, it should have it in the
  next version if its authors are serious about offering Common Lisp's
  /dynamic/ and /interactive/ development tradition.  However, it may be
  that it is not intended to be a living Common Lisp environment.  CLISP is
  not able to accomodate living Common Lisp applications, either, with its
  lack of ability to update class instances as required by the standard,
  but that does not seem to deter a significant number of people.  GCL is
  also unable to accomodate demanding Common Lisp users, so there does
  appear to be a market for dumbed-down and fake Common Lisp systems.

--
Erik Naggum, Oslo, Norway

That said, what's stopping an implementation's compiler from storing
this dependency information in (or with) the object code of a compiled
function, so that the function gets recompiled automatically if a
macro on which the function is dependent is changed?

There are some relatively easy ways to have your cake and eat it, too, in
many cases.  One way is to have your macro expand into a call to a helper
function that does most of the work:

(defmacro with-foo ((&rest options) &body body)
  `(with-foo-internal #'(lambda () ,@body) ,@options))

(defun with-foo-internal (body-fun &key ...)
  ...
  (funcall body-fun)
  ...)

Now you can redefine WITH-FOO-INTERNAL and it will affect all the uses.
The WITH-FOO macro is so trivial that it's unlikely you'll ever need to
modify it.

This technique also makes the macro hygienic automatically.

  And this was a counter-argument to what, precisely?  Should I shut up
  about where I think they should be going just because you -- what, felt
  bad or something?

| Given that the primary developers use it for their day-to-day work, it's
| unnecessarily provocative to say that it is "not intended to be a living
| Common Lisp environment."

  Yes, you were unnecessarily provoked.  That does not make what I said
  unnecessarily provocative.  My God, after all these years, people still
  cannot deal with "provocative" statements rationally.  Gimme a break!

| Obviously, it is so intended.

  *sigh*  It is the Common Lisp environment that is living when it adapts
  well to changes.  If you force compilation prematurely, you force your
  programmers to consider the cost of changing a macro, and this is anathema
  to the interactive, dynamic programming that Common Lisp is so good at
  supporting.  As I said, there is a market for Common Lisp environments
  that do not support this crucial element of Common Lisp, but people
  should know about alternatives and consider the fact that SBCL made an
  engineering decision between CMUCL and SBCL to force premature macro
  expansion.  So somebody made the wrong choice.

  Unnecessarily provocative, my ass.

--
Erik Naggum, Oslo, Norway

Firstly, on the subject of restoring non-eager
interpretation/compilation for the next release: given that according
to our (admittedly informal) release schedule sbcl-0.7.9 will be
released in the next ten to fifteen days, I can state with a fair
amount of confidence that it's unlikely that this functionality will
be included in the next release.  Sorry.

Secondly, yes, it was an engineering decision to remove the IR1
interpreter (and the byte compiler) from SBCL; this was not, however,
motivated by the attempt to cripple SBCL as a development environment,
but a step taken to improve the maintainability of the SBCL codebase
as a whole.  The point here being that I don't believe that there's
any political or technical problem with restoring this
interpreter-like functionality into SBCL; it shouldn't be
characterized as an irrevocable technical design decision.

However, there is the matter of priorities.  From the PRINCIPLES file
of the SBCL source distribution:
  1. conforming to the standard
  2. being maintainable
  3. being portable
  4. making a better programming environment
As far as I am aware, SBCL (in its treatment of macros in function
definitions) is conforming to the ANSI standard.  The possible
alternative behaviour pointed out by Erik is likewise conforming, of
course, which means that either behaviour would be fine by the first
guiding principle.  Thus, as long as the code implementing this
behaviour is maintainable (and doesn't impact on portability) it would
seem that we have a definite win.  However, I do not propose to spend
the next ten days of my time working to implement it, I'm afraid, as
there are a number of ANSI and AMOP compliance bugs that I view as
significantly more urgent than a programming convenience.  Speaking
for myself, I would at this moment rather work on ensuring that users
can have the confidence that SBCL implements the ANSI specification.

I hope that clarifies my point of view; please don't hesitate to ask
questions, and comments are always welcome.

For the lurkers: ANSI compatibility is an extremely high
priority for SBCL, including of course behavior which is
important for incremental development. Thus I was particularly
annoyed and provoked by Erik's comparisons to CLISP and gcl
above. (Lack of ability to update things as required by the
ANSI standard? Yikes!) Furthermore, between that and his
remarks in a subsequent message,
<news:3243804788259221@naggum.no>,  
  > As I said, there is a market for Common Lisp environments
  > that do not support this crucial element of Common Lisp, but
  > people should know about alternatives and consider the fact
  > that SBCL made an engineering decision between CMUCL and SBCL
  > to force premature macro expansion.  So somebody made the
  > wrong choice.
there seems to be the suggestion that SBCL has been backsliding
away from ANSI-specified behavior which supports incremental
development. This is not the case. There are various
shortcomings in SBCL, notably the way that I personally have
pessimized compiler performance and dropped popular extensions.
And we also have some long-standing ANSI-compliance problems
(dating back to the original CMU CL code) which we haven't
fixed. But other than temporary stumbles when we inadvertently
introduce temporary bugs, as far as I am aware SBCL's ANSI
compliance has only increased over time, with no significant
backsliding in any area, including this one.

Besides taking care not to backslide, we (originally me, but
over the last year mostly others) have been rather energetic
about improving ANSI compliance in many areas. As two examples
chosen to be relevant to incremental development, we fixed the
long-standing non-ANSI
  (compile (defun foo () nil))
  (compile (defun bar () (if (foo) :true :false)))
  (defun foo () 12)
  (bar)
  => :FALSE
over-eager type-inference behavior that we inherited from old
CMU CL, and fixed a recently-discovered PCL bug which occurs
when EXPORT operations are done on internal symbols sometime
after they're used in CLOS definitions.

So while SBCL may have ditched support for extras that Erik
likes very much ("crucial element", check) please do not go
away with the impression that we mess up ANSI-specified basics
like being able the redefinition of class instances mentioned
above.

Erik: You stand out from the crowd by being unusually smart,
knowledgeable, articulate, prominent, opinionated, and
undiplomatic, but you are by no means the first person to
criticize SBCL for not supporting a favorite allowed-by-ANSI
extension or idiosyncrasy, and you probably won't be the last.

If I understand correctly, you want macro redefinition DWIM in
interpreted code, so that as long as you don't COMPILE-FILE or
COMPILE, you don't need to think about rebuilding the world
from scratch when you redefine a macro. That does sound like
handy behavior, and it's certainly deliciously dynamic. If
there were a clean and simple way to make it happen, we might
well do it. (Though as explained by Christophe Rhodes in another
post, even then we probably wouldn't do it next release, i.e.
late this month, despite your perhaps-well-meant "if SBCL is
a reasonable Common Lisp environment" remark quoted above.)
Whether in reality we will actually do it, I don't know, since
  * Our abilities are not unlimited. (though still formidable:-)
  * I place some priority on minimizing the number of options and
    extras in the system, so having to control such behavior with
    a non-ANSI flag would be a little distasteful to me.
  * It's not immediately obvious to me that the semantics of this
    are well-specified. E.g. I have (in an application program) code
    implementing functor-like constructs using elaborate lexical
    closures which are then sometimes stuffed into (FDEFINITION 'FOO),
    and maybe if I thought about it I could find code within the
    implementation of SBCL which has similar issues, too. I'm not
    quite sure how to make macro redefinition DWIM on things like
    that. (Though maybe it's known and I'm just not aware of it, or
    maybe everything just works out naturally as long as the
    interpreter is cautious or simple-minded enough not to try to
    optimize away unused-under-current-macro-definitions elements
    of the lexical environment.)

Also, I suspect that you may be writing from incomplete
knowledge when you give the impression that the
macro-redefinition behavior in the old implementation of
interpreted functions was clean before "SBCL" -- in this case,
me -- "made an engineering decision" to punt the IR1 and byte
interpreters in sbcl-0.7.0. Perhaps in your programming style
you don't need to worry about stuff like the persistent-closure
issues above (where both "interpreters" did various
precompilation and caching including, IIRC, optimizing away
elements of the lexical environment based on what was needed by
the macro definition in use at initial "interpretation" time).
But even if that wouldn't have bitten you, I doubt you were in the
habit of taking care to execute EVAL:FLUSH-INTERPRETED-FUNCTION-CACHE
when you redefined a macro.

CMU CL developers: Now that I think of it, maybe %DEFMACRO or
(SETF MACRO-FUNCTION) or some such thing should automatically
call FLUSH-INTERPRETED-FUNCTION-CACHE? unless of course there's
some other kind of magic going on that I didn't find with
"egrep flush-interp" on recent CMU CL CVS source)

  That was a particularly depressing opening.

| Thus I was particularly annoyed and provoked by Erik's comparisons to
| CLISP and gcl above.

  In /addition/ to being egoistically concerned with your own feelings, how
  about also considering how annoyed and provoked I was when I noticed that
  SBCL had removed this crucial feature from CMUCL.  I have come to rely on
  the interpreted behavior of macros in Allegro CL because I have

| there seems to be the suggestion that SBCL has been backsliding away from
| ANSI-specified behavior which supports incremental development.

  You know, I consider ANSI compliance the baseline.  It is in my view quite
  irrelevant when arguing for a particular implementation over another.  It
  would be like marketing books with "now conforming to standard spelling
  and grammar" on the cover.  In other words, I do not look for compliance,
  I expect it, the same way I expect people to think and get annoyed when
  they do, instead of applauding them if they do like the touchy-feely guys.

| So while SBCL may have ditched support for extras that Erik likes very
| much ("crucial element", check) please do not go away with the impression
| that we mess up ANSI-specified basics like being able the redefinition of
| class instances mentioned above.

  I do not think there were any grounds to think that the annoyances of one
  implementation would be contagious just because it was juxtaposed to
  another implementation, so I do not believe anyone would go away with
  that impression, but had I known you were nervous about compliance, I
  would not have antagonized you on that aspect.  As far as I am concerned,
  CMUCL (and hence SBCL, which I have only recently installed for fun)
  stands head and shoulders above the other no-cost alternatives.

| Erik: You stand out from the crowd by being unusually smart,
| knowledgeable, articulate, prominent, opinionated, and undiplomatic, but
| you are by no means the first person to criticize SBCL for not supporting
| a favorite allowed-by-ANSI extension or idiosyncrasy, and you probably
| won't be the last.

  On my list of impressions that I want people to go away with are being
  unusually smart, knowledgeable, articulate, prominent, opinionated, and
  undiplomatic, but I have no intention of being remembered as the first or
  last person to criticize SBCL, so I mark that down as one success.  :)

| If I understand correctly, you want macro redefinition DWIM in interpreted
| code, so that as long as you don't COMPILE-FILE or COMPILE, you don't
| need to think about rebuilding the world from scratch when you redefine a
| macro.

  Precisely.

| That does sound like handy behavior, and it's certainly deliciously dynamic.

  Great that we see this the same way.

| If there were a clean and simple way to make it happen, we might well do it.

  Very good!  If you find that it is easier to keep a record of compiled
  functions that used a particular macro and report this upon redefinition
  of the macro in a way that could be used to recompile functions as needed,
  that would be even better because it would allow for automated tools to
  create patches and would influence the system-building features.

| Also, I suspect that you may be writing from incomplete knowledge [...]

  I started to learn Common Lisp for real (after on-and-off use and toying
  for at least a decade) in 1994 with CMUCL, but got my hands on Allegro CL
  and found that it was really remarkably different and so much better than
  CMUCL for software development.  I have stayed with it since, thanks in
  large part to the great people at Franz Inc.

--
Erik Naggum, Oslo, Norway           http://www.beltwaybuzz.com/Story%20Four.htm