In general, how can you free memory "manually", when you can't wait for the
gc?

TIA

  By calling a stack-allocating function yourself, presumably, but this
  is such an odd request that I doubt that you have asked for what you
  really need.

| It's a class that uses some windows resources that *must* be freed
| when leaving the function body where the class is used.

  What does "free" mean?  Objects on the stack are not _freed_ at all.
  The bytes of memory that made them up are still there until they are
  overwritten with something else.  If you have any references to that
  data when you leave the function, you're in _deep_ trouble.  So what I
  think you must mean is that you want to guarantee that there are no
  references to your data.  I can't see why that is a problem that does
  not have an obvious solution: Just the variables/slots/whatever that
  holds any such references to some other value, or nil, or whatever,
  and you no longer have any way of finding the data.

| In general, how can you free memory "manually", when you can't wait
| for the gc?

  Well, describe what it means to free it, and I'm sure it can be done.

Maybe this is a newbie's mistake, but I still don't feel safe to let the GC
handle "resource management" (not just memory management): file handles,
brushes, db connections, etc... that must be released asap because they are
scarce or that must be released in a precise order (that's the specific
problem I'm having).

(unwind-protect
  (progn
    ... code in the body ...)
  ... code that must be run on exit ...)

Or, assuming that the operations to get the thing are ALLOCATE-CRUN
and the operation to destroy it are DESTROY-CRUN you can write
something like this. (I am assuming that ALLOCATE-CRUN succeeds or
signals an error and so on)

 (let ((secret nil))
   (unwind-protect
       (let ((crun (setf secret (allocate-crun))))
         ... use crun ...)
     (when secret (destroy-crun secret))))

If you do this more than once you probably want to write a macro so
you have some syntax to make it easy to see what is going on:

 (defmacro with-crun ((var) &body body)
   (let ((sv (gensym)))
     `(let ((,sv nil))
        (unwind-protect
            (let ((,var (setf ,sv (allocate-crun))))
              ,@body)
          (when secret (destroy-crun secret))))))

 (with-crun (x)
   ... use X ...)

You are correct that it's inappropriate to let the GC do this kind of
resource management: there are plenty of cases where you need to
*know* when some kind of `finished-with-x' action happens, or that
such actions happen in a given order, and the GC can generally not
offer those kinds of promises.  However Lisp offers plenty of ways of
ensuring that they happen, and probably better ways than many
languages, as it decouples them from the low-level issue of memory
management.

The integration with e.g. the graphics (windows) environment is often
shallow, so that the resources for any handles you get via Lisp are
allocated elsewhere, and must be manually "freed". As Erik says, it can
be done, as there is (should be) a "release a resource" function
corresponding to the one you used to get the resource. But you must
build this kind of resource management into your application, since the
raw Lisp has no way to do it for you automatically.

I really believe that given the current abundance of memory the GC is
the best you can do unless you are working in an "embedded" framework,
where the enginners still think that adding an extra 0.5USD of memory
to their chips would cause failure in the market (therefore forcing
longer development and debugging times mostly due to memory management
issues, in spite of all the "time-to-market" hoopla.)

"C programmers think memory management is too important to leave it to
the system.  Lisp programmer think memory management is too important
to leave it to the programmer." (I believe this is a quote from
Stroustroup, though I do not know whether it is his own).

Anyway, this is CLL :)  And C/C++ programmer (including the designers
of COM/OLE at MS) then go on to reinvent the wheel with reference
counting. :)

Cheers

  Oh, I see.  I should have realized that when C++ and Java programmers
  talk about objects that are "freed", freeing _memory_ is nowhere near
  what they really talk about, they talk about running destructors, and
  I got a little hung up in terminology and your need to allocate on the
  stack and talk about the GC, which confused me.  Sorry about that.

  One does these things with macros that utilize unwind-protect to run
  cleanup-forms in Lisp, and Tim's answer was right on the mark.  Use a
  macro around the form that allocates, binds to a variable, and cleans
  up, but don't worry about the memory.

| It cannot be delayed until the GC starts.

  Well, what happens at GC time is that memory that is not referenced by
  anything can be reused for new objects.  This is strictly about
  freeing _memory_ and has absolutely nothing to do with destructors.
  The objects that GC sees are live objects, not dead ones.

  There is an exception to this, of course, called finalization.  It is
  code triggered after the GC discovers that there is only one reference
  to the object, and that is from the finalization.  Most Common Lisp
  implementations also have "weak references", references that are
  changed to nil when they are the last reference.

| I could do this explicitly (and this is the solution recommended by MS
| for Java, so I guess there must be some better way ;-), but it seems
| ugly and error prone.

  Yup, it would be in a language without macros and unwind-protect.

| Maybe this is a newbie's mistake, but I still don't feel safe to let
| the GC handle "resource management" (not just memory management): file
| handles, brushes, db connections, etc... that must be released asap
| because they are scarce or that must be released in a precise order
| (that's the specific problem I'm having).

  No, don't just feel unsafe, feel _scared_!  You're quite right that
  doing resource management with GC is wrong.  It's a pity that C++ and
  Java conflated the management of memory and other resources, as memory
  essentially is free and everything else is not, leading to radically
  different allocation, deallocation, and management strategies for the
  two kinds of resources.  Oh, it just occurred to me that maybe these
  languages are really based in the assumption that memory is _also_
  scarce.  Geez, _that's_ a flashback to the 50's!  On with your white
  lab coats and roll in the decks of cards with the Java program from
  the browser request made last week.  Crank up the power to that extra
  bank of 16K core memory and start the espresso program on the coffee
  system, too.  It's gonna be a _long_ night.  This code forgot to
  deallocate!

  unwind-protect is your friend here, but most people find that using it
  gives your code a low-level feel. Let fancy macros like with-open-file
  does the whole resource management cycle for you, from opening the
  stream to guaranteeing that it is closed when you leave.  You don't
  even see the memory management involved, of course.  That's the bonus.

  I think it is important to realize that GC doesn't _free_ memory or
  objects.  A transitive, active verb here implies something that exists
  and that something is done to it.  Not so.  Unreferenced memory does
  not exist as far as (access to) objects is concerned.  The reason it
  is not usable for new allocations is that even the allocator doesn't
  know it exists.  GC makes such unreferenced memory usable, again, but
  at this point, we have an essentially uninitialized resource that
  cannot possibly keep track of what it was last used for.  Let's call
  this _abandoned_ objects and memory.

  This is in sharp contrast to "freeing" an object, an active operation
  on an object that still exists qua object.  When the free function
  puts a chunk of bytes back on the free list used by malloc, the memory
  never _ceases_ to be referenced.  It is in fact an _error_ for memory
  to be abandoned in such a system!  (It's 3:30 AM.  Do you know where
  your memory is?  My God, it must be hard to live in the C++ mindset,
  parenting _memory_ as if it were a scarce resource.  Sheesh!)

  I supposed that if you think only in terms of referenced memory and
  that memory always has a structure, it will be very ahrd to switch to
  think in terms of unreferenced memory and allowing yourself to let go
  of memory, since that used to be a source of serious problems and
  outright pain and suffering during development.  If you aren't aware
  of what hurts, you're likely to fear something else that appears to
  coincide with it.  (Such as people who can't be bothered to figure out
  the difference between posting to comp.lang.lisp and being stupid.)

| And Lisp people *also* confuse it with storage management and since GC
| does largely solve that problem assume that these other problems must
| be solved too.

  Let's find a way to help Lisp people overcome that confusion, since GC
  doesn't solve any other problem than how to reclaim abandoned memory.
  If something else happens at GC time, I think it needs to be spelled
  out that there is a (very significant) convenience factor in doing it
  at that time, not an intrinsic relationship between the reclamation of
  abandoned memory and the more complex ways through which objects may
  be discovered to have been abandoned after all the next time around.

  Example: You don't have to reclaim abandoned memory to find that some
  object has no other references than that from the finalization list,
  it's just massively more efficient to mark the old objects you copy in
  a copying garbage collector so you can walk over the list of objects
  to finalize and see if anyone of them are still unmarked, at which
  point you call the finalization code associated with those objects.

But if you sell 10 million of the chips next year, that's $5 million
extra.  That might have paid for all of the development costs and then
some.  And if you sell that many the year after, that's pure profit.

It has to be carefully balanced with the time it takes to develop that
software to save that memory, of course.  That's the trick.

Actually, this _shouldn't_ be a problem for Java programmers; there are no
destructors in Java.  Unfortunately, a lot of Java programmers (even book
authors) seem to be using finalizers as destructors.

Java also has a try/finally clause that works like unwind-protect, but of
course, you can't use macros to hide the details.

some things people said there that seem correct:

. finalizers should never be used as the _only_ way to return some
  scarce resource to the system; there always should be an explicit
  mechanism to return the resource.

. there's no such thing as timely finalization -- you should never
  expect it.  this is a corollary from the previous and the next
  point.

. things get even more interesting when the finalization system
  releases objects in topological order (which is a nice safety
  feature, as you can be assured that you can do things on some
  complex unreachable object and not be afraid that some parts of it
  are already finalized).  apart from the problem of cycles (which
  doesn't seem to occur very often, and cycles can usually be easily
  detected (by the GC) and broken (by the user, after seeng a
  warning)), this also has the effect of postponing finalization of
  depended-on objects until some future GC invocation, which, of
  course, has interesting interactions with tenuring.

that said, I don't think you could really argue that some finalizers
will "never" run.  I don't know of any real-world GC-backed system
that relies on ephemeral collection exclusively.  usually the full GC
gets run periodically, too.

--
(only legal replies to this address are accepted)

Cheers

Of course, this is just my 0.5USD opinion.  You can have 10 millions
of 'em :)

Cheers

    >> It has to be carefully balanced with the time it takes to develop that
    >> software to save that memory, of course.  That's the trick.

    Marco> I have heard this argument many times. My experience is that this is
    Marco> not taken into account and there is no balancing act going on.

I have a feeling that what happens is that the amount of memory on the
chip is decided very early because it takes a long time to layout the
chip, fab it, test it, etc.  At that point, it seems like a reasonable
thing to do because the SW effort to support it doesn't look so big.
By the time the chip has arrived, the SW requirements have ballooned
so what was easy is now very hard.  And then the product is late, you
only sell 100,000 instead of 10 million, and you lose your shirt and
then some on a product that nobody wants anymore.... :-(

    Marco> Of course, this is just my 0.5USD opinion.  You can have 10 millions
    Marco> of 'em :)

Shall I send you my account number at my Swiss bank? :-)

Ray

  You're right!  I had actually missed that, and have been thinking in
  destructor terms myself.  Thanks!   Goes to show how little I use them.

| Java also has a try/finally clause that works like unwind-protect, but
| of course, you can't use macros to hide the details.

  I think the ugliness of such verbose forms deters people from doing
  the right thing.  Proper handling of exceptional situtations in Java
  is also too verbose for my comfort.  Maybe I'm just spoiled.