closing files by gc
Tom Lord
many:
[Don't rely on GC to close your files. It's just asking
to lose.]
Sure, 98.45% of the time.
But there's some subtleties.
a) What if my open files are hidden behind some abstraction?
I expect "widgets" to be GC'ed, but the widget holds open
files behind the barrier. I'm supposed to know this?
b) I divide my reliance on GC into two parts: parts of my program I
just Don't Worry About It and then see how memory usage comes out.
Other parts of my program, I keep very careful control over live
references, make reasonable assumptions about how GC will behave,
and then rely on those assumptions. I think that's perfectly reasonable
programming -- but it's also part of the reason I don't like systems
that use conservative or some kinds of generational GC.
c) It's not just files. There's lots of "precious resources" -- big
strings, for example. There isn't always an analog of "close the file"
to release precious resources. GC better have some reliable properties.
DoS attacks? Sure. How about DEGoS attacks (degradation of service)?
-t
Alain Picard
> a) What if my open files are hidden behind some abstraction?
> I expect "widgets" to be GC'ed, but the widget holds open
> files behind the barrier. I'm supposed to know this?
No, but the widgets sure as hell are supposed to know it, so they
should provide a widger user level api which ensure that things
will "just work".
Maybe (WITH-WIDGETS (w) .... (display w) ... ) ?
Kent M Pitman
> [This picks up on a semantic subthread of the "what is false" thread.]
>
>
> many:
> [Don't rely on GC to close your files. It's just asking
> to lose.]
>
> Sure, 98.45% of the time.
>
> But there's some subtleties.
>
>
> a) What if my open files are hidden behind some abstraction?
> I expect "widgets" to be GC'ed, but the widget holds open
> files behind the barrier. I'm supposed to know this?
No, but the widget-writer is.
I don't personally like the idea of a widget arranging closure of its
file due to GC. This means that if the widget is doing I/O, the
buffers might not be flushed properly. There's no guarantee that an
implementation will do a clean close. It might even do an abort close,
deleting the file... Relying on this to happen right seems like
sloppy programming.
If you disagree, I think you need to suggest a worked example so that
we can talk specifics and alternatives. For example, I can't tell if
the widget also has a process or, if not, by what means the widget
gains periodic control to do any I/O.
> b) I divide my reliance on GC into two parts: parts of my program I
> just Don't Worry About It and then see how memory usage comes out.
You might not worry about them as a client, but as programmers we need to
always be talking from the point of view of who programmed them. So if
that's not you, you may not have to care, but you also should not be in
this discussion. If you did program them, then you are obliged to care.
> Other parts of my program, I keep very careful control over live
> references, make reasonable assumptions about how GC will behave,
> and then rely on those assumptions. I think that's perfectly reasonable
> programming -- but it's also part of the reason I don't like systems
> that use conservative or some kinds of generational GC.
You are making a LOT of assumptions about the GC here. e.g., some
GC's take hours to run, while others take fractions of a second.
In some systems, GC can run in foreground and in others the system
is ground to a halt by GC running. What a "reasonable assumption" is
depends GREATLY on context. You can limit yourself to certain
implementations, of course, but then your programs are relying on a
non-portability.
> c) It's not just files. There's lots of "precious resources" -- big
> strings, for example. There isn't always an analog of "close the file"
> to release precious resources. GC better have some reliable properties.
No idea what you're talking about here.
> DoS attacks? Sure. How about DEGoS attacks (degradation of service)?
I don't get your point here either. Can you elaborate?
Duane Rettig
> [This picks up on a semantic subthread of the "what is false" thread.]
>
>
> many:
> [Don't rely on GC to close your files. It's just asking
> to lose.]
>
> Sure, 98.45% of the time.
>
> But there's some subtleties.
>
>
> a) What if my open files are hidden behind some abstraction?
> I expect "widgets" to be GC'ed, but the widget holds open
> files behind the barrier. I'm supposed to know this?
You should only _expect_ behaviors that the writer(s) of your widgets
have specified. If the widget writers have specified that you must
close or otherwise release the widget, then not doing so is of course
a bug.
> b) I divide my reliance on GC into two parts: parts of my program I
> just Don't Worry About It and then see how memory usage comes out.
> Other parts of my program, I keep very careful control over live
> references, make reasonable assumptions about how GC will behave,
> and then rely on those assumptions. I think that's perfectly reasonable
> programming -- but it's also part of the reason I don't like systems
> that use conservative or some kinds of generational GC.
Resourcing of memory itself is not a problem at all, although one must
be careful when using resources (e.g. to override sluggish behavior in
a gc) not to create more problematic behaviors than those being solved.
> c) It's not just files. There's lots of "precious resources" -- big
> strings, for example. There isn't always an analog of "close the file"
> to release precious resources. GC better have some reliable properties.
In CL, strings do have such bounding forms; witness for example
WITH-INPUT-FROM-STRING and WITH-OUTPUT-TO-STRING - when these forms
are exitted the macros are free either to let the string-streams and/or
their strings (as appropriate) be gc'd or else put back into a resource
pool.
We should narrow down what we are _really_ talking about, here.
What does it mean to "close" a file? Does close mean destroy? Certainly
not, otherwise there would be no need for open-stream-p. No, a closed
stream is simply defined as one on which most of the stream operations
can no longer be performed. It is the closing of a stream which makes
it available for GC, not the other way around.
Closing in fact means disassociating it from the lower-level I/O
resource to which it had been connected.
So let's then talk about these "precious resources", and relate that
to GC: I think of GC as the reclaimer of the "memory" resource; a
resource that is usually fairly abundant but which also gets high usage.
In many operating systems, file descriptors are "precious" in that
there are relatively few of them, and when they are connected (i.e.
"opened") to streams, which are objects which consume memory, then
we have a disparity of resource availabilty between the number of
stream objects one can create in memory, and the number of file
descriptors one can use at one time. Thus, when many streams are
created, it is more likely that the file descriptors will run out
more quickly than memory. We have seen such situations in server
applications where the number of streams exceeds the file number
limit, and thus these streams must be explicitly closed or the GC
explicitly configured (usually by limiting it). If, however,
you can guarantee that your application's usage pattern will not
cause the file number limit to be exceeded, then you could let the
streams be finalized and let them close themselves when no longer
pointed to. But neither of these situations falls into the category
of "just Don't Worry About It"... in both cases you have things you
must consider. Perhaps, if you simply didn't worry about it, it
really turns out to be the case that you didn't test it enough, and
someday a user of your software will poke you with a complaint about
the operating system complaining about file descriptors being used
up...
Remember, a finalization allows code to run to release resources,
some precious, but the basic mechanism, the GC, is in fact a
resource manager, i.e. memory, which resource is likely to be more
abundant than any other resource you might attach a finalization to
anyway.
--
Duane Rettig du...@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Ray Blaak
> [Don't rely on GC to close your files. It's just asking
> to lose.]
>
> Sure, 98.45% of the time.
>
> But there's some subtleties.
By all means, arrange things to be released via GC, but only as a safety net.
For normal situations, expensive resource allocation should have a "manual"
release mechanism (e.g. close files, dispose windows/widgets, etc.).
The point is to do both, and then one gets the benefits of both.
--
Cheers, The Rhythm is around me,
The Rhythm has control.
Ray Blaak The Rhythm is inside me,
bl...@telus.net The Rhythm has my soul.
Kaz Kylheku
> [This picks up on a semantic subthread of the "what is false" thread.]
>
>
> many:
> [Don't rely on GC to close your files. It's just asking
> to lose.]
>
> Sure, 98.45% of the time.
>
> But there's some subtleties.
>
>
> a) What if my open files are hidden behind some abstraction?
> I expect "widgets" to be GC'ed, but the widget holds open
> files behind the barrier. I'm supposed to know this?
No; the widget is supposed to be written sanely, with respect to the
context of its use, and should manage the opening and closing of the
file accordingly.
The widget may have a good reason to keep the file open for its entire
lifetime, until a GC finalization routine is invoked. Or it may have a
good reason to open and close it in individual method calls. Or the
widget may expose some protocol (say transactions) that the
abstraction user must follow, and the management of the open file
handles can be tied to that. The files are opened when the transaction
begins, and closed when it ends, though the widget user isn't aware
that files are being used.
> b) I divide my reliance on GC into two parts: parts of my program I
> just Don't Worry About It and then see how memory usage comes out.
> Other parts of my program, I keep very careful control over live
> references, make reasonable assumptions about how GC will behave,
> and then rely on those assumptions. I think that's perfectly reasonable
> programming -- but it's also part of the reason I don't like systems
> that use conservative or some kinds of generational GC.
>
> c) It's not just files. There's lots of "precious resources" -- big
> strings, for example. There isn't always an analog of "close the file"
> to release precious resources. GC better have some reliable properties.
A precious resource is one that the program cares about. If it cares
about it, then it keeps a reference. If no more references exist to a
big string, then it's not a problem in the same way that an open file
can be a problem; when the space is needed, it will be reclaimed.
Among finalization routines, weak references and weak hash
tables---features that any decent Lisp implementation provides---I
think you are pretty much covered, in terms of being able to hack
against any GC-related situation.
One problem that can come up is when you have objects from two realms
tied together: the GC realm and the non-GC realm. Failing to collect
in the GC realm might cause an out-of-memory condition in the non-GC
realm due to the hoarding of references to the foreign objects. The
out-of-memory condition in that realm won't trigger GC which would
liberate that memory.
This could happen with open files: the system says that no more files
can be opened, yet the garbage collected environment has tons of
unreferenced objects containing open files.
You need to somehow hook the event ``no more files'' into triggering
garbage collection. A system design that allows advice code to be
hooked into resource-acquisition routines would make this easily
possible.
Tom Lord
this is really to everyone who replied.]
Grossly simplifying and summarizing various replies, people have said:
Close your files explicitly. Don't rely on GC to do it.
That's asking to lose.
Well, obviously. That's applicable in most situations and is the
right thing to do.
But it's not always that simple. I'll describe one instance of why
abstractly, trying to fill in enough details to convince you there are
plausibly instances.
Let's suppose that we have an abstract data type, T. Because its an
ADT, multiple implementations are possible. For example, T might be a
type representing numeric computations whose arbitrary degree of
precision is determined on-demand.
In addition, we have a separate library A, that manipulates objects
that may be of type T. It might do this explicitly, knowing something
about the ADT, or it might do it generically (T objects are just
objects to A). It is a property of A that the lifetimes of the
objects it manipulates are managed by garbage collection. The
functionality in A doesn't lend itself to a `with-foo' style
interface; there is no sensible way in A to add `close-foo' to the
interface. For example, A might be a library for building, mutating,
and querying a certain class of graphs.
Next we have a particular implementation B of our type T -- an exotic
implementation in which a B object involves both open files and
references to a graph that may include instances of the object itself.
Perhaps, to continue the example and explain the need for open files,
this implementation is participating in a distributed computation, or
perhaps it is engaging in database transactions against a scientific
data set. Perhaps, to explain the need for the graph reference, the
computations B objects are performing is solving some graph of
constraints.
Now I want to write a program P that builds graphs of the sort managed
by A, labeling edges or nodes with values of the sort described by ADT
T. Some of these T values I want to be instances of type B, with
their open files, and those instances will refer to this same A-graph.
Continuing the example: I am able to prove, using graph theory and
some details of the task performed by P, that the number of properly
live instances of B objects in my A-graphs will always be small --
will always be below the system resource limits. At the same time,
I expect my program to create many B objects over its run, adding them
to the graph, and letting them drop away later.
In this case, it's perfectly reasonable for me to expect my open files
(in B instances) to be closed by GC, and to never have my program
fail by having too many open files. This reasonable expectation
should be true for all good implementations.
So, if your collector can't meet that reasonable expectation, I don't
think it's a good collector -- at least not for a general purpose lisp
or scheme. Conservative collectors and collectors that do not trigger
a full collection to avoid failing to open a file both fail to meet
this test of reasonableness.
It's also desirable in some circumstances for me to have an even
stronger expectation, though one that I'm willing to say needs to be
met by only _some_ good implementations: that not only will I not run
out of open files, but that the number of open files will always be
within a small constant factor of the number of properly live open
files needed by my A-graph. Not every good collector needs this weak
real-time property, but many should have it.
If your collector can't meet this second reasonable expectation --
that's a warning label to put on it. I guess some kinds of
generational GC need this label.
-t
Joe Marshall
> In this case, it's perfectly reasonable for me to expect my open
> files (in B instances) to be closed by GC, and to never have my
> program fail by having too many open files. This reasonable
> expectation should be true for all good implementations.
Since we're making demands upon the software, let me make some.
It seems as reasonable to me that a good OS would have sufficient
resources to let a program open as many files as it wants. In fact,
why should you need to `open' a file at all? It's just a collection
of bits somewhere, so why can't you just access them by name?
I'm not sure that the GC *ought* to take responsibility for a failing
of the OS.
Barry Margolin
Tom Lord <lo...@emf.emf.net> wrote:
>In this case, it's perfectly reasonable for me to expect my open files
>(in B instances) to be closed by GC, and to never have my program
>fail by having too many open files. This reasonable expectation
>should be true for all good implementations.
While you may consider this expectation reasonable, there's nothing in the
language specification that requires it. I'm pretty sure there are a
number of popular implementations that don't meet it.
The simple fact is that Common Lisp doesn't portably provide the underlying
mechanisms you need to create implementations of an ADT that needs
"closing" without exposing that as part of its generic API. So if there's
a possibility that some of the implementations of your ADT might need this,
you must make it a requirement that the caller perform this operation. The
non-exotic implementations can simply make it a no-op, but it's there as a
hook for the ones that need it. But expecting the system to do it
automatically for you, while desirable, is not supported by the language.
A number of implementations provide "finalizers", which provide a hook into
the GC so that you can make certain actions happen when a GC occurs. But I
don't think they provide a way for you to trigger a new GC automatically
when some resource is exhausted (you might be able to establish a condition
handler for the error that indicates that OPEN failed because it ran out of
file descriptors, but this will definitely not be portable, and you'll have
to wrap it around anything that might open a file).
--
Barry Margolin, bar...@genuity.net
Genuity, Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
Barry Margolin
The reason GC exists at all is because we know that Lisp systems are
embedded in operating systems that have various kinds of resource limits.
The one that GC traditionally deals with is memory (real or virtual,
depending on the system), but we know that almost all operating systems
also have other limits, such as simultaneous open files. So it's not
completely unreasonable to expect GC to deal with this other resource limit
for us.
The reason we use GC to manage memory and not files is because memory
allocation happens much more frequently than opening files, and it's very
common to have complex networks of memory references. We know from
experience that keeping track of this is error-prone; memory leaks and
dangling references are well-known problems in programs written in
languages that require manual memory management. Micro-management of
memory is difficult for application programmers, but we've discovered that
it's relatively easy for an automated GC to do (although you can still have
memory leaks, due to failing to clear references to objects that are not
really being used any more).
Open file leaks are not unheard of, but they're not generally considered
the serious problem that memory management is. I think they're usually due
to high-level logic errors, and are relatively easy to fix. For instance,
you have a loop that opens a file each time through, and you simply forgot
to close it at the end of the iteration. It's relatively rare that an open
file's extent is complicated in the same way that a reference to a memory
object might be (Tom's strawman example of a stream hidden within an ADT is
one of those rare occurrences). So the need for stream GC was never felt
to be as great as memory GC, and it wasn't made a requirement.
Another reason why I think stream GC is not required is because the precise
semantics are not clear. If an open file is abandoned like this, should it
be considered a normal close or an abortive one?
Pascal Costanza
> It seems as reasonable to me that a good OS would have sufficient
> resources to let a program open as many files as it wants. In fact,
> why should you need to `open' a file at all? It's just a collection
> of bits somewhere, so why can't you just access them by name?
>
> I'm not sure that the GC *ought* to take responsibility for a failing
> of the OS.
That's an interesting idea. What about the following scheme?
(defun access (file)
(unless (%openp file)
(unless (file-handles-available)
(close-least-recently-used-file))
(%open file))
(mark-as-recently-used file))
(defun open (file)
(access file))
(defun read-byte (file)
(access file)
(%read ...))
This should give the illusion of an unlimited amount of file handles.
(Of course, performance would degrade in extreme cases, and the scheme
needs to be fine-tuned for multi-threading.)
Opinions?
Pascal
--
Given any rule, however ‘fundamental’ or ‘necessary’ for science, there
are always circumstances when it is advisable not only to ignore the
rule, but to adopt its opposite. - Paul Feyerabend
Kent M Pitman
> Let's suppose that we have an abstract data type, T. Because its an
> ADT, multiple implementations are possible. For example, T might be a
> type representing numeric computations whose arbitrary degree of
> precision is determined on-demand.
>
> In addition, we have a separate library A, that manipulates objects
> that may be of type T. It might do this explicitly, knowing something
> about the ADT, or it might do it generically (T objects are just
> objects to A). It is a property of A that the lifetimes of the
> objects it manipulates are managed by garbage collection. The
> functionality in A doesn't lend itself to a `with-foo' style
> interface; there is no sensible way in A to add `close-foo' to the
> interface. For example, A might be a library for building, mutating,
> and querying a certain class of graphs.
I'd suggest that if it's a long-lived thing like this that it should be
registered in some sort of lookup table and that de-listing it with the
lookup table should cause it to go away. I'm not convinced that a
'library' that is only known by 'word of mouth' and not by 'white pages
lookup' is a realistic example. And if it is known by 'white pages lookup'
it will persist as long as the white pages do.
> Next we have a particular implementation B of our type T -- an exotic
> implementation in which a B object involves both open files and
> references to a graph that may include instances of the object itself.
> Perhaps, to continue the example and explain the need for open files,
> this implementation is participating in a distributed computation, or
> perhaps it is engaging in database transactions against a scientific
> data set. Perhaps, to explain the need for the graph reference, the
> computations B objects are performing is solving some graph of
> constraints.
>
> Now I want to write a program P that builds graphs of the sort managed
> by A, labeling edges or nodes with values of the sort described by ADT
> T. Some of these T values I want to be instances of type B, with
> their open files, and those instances will refer to this same A-graph.
>
> Continuing the example: I am able to prove, using graph theory and
> some details of the task performed by P, that the number of properly
> live instances of B objects in my A-graphs will always be small --
> will always be below the system resource limits. At the same time,
> I expect my program to create many B objects over its run, adding them
> to the graph, and letting them drop away later.
>
> In this case, it's perfectly reasonable for me to expect my open files
> (in B instances) to be closed by GC, and to never have my program
> fail by having too many open files. This reasonable expectation
> should be true for all good implementations.
This is a false assumption. Just because you drop pointers to
something, you should NOT assume they are going to be GC'd. The GC
may only need to run when there is a lack of storage. Consequently,
you might have many old pointers you no longer touch that you think
are GC'd but that are really just candidates for GC. And it's not at
all reasonable to assume that magic should prevail.
> So, if your collector can't meet that reasonable expectation, I don't
> think it's a good collector -- at least not for a general purpose lisp
> or scheme.
Some GC's do gain power by keeping the working set small, but even those
work in quanta bigger than 1 and do not assume that the mere availability
of an item to be gc'd means that thing will eventually be shrinkwrapped
out of the system.
But in general, a GC is responsible for (and its goodness is measured
by) making sure that an attempt to cons succeeds. If that happens, the
GC is a good one. You can create some other definition of a collector,
but that's not been the one Lisp has had for ages.
> Conservative collectors and collectors that do not trigger
> a full collection to avoid failing to open a file both fail to meet
> this test of reasonableness.
You can say this all you like, and maybe some vendor will buy into
what you're saying. But don't confuse that with saying there is a
canonical notion of goodness such that your notion is the one.
The reason it's not specified in the spec is to allow the details to be
market driven, so it's actually intended by the language that people can
speak up about what matters to them, as you have. I'm not trying to say
that your concern is not a legitimate one. I'm just trying to say that
it would be better for your terminology to say "GC's of this class are
not useful to me" rather than saying "GC's ought to be said to be not
useful if they don't satisfy my particular need". The latter is quite
an egocentric remark.
> It's also desirable in some circumstances for me to have an even
> stronger expectation, though one that I'm willing to say needs to be
> met by only _some_ good implementations: that not only will I not run
> out of open files, but that the number of open files will always be
> within a small constant factor of the number of properly live open
> files needed by my A-graph. Not every good collector needs this weak
> real-time property, but many should have it.
>
> If your collector can't meet this second reasonable expectation --
> that's a warning label to put on it. I guess some kinds of
> generational GC need this label.
I disagree that this requires a "warning".
I think it's fine if your LAYERED product provides a warning saying it
won't work well with GC's that don't do this because that warning would
be about something you have specified. But I don't think it's reasonable
to require a product that conforms to a standard to warn that it doesn't
conform to some other thing that it never purported to conform to in the
first place.
It's fine with me if GC's that do what you like get marked with the Tom
Lord seal of approval.
I hope you understand that all of my remarks here about expectation
setting, and that while expectations are relative, some of the things
to which they are relative are already fixed in some of the scenarios
you're talking about, and you are pretending they are not. The CL
spec not constraining the GC is not license for you to say "they
probably just left out that verbiage--I'll tell you what they should
have said"; instead, it means "there is wide latitude".
The programming style you are advocating is not one I would recommend to
any student of mine. In general, I never recommend that people adopt
styles that fight against the rules of the framework in which the person
is going to program.
I also don't think the example you gave is really a very useful one.
It still doesn't tell me what the objects are or why the objects have a
file open, so I can't analyze whether there is a better way to do things.
It's not that I don't think that it's possible for there to be a situation
that is programmed as you say; what I'm not sure of is that there is a
real world situation that requires such pathological programming. I don't
like the idea of allowing a problematic programming paradigm to lead the
discussion; I'd feel better hearing that there was some real world situation
which had no other way to be programmed. And I just don't see it yet.
Kent M Pitman
> lo...@emf.emf.net (Tom Lord) writes:
>
> > In this case, it's perfectly reasonable for me to expect my open
> > files (in B instances) to be closed by GC, and to never have my
> > program fail by having too many open files. This reasonable
> > expectation should be true for all good implementations.
>
> Since we're making demands upon the software, let me make some.
>
> It seems as reasonable to me that a good OS would have sufficient
> resources to let a program open as many files as it wants. In fact,
> why should you need to `open' a file at all? It's just a collection
> of bits somewhere, so why can't you just access them by name?
It's a poor man's transaction system. Ideally you're grabbing a read or
write lock on the file so you get a consistent view. Yes, yes, Unix
probably allows the file to change out from underneath you when you
think you have such a lock... sigh.
> I'm not sure that the GC *ought* to take responsibility for a failing
> of the OS.
Neither should the file system have to suffer the embarrassment of having
open not really lock the file...
Then again, perhaps this is the proverbial dual between the unstoppable
force and the impenetrable barrier. One programmer may have the desire
both to forcibly lock the file so no one else can modify it while another
has the desire to modify it even if someone has it locked...
Kent M Pitman
> In article <isxqnc...@ccs.neu.edu>, Joe Marshall <j...@ccs.neu.edu> wrote:
> >lo...@emf.emf.net (Tom Lord) writes:
> >
> >> In this case, it's perfectly reasonable for me to expect my open
> >> files (in B instances) to be closed by GC, and to never have my
> >> program fail by having too many open files. This reasonable
> >> expectation should be true for all good implementations.
> >
> >Since we're making demands upon the software, let me make some.
> >
> >It seems as reasonable to me that a good OS would have sufficient
> >resources to let a program open as many files as it wants. In fact,
> >why should you need to `open' a file at all? It's just a collection
> >of bits somewhere, so why can't you just access them by name?
> >
> >I'm not sure that the GC *ought* to take responsibility for a failing
> >of the OS.
>
> The reason GC exists at all is because we know that Lisp systems are
> embedded in operating systems that have various kinds of resource limits.
> The one that GC traditionally deals with is memory (real or virtual,
> depending on the system), but we know that almost all operating systems
> also have other limits, such as simultaneous open files. So it's not
> completely unreasonable to expect GC to deal with this other resource limit
> for us.
Well, indeed. On an OS in which opening the file holds it safe
against modification, deletion, recreation, or the like, then the
"resource" that is being competed for is not "number of open file
channels" but "simultaneous access at all". If you're going to have a
file system in which you can prove no two people will ever access the
same file, it's going to be kind of lonely and pointless. The whole
reason we HAVE file systems is so that one person can name a name and
another person can hear the name and get to the same data...
> Another reason why I think stream GC is not required is because the precise
> semantics are not clear. If an open file is abandoned like this, should it
> be considered a normal close or an abortive one?
I agree that this is a key issue not addressed in the spec and therefore
something serious programmers should care about. I am surprised I haven't
heard any pleading here for a FINALIZE-OBJECT method in which someone could
control this behavior precisely. That wouldn't address the issue of getting
the object finalized aggressively, but at least it would make sure that when
the long-awaited event happened, the expected action occurred.
Tim Bradshaw
> Another reason why I think stream GC is not required is because the precise
> semantics are not clear. If an open file is abandoned like this, should it
> be considered a normal close or an abortive one?
I think the general thing that makes GC work, is that it's invisible
to the program and (almost) everyone else. Once the program has no
references to some memory, then the GC can quietly recycle it, or not,
and really no-one knows. In particular the act of recycling a bit of
memory doesn't have any observable effects (modulo real-timeness
issues) - once there are no references nothing bad can happen.
But files or streams in general are not like this at all. The act of
`reclaiming' a stream by closing it can have very major effects:
buffers get flushed which can cause disks to fill (and not just disks
on your machine), locks on the file get released causing all sorts of
excitement, network connections get unilaterally closed causing yet
more excitement. Errors can happen during all these processes which
need to be dealt with, but by whom? By the GC?
Someone is now going to say `but none of this should happen - there
should always be space on the disk preallocated for file buffers, and
everyone knows that Zagbert's stateless file locking system (which was
even partly implemented once, or so he said) solves the file locking
problem. And anyway none of this would happen if you used a *real* OS
like, say this obscure ITS variant I hacked up one night (I'd show it
to you know, but I lost it because the disk filled up when I tried to
write out the file buffers)! And who needs files and network streams
anyway? We didn't have files in my day, oh no, we counted ourselves
lucky to have bits, I tell you! We never had enough parens...'. So
that's, what? 8 intractable problems which, if we could solve them, we
could use GC for files...
But, in the world we actually live in rather than some lisp hacker's
alternative universe, files and network connections have
characteristics which are almost totally unlike memory. Why is it
surprising (not to Barry, but apparently to some other people) that GC
is a good approach for one, but a really, really bad for the other?
--tim
Kent M Pitman
> [...] What about the following scheme?
>
> (defun access (file)
> (unless (%openp file)
> (unless (file-handles-available)
> (close-least-recently-used-file))
> (%open file))
> (mark-as-recently-used file))
>
> (defun open (file)
> (access file))
>
> (defun read-byte (file)
> (access file)
> (%read ...))
>
> This should give the illusion of an unlimited amount of file
> handles. (Of course, performance would degrade in extreme cases, and
> the scheme needs to be fine-tuned for multi-threading.)
>
> Opinions?
In a file system that has no actual locks anyway, I suppose this isn't
awful. Though it assumes that everyone who is competing for locks is
inside this universe--it probably doesn't interact well with other
(e.g., non-Lisp) processes.
But some file systems actually lock the file while open and some programs
WANT that lock. Imagine the above as a proposed solution for some kind of
database deadlock and ask yourself if the right thing is simply to drop your
lock randomly when someone else wants it and you probably see the problem.
Pascal Costanza
> Pascal Costanza <cost...@web.de> writes:
>
>
>>[...] What about the following scheme?
>>
>>(defun access (file)
>> (unless (%openp file)
>> (unless (file-handles-available)
>> (close-least-recently-used-file))
>> (%open file))
>> (mark-as-recently-used file))
>>
>>(defun open (file)
>> (access file))
>>
>>(defun read-byte (file)
>> (access file)
>> (%read ...))
>>
>>This should give the illusion of an unlimited amount of file
>>handles. (Of course, performance would degrade in extreme cases, and
>>the scheme needs to be fine-tuned for multi-threading.)
>>
>>Opinions?
[...]
> some file systems actually lock the file while open and some programs
> WANT that lock. Imagine the above as a proposed solution for some kind of
> database deadlock and ask yourself if the right thing is simply to drop your
> lock randomly when someone else wants it and you probably see the problem.
Yes, I realized that just before I went to bed.
Thanks for pointing this out.
Pascal
--
Pascal Costanza University of Bonn
mailto:cost...@web.de Institute of Computer Science III
http://www.pascalcostanza.de Römerstr. 164, D-53117 Bonn (Germany)
Joe Marshall
> If you're going to have a file system in which you can prove no two
> people will ever access the same file, it's going to be kind of
> lonely and pointless. The whole reason we HAVE file systems is so
> that one person can name a name and another person can hear the name
> and get to the same data...
Then why do non-networked single user machines have files?
I like the `poor man's transaction and persistence' premise.
Barry Margolin
Replace "person" with "entity" in Kent's paragraph. The entities may be
people, applications, or temporally separated invocations of the same
application.
Barry Margolin
Pascal Costanza <cost...@web.de> wrote:
>Joe Marshall wrote:
>
>> It seems as reasonable to me that a good OS would have sufficient
>> resources to let a program open as many files as it wants. In fact,
>> why should you need to `open' a file at all? It's just a collection
>> of bits somewhere, so why can't you just access them by name?
>>
>> I'm not sure that the GC *ought* to take responsibility for a failing
>> of the OS.
>
>That's an interesting idea. What about the following scheme?
>
>(defun access (file)
> (unless (%openp file)
> (unless (file-handles-available)
> (close-least-recently-used-file))
> (%open file))
> (mark-as-recently-used file))
>
>(defun open (file)
> (access file))
>
>(defun read-byte (file)
> (access file)
> (%read ...))
>
>This should give the illusion of an unlimited amount of file handles.
>(Of course, performance would degrade in extreme cases, and the scheme
>needs to be fine-tuned for multi-threading.)
It doesn't work very well in cases where the OS's file handle keeps state
information about the file. This is often the case when the OS allows
access to non-file devices via the filesystem.
For instance, if the "file" is a terminal device, closing it might cause
the modem to hang up. If it's a tape, it may rewind.
And even in the case of plain files, the OS often keeps track of the file
position, although in this case the Lisp runtime system could do an
automatic seek when it reopens it.
Kent M Pitman
> In article <7ke6m7...@ccs.neu.edu>, Joe Marshall <j...@ccs.neu.edu> wrote:
> >Kent M Pitman <pit...@world.std.com> writes:
> >
> >> If you're going to have a file system in which you can prove no two
> >> people will ever access the same file, it's going to be kind of
> >> lonely and pointless. The whole reason we HAVE file systems is so
> >> that one person can name a name and another person can hear the name
> >> and get to the same data...
> >
> >Then why do non-networked single user machines have files?
> >
> >I like the `poor man's transaction and persistence' premise.
>
> Replace "person" with "entity" in Kent's paragraph. The entities may be
> people, applications, or temporally separated invocations of the same
> application.
Or temporally separated invocations of the same person. ;)
Thanks, Barry. Just the right answer. So much more concise than me.
Tom Lord
[The whole thread has degnerated into incoherence. I don't expect
replies. Random replies of my own follow and if they happen to
"connect" in a productive way: great! But that's not the expected
outcome. Consider it "quasi-poetry" based on the assumption that we
all pretty much agree about the structure of the problem space --
and "get high" (so to speak) on contemplating it.]
Joe Marshall:
It seems as reasonable to me that a good OS would have
sufficient resources to let a program open as many files as it
wants.
I strongly disagree. I want my kernel to be simple. Simple means, in
part, that it can presume I don't want it to solve non-universal
problems that can be solved in user space. Number of open files is
one such problem. And, if there is such a thing as a literary
reference in C.S., the "PC loser-ing" problem is another example.
Kent Pitman:
It's a poor man's transaction system. Ideally you're grabbing
a read or write lock on the file so you get a consistent view.
Yes, yes, Unix probably allows the file to change out from
underneath you when you think you have such a lock... sigh.
"Ideally"? No. Unix (in sophisticated practice, not theory - the
formal standards are horrid) has the wonderful property of reflecting
the underlying physics of the situation, leaving non-universal details
up to user-space. It isn't perfect in that regard, but it's pretty
good.
Tertiary storage inherently has the nasty properties that unix
syscalls reveal. Idealizing here, as your comment suggests doing,
amounts to oversimplifying.
Barry Margolin:
Tom's strawman example of a stream hidden within an ADT is one
of those rare occurrences)
Rare, sure. "Strawman"? No. Why do you think so?
If an open file is abandoned like this, should it be
considered a normal close or an abortive one?
This reminds me of the "close on exec?" problem in unix. In other
words, it is reasonable to say that it should be a per-stream
parameter.
Tim Bradshaw:
I think the general thing that makes GC work, is that it's
invisible to the program and (almost) everyone else. Once the
program has no references to some memory, then the GC can
quietly recycle it, or not, and really no-one knows. In
particular the act of recycling a bit of memory doesn't have
any observable effects (modulo real-timeness issues) - once
there are no references nothing bad can happen.
But files or streams in general are not like this at all.
A model of the semantics of a program identifies a set of (in
principle, modulo "solving computability") referencable objects.
GC's job is to preserve those object and release all others.
All you've done with your "invisibility" analysis is, well, nothing.
Process memory, viewed as a resource, is not really all that different
from open files.
Barry Margolin:
While you may consider this expectation reasonable, there's
nothing in the language specification that requires it. I'm
pretty sure there are a number of popular implementations that
don't meet it.
No doubt.
The simple fact is that Common Lisp doesn't portably provide
the underlying mechanisms you need to create implementations
of an ADT that needs "closing" without exposing that as part
of its generic API.
Yes, that's a bug.
Kent Pitman:
Just because you drop pointers to something, you should NOT
assume they are going to be GC'd. The GC may only need to run
when there is a lack of storage.
Yes, that's a bug.
The reason it's not specified in the spec is to allow the
details to be market driven,
Yes, that's a bug.
I'm not trying to say that your concern is not a legitimate
one.
Completely clear. We are somewhat speaking at cross purposes. I
don't think we have any real disagreement except about how to speak
about the issues in a public forum.
I'm just trying to say that it would be better for your
terminology to say "GC's of this class are not useful to me"
rather than saying "GC's ought to be said to be not useful if
they don't satisfy my particular need". The latter is quite
an egocentric remark.
Please don't psychologize my remarks unless you want me to
psychologize yours.
I don't think it's reasonable to require a product that
conforms to a standard to warn that it doesn't conform to some
other thing that it never purported to conform to in the first
place.
And, on the topic of psychologizing, that's one point where'd I'd
start.
-t
Tim Bradshaw
> All you've done with your "invisibility" analysis is, well, nothing.
> Process memory, viewed as a resource, is not really all that different
> from open files.
Oh dear. The issue you've missed is that, if you want to model a
program which involves streams, you have to model all other processes
which may be aware of these streams. Before a GC can recycle some
memory it needs to establish that no-one will know if it does: it can
do this because it has total knowledge about the use of memory in the
program. Before a GC can recycle a stream, it needs to establish that
nothing will change if it does. It can do this only if it has total
knowledge of *all* the processes which may be aware of that stream.
Stream GC is distributed object GC, because streams are objects which
are distributed. Oh well, never mind.
--tim
Kent M Pitman
> I'm just trying to say that it would be better for your
> terminology to say "GC's of this class are not useful to me"
> rather than saying "GC's ought to be said to be not useful if
> they don't satisfy my particular need". The latter is quite
> an egocentric remark.
>
> Please don't psychologize my remarks unless you want me to
> psychologize yours.
>
>
> I don't think it's reasonable to require a product that
> conforms to a standard to warn that it doesn't conform to some
> other thing that it never purported to conform to in the first
> place.
>
>
> And, on the topic of psychologizing, that's one point where'd I'd
> start.
This wasn't intended to be a remark about your psychology. I didn't say
what motivates you. I didn't call you good or bad. I merely remarked
on your choice of phrasing. You might think that superficial (though
I would disagree) but I certainly did not intend it to be, nor do I
think it was, ad hominem.
I wouldn't have picked at your remark if you had prefaced it with
"I think" to remind yourself and others that it was an opinion.
I included such a caveat in my remark to remind myself and others
that my notion of what is reasoanble is "just what I think" not
some kind of law. You're welcome to "psychologize" that if you like.
I'm arguing for a pluralistic world, and for people to try to remark
on it on a regular basis because leaving out such qualifying remarks
tends to lead to people fighting to the death over matters of no
importance.
If you were trying to say "there is only one good way to do this",
that's a problem. It appears to those not familiar with how things
work that you are making an assertion of fact, and it calls upon those
with conflicting views to log their opinions so that someone diving
through with google doesn't find your remark undebated and think it is
accepted fact. If you were trying to say "In my opinion, there is
only one good way to do this", there is nothing to contest. You
certainly have that opinion, and you certainly have the right to voice
it. I find it interesting, and to the extent that it is mere opinion,
I'm glad you voiced it. To the extent that you appear to be asserting
there is no room for alternate opinions, then I and others will
likely continue to assert otherwise.
It's possible that you think all of your remarks are automatically
preferenced with "I think" merely because of the "From: Tom Lord" in
the message header. The problem with that is that such a way of
interpreting newsgroup posts means there is no way to ever state an
objective fact. There is a school of thought that says that this is
right, and that in fact all things, even facts, are merely opinions.
I personally find that school of thought "less than useful". True to
the world or not, I like the absraction of intent behind assertions
of fact.
Ray Blaak
> Kent Pitman:
>
> Just because you drop pointers to something, you should NOT
> assume they are going to be GC'd. The GC may only need to run
> when there is a lack of storage.
>
> Yes, that's a bug.
That depends on your goals.
Immediate cleanup of dropped pointers can perform badly in conjunction with
strong real time requirements, for example, because of the cascading cleanups
that might further result.
Amortizing cleanups works better in this situation. But this makes
GC-controlled resource cleanup fair poorly.
There is no one right answer, especially with garbage collection. The ideal
behaviour depends greatly on the application.
Andy Freeman
> It doesn't work very well in cases where the OS's file handle keeps state
> information about the file. This is often the case when the OS allows
> access to non-file devices via the filesystem.
It doesn't even work for boring filesystem files, at least under Unix
and probably other OSs as well.
Why? Because Unix lets one delete/unlink open files. The relevant handles
continue to work until closed and then the file's contents may become
inaccessible. I'm not sure when the association between those contents and
the file name disappears, but it's unlikely to be around when needed by
a runtime system that closes and reopens files on its own, behind the
application's back.
And, no, there's no way for the run-time system to know that this is a
consideration because the unlink may occur outside its scope.
It isn't enough to check whether an unlink has occurred before closing
because the unlink may occur while the runtime has it closed. And, even
if the run-time could know which files it can't close, the "as many open
files as you'd like" illusion can't be maintained.
-andy
Joe Marshall
> Joe Marshall:
>
> It seems as reasonable to me that a good OS would have
> sufficient resources to let a program open as many files as it
> wants.
>
>
> I strongly disagree. I want my kernel to be simple. Simple means, in
> part, that it can presume I don't want it to solve non-universal
> problems that can be solved in user space. Number of open files is
> one such problem.
So you prefer the `90% solution' to `the right thing'?
> And, if there is such a thing as a literary reference in C.S., the
> "PC loser-ing" problem is another example.
In what way is it an example? Do you believe that the standard unix
mechanism of returning EINTR to the user process is preferrable to
making system calls appear to be atomic? I certainly don't think so.
> Kent Pitman:
>
> It's a poor man's transaction system. Ideally you're grabbing
> a read or write lock on the file so you get a consistent view.
> Yes, yes, Unix probably allows the file to change out from
> underneath you when you think you have such a lock... sigh.
>
>
> "Ideally"? No. Unix (in sophisticated practice, not theory - the
> formal standards are horrid) has the wonderful property of reflecting
> the underlying physics of the situation, leaving non-universal details
> up to user-space. It isn't perfect in that regard, but it's pretty
> good.
But the underlying physics provides a terrible model. One important
reason for having an OS is to abstract away the hardware. Another is
to abstract away the complexities surrounding multiple simultaneous
threads of control.
> Tertiary storage inherently has the nasty properties that unix
> syscalls reveal. Idealizing here, as your comment suggests doing,
> amounts to oversimplifying.
There are better solutions than what unix syscalls provide! (And many
of them predate unix.) Sure, there are certain things that should not
be abstracted away --- for instance, the `end-to-end-ness' of a
network connection --- but things the size of wired tables within the
kernel should be completely hidden from the user.
> A model of the semantics of a program identifies a set of (in
> principle, modulo "solving computability") referencable objects.
> GC's job is to preserve those object and release all others.
Actually, this is a conservative approximation. (Not to be confused
with conservative GC). The semantics of a program define the set of
all objects that *will* be used by the continuation at any point in
the program. All other objects --- the ones that *will not* be used
in the future --- can be recycled. Of course, this is a
non-computable property, so we settle for a more conservative approach
and assume that all *reachable* objects (the ones that *may* be used
in the future) must be retained. One could imagine a GC that was able
to make use of certain information that the compiler leaves around to
come up with a tighter set of constraints that would allow it to
collect reachable, but never-to-be-used storage.
Tom Lord
Here's a summary/restatement/expansion of my little argument about
closing files from GC:
A) Some otherwise reasonable programs can not be written (cleanly and
simply) unless the program can rely on GC closing files. The
program I sketched supports this conclusion.
B) Closing streams in response to their having been GC'ed is easy to
implement. In fact, I know of several Scheme systems that do this.
Some of them make it a per-port option. I have trouble imagining a
modern implementation that does not already support this behavior,
if only indirectly through finalization mechanisms.
C) Running a full GC when an open fails because of too many open
streams, in an attempt to reclaim and close some streams, is also
easy to implement. Again, I know of at least some systems that
already do this. I can imagine why this behavior would not
_always_ be desirable, so I can see arguments for making it a
dynamic and/or per-open option.
D) Closing streams from GC, and collecting when streams appear to have
been exhausted, at least as optional behaviors, are both cleanly
compatible extensions to a system which does not already have these
properties. It has a clear and useful meaning and it has no impact
on programs which do not rely on this behavior. Yes, there are
questions to be answered, such as how asynchronous errors can be
handled. Yes, in the context of the CL and Scheme standards,
collecting on stream/port exhaustion could (for performance
reasons) be reasonably made an available but non-default
behavior.
Since it easy to do, clean, and useful, I conclude that it should be
done. For language designers and implementors (A) provides the best
motive, and (B), (C), and (D) provide the opportunity.
Kent: I think that that is reasonably called an objective conclusion
(or at least a refutable argument that attempts to reach an objective
conclusion) -- not an opinion. Your response has, however, forced me
to state the argument far more clearly (I hope).
Various People: Yes, many good programs do not need this feature.
Yes, as a very commonly applicable design pattern, programs do well to
avoid this feature. Yes, neither the CL nor Scheme standards
guarantee this behavior. No, none of that refutes the argument that
this is an objectively desirable feature.
It's not something I want to "fight to the death over" as Kent put
it. I just thought that point (A) in particular was a mildly
interesting observation with some interesting conclusions. I became
interested in trying to articulate (A) in response to discussions
elsewhere about the risks of conservative GC, and its appropriateness
for use in some situations.
Finally, I think this is also an interesting observation:
Tim Bradshaw:
The issue you've missed is that, if you want to model a
program which involves streams, you have to model all other
processes which may be aware of these streams. Before a GC
can recycle some memory it needs to establish that no-one will
know if it does: it can do this because it has total knowledge
about the use of memory in the program. Before a GC can
recycle a stream, it needs to establish that nothing will
change if it does. It can do this only if it has total
knowledge of *all* the processes which may be aware of that
stream.
I see. For example, the remote end of a stream, if it is a process,
may notice that the stream has been closed. There initially seems to
be nothing analogous with regards to memory.
It's interesting that purely-memory GC has that property, but I don't
think it's useful. I see no a priori reason why it is important to
preserve that property.
In this particular case, if the application writer is going to trust
GC to close some stream, then presumably (from the lisp
implementation's perspective), the application writer has taken the
side effect into account.
Memory is not necessarily so different. For example, if some of the
memory in question is shared with another process, being used for
communication between them, then the "complete model" for memory-GC
would have to include information about how the state of that memory
is interpreted (is there a semaphore there? what state should it be
in?). You could say, well, shared memory should always be a special
object -- never a seemingly ordinary string. I'd counter that that
would be an artificial and inconvenient limitation. We could go round
and round on that.
What about other resources? If an implementation supports having lots
of lightweight threads, should those (at least optionally) be
anonymous and collected? Or does their potential side effects on the
environment preclude that because the GC can not model the meaning of
those side effects?
-t
Kenny Tilton
just memory", it would make more sense (for some new language). But this
title has that backwards. the subject as is whines "why doesn't this
memory management mechanism do what I want with non-memory?". Hello?
Dude, mark-and-sweep seems to be the cat's meow for GC. That's all ya
need. But you seem to need it for open files and big strings.
Congratulations on your great idea for a new language! One which can ask
allocated memory not just what other memory it references, but what
other resources it references. files, strings, whatever. resources to be
defined by you, in your new language.
But please don't fuck up a purely memory-oriented GC by asking it to
manage resources it never thought about. Can you say "category error"?
Sher ya can.
Finalization? Nah, memory GC is cake because it is expressly limited to
and affects only memory as reachable by Lisp. mark-n-sweep. But Lisp
talks to other languages, and my processs may be using resources Lisp
memory does not knwo about it. Pleeeeze don't screw up a nice mechanism
by asking it to manage things it cannot know about.
--
kenny tilton
clinisys, inc
http://www.tilton-technology.com/
---------------------------------------------------------------
"Cells let us walk, talk, think, make love and realize
the bath water is cold." -- Lorraine Lee Cudmore
Scott Schwartz
(closing files by gc) take a look at Bell Labs' Limbo programming
language and Inferno operating system.
Tom Lord
resource, not just memory", it would make more sense (for some
new language)
First, it is your view that CL is now fixed in stone, no longer ammenable
to extension or improvement?
Second, maybe I missed some list charter somewhere - but "lisp != CL"
and "new language" is part of what lisp is about -- we hack at all
levels, including the dialect/implementation.
But please don't fuck up a purely memory-oriented GC by asking
it to manage resources it never thought about. Can you say
"category error"? Sher ya can.
I'm curious: are there really "purely memory-oriented GC"
implementations other than the one in GNU Emacs? Every other one I've
encountered (which is far from all or even most of them) have had some
type-specific free-ing code and/or finalization mechanisms and/or weak
references (a finalization precursor) for freeing non-memory resources
associated with freed objects. Emacs is even capable, internally, of
freeing non-memory resources -- but last I checked the semantics of
windows, subprocesses, buffers and so forth scrupulously avoided using
that capability by keeping global, life-preserving lists of these
things and only allowing deletion from these lists by `close-'-like
procedures. This limitation in Emacs, meanwhile, makes some kinds of
reasonable programs impractical to write in Emacs Lisp -- though that
is an arguably reasonable limitation to impose on a text editor.
One which can ask
allocated memory not just what other memory it references, but
what other resources it references. files, strings, whatever.
Um, aren't "strings" in the category "memory"? I don't see the
category error there and, indeed, I don't recall that strings have to
be explicitly deallocted in CL.
And the difference between strings and files is a slippery one: I can
imagine wanting to treat memory-mapped files as strings, in which case
they would refer to open files. In my Scheme dialect, strings can
share memory with I/O buffers of low-level C libraries.
Outside of a very controlled, limited environment, such as the Emacs
run time environment, "purely memory-oriented" is a terrible
limitation.
Finalization? Nah, memory GC is cake because it is expressly
limited to and affects only memory as reachable by
Lisp. mark-n-sweep.
Mark and sweep is a technique for discovering which objects are live,
and which not. Finalization is a functionality that can be imposed on
otherwise no-longer-live objects. These are orthogonal concepts.
There is no contradiction between them.
Dude, mark-and-sweep seems to be the cat's meow for GC. That's
all ya need.
(Assuming you mean the whole family of mark-and-sweep algorithms):
yes, I think that's _nearly_ true. Copying and generations are
overrated (but pretty). But that's a whole 'nother discussion, dude.
-t
Tom Lord
> means, in part, that it can presume I don't want it to solve
> non-universal problems that can be solved in user space.
> Number of open files is one such problem.
So you prefer the `90% solution' to `the right thing'?
No. I think that unix provides one of the better approximations of
`the right thing'.
The unix filesystem interface is a pretty good set of primitives. It
is useful in and of itself, relatively easy to implement, and in
combination with IPC and authentication facilities, it gives you a
sufficient basis set for implementing file-like storage with stronger
guarantees.
> And, if there is such a thing as a literary reference in
> C.S., the "PC loser-ing" problem is another example.
In what way is it an example? Do you believe that the
standard unix mechanism of returning EINTR to the user process
is preferrable to making system calls appear to be atomic? I
certainly don't think so.
In some programs it _is_ preferable. In programs where it is not, the
same effect can be achieved by a layer of user-space code over the
EINTR mechanism. As with filesystems, unix has provided a small, simple
basis set, useful in and of itself, and suitable for implementing
more complicated behaviors.
[de facto unix filesystem semantics reflect physics]
But the underlying physics provides a terrible model. One
important reason for having an OS is to abstract away the
hardware.
Abstracting away the difference between an IDE disk and a SCSI disk
is very useful. Abstracting away physical properties that are invariant
across pretty much all systems is excessive.
There are better solutions than what unix syscalls provide!
(And many of them predate unix.) Sure, there are certain
things that should not be abstracted away --- for instance,
the `end-to-end-ness' of a network connection --- but things
the size of wired tables within the kernel should be
completely hidden from the user.
I'm all for eliminating arbitrary limits in the kernel when it can be
done without making the kernel much more complicated or significantly
slower. That would eliminate part, but not all, of the reasons for
closing streams from GC. As I understand it, incremental improvements
in this area are common on both BSD-based and Linux kernels.
> A model of the semantics of a program identifies a set of
> (in principle, modulo "solving computability") referencable
> objects. GC's job is to preserve those object and release
> all others.
Actually, this is a conservative approximation. (Not to be
confused with conservative GC). The semantics of a program
define the set of all objects that *will* be used by the
continuation at any point in the program. All other objects
--- the ones that *will not* be used in the future --- can be
recycled. Of course, this is a non-computable property, so we
settle for a more conservative approach and assume that all
*reachable* objects (the ones that *may* be used in the
future) must be retained. One could imagine a GC that was
able to make use of certain information that the compiler
leaves around to come up with a tighter set of constraints
that would allow it to collect reachable, but never-to-be
I don't think you disagreed with me really.
As an aside, it would be useful for standards to state a minimum
"conservative approximation" of GC that all implementations provide.
It would have to be _very_ conservative, though, in order not to
preclude implementations from having introspective features, debuggers,
and such.
-t
Kenny Tilton
Tom Lord wrote:
> I think if the subject of this thread were "GC for any
> resource, not just memory", it would make more sense (for some
> new language)
>
>
> First, it is your view that CL is now fixed in stone, no longer ammenable
> to extension or improvement?
?? I was not kidding when I said it was a good idea, just don't ask
memory GC to do it. I am just saying that unless your improvement is
cast properly as a grander, more general category of GC in which
mark-and-sweep works for my entire process (not just Lisp code) and for
arbitrary other resources (not just memory), then we run the risk of
cocking things up by asking too much of GC.
We engineers get cute like that all the time, trying to get something
for nothing. ie, Gee, I can't figure out when to close my files... hey!
let GC do it! But then it turns out some bozo installed enough RAM to
choke a horse and then (1) file stayed open waaay too long to satisfy
some other requirement. Oh! OK! then we just... nahhh.
Here's the alternative, picking a project at random: I feel a little
guilty about my Cells scheme because, when it is used to build
interesting tree structures of Cell-powered Model instancess (ie, all
the time) one has to ensure a new instance goes thru both "to-be" and
"not-to-be" methods on the way in and out of the model. I do not feel
too guilty because, effectively, it's all done reliably by a single
around setter method on a single "children" slot. I /never/ get into
trouble, because the only way to get into trouble would be to use (setf
slot-value) as a backdoor.
In your motivating example you talked about file-happy instances of type
B just "dropping away". There's your problem. Fix that. Work out a nice
manageable scheme for them to explicitly leave the game and handle the
file closes yourself. Don't ask a memory management scheme to manage
files. It's like asking Hal to lie. He didn't know how, look what happened.
Yeah Lisp is extensible, but it is also very powerful, so before asking
for extensions make sure Lisp as is will not suffice.
Thomas F. Burdick
> Lord wrote:
> > I think if the subject of this thread were "GC for any
> > resource, not just memory", it would make more sense (for some
> > new language)
> >
> >
> > First, it is your view that CL is now fixed in stone, no longer ammenable
> > to extension or improvement?
>
> ?? I was not kidding when I said it was a good idea, just don't ask
> memory GC to do it. I am just saying that unless your improvement is
> cast properly as a grander, more general category of GC in which
> mark-and-sweep works for my entire process (not just Lisp code) and for
> arbitrary other resources (not just memory), then we run the risk of
> cocking things up by asking too much of GC.
While he's saying that the GC must do this and should do that, what he
wants is a reference counter, not a GC. Part of the reason this
thread is so tiresome is that one participant is claiming that all
reasonable GC should do things one way, and that such-and-such should
be considered a bug -- when the values the GC is being judged by are
the values of a reference counter, not a GC.
And for general resource management, reference counting works great.
There's no ambiguity as to when a file is closed, say; it's closed
exactly when the last reference is dropped. It just happens to be a
shitty way to manage memory.
--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'
Tim Bradshaw
> Memory is not necessarily so different. For example, if some of the
> memory in question is shared with another process, being used for
> communication between them, then the "complete model" for memory-GC
> would have to include information about how the state of that memory
> is interpreted (is there a semaphore there? what state should it be
> in?).
Well, yes. One of the classic non-GC'd language bugs is freeing
something which is in fact still in use by someone else, causing (if
you're very lucky) some kind of exception, or (if you're not) silent
failure. So I think that, if you have shared memory (either
implicitly, as for instance when you give chunks of memory to
something in an FFI, or explicitly, as OS-shared memory say), then if
you unilaterally free it you need to be sure that whatever is sharing
it can both detect this freeing (by getting an exception on next
access, say), and is willing to handle the consequences. I guess the
normal way to do that would be to have some kind of protocol where you
(the programs, not the GC) communicate and agree their intent to stop
using the shared memory.
> You could say, well, shared memory should always be a special object
> -- never a seemingly ordinary string. I'd counter that that would
> be an artificial and inconvenient limitation. We could go round and
> round on that.
Why would it need to be special? You can just allocate a string (or
whatever you need) from a special area of memory. Once you have it
it's just like any other object modulo GC behaviour. This is, of
course, what systems actually do at present.
--tim
Duane Rettig
> Here's a summary/restatement/expansion of my little argument about
> closing files from GC:
>
>
> A) Some otherwise reasonable programs can not be written (cleanly and
> simply) unless the program can rely on GC closing files. The
> program I sketched supports this conclusion.
What sketch is that? I googled in this thread and the "what is false"
thread from which this one was derived, and could not find any such
sketch.
I dispute your premise. We may have different ideas about what is
"clean" and "simple", but I have never seen a problem that cannot
be done reasonably well without gc'ing file descriptors but which can
be done by gc'ing them. Recall Einstein's admonition: "Make everything
as simple as possible, and no simpler". We should discuss some of the
real issues of trying to gc file descriptors, and what edge conditions
such a design might run into.
> B) Closing streams in response to their having been GC'ed is easy to
> implement.
I dispute that it is easy to implement. Have you implemented one
yourself? Or do you have statements from the implementors that state
that it was easy to do?
> In fact, I know of several Scheme systems that do this.
Scheme systems tend to be used in academic situations, where edge
conditions are not fully tested. I suspect that there are serious
holes in these implementations.
> Some of them make it a per-port option. I have trouble imagining a
> modern implementation that does not already support this behavior,
> if only indirectly through finalization mechanisms.
Finalization does _not_ support such behavior, because it only guarantees
that the finalization will happen when the gc is run (and for generational
gcs, this may mean a full gc). If the gc (which is a memory reclaiming
mechanism) doesn't key off of other resources than memory, then you have
no guaranteed that the finalization will occur before you run out of that
resource. Which brings us to the question of tripping a gc on file
descriptors:
> C) Running a full GC when an open fails because of too many open
> streams, in an attempt to reclaim and close some streams, is also
> easy to implement. Again, I know of at least some systems that
> already do this. I can imagine why this behavior would not
> _always_ be desirable, so I can see arguments for making it a
> dynamic and/or per-open option.
You really should not keep saying that these things are "easy to
implement" without providing your credentials for saying so.
I dispute the claim. Some systems may do this, but it is certainly not
an easy thing to get right.
Consider: file descriptors are a resource which
- are globally used - they will be used by the Lisp and by C code.
- are not predictive. You can usually guess which file descriptor
you are going to next get (it is usually the first sequential
integer that is not currently open, though sometimes there is a
delay when a socket port is closed before the port number is
reallocated), but there is no actual guarantee of how the allocation
occurs.
- have no allocation hooks. A program cannot register a hook which
will run either before a file is opened, or an allocation function
which will then allocate and provide the allocated file number.
Thus, there is no way for Lisp to perform a gc when a C or other non-lisp
section of the program happens to be the code which fails to allocate
a file descriptor.
> D) Closing streams from GC, and collecting when streams appear to have
> been exhausted, at least as optional behaviors, are both cleanly
> compatible extensions to a system which does not already have these
> properties. It has a clear and useful meaning and it has no impact
> on programs which do not rely on this behavior. Yes, there are
> questions to be answered, such as how asynchronous errors can be
> handled. Yes, in the context of the CL and Scheme standards,
> collecting on stream/port exhaustion could (for performance
> reasons) be reasonably made an available but non-default
> behavior.
What does "exhaustion" mean, here? I suppose you could think of a
socket as being exhausted when it receives a "connection reset by peer"
signal, but otherwise it tends to remain open unless and until both sides
agree that the connection need no longer stay open. (this SIGPIPE signal
is in fact a good reason to automatically close a stream, but it has
nothing at all to do with GC). And file streams are not "exhausted"; even
if a read on a file yields EOF, any stream which has position information
may change its position at any time. So I submit that the only kind of
"exhaustion" that makes sense is one where the stream is no longer
referenced, which gets back to being a memory-collection issue. Others
have already stated on this thread that redemption of the file numbers
in such a situation by using finalization is a fine safety net in
addition to explicit management, but it cannot make guarantees that
there will not be resoures used by otherwise dead streams, and so should
not be the primary mechanism.
> Various People: Yes, many good programs do not need this feature.
> Yes, as a very commonly applicable design pattern, programs do well to
> avoid this feature. Yes, neither the CL nor Scheme standards
> guarantee this behavior. No, none of that refutes the argument that
> this is an objectively desirable feature.
I don't dispute this. What I dispute is that:
1. It is easy to do correctly
2. There are programs which absolutely need this feature.
For #1, please post your design outlining the ease in which file
descriptors can be reliably used to trigger gcs. [This is _not_
a rhetorical challenge; altough I doubt that it can be done easily
and portably, I am a reasonable designer, and if you can show me
the goods, I will snap it up quickly and with much grattitude]
For #2, I must have missed the outline of the program which must have
this feature; please repost.
> What about other resources? If an implementation supports having lots
> of lightweight threads, should those (at least optionally) be
> anonymous and collected? Or does their potential side effects on the
> environment preclude that because the GC can not model the meaning of
> those side effects?
I think we should stick to one resource at a time. If we cannot agree
on open files, how will we even be able to start conversing about
gc'ing threads?
--
Duane Rettig du...@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Tom Lord
for different reasons than what that person said. I was thinking of
starting a new one on the general topic of design patterns, and the
specific topic of "subtle aspects of GNU Emacs that make it good".
Yes, no, maybe?
Last reply unless something really interesting comes up:
> A) Some otherwise reasonable programs can not be written (cleanly and
> simply) unless the program can rely on GC closing files.
> The program I sketched supports this conclusion.
What sketch is that?
Enclosed, but you shouldn't need it. You can easily make one for
yourself.
Programs build data structures that can be described as graphs of
objects with references to one another. Freeing nodes of those
graphs gets arbitrarilly hairy, depending on the structure of the
graph and how it evolves. Find a natural reason to store non-memory
resources in the nodes, and then you have a reason to want GC for
those resources.
-t
The particular instance I sketched was:
Subject: Re: closing files by gc
Close your files explicitly. Don't rely on GC to do it.
That's asking to lose.
Well, obviously. That's applicable in most situations and is the
right thing to do.
But it's not always that simple. I'll describe one instance of why
abstractly, trying to fill in enough details to convince you there are
plausibly instances.
Let's suppose that we have an abstract data type, T. Because its an
ADT, multiple implementations are possible. For example, T might be a
type representing numeric computations whose arbitrary degree of
precision is determined on-demand.
In addition, we have a separate library A, that manipulates objects
that may be of type T. It might do this explicitly, knowing something
about the ADT, or it might do it generically (T objects are just
objects to A). It is a property of A that the lifetimes of the
objects it manipulates are managed by garbage collection. The
functionality in A doesn't lend itself to a `with-foo' style
interface; there is no sensible way in A to add `close-foo' to the
interface. For example, A might be a library for building, mutating,
and querying a certain class of graphs.
Next we have a particular implementation B of our type T -- an exotic
implementation in which a B object involves both open files and
references to a graph that may include instances of the object itself.
Perhaps, to continue the example and explain the need for open files,
this implementation is participating in a distributed computation, or
perhaps it is engaging in database transactions against a scientific
data set. Perhaps, to explain the need for the graph reference, the
computations B objects are performing is solving some graph of
constraints.
Now I want to write a program P that builds graphs of the sort managed
by A, labeling edges or nodes with values of the sort described by ADT
T. Some of these T values I want to be instances of type B, with
their open files, and those instances will refer to this same A-graph.
Continuing the example: I am able to prove, using graph theory and
some details of the task performed by P, that the number of properly
live instances of B objects in my A-graphs will always be small --
will always be below the system resource limits. At the same time,
I expect my program to create many B objects over its run, adding them
to the graph, and letting them drop away later.
In this case, it's perfectly reasonable for me to expect my open files
(in B instances) to be closed by GC, and to never have my program
fail by having too many open files. This reasonable expectation
should be true for all good implementations.
So, if your collector can't meet that reasonable expectation, I don't
think it's a good collector -- at least not for a general purpose lisp
or scheme. Conservative collectors and collectors that do not trigger
a full collection to avoid failing to open a file both fail to meet
this test of reasonableness.
It's also desirable in some circumstances for me to have an even
stronger expectation, though one that I'm willing to say needs to be
met by only _some_ good implementations: that not only will I not run
out of open files, but that the number of open files will always be
within a small constant factor of the number of properly live open
files needed by my A-graph. Not every good collector needs this weak
real-time property, but many should have it.
If your collector can't meet this second reasonable expectation --
that's a warning label to put on it. I guess some kinds of
generational GC need this label.
-t
Kenny Tilton
Tom Lord wrote:
> Someone commented that this thread has become tedious.
Yup, looks like deadly embrace. Of course if you want to admit you are
wrong... :)
No, seriously, i got interested when I realized this is a classic case
of how programmers (and others) get into trouble. They get cute and try
to reuse some tool for purposes never intended. Implementation
stumbles--no problem! We'll set a flag and clear it at the next null
event! Too late? I know, we'll... unfortunately never notice that each
speed bump is a manifestation of some specific way the new role is
inappropriate for the tool. ie, these are not just the usual
problems-to-be-solved, these are Messages From God to "stop, turn back,
wrong way, do not proceed, what part of 'abandon all hope' do you not
understand?"
> I agree, but
> for different reasons than what that person said. I was thinking of
> starting a new one on the general topic of design patterns, and the
> specific topic of "subtle aspects of GNU Emacs that make it good".
> Yes, no, maybe?
>
>
> Last reply unless something really interesting comes up:
>
> > A) Some otherwise reasonable programs can not be written (cleanly and
> > simply) unless the program can rely on GC closing files.
> > The program I sketched supports this conclusion.
>
> What sketch is that?
>
> Enclosed, but you shouldn't need it. You can easily make one for
> yourself.
No, that is the whole problem, your example is not persuasive, and some
of us hear think flaws which appear only in bogus examples are not
flaws. Not that I am saying it is a flaw that Lisp GC does not do
arbitrary work for you which you choose post hoc to assign to its designers.
>
> Programs build data structures that can be described as graphs of
> objects with references to one another. Freeing nodes of those
> graphs gets arbitrarilly hairy
I am thinking there is one slot for which there is one setter to
specialize with an after method. Whoa, i need a break! :)
, depending on the structure of the
> graph and how it evolves. Find a natural reason to store non-memory
> resources in the nodes, and then you have a reason to want GC for
> those resources.
Who needs a reason? Memory GC is great, it would be great to have a
super GC which watched and managed other resources. Your mistake is in
faulting the sub GC for not doing your file management for you.
So you may as well give up on finding an example, that's not what GC
does. And while we're at it, saying GCs should come with a warning label
was really a bit much.
Tom Lord
No, that is the whole problem, your example is not persuasive,
Potentially fair and interesting (but clearly wrong). If it isn't
persuasive, please state why, exactly?
Also: you ignored my question - do you know of any "pure memory GC"
aside from the one in GNU Emacs? If your GC has finalizers, weak
references, closes ports, or kills threads or engines -- it isn't
"pure memory". The original motivation for the example was
conservative GC, which is so broken that you can't count on _any_
particular object _ever_ being collected.
-t
Matthew Danish
> conservative GC, which is so broken that you can't count on _any_
> particular object _ever_ being collected.
If you have a better solution for a garbage collector for C/C++, or on a
register-starved architecture, please let us know. Better yet, donate
an implementation so everyone can be happy.
Maybe you can't _count_ on any object being GCed but practically
speaking, does it matter?
--
; Matthew Danish <mda...@andrew.cmu.edu>
; OpenPGP public key: C24B6010 on keyring.debian.org
; Signed or encrypted mail welcome.
; "There is no dark side of the moon really; matter of fact, it's all dark."
Basile STARYNKEVITCH
Matthew> On Mon, Dec 23, 2002 at 07:29:13AM -0000, Tom Lord wrote:
>> conservative GC, which is so broken that you can't count on
>> _any_ particular object _ever_ being collected.
Matthew> If you have a better solution for a garbage collector for
Matthew> C/C++, or on a register-starved architecture, please let
Matthew> us know. Better yet, donate an implementation so
Matthew> everyone can be happy.
I don't know if it is better, but there exists some exact (or precise)
garbage collectors for C or C++. For example, my Qish runtime on
http://starynkevitch.net/Basile/qishintro.html and on
http://freshmeat.net/projects/qish
Qish is a precise, generational copying GC for C (with finalized
objects). It is open-source under LGPL. Of course being not
conservative but precise, Qish requires therefore a precise style of
coding - which translate the requirements that every pointer should be
precisely known to the GC. Qish could be useful in C++
Precise GC for C do have drawbacks: since they are precise, and since C
does not have the notion of garbage-collected pointers, you have to
follow some very specific coding style which is a constraint on the
coder.
Regards.
--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net
alias: basile<at>tunes<dot>org
8, rue de la Faïencerie, 92340 Bourg La Reine, France
Tom Lord
me:
> [this all started from a consieration of] conservative GC,
> which is so broken that you can't count on
> _any_ particular object _ever_ being collected.
matthew danish:
Maybe you can't _count_ on any object being GCed but
practically speaking, does it matter?
Let's see now, possible replies:
A) And thus, the circle is complete.
B) AAAUUUUGHHH! [insert lucy/charlie/football image]
C) I think Alan Watts conclusively related all "does it matter"
questions to wave mechanics. Regardless, chem-lab jars _do_
make good kitchen equipment.
D) It's all good. It's cool.
E) Wouldn't "what's good about Emacs?" be more interesting? I mean
sure, there's obvious things like doc-strings and `interactive',
but then there's subtle thing, like "pure-memory" GC,
user-perspective programming primitives, buffer-based scope,
data-driven redisplay, and repl-based interaction and flow control.
Xemacs blew it by making events and keymaps something other than
basic lisp types (consonance is a virtue). But what else? What
haven't I thought of that make Emacs good.
too bored for my own good,
-t
Duane Rettig
> Someone commented that this thread has become tedious. I agree, but
> for different reasons than what that person said.
I wasn't the one who said this thread has become tedious. If this
thread is tedious for you, why do you continue it?
> I was thinking of
> starting a new one on the general topic of design patterns, and the
> specific topic of "subtle aspects of GNU Emacs that make it good".
> Yes, no, maybe?
No, thank you.
> Last reply unless something really interesting comes up:
>
> > A) Some otherwise reasonable programs can not be written (cleanly and
> > simply) unless the program can rely on GC closing files.
> > The program I sketched supports this conclusion.
>
> What sketch is that?
>
> Enclosed, but you shouldn't need it. You can easily make one for
> yourself.
No, thank you. I prefer believing my own position, rather than to
try to refute it.
Below, you quote the answer to my first question:
> The particular instance I sketched was:
>
> Subject: Re: closing files by gc
>
>
> Close your files explicitly. Don't rely on GC to do it.
> That's asking to lose.
>
> Well, obviously. That's applicable in most situations and is the
> right thing to do. [ ... ]
Yes, I apologize; I passed over this as an example of a reasonable
program that met my criteria (which I quote from my post: "I have
never seen a problem that cannot be done reasonably well without
gc'ing file descriptors but which can be done by gc'ing them"),
simply because I only saw this example as a bad design which had a
great potential for bugs. In my mind being a bad design disqualifies
it as having the potential of resulting in a reasonable program.
[You might want to ask me for the reasons why I consider your design
to be bad. It is not part of this discussion, but some of it may
become apparent from the discussion below]
> Programs build data structures that can be described as graphs of
> objects with references to one another. Freeing nodes of those
> graphs gets arbitrarilly hairy, depending on the structure of the
> graph and how it evolves. Find a natural reason to store non-memory
> resources in the nodes, and then you have a reason to want GC for
> those resources.
As my own assertion (repeated above) claims, there is no more reason
to want GC to handle these resources than to handle them explicitly.
This is true whether or not the reason you have to store non-memory
resources in the nodes is natural or not.
Other than the answer to my question about what sketch you had provided,
you failed to answer _any_ of my other questions and assertions,
especially those in which I refute your assertion that making such a
GC would be "easy". OK, so you don't want to answer these, for whatever
reason. Perhaps you're even now preparing your answer.
Meanwhile, let's move on to the answer you did give me:
> The particular instance I sketched was:
>
> Subject: Re: closing files by gc
>
>
> Close your files explicitly. Don't rely on GC to do it.
> That's asking to lose.
>
> Well, obviously. That's applicable in most situations and is the
> right thing to do.
>
> But it's not always that simple. I'll describe one instance of why
> abstractly, trying to fill in enough details to convince you there are
> plausibly instances.
>
>
> Let's suppose that we have an abstract data type, T. Because its an
> ADT, multiple implementations are possible. For example, T might be a
> type representing numeric computations whose arbitrary degree of
> precision is determined on-demand.
>
> In addition, we have a separate library A, that manipulates objects
> that may be of type T. It might do this explicitly, knowing something
> about the ADT, or it might do it generically (T objects are just
> objects to A). It is a property of A that the lifetimes of the
> objects it manipulates are managed by garbage collection. The
> functionality in A doesn't lend itself to a `with-foo' style
> interface; there is no sensible way in A to add `close-foo' to the
> interface. For example, A might be a library for building, mutating,
> and querying a certain class of graphs.
I assume here that your "close-foo" reference is a reference to the
portions of B that contain open files, described below. You are
asserting that a solution that involves the imaginary GC (the one
which is triggerred on lack of file descriptors) can be constructed,
whereas an explicit deallocation solution cannot be done in a sensible
way.
> Next we have a particular implementation B of our type T -- an exotic
> implementation in which a B object involves both open files and
> references to a graph that may include instances of the object itself.
> Perhaps, to continue the example and explain the need for open files,
> this implementation is participating in a distributed computation, or
> perhaps it is engaging in database transactions against a scientific
> data set. Perhaps, to explain the need for the graph reference, the
> computations B objects are performing is solving some graph of
> constraints.
>
> Now I want to write a program P that builds graphs of the sort managed
> by A, labeling edges or nodes with values of the sort described by ADT
> T. Some of these T values I want to be instances of type B, with
> their open files, and those instances will refer to this same A-graph.
You are asserting the existence of circularities, here. OK.
> Continuing the example: I am able to prove, using graph theory and
> some details of the task performed by P, that the number of properly
> live instances of B objects in my A-graphs will always be small --
> will always be below the system resource limits. At the same time,
> I expect my program to create many B objects over its run, adding them
> to the graph, and letting them drop away later.
Here is where the problem comes in; you have just asserted here that
the number of open files is bounded and provable, but that implies
that the maximum number of open files is computable. This means that
any one time there must be a way of computing _why_ the number of
open files has a limit, and thus it is possible to compute _which_
open files are not necessary to keep open and can thus be explicitly
closed. This conflicts with your original assertion that explicit
closing cannot be done in a sensible way.
Now, I agree that computing which files can be closed might be a
daunting task, depending on your actual task and design, but I
believe that this does not disqualify the possibility for doing
this without also disqualifying your assertion as to having a proof
for the bounding of live B objects (which translates, presumably, to
a bounding of open files).
Having thus no proof of bounding, there is no guarantee that the
number of files will not exceed the file limit set by the operating
system.
> In this case, it's perfectly reasonable for me to expect my open files
> (in B instances) to be closed by GC, and to never have my program
> fail by having too many open files. This reasonable expectation
> should be true for all good implementations.
I assume by "open files in B instances", you really mean only those
which are no longer in use.
I can easily make your program fail. Try this experiment. On
any linux box, run csh(1) to get a csh prompt. Type "limit" and a
return to get a listing of resource availability, including
"descriptors", which refers to the number of file descriptors that can
be open at a time. Type "limit descriptors 1" and a return, followed
by "limit", and watch what happens (csh will fail).
If you then take your application, and calculate the maximum number N of
file descriptors it will ever need (using your purported proof), and
then perform the above kind of limit command using N-1 as the new
limit, and then run your program, it will fail either by having too many
open files, or by an infinite loop in a GC which keeps running because
there are not enough file resources.
In fact, give me any such maximum, and I can trivially give you a
resource limit which causes your program to fail just as miserably
as (and possibly in an even worse way than) the same program which
explicitly manages file descriptors.
> So, if your collector can't meet that reasonable expectation, I don't
> think it's a good collector -- at least not for a general purpose lisp
> or scheme. Conservative collectors and collectors that do not trigger
> a full collection to avoid failing to open a file both fail to meet
> this test of reasonableness.
>
> It's also desirable in some circumstances for me to have an even
> stronger expectation, though one that I'm willing to say needs to be
> met by only _some_ good implementations: that not only will I not run
> out of open files, but that the number of open files will always be
> within a small constant factor of the number of properly live open
> files needed by my A-graph. Not every good collector needs this weak
> real-time property, but many should have it.
>
> If your collector can't meet this second reasonable expectation --
> that's a warning label to put on it. I guess some kinds of
> generational GC need this label.
Kent has already given you some good responses to these assertions,
so I will decline to comment.
Matthew Danish
> Let's see now, possible replies:
[elided]
I'm not sure what to make of these responses, perhaps you have a slight
misunderstanding?
> E) Wouldn't "what's good about Emacs?" be more interesting? I mean
And what is preventing you from starting a new thread? Better than
whining about it in some other one.
> too bored for my own good,
Then cut the crap and go write some code, read a book, watch a movie,
or anything.
Pascal Costanza
> Precise GC for C do have drawbacks: since they are precise, and since C
> does not have the notion of garbage-collected pointers, you have to
> follow some very specific coding style which is a constraint on the
> coder.
...and they don't seem to buy you much. Here is a paper that
investigates this issue: http://citeseer.nj.nec.com/446175.html
Pascal
--
Pascal Costanza University of Bonn
mailto:cost...@web.de Institute of Computer Science III
http://www.pascalcostanza.de Römerstr. 164, D-53117 Bonn (Germany)
Tom Lord
As my own assertion (repeated above) claims, there is no more
reason to want GC to handle these resources than to handle
them explicitly.
Look at it this way: why do you want GC for memory? Because some good
programming techniques require run-time liveness analysis to decide
what memory can be freed (neither explicit free or ref counting cut
it). So why should memory be priveleged as the resource managed by
that run-time liveness analysis? It shouldn't (and usually isn't).
There was one good proposal for why memory should be so privileged --
the "invisibility analysis". But it didn't really hold up under
scrutiny.
If this thread is tedious for you, why do you continue it?
At the moment, because I'm babysitting a test that takes about 6 or 7
hours. Largely, I feel like I'm just a part of a choir reciting canon
here, and I'm lucky enough to have the role of stating fairly obvious
things while some interesting people have generously taken the role
of saying subtly wrong things by way of counterpoint. But usenet's a
very noisy channel so one can never be quite sure.
-t
Duane Rettig
> If this thread is tedious for you, why do you continue it?
>
> At the moment, because I'm babysitting a test that takes about 6 or 7
> hours. Largely, I feel like I'm just a part of a choir reciting canon
> here, and I'm lucky enough to have the role of stating fairly obvious
> things while some interesting people have generously taken the role
> of saying subtly wrong things by way of counterpoint. But usenet's a
> very noisy channel so one can never be quite sure.
I see; in your too-bored tedium you're trolling. Well, OK, then,
Goodbye. Have a nice time babysitting.
Joe Marshall
> > I strongly disagree. I want my kernel to be simple. Simple
> > means, in part, that it can presume I don't want it to solve
> > non-universal problems that can be solved in user space.
> > Number of open files is one such problem.
>
> So you prefer the `90% solution' to `the right thing'?
>
>
> No. I think that unix provides one of the better approximations of
> `the right thing'.
We'll have to agree to disagree, then. I think the unix file system
isn't worthy of the name `file system'.
Kaz Kylheku
> As my own assertion (repeated above) claims, there is no more
> reason to want GC to handle these resources than to handle
> them explicitly.
>
> Look at it this way: why do you want GC for memory? Because some good
> programming techniques require run-time liveness analysis to decide
> what memory can be freed (neither explicit free or ref counting cut
> it).
Or: because some good programming techniques require the illusion of
infinite memory.
> So why should memory be priveleged as the resource managed by
> that run-time liveness analysis? It shouldn't (and usually isn't).
Here is a clue: memory is not involved in externally visible effects
of a program. It's just some stuff that is needed to achieve its
internal semantics.
Other resources are involved in world interaction. The way in which
they are manipulated constitutes a program's visible behavior. Closing
a network connection, opening a file, unlocking a system-wide
mutex---these actions are visible outside of the program. Therefore
the program usually must do these things in a proper order and in a
timely fashion.
The effect of consuming and recycling memory only becomes externally
visible when the supply runs low.
There is a big difference between closing a file, and freeing the
memory associated with keeping track of an open file. Closing a file
can only take place while that in-memory object exists and is
reachable; it's a side effect action that may involve I/O, and the
release of an exclusive file lock. It's not in itself an act of
internal storage reclamation; it's an act of communication with the
environment.
If a situation warrants that the communication can be delayed by
associating it with garbage collection actions, then by all means, why
not. But the convenience of that assocation does not make the two
equal.
Kaz Kylheku
> reason to want GC to handle these resources than to handle
> them explicitly.
>
> Look at it this way: why do you want GC for memory? Because some good
> programming techniques require run-time liveness analysis to decide
> what memory can be freed (neither explicit free or ref counting cut
> it). So why should memory be priveleged as the resource managed by
> that run-time liveness analysis? It shouldn't (and usually isn't).
I have this bit to add, to my other response.
In many systems interfaces, the liberation of a memory object is
conflated with a state change of a resource. For instance a system
call that liberates an in-memory file descriptor also has the effect
of closing a file.
But there are examples that don't follow this. Berkeley sockets can be
subject to a shutdown() which closes the connection, without
discarding the socket descriptor. (Moreover, when a socket is subject
to close(), the connection actually endures; the operating system
closes it in the background). Another example might be some interface
for a global lock; if the handle to the shared mutex is closed, then
the mutex might stay locked, or an ``abandoned'' error may be signaled
to the other processes that were trying to acquire the mutex. Windows
mutexes are this way. Releasing the mutex and destroying the mutex
handle are quite different actions.
If a socket descriptor is associated with a garbage-collected object,
one possible way to do it is this: require the program logic to call
the method which triggers a shutdown() call, and use GC finalization
to call close(), to seal any possible descriptor leaks. Closing the
connection is simply an act of communication which changes the state
of the resource. The descriptor is then a husk which occupies memory.
Freeing that memory may be delayed; the communication which closes the
connection may not.
Similarly, I would require the program logic to release a lock, but
might use the GC finalizer to get rid of the handle. Releasing a lock
cannot be delayed. Doing so might introduce performance problems and
even deadlocks! One process may wait for another to do something, but
that other one is waiting on a lock that is held by a reference within
the first one's garbage pool.
The operating system descriptor or handle object is simply a
foreign-domain memory resource that is associated with a Lisp-domain
resource. It's useful to tie the two together for the purposes of
garbage collection---with the risk that if the former domain runs out
of space, there might not be any event propagation to trigger the
garbage collection in the latter space which would lead to a
liberation in the former space.
The same thing is done with other foreign-domain resources---for
example, objects allocated within a C++ library using operator new, or
within a C library using malloc(). If these are tied to Lisp objects,
then a finalizer can be set up to do the foreign call that will invoke
operator delete, or free(). The malloc or new mechanisms can be
overridden so that if the fail, GC is triggered in the Lisp domain,
and then they are tried again.
Kenny Tilton
Tom Lord wrote:
....>
> There was one good proposal for why memory should be so privileged --
> the "invisibility analysis". But it didn't really hold up under
> scrutiny.
...
> ... I'm lucky enough to have the role of stating fairly obvious
> things while some interesting people have generously taken the role
> of saying subtly wrong things by way of counterpoint.
I don't know about anyone else, but my troll-detector just went off.
:)
meanwhile, after a two-three day ground fight Cello is now doing
arbitrary bitmapped or outline fonts within the context of FreeGlut
(enhanced with new code). It's fun (kinda) to be doing C again. The bad
news is I only added support for fonts for win32, the good news is that
X and the Mac both offer the same API (xgl* and agl* vs wgl*) to build
display lists OpenGL can eat.
Seems the main problem was my OpenGL ignorance. Only solved it by
finding a worked GLUT example that also used wglUseFontOutlines and then
massively and incrementally borrowing from that. Got crazy so by the
time it worked Lord knows which change was the fix. gl-scalef?
gl-perspective? the viewport bounds? gl-translatef?
All by way of explanation as to why i am now going to treat myself to
working linearly thru some Ogl tutorials. 3gl is a ball, and I am
getting badly stuck too often because I have no idea what I am doing.
Tom Lord
It was not my intention to "troll". Actually I'm surprised at the
volume of replies (some of which are obvious single-target trolls :-).
One thing that I learned from the thread is that it might be worth
trying to formalize an understanding of the intution that some
algorithms which need graph-tracing for resource mgt. can't be
simply rewritten to avoid that need. It's an instance of the
"Those who don't use lisp are are condemned to reimplement it,
poorly" family of adages.
-t
Tim Bradshaw
> Also: you ignored my question - do you know of any "pure memory GC"
> aside from the one in GNU Emacs? If your GC has finalizers, weak
> references, closes ports, or kills threads or engines -- it isn't
> "pure memory". The original motivation for the example was
> conservative GC, which is so broken that you can't count on _any_
> particular object _ever_ being collected.
I don't see the relevance of whether a GC is `pure memory' or not to
anything. Sure, most GCs provide finalization (and weak pointers,
pre-region-GC behaviour &c &c), but that is in no way related to
whether the use of (say finalization) as the *preferred* way to close
files (say) rather than a last resort recovery technique is a good
thing. The car I drive has airbags, and that's a good feature, but
that *doesn't* mean I should drive as if I expect to use them.
--tim
Kenny Tilton
> * Tom Lord wrote:
>
>
>>Also: you ignored my question -
Sorry, I did not realize you had miscontrued finalization to mean "I'll
help you manage file closing". But in general you yourself are so purely
non-responsive to what others write that I am not even going to finish
this sentence.
:)
Damien Kick
> lo...@emf.emf.net (Tom Lord) writes:
> > [Don't rely on GC to close your files. It's just asking
> > to lose.]
> >
> > Sure, 98.45% of the time.
> >
> > But there's some subtleties.
> > [examples where releasing via GC is desirable]
>
> By all means, arrange things to be released via GC, but only as a
> safety net.
>
> For normal situations, expensive resource allocation should have a
> "manual" release mechanism (e.g. close files, dispose
> windows/widgets, etc.).
>
> The point is to do both, and then one gets the benefits of both.
Somewhat recently, I was having a private e-mail exchange about a
small chunk of Lisp code that I had posted to the newsgroup to solicit
comments from anyone willing to give me a few pointers. One of the
things that was discussed was a macro that I had originally written as
something like the following.
(defmacro with-foo (bar baz &rest body)
(let ((,bar (acquire-foo ,baz)))
;; 1. Assuming that ,bar has been initialized if we get
;; further than this and that we're guaranteed to reach
;; the unwind-protect
(unwind-protect
(progn ,@body)
(when (foop ,bar) (release-f00 ,bar)))))
It was mentioned to me that my assumption might not hold true if some
kind of signal is raised between the binding of ',bar' to a value and
the evaluation of the unwind-protect form. It was suggested to me
that I rewrite with-foo to be something like the following (mistakes
are my own).
(defmacro with-foo (bar baz &rest body)
(let ((,bar nil))
(unwind-protect
(progn
(setf ,bar (acquire-foo ,baz))
;; If we get this far, the acquire-foo is within
;; the protected form. If some signal occurs
;; between the setf and the body, release-foo is
;; still evaluated.
,@body)
(when (and ,bar (foop ,bar))
(release-foo ,bar)))))
I had modeled my with-foo on the expansion I got from with-open-file
from the two Lisp implementations I have used. For example, Allegro
CL Trial Edition 6.2 gives me the following:
PG-USER(15): (macroexpand-1 '(with-open-file (str) nil))
(LET ((STR (OPEN)) (#:G421 T))
(UNWIND-PROTECT
(MULTIPLE-VALUE-PROG1 (PROGN NIL) (SETQ #:G421 NIL))
(WHEN (STREAMP STR) (CLOSE STR :ABORT #:G421))))
T
PG-USER(16):
And CMUCL 18d gives me the following:
* (macroexpand-1 '(with-open-file (str) nil))
(LET ((STR (OPEN)) (#:G851 T))
(UNWIND-PROTECT (MULTIPLE-VALUE-PROG1 (PROGN NIL) (SETQ #:G851 NIL))
(WHEN STR (CLOSE STR :ABORT #:G851))))
T
*
When I mentioned this, the response was that the GC knows how to close
streams and so with-open-file does not need to worry about something
happening between the let form and the unwind-protect form. Now, I'm
a little confused by how some of the things I've been reading in this
thread relate to this analysis of the with-open-file expansion. In
particular, I'm lead to believe by some of Duanne Rettig's comments,
for example, that the ACL GC does not close open file streams--I don't
remember his having rewriting this explicitly but I've taken what I
understand to be his disapproval of the idea of the GC closing file
streams to imply that this is the case for ACL--nor would he find that
to be a worthwhile extension. If the ACL GC does not handle closing
file streams, how does it handle the case in which some kind of
(presumably asynchronous) signal occurs between the binding of ',bar'
and the protected form? My original with-foo code was meant to run in
CMU CL and I'm assuming that the person with whom I was corresponding
knows something about the way in which CMU CL handles file streams;
i.e. it has some implementation dependent extensions to guarantee that
file streams will eventually be closed if something goes wrong in the
same spirit as Ray Blaak's comment about "getting the best of both
worlds." Is my understanding of the situation regarding CMU CL
correct? What about the case of ACL? Should I assume that looking at
the expansions of such things as with-open-file is a bad way to learn
to model my own macros because their correct behavior might rely on
extensions with which I am unfamiliar, e.g. the situation with CMU CL
arranging so that the GC closes file streams?
Duane Rettig
> Ray Blaak <bl...@telus.net> writes:
>
> > lo...@emf.emf.net (Tom Lord) writes:
> > > [Don't rely on GC to close your files. It's just asking
> > > to lose.]
> > >
> > > Sure, 98.45% of the time.
> > >
> > > But there's some subtleties.
> > > [examples where releasing via GC is desirable]
> >
> > By all means, arrange things to be released via GC, but only as a
> > safety net.
> >
> > For normal situations, expensive resource allocation should have a
> > "manual" release mechanism (e.g. close files, dispose
> > windows/widgets, etc.).
> >
> > The point is to do both, and then one gets the benefits of both.
Please note at the outset that I agree with Ray Blaak's idea to use
both mechanisms, the automatic method as a safety net for the normal
manual method.
> Somewhat recently, I was having a private e-mail exchange about a
> small chunk of Lisp code that I had posted to the newsgroup to solicit
> comments from anyone willing to give me a few pointers. One of the
> things that was discussed was a macro that I had originally written as
> something like the following.
>
> (defmacro with-foo (bar baz &rest body)
> (let ((,bar (acquire-foo ,baz)))
> ;; 1. Assuming that ,bar has been initialized if we get
> ;; further than this and that we're guaranteed to reach
> ;; the unwind-protect
> (unwind-protect
> (progn ,@body)
> (when (foop ,bar) (release-f00 ,bar)))))
>
> It was mentioned to me that my assumption might not hold true if some
> kind of signal is raised between the binding of ',bar' to a value and
> the evaluation of the unwind-protect form.
In general, in Allegro CL, any such window for interrupts is smaller than
you might think, since the Lisp only allows interrupts to be synchronized
at particular "safe" spots, and those safe spots can be extended with a
without-interrupts form. The interrupt-synchronization spots do not
include returns from functions, nor do they include the starts of
unwind-protects. The synch points occur at the beginnings of functions and
at the ends of loops. Therefore, although there are possible leaks which
can occur in the above, the above code is no safer than the code below:
> It was suggested to me
> that I rewrite with-foo to be something like the following (mistakes
> are my own).
>
> (defmacro with-foo (bar baz &rest body)
> (let ((,bar nil))
> (unwind-protect
> (progn
> (setf ,bar (acquire-foo ,baz))
> ;; If we get this far, the acquire-foo is within
> ;; the protected form. If some signal occurs
> ;; between the setf and the body, release-foo is
> ;; still evaluated.
> ,@body)
> (when (and ,bar (foop ,bar))
> (release-foo ,bar)))))
Let me explain: Note that although acquire-foo is executed within the
unwind-protect form, the value to be returned from acquire-foo (i.e. the
resource needing deallocation) is in fact not protected _until_ it is
actually assigned to the bar gensym. This means that if acquire-foo is
defined thus:
[note the lack of an enclosing without-interrupts form around the
whole body of the function]:
(defun acquire-foo (x)
(let ((res (allocate-some-foo x)))
(initialize-a-foo x)
res))
then at entry to initialize-a-foo, a foo has been allocated, but
any interrupt at that point which results in a throw out through
the with-foo body will _not_ deallocate the foo resource, because
although the unwind-protect forces the cleanup form to be run, no
deallocation takes place, because the bar gensym has not been set.
This is no different than the previous code with the allocation
outside of the unwind-protect form.
> I had modeled my with-foo on the expansion I got from with-open-file
> from the two Lisp implementations I have used. For example, Allegro
> CL Trial Edition 6.2 gives me the following:
>
> PG-USER(15): (macroexpand-1 '(with-open-file (str) nil))
> (LET ((STR (OPEN)) (#:G421 T))
> (UNWIND-PROTECT
> (MULTIPLE-VALUE-PROG1 (PROGN NIL) (SETQ #:G421 NIL))
> (WHEN (STREAMP STR) (CLOSE STR :ABORT #:G421))))
> T
> PG-USER(16):
>
> And CMUCL 18d gives me the following:
>
> * (macroexpand-1 '(with-open-file (str) nil))
> (LET ((STR (OPEN)) (#:G851 T))
> (UNWIND-PROTECT (MULTIPLE-VALUE-PROG1 (PROGN NIL) (SETQ #:G851 NIL))
> (WHEN STR (CLOSE STR :ABORT #:G851))))
> T
> *
>
> When I mentioned this, the response was that the GC knows how to close
> streams and so with-open-file does not need to worry about something
> happening between the let form and the unwind-protect form.
The responder that said this was incorrect, unless there was also
something also said about finalizing the object.
> Now, I'm
> a little confused by how some of the things I've been reading in this
> thread relate to this analysis of the with-open-file expansion. In
> particular, I'm lead to believe by some of Duanne Rettig's comments,
> for example, that the ACL GC does not close open file streams
That is correct.
> I don't
> remember his having rewriting this explicitly but I've taken what I
> understand to be his disapproval of the idea of the GC closing file
> streams to imply that this is the case for ACL--nor would he find that
> to be a worthwhile extension.
I can understand where you got this idea, but if you read over what I
wrote, you'll find that that is not what I said. Specifically, I don't
disapprove of adding a finalization which will allow a GC to close a
file if it had not been closed before (i.e. as a safety net); my
warning is only against any programmer _counting_ on a gc closing a
file as its sole means of retrieving the file descriptor. If 98.45%
of the time, files are closed explicitly, and the last 1.55% of the time
are handled by a global-gc, then all of the cases are covered and there
is probably little chance of long-term leakage to the point where the
system resource limit for file descriptors is reached. But as a user,
you would have to add these finalizations yourself to the stream
objects; Allegro CL won't do it automatically for you.
> If the ACL GC does not handle closing
> file streams, how does it handle the case in which some kind of
> (presumably asynchronous) signal occurs between the binding of ',bar'
> and the protected form?
As demonstrated above, the real potential problem is _before_ the
assignment to ',bar' - after the assignment there are no interrupt-check
points in the code which will cause the leak you are concerned about.
However, before the assignment, the unwind-protect is simply not going
to help you, because the cleanup form doesn't have the object yet.
> My original with-foo code was meant to run in
> CMU CL and I'm assuming that the person with whom I was corresponding
> knows something about the way in which CMU CL handles file streams;
> i.e. it has some implementation dependent extensions to guarantee that
> file streams will eventually be closed if something goes wrong in the
> same spirit as Ray Blaak's comment about "getting the best of both
> worlds." Is my understanding of the situation regarding CMU CL
> correct? What about the case of ACL?
I don't know how CMUCL handles interrupts, so I can't comment on that,
nor do I now if CMUCL automatically places finalizations on open
streams. It certainly can't hurt to put a finalization on the ',bar'
resource, as long as you can get the assignment done before the interrupt
is processed. But in ACL, if you get that far it won't matter anyway,
since the resource is then as good as protected, anyway.
> Should I assume that looking at
> the expansions of such things as with-open-file is a bad way to learn
> to model my own macros because their correct behavior might rely on
> extensions with which I am unfamiliar, e.g. the situation with CMU CL
> arranging so that the GC closes file streams?
If CMUCL arranges for file streams to be closed by gc when the stream
is unreferenced, then yes, it would be bad to use with-open-file as a
model. However, you can use other with-* macros as models, and
with-open-file is certainly ok to use as a model in Allegro CL. What
you do have to understand, if you're concerned about it, is the
interrupt model, and how asynchronous interrupts are synchronized
in each of your vendors' Lisps.
Ray Blaak
> My original with-foo code was meant to run in CMU CL and I'm assuming that
> the person with whom I was corresponding knows something about the way in
> which CMU CL handles file streams; i.e. it has some implementation dependent
> extensions to guarantee that file streams will eventually be closed if
> something goes wrong in the same spirit as Ray Blaak's comment about
> "getting the best of both worlds." Is my understanding of the situation
> regarding CMU CL correct?
If you are not sure then close/dispose/clean up what you need to explicitly in
your with-foo macro. If you need to take signals into account, then do so.
Your resulting code will be more portable, correct, safer, and serve as a
better learning example for the next newbie.
An implementation's built-in macros are quite naturally allowed to make
assumptions about the implementation's abilities, so no, I would not rely on
them as a guide for anything except how to do things with your particular
implementation.
--
Cheers, The Rhythm is around me,
The Rhythm has control.
Ray Blaak The Rhythm is inside me,
bl...@telus.net The Rhythm has my soul.
Tom Lord
>> both.
Please note at the outset that I agree with Ray Blaak's idea
to use both mechanisms, the automatic method as a safety net
for the normal manual method.
So, about this "safety net" idea:
One of the things I've noticed while using GC for memory reclamation
is that it's algorithmically liberating. It's not just that I don't
have to put `free' statements in my programs: it's that I can write
programs in which there is no point in the program text where `free'
statements would go. The control flow of the algorithm doesn't have
to converge in any useful way with the points at which various data
structures can be freed -- GC is kind of a separate thread that is
watching for those points dynamically by doing graph tracing. Surely
I'm not alone in noticing that.
From there, it's just a short stumble to thinking about data
structures that include open files and algorithms that don't converge
in any useful way with the points at which those files can be closed.
In just the same way that I need GC to free memory for some algorithms
-- I need GC to close files.
In other words: it isn't just a "safety net" to compensate for sloppy
programming (like `with-foo' implementations that aren't quite
right). In fact, for some algorithms, it's essential.
All the people advising "Hey, just use `with-foo'" -- well, as a rule
of thumb, that's fine. But as an absolute rule it means "Hey, there
are some algorithms you can't code in lisp" and that's just icky.
Geeze, isn't this completely obvious? Y'all are writing satire,
right?
As far as I know, I'm responsible (sorry) for starting the whole
"closing files with GC" thread. As a point of interest, my attention
was first drawn to the issue by contemplating the shortcomings of
conservative GC, including the RISKS it creates in applications which
rely upon it.
Even if you only believe that GC-based file closing is a safety-net --
there are still RISKS, there.
-t
Ray Blaak
> One of the things I've noticed while using GC for memory reclamation
> is that it's algorithmically liberating. It's not just that I don't
> have to put `free' statements in my programs: it's that I can write
> programs in which there is no point in the program text where `free'
> statements would go. The control flow of the algorithm doesn't have
> to converge in any useful way with the points at which various data
> structures can be freed -- GC is kind of a separate thread that is
> watching for those points dynamically by doing graph tracing. Surely
> I'm not alone in noticing that.
>
> From there, it's just a short stumble to thinking about data
> structures that include open files and algorithms that don't converge
> in any useful way with the points at which those files can be closed.
> In just the same way that I need GC to free memory for some algorithms
> -- I need GC to close files.
What you want is extremely desirable. Making it possible would mean making
decisions for various tradeoffs (performance, safely, predictability).
> All the people advising "Hey, just use `with-foo'" -- well, as a rule
> of thumb, that's fine. But as an absolute rule it means "Hey, there
> are some algorithms you can't code in lisp" and that's just icky.
Perhaps one could write (in Lisp of course) simultaneous GC mechanisms, or
equivalently, GC such that different kinds of resources are collected in an
"optimal" manner.
For example, one could do memory reclamation the usual (amortized?) way, file
and streams "immediately" when they go out of (all) scopes, etc.
One would still have the performance predictability issue, though, which could
cause problems for real time apps.
Erik Naggum
| In other words: it isn't just a "safety net" to compensate for
| sloppy programming (like `with-foo' implementations that aren't
| quite right). In fact, for some algorithms, it's essential.
|
| All the people advising "Hey, just use `with-foo'" -- well, as a
| rule of thumb, that's fine. But as an absolute rule it means "Hey,
| there are some algorithms you can't code in lisp" and that's just
| icky.
|
| Geeze, isn't this completely obvious? Y'all are writing satire,
| right?
What I find rather obvious is that any stream you read from may be
closed when you hit end-of-file, regardless of where this happens.
Users of a stream can test for open-ness with `open-stream-p´,
which is quite remarkably different from testing for end-of-file.
--
Erik Naggum, Oslo, Norway
Act from reason, and failure makes you rethink and study harder.
Act from faith, and failure makes you blame someone and push harder.
Eric Marsden
dk> something about the way in which CMU CL handles file streams;
dk> i.e. it has some implementation dependent extensions to guarantee that
dk> file streams will eventually be closed if something goes wrong in the
dk> same spirit as Ray Blaak's comment about "getting the best of both
dk> worlds."
CMUCL provides a non-standard extension called finalization, that is
used to close the underlying file descriptor when a
file-descriptor-stream is reclaimed by the garbage collector. This
extension is documented in the CMUCL User's Manual:
<URL:http://cvs2.cons.org/ftp-area/cmucl/doc/cmu-user/extensions.html#toc31>
It's easy to see it in action:
,----
| USER> (open "/tmp/unused" :direction :output)
| #<Stream for file "/tmp/unused">
| USER> ext::*objects-pending-finalization*
| ((#<Weak Pointer: #<Stream for file "/tmp/unused">> . #<Closure Over Function "DEFUN MAKE-FD-STREAM" {481FAB09}>)
| (#<Weak Pointer: #<Stream for the Terminal>> . #<Closure Over Function "DEFUN MAKE-FD-STREAM" {48078239}>))
| USER> (gc :full t)
| nil
| USER> (gc :full t)
| nil
| USER> (gc :full t)
| ** Closed file descriptor 5
| nil
`----
The stream wasn't reclaimed by the first two invocations of the
garbage collector, because it was still accessible via the listener's
*/**/*** history variables.
--
Eric Marsden <URL:http://www.laas.fr/~emarsden/>
Duane Rettig
> >>>>> "dk" == Damien Kick <dki...@email.mot.com> writes:
>
> dk> something about the way in which CMU CL handles file streams;
> dk> i.e. it has some implementation dependent extensions to guarantee that
> dk> file streams will eventually be closed if something goes wrong in the
> dk> same spirit as Ray Blaak's comment about "getting the best of both
> dk> worlds."
>
> CMUCL provides a non-standard extension called finalization, that is
> used to close the underlying file descriptor when a
> file-descriptor-stream is reclaimed by the garbage collector. This
> extension is documented in the CMUCL User's Manual:
>
> <URL:http://cvs2.cons.org/ftp-area/cmucl/doc/cmu-user/extensions.html#toc31>
Allegro CL also provides finalizations:
http://www.franz.com/support/documentation/6.2/doc/gc.htm#finalizations-2
but we don't automatically attach them to streams, due to their GC
overhead. Finalizations are never automatically attached to any
Lisp object in Allegro CL. A user must decide that a finalization
is desired and explicitly request it.
> It's easy to see it in action:
>
> ,----
> | USER> (open "/tmp/unused" :direction :output)
> | #<Stream for file "/tmp/unused">
> | USER> ext::*objects-pending-finalization*
> | ((#<Weak Pointer: #<Stream for file "/tmp/unused">> . #<Closure Over Function "DEFUN MAKE-FD-STREAM" {481FAB09}>)
> | (#<Weak Pointer: #<Stream for the Terminal>> . #<Closure Over Function "DEFUN MAKE-FD-STREAM" {48078239}>))
> | USER> (gc :full t)
> | nil
> | USER> (gc :full t)
> | nil
> | USER> (gc :full t)
> | ** Closed file descriptor 5
> | nil
> `----
>
> The stream wasn't reclaimed by the first two invocations of the
> garbage collector, because it was still accessible via the listener's
> */**/*** history variables.
Note that a full GC is required for a guarantee, and only after all
references to the object are known gone (although hopefully in this
particular case the stream just allocated was not old enough to really
require a full GC). How often does CMUCL do a full GC? Is it automatic?
Duane Rettig
> overhead. Finalizations are never automatically attached to any
> Lisp object in Allegro CL. A user must decide that a finalization
> is desired and explicitly request it.
Oops. Never say "never"; I should know that. To qualify this brash
statement, I must admit that
- An extension to make-array is the :allocation keyword. If that
value is specified as :static-reclaimable, then a finalization is
indeed scheduled to deallocate the C portion of the array when the
Lisp portion is no longer referenced. (I almost don't consider this
an exception to the rule, since a user has to specify the option
in order to get the finalization).
- Some Franz add-ons like Aserve and our OLE product do use
finalizations (aserve uses finalizations as a safety net for socket
file descriptors in precisely the fashion that has been described
earlier in this thread).
Kent M Pitman
> Even if you only believe that GC-based file closing is a safety-net --
> there are still RISKS, there.
Risks are greatly magnified if you pretend you have more promised to you
than you do. A person who is told he has an unreliable tool can still do
good things with it if he treats the tool with the care it requires. The
internte itself is built up in layers from "unreliable" to "reliable"
substrates. But it is done by plainly understanding what is reliable and
what is not, and not by treating the unreliable substrate as if more were
promised than was.
And yes, even with the so-called reliable substrate, there are risks.
But that's always going to be true, so is by itself a slightly boring
claim. If someone offers you a risk-free programming substrate, especially
one that involves multi-tasking and (effective or actual) non-determinism,
eye it skeptically.
Joe Marshall
Although the two versions are equivalent under AllegroCL, it is
because of an implementation detail that is specific to AllegroCL: to
wit, that interrupts are only checked at safe spots, and are not
checked on function return or unwind-protect starts.
However, this isn't necessarily the case in other CommonLisp
implementations.
Sam Steingold
> * On the subject of "Re: closing files by gc"
> * Sent on Thu, 09 Jan 2003 17:00:01 GMT
> * Honorable Duane Rettig <du...@franz.com> writes:
>
> Eric Marsden <emar...@laas.fr> writes:
>
> > >>>>> "dk" == Damien Kick <dki...@email.mot.com> writes:
> >
> > dk> something about the way in which CMU CL handles file streams;
> > dk> i.e. it has some implementation dependent extensions to guarantee that
> > dk> file streams will eventually be closed if something goes wrong in the
> > dk> same spirit as Ray Blaak's comment about "getting the best of both
> > dk> worlds."
> >
> > CMUCL provides a non-standard extension called finalization
>
> Allegro CL also provides finalizations:
> http://www.franz.com/support/documentation/6.2/doc/gc.htm#finalizations-2
> but we don't automatically attach them to streams
CLISP also provides finalizations:
<http://clisp.cons.org/impnotes/final.html>
CLISP closes streams when they are GCed using special GC
trickery (not a finalizer).
CLISP attaches a finalizer to each SOCKET-SERVER.
--
Sam Steingold (http://www.podval.org/~sds) running RedHat8 GNU/Linux
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.palestine-central.com/links.html>
Daddy, why doesn't this magnet pick up this floppy disk?
Damien Kick
> Duane Rettig <du...@franz.com> writes:
>
> > Damien Kick <dki...@email.mot.com> writes:
> >
> > > [...] One of the things that was discussed was a macro that I
> > > had originally written as something like the following. [...]
> >
> > > It was suggested to me that I rewrite with-foo to be something
> > > like the following (mistakes are my own). [...]
>
> Although the two versions are equivalent under AllegroCL, it is
> because of an implementation detail that is specific to AllegroCL:
> to wit, that interrupts are only checked at safe spots, and are not
> checked on function return or unwind-protect starts.
Fair enough.
> However, this isn't necessarily the case in other CommonLisp
> implementations.
What does CL have to say about the subject? Is there any asynchronous
signal (and I use the term signal here in a loose sense) mechanism
within CL iteself; i.e. does not require that one use extensions to
generate these signals. If I understand Duane's point about it being
possible to have interrupts cause problems during 'acquire-foo', then
the only truly iron-clad solution would require that one start using
extensions. For example, if I was using CMU CL, I would have to start
using 'without-interrupts'. Now, if I only need to worry about such
an interrupt occuring only if I start using extensions in the first
place, then this seems to me to be quite expected. In other words, if
I am using CMU CL and, for example, I start forking processes that are
going to be using 'unix-kill' to send asynchronous signals to their
parents, then I would expect to have to use 'without-interrupts' to
protect critical regions. However, if I'm writing a program that only
uses symbols in the CL package and I still need to use extensions to
write an iron clad 'with-foo', this seems to be to be quite unexpected
and undesirable. Does my question/concern make sense?
Damien Kick
> Joe Marshall <j...@ccs.neu.edu> writes:
>
> > Duane Rettig <du...@franz.com> writes:
> >
> > > Damien Kick <dki...@email.mot.com> writes:
> > >
> > > > [...] One of the things that was discussed was a macro that I
> > > > had originally written as something like the following. [...]
> > >
> > > > It was suggested to me that I rewrite with-foo to be something
> > > > like the following (mistakes are my own). [...]
> >
> > Although the two versions are equivalent under AllegroCL, it is
> > because of an implementation detail that is specific to AllegroCL:
> > to wit, that interrupts are only checked at safe spots, and are not
> > checked on function return or unwind-protect starts.
>
> Fair enough.
>
> > However, this isn't necessarily the case in other CommonLisp
> > implementations.
>
> What does CL have to say about the subject? [...] Does my
> question/concern make sense?
Sorry to respond to my own post but I thought that this would clarify
my question/concern. Consider the following code.
(defpackage "PLAYGROUND"
(:nicknames "PG"))
(in-package "PG")
(defvar *time-to-sleep* 0)
(defstruct foo
id-number)
(defun acquire-foo (id-number)
(let ((value (make-foo :id-number id-number)))
(format t "acquire foo ~A~%" (foo-id-number value))
(sleep *time-to-sleep*)
value))
(defun release-foo (foo)
(progn
(format t "release foo ~A~%" (foo-id-number foo))))
(defmacro with-foo ((bar baz) &rest body)
`(let ((,bar (acquire-foo ,baz)))
(unwind-protect
(progn ,@body)
(when (foo-p ,bar)
(release-foo ,bar)))))
(defun test-with-foo ()
(with-foo (x 69)
(format t "use foo ~A~%" (foo-id-number x))))
(defmacro with-foo-1 ((bar baz) &rest body)
`(let (,bar)
(unwind-protect
(progn
(setf ,bar (acquire-foo ,baz))
,@body)
(when (foo-p ,bar)
(release-foo ,bar)))))
(defun test-with-foo-1 ()
(with-foo (x 71)
(format t "use foo ~A~%" (foo-id-number x))))
Now, as I was using CMU CL, I tried it there first, setting
*time-to-sleep* to something large enough so that I could interrupt
execution.
* (setf pg::*time-to-sleep* 100)
100
* (pg::test-with-foo)
acquire foo 69
Interrupted at #xFF215F54.
Restarts:
0: [CONTINUE] Return from BREAK.
1: [ABORT ] Return to Top-Level.
Debug (type H for help)
(UNIX::SIGINT-HANDLER #<unavailable-arg>
#<unavailable-arg>
#.(SYSTEM:INT-SAP #xFFBEF1A0))
0] 1
* (pg::test-with-foo-1)
acquire foo 71
Interrupted at #xFF215F54.
Restarts:
0: [CONTINUE] Return from BREAK.
1: [ABORT ] Return to Top-Level.
Debug (type H for help)
(UNIX::SIGINT-HANDLER #<unavailable-arg>
#<unavailable-arg>
#.(SYSTEM:INT-SAP #xFFBEF1A0))
0] 1
*
As Duanne mentioned previously in the thread, these both "leaked" a
foo. I resorted to using 'without-interrupts'.
(defmacro with-foo-cmucl ((bar baz) &rest body)
`(sys:without-interrupts
(let ((,bar (acquire-foo ,baz)))
(unwind-protect
(sys:with-interrupts
(progn ,@body))
(when (foo-p ,bar)
(release-foo ,bar))))))
(defun test-with-foo-cmucl ()
(with-foo-cmucl (x 72)
(format t "use foo ~A~%" (foo-id-number x))))
And, apparently, I finally get my iron-clad 'with-foo'.
* (pg::test-with-foo-cmucl)
acquire foo 72
Interrupted at #x40406860.
Restarts:
0: [CONTINUE] Return from BREAK.
1: [ABORT ] Return to Top-Level.
Debug (type H for help)
(UNIX::SIGINT-HANDLER #<unavailable-arg>
#<unavailable-arg>
#.(SYSTEM:INT-SAP #xFFBEF2C8))
0] 1
release foo 72
*
Oddly, enough, I'm seeing the following behavior with ACL Trial
Edition 6.2 (on Windows), interrupting the tests by selecting "Run ->
Interrupt" and then trying all the various options.
PG(27): (in-package :pg-user)
#<The PLAYGROUND-USER package>
PG-USER(28): pg::*time-to-sleep*
0
PG-USER(29): (setf pg::*time-to-sleep* 100)
100
PG-USER(30): (test-with-foo)
Error: attempt to call `TEST-WITH-FOO' which is an undefined function.
[condition type: UNDEFINED-FUNCTION]
PG-USER(31): (pg::test-with-foo)
acquire foo 69
PG-USER(32): (pg::test-with-foo-1)
acquire foo 71
PG-USER(33):
And so I tried to write an iron-clad version for ACL.
(defmacro with-foo-acl ((bar baz) &rest body)
`(mp:without-scheduling
(let ((,bar (acquire-foo ,baz)))
(unwind-protect
(mp:process-allow-schedule
(progn ,@body))
(when (foo-p ,bar)
(release-foo ,bar))))))
(defun test-with-foo-acl ()
(with-foo-acl (x 73)
(format t "use foo ~A~%" (foo-id-number x))))
This did not seem to do the trick for me, either.
PG-USER(33): (pg::test-with-foo-acl)
acquire foo 73
PG-USER(34):
And so, how is one supposed to be able to write an iron-clad
'with-foo' without resorting to extensions? In my test runs, I was
using what I consider to be "standard" Common Lisp, i.e. just the REPL
and the COMMON-LISP package, and I could not seem to write a
'with-foo' that worked with only symbols from the COMMON-LISP package.
I realize that it is probably because I am doing something incorrect.
What should I be doing differently? Is my expectation to be able to
write an iron-clad 'with-foo' only using symbols from the COMMON-LISP
package as long as my test cases and/or code does not stray outside of
the COMMON-LISP package somehow misguided? For example, the behavior
of the REPL is not really specified in the HyperSpec (is it?) and so
the way that COMMON-LISP code interacts with, for example,
interrupting the top level is probably undefined, yes? However, it
wasn't like I was using multiple threads/processes to deliver
asynchronous signals. I was restricting myself to "standard" Common
Lisp (wasn't I?) and did not seem to be able to write an iron-clad
'with-foo' using just "standard" Common Lisp. This surprises me.
(Well, that I couldn't do it doesn't surprise me but rather my
for-the-moment-until-somebody-shows-me-my-errors conclusion that it
can't be done surprises me.)
Joe Marshall
> And so, how is one supposed to be able to write an iron-clad
> 'with-foo' without resorting to extensions?
Unfortunately, you cannot.
In my opinion, however, Common Lisp extensions that add multitasking
*ought* to ensure that UNWIND-PROTECT remains `iron-clad'. Section
1.6 of the hyperspec states:
``An implementation can have extensions, provided they do not alter
the behavior of conforming code and provided they are not explicitly
prohibited by this standard.''
And it seems to me that the behavior of conforming code is altered if
it is possible to `break' UNWIND-PROTECT.
There are ways to fix UNWIND-PROTECT so that it *is* `iron-clad'. The
easiest trick is to disable interrupts:
(defmacro iron-clad-unwind-protect (protected-form &body cleanup-forms)
(let ((interrupt-enables (gensym "INTERRUPT-ENABLES-")))
`(MULTIPLE-VALUE-PROG1
(LET ((,interrupt-enables
#+allegro EXCL::*WITHOUT-INTERRUPTS*
#+lispworks SYS::*IN-NO-INTERRUPTS*)
#+allegro (EXCL::*WITHOUT-INTERRUPTS* t)
#+lispworks (SYS::*IN-NO-INTERRUPTS* 1))
(CL:UNWIND-PROTECT
(LET ((#+allegro EXCL::*WITHOUT-INTERRUPTS*
#+lispworks SYS::*IN-NO-INTERRUPTS* ,interrupt-enables))
#+lispworks
(UNLESS SYS::*IN-NO-INTERRUPTS*
(SYSTEM::WITHOUT-INTERRUPT-CHECK-FOR-INTERRUPTS))
,protected-form)
;; interrupts are off during cleanup
,@cleanup-forms))
#+lispworks
(UNLESS SYS::*IN-NO-INTERRUPTS*
(SYSTEM::WITHOUT-INTERRUPT-CHECK-FOR-INTERRUPTS)))))
But this relies on two non-portable details of implementation: the
fact that special variable binding and unbinding is atomic with
respect to interrupts, and that interrupt enables are controlled via
the value of a special variable. (Fortunately, both of these details
are common implementation techniques.)
I have argued this issue extensively. Google the news on
`EXCL::*WITHOUT-INTERRUPTS*'
Duane Rettig
> Damien Kick <dki...@email.mot.com> writes:
>
> > And so, how is one supposed to be able to write an iron-clad
> > 'with-foo' without resorting to extensions?
>
> Unfortunately, you cannot.
Agreed.
> In my opinion, however, Common Lisp extensions that add multitasking
> *ought* to ensure that UNWIND-PROTECT remains `iron-clad'. Section
> 1.6 of the hyperspec states:
>
> ``An implementation can have extensions, provided they do not alter
> the behavior of conforming code and provided they are not explicitly
> prohibited by this standard.''
Also agreed. However, multitasking has little to do with it; the
breakage you are apparently concerned about can occur even in the
absence of multitasking.
> And it seems to me that the behavior of conforming code is altered if
> it is possible to `break' UNWIND-PROTECT.
Again, agreed, but I have a feeling our definitions of 'break' might
be different. My definition of breakage is based solely on what
the CL Spec requires.
> There are ways to fix UNWIND-PROTECT so that it *is* `iron-clad'. The
> easiest trick is to disable interrupts:
>
> (defmacro iron-clad-unwind-protect (protected-form &body cleanup-forms)
> (let ((interrupt-enables (gensym "INTERRUPT-ENABLES-")))
> `(MULTIPLE-VALUE-PROG1
> (LET ((,interrupt-enables
> #+allegro EXCL::*WITHOUT-INTERRUPTS*
> #+lispworks SYS::*IN-NO-INTERRUPTS*)
> #+allegro (EXCL::*WITHOUT-INTERRUPTS* t)
> #+lispworks (SYS::*IN-NO-INTERRUPTS* 1))
> (CL:UNWIND-PROTECT
> (LET ((#+allegro EXCL::*WITHOUT-INTERRUPTS*
> #+lispworks SYS::*IN-NO-INTERRUPTS* ,interrupt-enables))
> #+lispworks
> (UNLESS SYS::*IN-NO-INTERRUPTS*
> (SYSTEM::WITHOUT-INTERRUPT-CHECK-FOR-INTERRUPTS))
> ,protected-form)
> ;; interrupts are off during cleanup
> ,@cleanup-forms))
> #+lispworks
> (UNLESS SYS::*IN-NO-INTERRUPTS*
> (SYSTEM::WITHOUT-INTERRUPT-CHECK-FOR-INTERRUPTS)))))
This is a very nice macro, and it reduces the size of one of the holes
in typical UNWIND-PROTECT usage. However, I see two problems with
it:
1. It does not solve Damien's problem, and may introduce deadlocks.
2. It implements more than the spec requires.
For #1, let's go back to Damien's rewritten macro, with my own modification
to use your macro:
(defmacro with-foo (bar baz &rest body)
(let ((,bar nil))
(iron-clad-unwind-protect
(progn
(setf ,bar (acquire-foo ,baz))
;; If we get this far, the acquire-foo is within
;; the protected form. If some signal occurs
;; between the setf and the body, release-foo is
;; still evaluated.
,@body)
(when (and ,bar (foop ,bar))
(release-foo ,bar)))))
Is this ironclad? Certainly, there will be no interrupts that occur
which are going to cause a throw out of the protected-form, and that
may be a good thing (or not?). Suppose acquire-foo has two components:
allocate-foo and initialize-foo. If allocation of a foo is successful,
but initialization of that foo fails for any reason, then it may be
that ERROR will be called, which could result in a throw out of the
with-foo form. This leaks a foo resource, because it had never yet
been assigned to the ,bar variable, and so although the cleanup form
runs (as required), that form ends up doing nothing, because ,bar
is still nil. And this problem can occur regardless of whether there
are multiple threads (or indeed even if multiprocessing is not even
turned on), and regardless of the without-interrupts protection.
OK, so let's tighten things up a little further and work on acquire-foo
so that it will never call ERROR. Further, let's reintroduce multiple
threads, so the without-interrupts protection is useful to disallow
any scheduling that might cause a throw out of the with-foo form.
Unfortunately, if a foo is a limited resource, that same without-interrupts
which protects against non-local transfers is also shielding any other
processes from giving up their foo resources, and so may result in a
deadlock situation if foos are in short supply.
One way around this situation is to allow acquire-foo to return nil
instead of waiting for one to become available. This then means that
the ,@body forms would have to allow for the contingency that ,bar
is nil. If the ,@body must have a foo resource to operate, then I
believe (but have not worked it out to that depth) that the only
reasonable thing to do is to generate an error at that point and to
return from the with-foo form. My reasoning is this: we are still
in a without-interrupts situation, so we have little chance of getting
a new foo, unless there is some foo-arbiter we can call that will
force a foo to become available, although that might be hard within
a without-interrupts context. And if we temporarily set
*without-interrupts* to nil in order to allow the release of a foo,
then we are back to the situation where the foo allocation might be
unprotected. At any rate, we would be working against the grain of
your iron-clad-unwind-protect, which doesn't protect quite enough,
and at the same time protects too much.
For #2, let's go back to the spec:
http://www.franz.com/support/documentation/6.2/ansicl/dictentr/unwind-p.htm
"Unwind-protect evaluates protected-form and guarantees that cleanup-forms
are executed before unwind-protect exits, whether it terminates normally or
is aborted ..."
Unwind-protect isn't guaranteeing that the protected-form will execute
to completion (there is no way to guarantee that). Instead, what it is
guaranteeing is that _when_ the protected form is ready to return by _any_
means, the cleanup forms will be executed before that return occurs.
Now, perhaps you are reading something into the third paragraph:
"unwind-protect protects against all attempts to exit from protected-form,
including <list of non-local control transfers>". If you are interpreting
"protects against all attempts to exit" as meaning "forces the rest of
the protected-form to execute", then that would be an extremely hard
requirement to implement. Instead, I believe that "protects against exit"
means "doesn't complete the exit until the cleanups have executed".
Your "iron-clad" uwp form does indeed try to ensure that the protected
form is actually completed. However:
1. It isn't what the spec requires.
2. It doesn't force the protected-form to be completed in _all_ cases.
> But this relies on two non-portable details of implementation: the
> fact that special variable binding and unbinding is atomic with
> respect to interrupts, and that interrupt enables are controlled via
> the value of a special variable. (Fortunately, both of these details
> are common implementation techniques.)
Minor nit; unbinding is atomic, but binding is not (bindstack resources
may overflow during binding, which may require gc, etc). However, this
does not affect your argument, since it is unbinding which occurs in
critical sections wrt interrupts.
> I have argued this issue extensively. Google the news on
> `EXCL::*WITHOUT-INTERRUPTS*'
--
Joe Marshall
> Joe Marshall <j...@ccs.neu.edu> writes:
>
> > Damien Kick <dki...@email.mot.com> writes:
> >
> > > And so, how is one supposed to be able to write an iron-clad
> > > 'with-foo' without resorting to extensions?
> >
> > Unfortunately, you cannot.
>
> Agreed.
>
> > In my opinion, however, Common Lisp extensions that add multitasking
> > *ought* to ensure that UNWIND-PROTECT remains `iron-clad'. Section
> > 1.6 of the hyperspec states:
> >
> > ``An implementation can have extensions, provided they do not alter
> > the behavior of conforming code and provided they are not explicitly
> > prohibited by this standard.''
>
> Also agreed. However, multitasking has little to do with it; the
> breakage you are apparently concerned about can occur even in the
> absence of multitasking.
> ... I have a feeling our definitions of 'break' might be different.
I should be explicit on exactly the kind of breakage I am concerned
about. I want to write a piece of code that acts as follows: I wish
to provide an API that allows a `client' to run a segment of code in a
particular dynamic context (i.e., I will establish some sort of
context, run the client code, then disestablish that context.)
We'll assume the following:
1. I have access to the entire implementation of the
context-manipulating code. I have tested it to ensure it
is bug-free (within the contract of the API).
2. Additionally, let us assume that the context-manipulation code
does not involve synchronization between multiple threads or
non-local exits.
3. The `client' code is unavailable to me, may be buggy or
deliberately perverse.
First, let me address your examples in this simple case, then let me
extend them to the more complex case.
>> [macro omitted]
> This is a very nice macro, and it reduces the size of one of the holes
> in typical UNWIND-PROTECT usage.
Thank you!
> However, I see two problems with it:
>
> 1. It does not solve Damien's problem, and may introduce deadlocks.
Damien's problem is solved by a simple application of the macro:
(defmacro with-foo ((bar baz) &rest body)
(let ((baz-temp (gensym "BAZ-")))
;; evaluate BAZ *outside* the context
`(LET ((,baz-temp ,baz)
(,bar NIL))
(IRON-CLAD-UNWIND-PROTECT
(PROGN (EXCL:WITHOUT-INTERRUPTS
(SETQ ,bar (ACQUIRE-FOO ,baz-temp)))
,@body)
(WHEN (FOO-P ,bar)
(RELEASE-FOO ,bar))))))
(as an aside, with a little clever shuffling of bindings you can avoid
the nested call to WITHOUT-INTERRUPTS.)
> Suppose acquire-foo has two components: allocate-foo and
> initialize-foo. If allocation of a foo is successful, but
> initialization of that foo fails for any reason, then it may be that
> ERROR will be called, which could result in a throw out of the
> with-foo form. This leaks a foo resource, because it had never yet
> been assigned to the ,bar variable, and so although the cleanup form
> runs (as required), that form ends up doing nothing, because ,bar is
> still nil. And this problem can occur regardless of whether there
> are multiple threads (or indeed even if multiprocessing is not even
> turned on), and regardless of the without-interrupts protection.
Assumption 1 is that I have access to the context manipulation code,
thus I assume that I could write ACQUIRE-FOO in such a way as to avoid
this problem.
> OK, so let's tighten things up a little further and work on acquire-foo
> so that it will never call ERROR. Further, let's reintroduce multiple
> threads, so the without-interrupts protection is useful to disallow
> any scheduling that might cause a throw out of the with-foo form.
> Unfortunately, if a foo is a limited resource, that same without-interrupts
> which protects against non-local transfers is also shielding any other
> processes from giving up their foo resources, and so may result in a
> deadlock situation if foos are in short supply.
Again, assumption 1 is that I can write ACQUIRE-FOO in such a way as
to avoid this problem. Alternatively, the API could be defined such
that an error is thrown from within ACQUIRE-FOO. In either case, I
can ensure that the client code will not be run *unless* a foo is
properly acquired.
> One way around this situation is to allow acquire-foo to return nil
> instead of waiting for one to become available.
Yet another alternative. If there are not an unlimited number of FOOs
available, there should be a mechanism within the API to allow the
user to cope. Client code could do this:
(loop (with-foo (bar baz)
(when (foo-p bar)
....
(return)))
(sleep 1))
> At any rate, we would be working against the grain of your
> iron-clad-unwind-protect, which doesn't protect quite enough, and at
> the same time protects too much.
It is only supposed to protect against the case of the client code
attempting a non-local exit or against the user or other agent forcing
an asynchronous non-local exit. It is *part* of the resource
management solution, not *all* of it. It ensures that properly
written resource management code is robust in the face of improperly
written client code.
> 2. It implements more than the spec requires.
It has to since the spec does not address either asynchronous control
flow transfers or multiprocesses synchronization.
> For #2, let's go back to the spec:
> http://www.franz.com/support/documentation/6.2/ansicl/dictentr/unwind-p.htm
>
> "Unwind-protect evaluates protected-form and guarantees that cleanup-forms
> are executed before unwind-protect exits, whether it terminates normally or
> is aborted ..."
>
> Unwind-protect isn't guaranteeing that the protected-form will execute
> to completion (there is no way to guarantee that). Instead, what it is
> guaranteeing is that _when_ the protected form is ready to return by _any_
> means, the cleanup forms will be executed before that return occurs.
Unwind-protect *is* supposed to guarantee that if the protected form
has gained control, then control is passed to the cleanup forms before
continuing. In a Common Lisp that has no asynchronous control flow
changes (no multitasking or user interrupts), it is possible to
guarantee that the cleanup forms run to completion. (By assumption 1,
we have complete control over how the cleanup is implemented.)
However, if the Common Lisp in question supports user interrupts or
asynchronous control flow transfers, then the `standard'
UNWIND-PROTECT can be `broken' by an untimely interrupt. By wrapping
the UNWIND-PROTECT in a without-interrupts context, we can restore the
behavior to what it would have been in a non-enhanced lisp.
> Now, perhaps you are reading something into the third paragraph:
> "unwind-protect protects against all attempts to exit from protected-form,
> including <list of non-local control transfers>". If you are interpreting
> "protects against all attempts to exit" as meaning "forces the rest of
> the protected-form to execute", then that would be an extremely hard
> requirement to implement.
No. The client code is free to do whatever it wants, including
running to completion. Additionally, the client code is free to be
interrupted by the user and we'll allow multitasking to run.
> Instead, I believe that "protects against exit" means "doesn't
> complete the exit until the cleanups have executed".
This is my interpretation as well.
It is true that turning off interrupts completely is rather draconian,
but it is a simple and quick solution. What you *really* want to
disable is asynchronous control transfer. This is doable, but *much*
hairier and slower. If you have a simple piece of well-debugged
critical code, then I'd turn off interrupts.
-----
Incidentally, I'm afraid I'm not exactly happy with my macro. Here's
a better version. I've implemented iron-clad-unwind-protect* which
allows for the addition of a setup form to run in the uninterruptable
context.
(defmacro sans-interrupts (&body body)
"SANS-INTERRUPTS disables interrupts during execution of BODY.
Within BODY, the macro (RESTORE-INTERRUPTS &body body) can be
called to restore the interrupt state to what it was before."
(let* ((interrupt-enables (gensym "INTERRUPT-ENABLES-"))
(interrupt-control #+allegro 'EXCL::*WITHOUT-INTERRUPTS*
#+lispworks 'SYS::*IN-NO-INTERRUPTS*)
(disabling-value #+allegro t #+lispworks 1)
(interrupt-poll `(PROGN #+lispworks (UNLESS SYS::*IN-NO-INTERRUPTS*
(SYSTEM::WITHOUT-INTERRUPT-CHECK-FOR-INTERRUPTS)))))
`(MULTIPLE-VALUE-PROG1
(LET ((,interrupt-enables ,interrupt-control)
(,interrupt-control ,disabling-value))
(MACROLET ((RESTORE-INTERRUPTS (&BODY RESTORE-INTERRUPTS-BODY)
`(LET ((,',interrupt-control ,',interrupt-enables))
,',interrupt-poll
,@RESTORE-INTERRUPTS-BODY)))
,@body))
,interrupt-poll)))
(defmacro iron-clad-unwind-protect (protected-form &body cleanup-forms)
`(SANS-INTERRUPTS
(UNWIND-PROTECT
(RESTORE-INTERRUPTS ,protected-form)
,@cleanup-forms)))
(defmacro iron-clad-unwind-protect* (setup-form protected-form &body cleanup-forms)
`(SANS-INTERRUPTS
(UNWIND-PROTECT
(PROGN ,setup-form
(RESTORE-INTERRUPTS ,protected-form))
,@cleanup-forms)))
(defmacro with-foo ((bar baz) &rest body)
(let ((baz-temp (gensym "BAZ-")))
;; evaluate BAZ *outside* the context
`(LET ((,baz-temp ,baz)
(,bar NIL))
(IRON-CLAD-UNWIND-PROTECT*
(SETQ ,bar (ACQUIRE-FOO ,baz-temp))
(PROGN ,@body)
(WHEN (FOO-P ,bar)
(RELEASE-FOO ,bar))))))
Duane Rettig
I think we have pretty much come to terms, and in fact you have
enhanced your macro in such a way that will help Damien's problem,
and which satisfies most of my concern with the old version.
There are just two issues left, which I will explain after I
ask a question:
Joe Marshall <j...@ccs.neu.edu> writes:
> Duane Rettig <du...@franz.com> writes:
>
> > Unwind-protect isn't guaranteeing that the protected-form will execute
> > to completion (there is no way to guarantee that). Instead, what it is
> > guaranteeing is that _when_ the protected form is ready to return by _any_
> > means, the cleanup forms will be executed before that return occurs.
>
> Unwind-protect *is* supposed to guarantee that if the protected form
> has gained control, then control is passed to the cleanup forms before
> continuing. In a Common Lisp that has no asynchronous control flow
> changes (no multitasking or user interrupts), it is possible to
> guarantee that the cleanup forms run to completion. (By assumption 1,
> we have complete control over how the cleanup is implemented.)
>
> However, if the Common Lisp in question supports user interrupts or
> asynchronous control flow transfers, then the `standard'
> UNWIND-PROTECT can be `broken' by an untimely interrupt. By wrapping
> the UNWIND-PROTECT in a without-interrupts context, we can restore the
> behavior to what it would have been in a non-enhanced lisp.
The language you use in describing the 'breakage' of UNWIND-PROTECT,
plus the dichototomy you use in those CLs which support ashynchronous
interrupts and those which don't (I know of CLs which do not support
multitasking, but are there any which don't support asynch interrupts?)
leads me to believe that you consider it nonconformant (or at least
"wrong") for a CL to allow interrupts to allow a protected form in an
UNWIND-PROTECT to terminate early. Is that the case?
Damien Kick
> I think we have pretty much come to terms, and in fact you have
> enhanced your macro in such a way that will help Damien's problem,
> and which satisfies most of my concern with the old version. [...]
Unfortunately, I just posted more questions before I had a chance to
read these last two posts. I'll have to study Joe's enhanced iron
clad macro. Thank you, Joe and Duane, for the excellent posts.
Damien Kick
> Joe Marshall <j...@ccs.neu.edu> writes:
>
> > Damien Kick <dki...@email.mot.com> writes:
> >
> > > And so, how is one supposed to be able to write an iron-clad
> > > 'with-foo' without resorting to extensions?
> >
> > Unfortunately, you cannot.
>
> Agreed.
>
> > In my opinion, however, Common Lisp extensions that add multitasking
> > *ought* to ensure that UNWIND-PROTECT remains `iron-clad'. Section
> > 1.6 of the hyperspec states:
> >
> > ``An implementation can have extensions, provided they do not
> > alter the behavior of conforming code and provided they are
> > not explicitly prohibited by this standard.''
>
> Also agreed. However, multitasking has little to do with it; the
> breakage you are apparently concerned about can occur even in the
> absence of multitasking.
Actually, isn't my having interrupted the evaluation of
TEST-WITH-FOO-ACL involving multitasking just as much as if I were
writing code that used MP:PROCESS-RUN-FUNCTION even if the top level
is part of "standard" Common Lisp? In other words, is the behavior of
something like UNWIND-PROTECT in the presence of having an interrupt
generated at the top level actually part of "standard" Common Lisp?
> > And it seems to me that the behavior of conforming code is altered
> > if it is possible to `break' UNWIND-PROTECT.
>
> Again, agreed, but I have a feeling our definitions of 'break' might
> be different. My definition of breakage is based solely on what
> the CL Spec requires.
Well, personally, I just want to know how to write some Lisp code that
will allow me to obtain a FOO, use it if I get it, and make sure that
I give it back if I got it regardless of whether or not my use of it
is successful. Preferably, I would like to do this in as portable a
way as possible. In other words, if UNWIND-PROTECT is not the answer
in and of itself, I would be just as happy to know how to do it using
UNWIND-PROTECT as part of a larger form. I did not seem to be able to
do this with my WITH-FOO-ACL example. I haven't tried Joe's
IRON-CLAD-UNWIND-PROTECT, yet, but it seems to be based on the same
idea, i.e. disabling interrupts.
> > There are ways to fix UNWIND-PROTECT so that it *is* `iron-clad'. The
> > easiest trick is to disable interrupts:
>
> > (defmacro iron-clad-unwind-protect [...])
But wansn't that, i.e. disabling interrupts, what I tried to do in the
WITH-FOO-ACL form, albeit in a different way than Joe's
IRON-CLAD-UNWIND-PROTECT? Why did it not seem to work correctly?
> This is a very nice macro, and it reduces the size of one of the holes
> in typical UNWIND-PROTECT usage. However, I see two problems with
> it:
>
> 1. It does not solve Damien's problem, and may introduce deadlocks.
>
> 2. It implements more than the spec requires.
Would someone please suggest a version that would solve my problem for
ACL? What I had in WITH-FOO-CMUCL did seem to work for me on CMU CL;
i.e. I saw the message from RELEASE-FOO at the top level even if I
interrupted evaluation. I could not seem to write a version that
works for ACL and Duanne seems to think that Joe's solution has its
own problems.
Isn't this example of an ERROR between an ALLOCATE-FOO and an
INITIALIZE-FOO a bit of a red herring, though? Can we not assume that
ACQUIRE-FOO will not leak? I believe that you hint at this in the
following paragraph.
> OK, so let's tighten things up a little further and work on
> acquire-foo so that it will never call ERROR [and return nil
> instead]. [...]
This is morally equivalent to the case in which if ACQUIRE-FOO does
call ERROR, it'll take care of itself to make sure that any resources
acquired will be released; e.g. yet another UNWIND-PROTECT to protect
the ALLOCATE-FOO, matching it with a cleanup form of DEALLOCATE-FOO
(one would assume that DEALLOCATE-FOO would be half of a two-phase
RELEASE-FOO in this scheme), and insist that the hypothetical
two-phase ACQUIRE-FOO either wholly suceeds or wholly fails. I
realize that my example ACQUIRE-FOO had this two-phased approach so
that I could but a SLEEP in the middle to give me time to interrupt
evaluation at the top level but that was only because I didn't know of
a way to write something like the following.
(setf blatz
;; sleep here before assignment but after ACQUIRE-FOO
;; has finished successfully
(acquire-foo))
I was hoping that sleeping inside ACQUIRE-FOO would simulate the case
in which ACQUIRE-FOO had completed successfully but the evaluation was
interrupted before one could capture its return value. Having
ACQUIRE-FOO fail in the middle of a two-phase evaluation, e.g. your
ALLOCATE-FOO and INITIALIZE-FOO example, seems to be a different case
to me. Am I mistaken?
> [...] Further, let's reintroduce multiple threads, so the
> without-interrupts protection is useful to disallow any scheduling
> that might cause a throw out of the with-foo form. Unfortunately,
> if a foo is a limited resource, that same without-interrupts which
> protects against non-local transfers is also shielding any other
> processes from giving up their foo resources, and so may result in a
> deadlock situation if foos are in short supply.
<nod> Right, iff ACQUIRE-FOO waits until it can obtain a FOO. If it
does not, we have the following situation.
> One way around this situation is to allow acquire-foo to return nil
> instead of waiting for one to become available. This then means that
> the ,@body forms would have to allow for the contingency that ,bar
> is nil. [...]
Or even if ACQUIRE-FOO would ERROR if it could not acquire a FOO,
right? As long as ACQUIRE-FOO does not block waiting to obtain a foo,
whether or not it ERRORs or returns NIL, and as long as an exit on an
error condition cleans up after any partially completed work before
leaving, then we've a non-leaking ACQUIRE-FOO that will not cause
deadlock.
> [...] If the ,@body must have a foo resource to operate, then I
> believe (but have not worked it out to that depth) that the only
> reasonable thing to do is to generate an error at that point and to
> return from the with-foo form.
But that shouldn't be a problem, yes? If ACQUIRE-FOO can not obtain a
FOO, then indicating failure in some form seems to be a respectable
response.
> [...] My reasoning is this: we are still in a without-interrupts
> situation, so we have little chance of getting a new foo, unless
> there is some foo-arbiter we can call that will force a foo to
> become available, although that might be hard within a
> without-interrupts context. And if we temporarily set
> *without-interrupts* to nil in order to allow the release of a foo,
> then we are back to the situation where the foo allocation might be
> unprotected. [...]
Perhaps what I want to be able to implement my IRON-CLAD-WITH-FOO is
not an IRON-CLAD-UNWIND-PROTECT but an IRON-CLAD-SETF with which one
could write a cleanly protected form with UNWIND-PROTECT? Assuming
that ACQUIRE-FOO does not leak, then the following should be safe,
yes?
(let (x)
(unwind-protect
(progn
(can-not-interrupt-setf (acquire-foo))
...)
(release-foo)))
And then to allow the application handle what to do when ACQUIRE-FOO
was unable to obtain a FOO. Perhaps something like the following.
(I'm attempting to hypothetically solve the problem of the foo
acquisition broker (what Duanne called the FOO-ARBITER). Please
excuse the very UNIX-ish, Pthreads feel of the example but that is the
threading model with which I am most familiar.)
(let (x)
(unwind-protect
(progn
(mutex-lock foo-lock) ;; and interrupts disabled
(while (no-available-foo-p) ;; protected by foo-lock
(cond-wait foo-cond-var foo-lock))
;; with the mutex-wait we've reenabled interrupts
;; and unlocked the foo-lock. when we exit
;; cond-wait, interrupts are again disabled
(setf x (acquire-foo))
(mutex-unlock foo-lock) ;; and interrupts enabled
(do-something-with x)
(do-something-else-with x))
(when (foo-p x)
(release x))))
> [...] At any rate, we would be working against the grain of
> your iron-clad-unwind-protect, which doesn't protect quite enough,
> and at the same time protects too much.
If I understand Joe's IRON-CLAD-UNWIND-PROTECT correctly, it is
attempting to disable interrupts during the entire evaluation of the
protected form and that this is the problem that Duanne sees with it.
Am I correct in this reading? In my previous post, I was attempting
to write a version of WITH-FOO-ACL that would be morally equivalent to
the pseudo-code version I wrote above (only without the FOO-ARBITER)
in which interrupts are only disabled long enough to make certain that
assignment can not be interrupted so that wheter or not ACQUIRE-FOO is
successful then we are gauranteed to capture the result in X. Duanne,
would this not provide an iron clad WITH-FOO-ACL that did not have the
same problems as a solution based on Joe's IRON-CLAD-UNWIND-PROTECT?
However, my previous example did not seem to do that for ACL. Or, to
put it anyother way, what is the ACL version of the following CMU CL
form?
(sys:without-interrupts
(f (g)
(sys:with-interrupts (h)) ;; reenable interrupts but only
;; for H
(i)))
Joe Marshall
> There are just two issues left, which I will explain after I
> ask a question:
>
> The language you use in describing the 'breakage' of UNWIND-PROTECT,
> plus the dichototomy you use in those CLs which support ashynchronous
> interrupts and those which don't (I know of CLs which do not support
> multitasking, but are there any which don't support asynch interrupts?)
> leads me to believe that you consider it nonconformant (or at least
> "wrong") for a CL to allow interrupts to allow a protected form in an
> UNWIND-PROTECT to terminate early. Is that the case?
No. The protected form should be interruptable, even asynchronously.
However, the *cleanup* forms should not be. As per the hyperspec,
the cleanup forms themselves are not protected, so an error or a
non-local exit in the cleanup will cause the cleanup to not complete.
This is ok because I assume that as the author of the cleanup form I
have control over how it behaves. What I don't have control over is
asynchronous interrupts.
Let me clarify that, though. The cleanup forms do not necessarily
need to run in a `without-interrupts' context. The ideal case would
be that multitasking continues as usual, but asynchronous interrupts
to the particular stack group (thread) that is cleaning up would be
deferred until the cleanup is complete. I'm making the assumption
that as the author of the cleanup code, I can ensure that it is
error-free and completes in a timely manner.
-----
Incidentally, Liquid Common Lisp (nee Lucid) and Zetalisp (Lisp
Machine) both deferred asynchronous interrupts during unwind-protect
cleanup. ACL and Lispworks do not.
I have written some code that causes ACL to behave in this manner (use
handler-bind to catch excl:interrupt-signal). To facilitate
debugging, I followed Franz's lead so that 5 control-c's caught this
way cause the debugger to be entered anyway. A better solution would
be to enter the debugger immediately, but push a new unwind-protect
frame that continues the cleanup (really hairy).
I have a partial solution for Lispworks, but it doesn't work in *all*
cases. (There is some interaction with the window system that I don't
understand, and without the source code, it may take some time.)
Joe Marshall
> > Joe Marshall <j...@ccs.neu.edu> writes:
>
> > > There are ways to fix UNWIND-PROTECT so that it *is* `iron-clad'. The
> > > easiest trick is to disable interrupts:
> >
> > > (defmacro iron-clad-unwind-protect [...])
>
> But wansn't that, i.e. disabling interrupts, what I tried to do in the
> WITH-FOO-ACL form, albeit in a different way than Joe's
> IRON-CLAD-UNWIND-PROTECT? Why did it not seem to work correctly?
You turned off multitasking, but control-c was still being checked.
> If I understand Joe's IRON-CLAD-UNWIND-PROTECT correctly, it is
> attempting to disable interrupts during the entire evaluation of the
> protected form and that this is the problem that Duanne sees with it.
> Am I correct in this reading?
No, I was disabling interrupts during the entire *cleanup* form, not
the protected form. The funky way in which I was doing it (binding to
turn them off, rebinding to re-enable) ensures that the unwinder turns
off interrupts when it tries to throw past the frame.
Of course, there may be issues with turning off interrupts during the
cleanup, and there are solutions to these as well. The solutions are
very hairy, however.
Duane Rettig
> Duane Rettig <du...@franz.com> writes:
>
> > There are just two issues left, which I will explain after I
> > ask a question:
> >
> > The language you use in describing the 'breakage' of UNWIND-PROTECT,
> > plus the dichototomy you use in those CLs which support ashynchronous
> > interrupts and those which don't (I know of CLs which do not support
> > multitasking, but are there any which don't support asynch interrupts?)
> > leads me to believe that you consider it nonconformant (or at least
> > "wrong") for a CL to allow interrupts to allow a protected form in an
> > UNWIND-PROTECT to terminate early. Is that the case?
>
> No. The protected form should be interruptable, even asynchronously.
OK, then there was only one issue left, which was the name of the
macro (I explained the issue to Damien in a previous posting).
Erik Naggum
| Let me clarify that, though. The cleanup forms do not necessarily
| need to run in a `without-interrupts' context. The ideal case would
| be that multitasking continues as usual, but asynchronous interrupts
| to the particular stack group (thread) that is cleaning up would be
| deferred until the cleanup is complete. I'm making the assumption
| that as the author of the cleanup code, I can ensure that it is
| error-free and completes in a timely manner.
Some of this discussion appears to exhibit some assumptions about
what kind of code should be permissible in the cleanup forms of an
`unwind-protect´ form, from the implication that not everything
would be equally acceptable. Perhaps they should be made explicit?
Joe Marshall
> * Joe Marshall
> | Let me clarify that, though. The cleanup forms do not necessarily
> | need to run in a `without-interrupts' context. The ideal case would
> | be that multitasking continues as usual, but asynchronous interrupts
> | to the particular stack group (thread) that is cleaning up would be
> | deferred until the cleanup is complete. I'm making the assumption
> | that as the author of the cleanup code, I can ensure that it is
> | error-free and completes in a timely manner.
>
> Some of this discussion appears to exhibit some assumptions about
> what kind of code should be permissible in the cleanup forms of an
> `unwind-protect´ form, from the implication that not everything
> would be equally acceptable. Perhaps they should be made explicit?
The primary assumption is that the cleanup code will in fact clean up
if it is allowed to run. The cleanup code would be allowed to throw,
return-from, signal errors, etc. But whatever it does, it should be
run free of interference from user interrupts or ill-behaving code in
other processes so that if it is properly written and debugged, there
is no way whatsoever to prevent it from cleaning up.
However, if the cleanup code is run in some sort of protected context,
for example, where interrupts are off, it ought to be aware that some
actions may deadlock. Here's an example:
(defun example ()
(let ((foo nil))
(unwind-protect
(progn (setq foo (acquire-foo ...))
(perform-computation foo))
(when foo
(release-foo foo)
(format t "~&Released a foo."))))) ;; ****
The line with the asterisks looks innocuous enough, but it could be a
problem if UNWIND-PROTECT disabled multitasking. If *standard-output*
were bound to a window object, it might have to acquire a lock, but
with multitasking disabled, it could deadlock.
But this is a general issue with multitasking and not specific to
UNWIND-PROTECT.
Damien Kick
> Damien Kick <dki...@email.mot.com> writes:
>
> > > Joe Marshall <j...@ccs.neu.edu> writes:
> >
> > > > There are ways to fix UNWIND-PROTECT so that it *is* `iron-clad'. The
> > > > easiest trick is to disable interrupts:
> > >
> > > > (defmacro iron-clad-unwind-protect [...])
> >
> > But wansn't that, i.e. disabling interrupts, what I tried to do in
> > the WITH-FOO-ACL form, albeit in a different way than Joe's
> > IRON-CLAD-UNWIND-PROTECT? Why did it not seem to work correctly?
>
> You turned off multitasking, but control-c was still being checked.
Joe, thanks again for helping me with my question as much as you have.
It would seem that your IRON-CLAD-UNWIND-PROTECT* is turning off
multi-tasking but still allowing for C-c to be checked, too. When I
use the latest version of WITH-FOO that uses
IRON-CLAD-UNWIND-PROTECT*, I still see an "acquire foo" message
without a corresponding "release foo" message when it is interrupted
during evaluation.
PG(26): (in-package :pg-user)
#<The PLAYGROUND-USER package>
PG-USER(27): (with-foo (x 69)
(format t "use foo ~A~%" (foo-id-number x)))
acquire foo 69
use foo 69
release foo 69
NIL
PG-USER(28): (setq *time-to-sleep* 1000)
1000
PG-USER(29): (with-foo (x 69)
(format t "use foo ~A~%" (foo-id-number x)))
acquire foo 69
Break: Interrupting the process of the currently selected IDE Listener
(Listener 1), which is doing an evaluation.
PG-USER(30):
I'm still assuming that my ACQUIRE-FOO code with the sleep in the
middle of it is an acceptable simulation of having ACQUIRE-FOO
complete successfully but the interrupt occurring before the SETQ can
capture the result. Is this a good assumption?
If I was experimenting with some code and needed to interrupt it, the
evalution would still leak a FOO. How can one write a WITH-FOO so
that the result would be as follows?
PG-USER(29): (with-foo (x 69)
(format t "use foo ~A~%" (foo-id-number x)))
acquire foo 69
Break: Interrupting the process of the currently selected IDE Listener
(Listener 1), which is doing an evaluation.
release foo 69
PG-USER(30):
Would it not require that IRON-CLAD-UNWIND-PROTECT* disable all kinds
of interrupts during evaluation of its setup-form? Is this possible
with ACL? I realize that this might lead to other problems if the
setup-form is ill-behaved; i.e. it might be analgous to allowing a
UNIX process to ignore SIGKILL and, therefore, become completely
unstoppable. Of course, if all interrupts that might be raised from
the top level are disabled, one should still be able to kill the
entire Lisp engine from outside of itself. Would one have to make use
of finalization, assuming that it would work in this case, to close
all of the possible holes? If one would have to resort to using
finalization, would this not take the WITH-FOO idiom down a peg or
two?
Joe Marshall
> It would seem that your IRON-CLAD-UNWIND-PROTECT* is turning off
> multi-tasking but still allowing for C-c to be checked, too. When I
> use the latest version of WITH-FOO that uses
> IRON-CLAD-UNWIND-PROTECT*, I still see an "acquire foo" message
> without a corresponding "release foo" message when it is interrupted
> during evaluation.
Hmmm. I was pretty sure that EXCL::*WITHOUT-INTERRUPTS* really turned
them off. It is possible that something is turning them back on,
though.
Could you try this:
(defun acquire-foo (id-number)
(let ((value (make-foo :id-number id-number)))
(format t "Before acquiring interrupts are ~:[en~;dis~]abled.~%" excl::*without-interrupts*)
(format t "acquire foo ~A~%" (foo-id-number value))
(sleep *time-to-sleep*)
(format t "After acquiring interrupts are ~:[en~;dis~]abled.~%" excl::*without-interrupts*)
value))
> I'm still assuming that my ACQUIRE-FOO code with the sleep in the
> middle of it is an acceptable simulation of having ACQUIRE-FOO
> complete successfully but the interrupt occurring before the SETQ can
> capture the result. Is this a good assumption?
I'm not sure. The call to sleep might be the problem.
> If I was experimenting with some code and needed to interrupt it, the
> evalution would still leak a FOO. How can one write a WITH-FOO so
> that the result would be as follows?
>
> PG-USER(29): (with-foo (x 69)
> (format t "use foo ~A~%" (foo-id-number x)))
> acquire foo 69
> Break: Interrupting the process of the currently selected IDE Listener
> (Listener 1), which is doing an evaluation.
> release foo 69
> PG-USER(30):
>
> Would it not require that IRON-CLAD-UNWIND-PROTECT* disable all kinds
> of interrupts during evaluation of its setup-form? Is this possible
> with ACL?
It is possible with ACL (I've done it), but there are some subtleties
and no doubt I've missed one. Unfortunately, I don't read usenet on
a machine that has ACL on it, so I can't debug it easily.
> I realize that this might lead to other problems if the
> setup-form is ill-behaved; i.e. it might be analgous to allowing a
> UNIX process to ignore SIGKILL and, therefore, become completely
> unstoppable. Of course, if all interrupts that might be raised from
> the top level are disabled, one should still be able to kill the
> entire Lisp engine from outside of itself.
Yes. You could write a Lisp program that is completely
non-responsive (the OS could always kill it, though).
> Would one have to make use of finalization, assuming that it would
> work in this case, to close all of the possible holes?
No. We can get it to work without finalization.
Damien Kick
> Damien Kick <dki...@email.mot.com> writes:
>
> > It would seem that your IRON-CLAD-UNWIND-PROTECT* is turning off
> > multi-tasking but still allowing for C-c to be checked, too. When I
> > use the latest version of WITH-FOO that uses
> > IRON-CLAD-UNWIND-PROTECT*, I still see an "acquire foo" message
> > without a corresponding "release foo" message when it is interrupted
> > during evaluation.
>
> Hmmm. I was pretty sure that EXCL::*WITHOUT-INTERRUPTS* really turned
> them off. It is possible that something is turning them back on,
> though.
>
> Could you try this:
>
> (defun acquire-foo (id-number)
> (let ((value (make-foo :id-number id-number)))
> (format t "Before acquiring interrupts are ~:[en~;dis~]abled.~%" excl::*without-interrupts*)
> (format t "acquire foo ~A~%" (foo-id-number value))
> (sleep *time-to-sleep*)
> (format t "After acquiring interrupts are ~:[en~;dis~]abled.~%" excl::*without-interrupts*)
> value))
PG-USER(22): (with-foo (x 69) (format t "use foo ~A~%" (foo-id-number x)))
Before acquiring foo 69, interrupts are disabled.
acquire foo 69
After acquiring foo 69, interrupts are disabled.
use foo 69
release foo 69
NIL
PG-USER(23): (setq pg::*time-to-sleep* 1000)
1000
PG-USER(24): (with-foo (x 69) (format t "use foo ~A~%" (foo-id-number x)))
Before acquiring foo 69, interrupts are disabled.
acquire foo 69
Break: Interrupting the process of the currently selected IDE Listener
(Listener 1), which is doing an evaluation.
PG-USER(25):
Please excuse the following list of newbie questions...
Why is directly setting values for EXCL::*WITHOUT-INTERRUPTS* better
than using MP:PROCESS-ALLOW-SCHEDULE and either
EXCL:WITHOUT-INTERRUPTS (aka MP::WITHOUT-INTERRUPTS) or
MP:WITHOUT-SCHEDULING ? My first version of WITH-FOO-ACL attempted to
use these functions from the MULTIPROCESSING and EXCL packages (I'd
have look back to remember exactly which ones I tried) to disable and
then re-enable interrupts and, when they did not do the trick, I
assumed that Joe was using EXCL::*WITHOUT-INTERRUPTS* because it would
be better. However, using EXCEL::*WITHOUT-INTERRUPTS* directly does
not seem to work any better than my original WITH-FOO-ACL. Do these
functions indeed do the same thing as accessing
EXCL::*WITHOUT-INTERRUPTS* directly? It certainly seems to be the
case for EXCL:WITHOUT-INTERRUPTS from the following expansion.
PG-USER(36): (macroexpand-1 '(mp::without-interrupts nil))
(EXCL::PRESERVING-INTERRUPTS (LET ((EXCL::*WITHOUT-INTERRUPTS* T)) NIL))
T
PG-USER(37):
Again, I can't make any sense of the results of using DISASSEMBLE on
PROCESS-ALLOW-SCHEDULE but the description in the documentation
<http://www.franz.com/support/documentation/6.2/doc/pages/operators/mp/process-allow-schedule.htm>
makes me think that it just might be morally equivalent to setting
EXCL::*WITHOUT-INTERRUPTS* to NIL, though I am a bit confused by the
wording used therein. Are the functions equivalent? If they are,
would they not be preferred, for stylistic reasons, because they are
external symbols and, therefore, meant to be used outside of their
packages? And why is WITHOUT-INTERRUPTS not external to the
MULTIPROCESSING package? Is it because one should be using
MP:WITHOUT-SCHEDULING and it will decide on whether or not it can use
EXCL:WITHOUT-INTERRUPTS based on the threading model being used?
> > I'm still assuming that my ACQUIRE-FOO code with the sleep in the
> > middle of it is an acceptable simulation of having ACQUIRE-FOO
> > complete successfully but the interrupt occurring before the SETQ
> > can capture the result. Is this a good assumption?
> I'm not sure. The call to sleep might be the problem.
<shrug> I can't make any sense of "(disassemble 'sleep)". I see a
SYS::C_INTERRUPT in the comments somewhere. The name certainly
implies something is happening with interrupts. Perhaps interrupts
are re-enabled so that they can be used in the implementation of
MP:PROCESS-SLEEP (as the ACL documentation
<http://www.franz.com/support/documentation/6.2/doc/pages/operators/mp/process-sleep.htm>
tells us the CL:SLEEP is changed to use MP:PROCESS-SLEEP under the
hood). Totally guessing.
Damien Kick
> Damien Kick <dki...@email.mot.com> writes:
>
> > It would seem that your IRON-CLAD-UNWIND-PROTECT* is turning off
> > multi-tasking but still allowing for C-c to be checked, too. When I
> > use the latest version of WITH-FOO that uses
> > IRON-CLAD-UNWIND-PROTECT*, I still see an "acquire foo" message
> > without a corresponding "release foo" message when it is interrupted
> > during evaluation.
>
> Hmmm. I was pretty sure that EXCL::*WITHOUT-INTERRUPTS* really turned
> them off. It is possible that something is turning them back on,
> though.
>
> Could you try this:
>
> (defun acquire-foo (id-number)
> (let ((value (make-foo :id-number id-number)))
> (format t "Before acquiring interrupts are ~:[en~;dis~]abled.~%" excl::*without-interrupts*)
> (format t "acquire foo ~A~%" (foo-id-number value))
> (sleep *time-to-sleep*)
> (format t "After acquiring interrupts are ~:[en~;dis~]abled.~%" excl::*without-interrupts*)
> value))
PG-USER(22): (with-foo (x 69) (format t "use foo ~A~%" (foo-id-number x)))
Before acquiring foo 69, interrupts are disabled.
acquire foo 69
After acquiring foo 69, interrupts are disabled.
use foo 69
release foo 69
NIL
PG-USER(23): (setq pg::*time-to-sleep* 1000)
1000
PG-USER(24): (with-foo (x 69) (format t "use foo ~A~%" (foo-id-number x)))
Before acquiring foo 69, interrupts are disabled.
acquire foo 69
Break: Interrupting the process of the currently selected IDE Listener
(Listener 1), which is doing an evaluation.
PG-USER(25):
Joe Marshall
I'm going to take a wild guess that you are using
the Allegro IDE for development. (Some guess. It
says IDE in your output.)
At CII, we were using the Emacs interface.
Since the IDE is a windowing system, it probably has
a `message pump', and I bet that the message pump runs
in a separate thread even if interrupts are disabled.
Probably the control-c is caught in the message pump
below the level of Lisp, so without-interrupts is not
having an effect.
If you are not wedded to the IDE, try it in a non-IDE lisp.
If you are wedded to the IDE, we'll have to poke around
a lot more to find out how the control-c is delivered
to the appropriate lisp thread.
"Damien Kick" <dki...@email.mot.com> wrote in message news:ov3cnue...@email.mot.com...
Joe Marshall
The problem is in SLEEP.
When SLEEP is called, the current process is put on
a queue to be awoken at a later time. Then control
is transferred to the process scheduler. This, of course,
transfers control to the next waiting process, or does
some `idle' processing. Part of that `idle' processing
is handling control-c.
So rather than using SLEEP to test if there is a race
condition, you should write your own `time wasting'
routine. Here's one:
(defparameter *fibarg* 35)
(defun waste-time (seconds)
(labels ((fib (n)
(if (< n 2)
n
(+ (fib (1- n)) (fib (- n 2))))))
(dotimes (i seconds)
(fib *fibarg*))))
(adjust *fibarg* to suit your taste, and make sure
you compile waste-time)
Using this instead of SLEEP should show that the
IRON-CLAD-UNWIND-PROTECT* is working.
"Damien Kick" <dki...@email.mot.com> wrote in message news:ov3cnue...@email.mot.com...
Damien Kick
> Nope, I was wrong. The problem is in SLEEP.
<nod>
> When SLEEP is called, [... what sleep does ...].
Is this documented in a place available to a user of the trial edition
so that when someone tells me to RTFM in the future I at least know
where to find TFM? I couldn't find that level of detail in the
description of PROCESS-SLEEP
<http://www.franz.com/support/documentation/6.2/doc/pages/operators/mp/process-sleep.htm>.
> So rather than using SLEEP to test if there is a race condition, you
> should write your own `time wasting' routine. Here's one:
<nod> Indeed, this was the case. Again, thank you for being so
persistent with your help.
Joe Marshall
> "Joe Marshall" <prunes...@attbi.com> writes:
>
> > Nope, I was wrong. The problem is in SLEEP.
>
> <nod>
>
> > When SLEEP is called, [... what sleep does ...].
>
> Is this documented in a place available to a user of the trial edition
> so that when someone tells me to RTFM in the future I at least know
> where to find TFM? I couldn't find that level of detail in the
> description of PROCESS-SLEEP
> <http://www.franz.com/support/documentation/6.2/doc/pages/operators/mp/process-sleep.htm>.
I'm not sure. I disassembled the code for sleep to find out what it
was *really* up to.
> <nod> Indeed, this was the case. Again, thank you for being so
> persistent with your help.
You're welcome.