Continuations in Common Lisp (with apologies)
Robert Marlow
used to implement common lisps more often and still haven't been able to
figure it out. I was wondering if people here could shed some light for me.
I've read the FAQ here and the most it says is that it's an implementation
burden. Well, I'm curious about what the burden actually is. I understand
continuations with undefined extent can conflict with some features like
unwind-protect in user code but I can't see this being such a problem in
implementation code where there'd be few places both would need to be
used simultaneously. What are the difficulties of using continuations, or
rather, what are the advantages of using a stack-based approach? Am I
wrong that continuations would provide simplifications to some features
such as threads?
mmcconn...@yahoo.com
Common
Lisp. Your question is "why (not)", not "how", but I don't remember
how much
of the "why" Graham addresses. Anyway, the book is at
http://www.paulgraham.com/paulgraham/onlisp.html
Zach Beane
> I've been looking around trying to find the reason continuations aren't
> used to implement common lisps more often and still haven't been able to
> figure it out. I was wondering if people here could shed some light for me.
Christian Quiennec's "Lisp in Small Pieces" discusses the user
benefits and the implementor costs of continuations. It also covers a
number of other features that are divergent in Scheme and Common Lisp.
(I wish I grasped the material well enough to provide a summary
here. Sorry.)
Zach
Pascal Costanza
> I've been looking around trying to find the reason continuations aren't
> used to implement common lisps more often and still haven't been able to
> figure it out. I was wondering if people here could shed some light for me.
This issues has been discussed in c.l.l in the past, google for it.
As a rough summary:
- Continuations can only be implemented with some overhead.
- It's not clear what they are useful for.
- If there were a strong need for continuations, CL vendors would have
added them.
> I've read the FAQ here and the most it says is that it's an implementation
> burden. Well, I'm curious about what the burden actually is. I understand
> continuations with undefined extent can conflict with some features like
> unwind-protect in user code but I can't see this being such a problem in
> implementation code where there'd be few places both would need to be
> used simultaneously.
I don't think that that's the actual implementation burden, that's more
a semantical problem. The issues with conflicts between unwind-protect
and call/cc are not localizable - you don't know who will invoke a
continuation after you have captured it. The problem is that the
implementation cannot automatically decide whether pending
unwind-protect forms should be invoked or not once certain continuations
are invoked. See
http://www.nhplace.com/kent/PFAQ/unwind-protect-vs-continuations.html
for a better explanation.
> What are the difficulties of using continuations, or
> rather, what are the advantages of using a stack-based approach? Am I
> wrong that continuations would provide simplifications to some features
> such as threads?
If I understood some of the Scheme folks correctly, they are advocating
not to use continuations for implementing threads anymore. If I
understood it correctly, continuations can only help you to implement
variants of cooperative multithreading.
Pascal
Robert Marlow
half-way since nothing was really sinking in.
Rahul Jain
> I've been looking around trying to find the reason continuations aren't
> used to implement common lisps more often and still haven't been able to
> figure it out.
Maybe that's because they are.
--
Rahul Jain
rj...@nyct.net
Professional Software Developer, Amateur Quantum Mechanicist
Robert Marlow
> Robert Marlow <bobst...@bobturf.org> writes:
>
>> I've been looking around trying to find the reason continuations aren't
>> used to implement common lisps more often and still haven't been able to
>> figure it out.
>
> Maybe that's because they are.
Which implementations are you talking about?
Rahul Jain
Robert Marlow
> CMUCL and SBCL, at the least.
In that case what's the implementation burden of making these
continuations available with indefinite extent to end users?
Marco Antoniotti
What's the need?
Cheers
--
Marco
Wade Humeniuk
It could be HUGE. Say you have a socket open, transferring an
image of a DVD. Should a continuation be available if the
Lisp process exists and is then restarted? The continuation
is possibly incorrect since how would it know if the process
crashed, and your socket is now in use, and, and, ...
Indefinite extent is a pretty tall order. Though I think you
have in mind a much smaller idea of what a continuation is. :)
Another grossly exagerated idea:
Would a continuation keep a snapshot of DB's known state,
irrespective of any changes to it after the continuation
was created? (i.e. would you have to clone a Oracle database
for the continuations sake?)
Wade
Julian Stecklina
> Robert Marlow wrote:
>> I've been looking around trying to find the reason continuations aren't
>> used to implement common lisps more often and still haven't been able to
>> figure it out. I was wondering if people here could shed some light for me.
>
> This issues has been discussed in c.l.l in the past, google for it.
>
> As a rough summary:
>
> - Continuations can only be implemented with some overhead.
> - It's not clear what they are useful for.
I find them useful to simulate blocking I/O, though internally select
or kqueues are used. Ok, this implements some kind of cooperative
multithreading, but it is most likely more efficient than having
multiple OS threads.
Regards,
--
____________________________
Julian Stecklina / _________________________/
________________/ /
\_________________/ LISP - truly beautiful
Robert Marlow
> It could be HUGE. Say you have a socket open, transferring an
> image of a DVD. Should a continuation be available if the
> Lisp process exists and is then restarted? The continuation
> is possibly incorrect since how would it know if the process
> crashed, and your socket is now in use, and, and, ...
>
> Indefinite extent is a pretty tall order. Though I think you
> have in mind a much smaller idea of what a continuation is. :)
>
> Another grossly exagerated idea:
>
> Would a continuation keep a snapshot of DB's known state,
> irrespective of any changes to it after the continuation
> was created? (i.e. would you have to clone a Oracle database
> for the continuations sake?)
>
> Wade
What I have in mind is that such continuations are made available to the
user and it's up to the user to decide when it's a good time to use them.
Essentially, given that continuations can be useful for solving some
classes of problems it seems to me that users should be left to make up
their own minds when and when not to use them. Schemers (No, I'm not
advocating scheme) seem to manage with and appreciate continuations with
indefinite extent so I see no reason not to include them in Common Lisp.
UNLESS of course they're such an implementation burden it can't be seen as
worth hassle. Hence I'm curious what sort of implementation burdens they
raise.
John Thingstad
wrote:
> Pascal Costanza <p...@p-cos.net> writes:
>
>> Robert Marlow wrote:
>>> I've been looking around trying to find the reason continuations aren't
>>> used to implement common lisps more often and still haven't been able
>>> to
>>> figure it out. I was wondering if people here could shed some light
>>> for me.
>>
>> This issues has been discussed in c.l.l in the past, google for it.
>>
>> As a rough summary:
>>
>> - Continuations can only be implemented with some overhead.
>> - It's not clear what they are useful for.
>
> I find them useful to simulate blocking I/O, though internally select
> or kqueues are used. Ok, this implements some kind of cooperative
> multithreading, but it is most likely more efficient than having
> multiple OS threads.
>
> Regards,
Seems to me that depends on the threading model.
Lightweight threads (sharing process and memory) shouldn't have
much of a overhead compared to continuations.
This brings my memory back to windows 3.1 and non-preemptive
multitasking. In windows 95 performance went up considerable
as did the reliability.
Can't say I have used continuations in Scheme/Lisp, but I have
used them in Simula which has a similar model some years ago.
It is'nt something I miss much though.
--
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Rahul Jain
Infinite, assuming UNWIND-PROTECT is a desired operator.
Julian Stecklina
>> I find them useful to simulate blocking I/O, though internally select
>> or kqueues are used. Ok, this implements some kind of cooperative
>> multithreading, but it is most likely more efficient than having
>> multiple OS threads.
>>
>> Regards,
>
> Seems to me that depends on the threading model.
> Lightweight threads (sharing process and memory) shouldn't have
> much of a overhead compared to continuations.
Continuations in this application have the advantage that control is
only transferred from one "thread" to another, when it is logically
neccessary. I do not want to use them as general purpose thread
alternative.
On a sidenote: I think kernel scheduled threads have a MUCH higher
overhead the continuations. User-level threads could come close
depending on the implementation.
> This brings my memory back to windows 3.1 and non-preemptive
> multitasking. In windows 95 performance went up considerable
> as did the reliability.
See above: Continuations are no alternative to preemptible threads.
> Can't say I have used continuations in Scheme/Lisp, but I have
> used them in Simula which has a similar model some years ago.
> It is'nt something I miss much though.
They solve some problems very efficiently. Though the need for them is
not as great if you have closures.
cesur...@verizon.net
> > In that case what's the implementation burden of making these
> > continuations available with indefinite extent to end users?
>
> Infinite, assuming UNWIND-PROTECT is a desired operator.
Why do you think that?
Will
Brian Downing
<cesur...@verizon.net> wrote:
> Rahul Jain wrote:
> > > In that case what's the implementation burden of making these
> > > continuations available with indefinite extent to end users?
> >
> > Infinite, assuming UNWIND-PROTECT is a desired operator.
>
Unconstrained CALL/CC makes it impossible to know when to run
UNWIND-PROTECT cleanup forms. For example (in a mythical CL with
call/cc):
(let ((continuation nil)
(n 0))
(let ((file nil))
(unwind-protect
(progn
(setf file (open "/tmp/foo"))
(call/cc (lambda (k) (setf continuation k)))
(format file "N is ~D~%" n)
(incf n))
(if file (close file))))
(when (< n 10)
(funcall continuation nil)))
There's no hint as to when the last call to the continuation will be
made here, so it's not known when the UNWIND-PROTECT should run.
This example may look silly, but if you replace the UNWIND-PROTECT with
WITH-OPEN-FILE and move the CALL/CC into the guts of the write (to save
off the state of the thread while blocking on the write) it should be
obvious how this would suck.
http://www.ccs.neu.edu/home/dorai/uwcallcc.may.15/uwcallcc.html has some
implementations of constrained CALL/CC in Scheme that allow
UNWIND-PROTECT to work correctly.
-bcd
--
*** Brian Downing <bdowning at lavos dot net>
cesur...@verizon.net
> http://www.ccs.neu.edu/home/dorai/uwcallcc.may.15/uwcallcc.html
> has some implementations of constrained CALL/CC in Scheme that
> allow UNWIND-PROTECT to work correctly.
Dorai's tutorial shows there is no conflict between
UNWIND-PROTECT and full continuations. Furthermore
this fact has been known for 20 years [1], and has
been demonstrated by many systems that support both
full continuations and some variation of UNWIND-PROTECT.
My question was social, not technical. I am interested
in the processes by which myths arise and propagate
among programmer communities. Toward that end, I asked
Rahul Jain why he believes the disinformation he posted.
Will
[1] Daniel P. Friedman and Christopher T. Haynes
"Constraining Control". Proceedings of the Twelfth
Annual Symposium on Principles of Programming Languages.
January 1985.
Ron Garret
cesur...@verizon.net wrote:
> Brian Downing wrote:
> > http://www.ccs.neu.edu/home/dorai/uwcallcc.may.15/uwcallcc.html
> > has some implementations of constrained CALL/CC in Scheme that
> > allow UNWIND-PROTECT to work correctly.
>
> Dorai's tutorial shows there is no conflict between
> UNWIND-PROTECT and full continuations.
Actually, it specifically says the exact opposite:
"Once these new operators are defined, the original call/cc would have
to be retired so as not to interfere with the functioning of the new
operators."
rg
Pascal Costanza
> Brian Downing wrote:
>
>>http://www.ccs.neu.edu/home/dorai/uwcallcc.may.15/uwcallcc.html
>>has some implementations of constrained CALL/CC in Scheme that
>>allow UNWIND-PROTECT to work correctly.
>
> Dorai's tutorial shows there is no conflict between
> UNWIND-PROTECT and full continuations.
No, it doesn't. Someone has to declare somewhere whether pending protect
forms should be invoked or not when a continuation is invoked. So either
you declare this along with call/cc or with unwind-protect. That's
what Kent Pitman's paper says. Dorai just shows how one can fix call/cc
to do that.
Pascal
Michael Livshin
this is starting to resemble a semantic dispute. yes, one can define
some form of "full" (i.e. more powerful than just escape)
continuations that plays nicely with unwind-protect. no, call/cc as
defined by Scheme is not it.
seems to me you are in violent agreement.
--
In an experiment to determine the precise amount of beer required to
enjoy this film, I passed out. -- dave o'brien, on Highlander II
William D Clinger
> Dorai's tutorial shows there is no conflict between
> UNWIND-PROTECT and full continuations.
Ron Garret replied:
> Actually, it specifically says the exact opposite:
>
> "Once these new operators are defined, the original call/cc
> would have to be retired so as not to interfere with the
> functioning of the new operators."
You sound as though you believe Scheme's original call/cc is
the one and only operator that provides full continuations. That
is not true. Dorai's redefinition of call/cc also provides full
continuations. In the preliminary version of Dorai's tutorial [1],
Dorai explicitly stated that Kent Pitman's goal was coexistence
of full continuations with UNWIND-PROTECT:
> ...he proposes that one of two modified versions of call/cc
> should have been provided in the Scheme standard, which
> would then permit a suitable unwind-protect, while still
> providing full continuations.
Dorai's tutorial showed how this goal can be achieved for
either of Kent's two proposed semantics for UNWIND-PROTECT.
Friedman and Haynes had already shown this twenty years
earlier, for four possible semantics of UNWIND-PROTECT [3].
Many implementations of Scheme have provided full
continuations while also providing DYNAMIC-WIND,
which is a generalization of UNWIND-PROTECT. The
semantics of UNWIND-PROTECT can be implemented
by one page of portable Scheme code, while retaining
full continuations [4].
Many people have done this over the years. I did it back
in 1985 or 1986, using the Friedman and Haynes algorithm
to implement DYNAMIC-WIND, and then using DYNAMIC-WIND
to implement a version of UNWIND-PROTECT.
When I became aware that Kent Pitman was proclaiming
that what I and others had done was impossible, I told him
about this history and cited the Friedman and Haynes paper
in private email. He ignored these facts until Dorai had
published his refutation. I suspect that Kent Pitman was
the primary source of the disinformation that has been
repeated in this thread by Rahul Jain, Ron Garret, and
Pascal Costanza. Would you guys care to confirm that?
Will
[1] Dorai Sitaram. "Unwind-protect in portable Scheme."
Fourth Workshop on Scheme and Functional Programming.
November 7, 2003. Online at
http://www.ccs.neu.edu/home/dorai/uwcallcc/uwcallcc.html
[2] Dorai Sitaram. "Unwind-protect in portable Scheme."
Preliminary version of [1], dated 15 May 2003. Online at
http://www.ccs.neu.edu/home/dorai/uwcallcc.may.15/uwcallcc.html
[3] Daniel P. Friedman and Christopher T. Haynes.
"Constraining Control." Twelfth ACM Symposium on Principles
of Programming Languages, 1985, pp. 245-254.
[4] Will Clinger. Untitled page of Scheme code that implements
one of Kent Pitman's semantics for UNWIND-PROTECT in terms
of Scheme's DYNAMIC-WIND. Online at
http://www.ccs.neu.edu/home/will/Temp/uwesc.sch
[5] Kent Pitman. "UNWIND-PROTECT versus continuations."
Unpublished disinformation, online at
http://www.nhplace.com/kent/PFAQ/unwind-protect-vs-continuations.html
Bulent Murtezaoglu
[...]
WDC> [5] Kent Pitman. "UNWIND-PROTECT versus continuations."
WDC> Unpublished disinformation, online at
WDC> http://www.nhplace.com/kent/PFAQ/unwind-protect-vs-continuations.html
Not that I wish to be a party to this beyond curiosity but: Is the
'disinformation' annotation for this link still your opinion in view
of what that link now says? (There's an update and links to two
publications (yours and DS's) alongside his original article). It
seems the update document states an opinion on the bigger picture rather
than misinform about what's possible.
cheers,
BM
William D Clinger
> yes, one can define some form of "full" (i.e. more powerful
> than just escape) continuations that plays nicely with
> unwind-protect.
Unfortunately, Michael then wrote:
> no, call/cc as defined by Scheme is not it.
That is technically true, but misleading. It is true only
because Scheme does not define any standard procedure named
call/cc. Scheme's call-with-current-continuation procedure
is standard, and plays nicely with unwind-protect.
Dorai's tutorial implementations must redefine Scheme's
standard call-with-current-continuation procedure in order
to work. Serious implementations of unwind-protect in Scheme
use dynamic-wind, and do not have to redefine the standard
call-with-current-continuation procedure. My implementation
of unwind-protect in terms of dynamic-wind shows how this is
done.
Yes, Kent Pitman claims that my implementation redefines
call-with-current-continuation. A look at my code will
show that Kent's claim is untrue. With my implementation,
Scheme's standard call-with-current-continuation remains
available to the programmer.
I suspect that Kent was misled by the assignment to call/cc
at the end of my code. Let me remind everybody that call/cc
is not one of Scheme's standard procedure; it is just a name
that Dorai chose for one of his variables. The assignment
to that variable in my code allowed my code to use Dorai's
test harness, and served no other purpose.
Will
Pascal Costanza
> When I became aware that Kent Pitman was proclaiming
> that what I and others had done was impossible, I told him
> about this history and cited the Friedman and Haynes paper
> in private email. He ignored these facts until Dorai had
> published his refutation. I suspect that Kent Pitman was
> the primary source of the disinformation that has been
> repeated in this thread by Rahul Jain, Ron Garret, and
> Pascal Costanza. Would you guys care to confirm that?
Yes, confirmed for my case. Don't blame Kent, though. It was my mistake.
The OP asked about continuations, and not specifically about call/cc,
and I have made the mistake to assume he meant call/cc. Thanks for
pointing this out.
I can't speak for the others.
Pascal
Pascal Costanza
> Scheme's call-with-current-continuation procedure
> is standard, and plays nicely with unwind-protect.
I'd like to check my understanding: Your implementation of call/cc-new,
for example, plays well with your implementation of unwind-protect.
However, if some application code that uses your implementation tries to
interact with some third library that uses a different approach (say, it
uses plain call/cc for simple exception handling), then there may be a
problem, right?
Or is there a way to control this further? Or do Schemers simply not use
call/cc and/or dynamic-wind in their raw form in practice?
Pascal
William D Clinger
versus continuations" [1], which I called "disinformation",
Bulent Murtezaoglu asked:
> Not that I wish to be a party to this beyond curiosity but:
> Is the 'disinformation' annotation for this link still your
> opinion in view of what that link now says?
Yes. For example, Kent Pitman is still claiming that Scheme's
standard call-with-current-continuation procedure is redefined
by my implementation of unwind-protect. That is untrue, and
it is an important point that I and others have made to Kent
on countless occasions. That Kent continues to spread such
disinformation is irresponsible.
It is a fact that Scheme's standard DYNAMIC-WIND procedure
can be used to define UNWIND-PROTECT, without redefining
any of Scheme's standard procedures.
This has been known for twenty years.
In the notes that Kent added to his web page on the day
that Dorai presented his refutation of Kent's claims at
the Scheme workshop, Kent presents several quibbles, most
of which are baseless. All but one of those quibbles is
qualified by weasel words like "should mostly work", "I'm
troubled", "I have some concern", "I have some queasiness",
"it seems to me", "I...cringe" [1]. The only quibble that
is not so qualified is Kent's claim that my implementation
redefines call-with-current-continuation, and that claim
is flatly untrue. These are the words of a man who would
rather mislead readers than admit his mistakes.
Let me add here that I do not believe my implementation
[2] does what Kent would want for continuations created
by Scheme's standard call-with-current-continuation.
That was a deliberate bit of sneakiness; once my code
is understood, the problem becomes obvious---as does
the way to fix it. I asked Kent to let me know whether
I had gotten that part of the semantics right, but he
never responded to my question. When he made his false
claim about redefinition, it became clear that he had
forgotten about the question. To me, this all implies
he couldn't be bothered to understand the issues on
which he pretends to be such an expert.
We cannot eliminate disinformation from Usenet or from
the World-Wide Web, but we can identify it as such, and
refute it.
Will
[1] Kent Pitman. "UNWIND-PROTECT vs Continuations."
Online at
http://www.nhplace.com/kent/PFAQ/unwind-protect-vs-continuations.html
[2] Will Clinger. Untitled page of Scheme code, at
http://www.ccs.neu.edu/home/will/Temp/uwesc.sch
William D Clinger
> However, if some application code that uses your implementation
> tries to interact with some third library that uses a different
> approach (say, it uses plain call/cc for simple exception
> handling), then there may be a problem, right?
Yes, but that's only because, as a private joke, and to
make my code more symmetrical and more like Dorai's, I
defined both call/cc-escaping and call/cc-full, while
inserting a deliberate misfeature into my code to see
whether Kent was paying attention. (Let me congratulate
you, Pascal, for being the first Common Lisper to spot it!)
If you change my code [1] so that the "full?" variable is
initialized to true, change the code for call/cc-escaping
to set it to false and then back to true, and change the
code for call/cc-full to leave that variable completely
alone, then call/cc-full will behave exactly like Scheme's
standard call-with-current-continuation (modulo multiple
return values, which would obscure the code without adding
any new insight), and both will do the right thing with
respect to unwind-protect.
In other words, call/cc-full is entirely unnecessary.
Scheme's standard call-with-current-continuation procedure
should be used instead. Then there is no problem with
legacy code.
This is well-known in the Scheme and research communities,
so they were in on the joke. I apologize to anyone whose
understanding of my code was hindered by the joke, but I
had already shown Kent how to implement unwind-protect the
easy way, and he had rejected that implementation on vague
grounds that turned out to have nothing to do with the
major issues. By the time I wrote this code [1], which
matched Dorai's understanding of Kent's vague semantics,
I felt I had earned the right to a bit of fun.
Will
Bruce Stephens
> [...] If you change my code [1] so that the "full?" variable is
> initialized to true, change the code for call/cc-escaping to set it
> to false and then back to true, and change the code for call/cc-full
> to leave that variable completely alone, then call/cc-full will
> behave exactly like Scheme's standard call-with-current-continuation
> (modulo multiple return values, which would obscure the code without
> adding any new insight), and both will do the right thing with
> respect to unwind-protect.
>
> In other words, call/cc-full is entirely unnecessary. Scheme's
> standard call-with-current-continuation procedure should be used
> instead. Then there is no problem with legacy code.
>
> This is well-known in the Scheme and research communities, so they
> were in on the joke. I apologize to anyone whose understanding of
> my code was hindered by the joke, but I had already shown Kent how
> to implement unwind-protect the easy way, and he had rejected that
> implementation on vague grounds that turned out to have nothing to
> do with the major issues. By the time I wrote this code [1], which
> matched Dorai's understanding of Kent's vague semantics, I felt I
> had earned the right to a bit of fun.
Ah, I think I understand. So (presuming one does as you say, and
remove call/cc-full), what you're really doing is defining (using the
standard call-with-current-continuation) call/cc-escaping---a way to
construct escape procedures---together with unwind-protect which
carefully only evaluates its cleanup bodies once. It might be worth
rearranging the code just to make that more clear to those of us not
in the relevant communities.
Presuming I understand dynamic-wind approximately correctly, that
means that if I jump out of an unwind-protect and then jump back in
again (using continuations), then the cleanup bodies will have been
evaluated (your implementation prevents them from being evaluated
again). So if I'm using unwind-protect to implement with-open-file or
something then that's not necessarily going to give the semantics that
I might expect (whereas in Common Lisp if I escape from inside and the
file is closed then that's OK because there's no way for me to jump
back in)?
William D Clinger
> Ah, I think I understand. So (presuming one does as you say,
> and remove call/cc-full), what you're really doing is defining
> (using the standard call-with-current-continuation)
> call/cc-escaping---a way to construct escape
> procedures---together with unwind-protect which
> carefully only evaluates its cleanup bodies once.
Exactly.
> It might be worth rearranging the code just to make that more
> clear to those of us not in the relevant communities.
Done. While cleaning up the code to remove the joke, I noticed one
other change that had to be made. The more serious code is now at
http://www.ccs.neu.edu/home/will/UWESC/uwesc.sch and I have
added this URL to the top of the old version in my Temp directory.
> So if I'm using unwind-protect to implement with-open-file
> or something then that's not necessarily going to give the
> semantics that I might expect (whereas in Common Lisp if I
> escape from inside and the file is closed then that's OK
> because there's no way for me to jump back in)?
That's true. IIRC, the first implementation that I showed Kent
would not have allowed you to jump back into an unwind-protect
under any circumstances, and he objected to that, arguing for
the semantics that Dorai and I implemented. I still think it
makes more sense to disallow jumps back into an unwind-protect,
because that's the semantics that Common Lisp programmers would
expect. That semantics would involve a very minor tweak on the
code whose URL I gave above.
Will
William D Clinger
Bruce Stephens wrote:
> Presuming I understand dynamic-wind approximately correctly,
> that means that if I jump out of an unwind-protect and then
> jump back in again (using continuations), then the cleanup
> bodies will have been evaluated (your implementation prevents
> them from being evaluated again).
Actually, Kent wanted to allow full continuations to jump out
of the unwind-protect without evaluating the cleanup forms,
and also to jump back into the unwind-protect provided the
cleanup forms have not yet been evaluated. He motivated
this with his movie ticket example.
Once the cleanup forms have been evaluated, I don't think
Kent's semantics or my code allows you to jump back into
the unwind-protect. I haven't actually proved that invariant,
but I think it's true for my code. If so, then I can simplify
the code for unwind-protect-proc to
(set! unwind-protect-proc
(lambda (body postlude)
(let ((exited? #f) ; have we done the postlude?
(normal-exit? #f)) ; is this a normal exit?
(dynamic-wind
(lambda () (if exited? (errorproc)))
(lambda () (let ((v (body))) (set! normal-exit? #t) v))
(lambda ()
(if (and (not exited?)
(or normal-exit? (not full?)))
(begin (set! exited? #t)
(postlude))))))))
If I'm wrong about this, and the invariant isn't true for my
code, it might still be true for the semantics that Kent
actually wants, in which case the above shows how my code
could be changed to implement that semantics.
Will
Ron Garret
"William D Clinger" <cesur...@verizon.net> wrote:
> Concerning Dorai's tutorial [1,2], I wrote:
> > Dorai's tutorial shows there is no conflict between
> > UNWIND-PROTECT and full continuations.
>
> Ron Garret replied:
> > Actually, it specifically says the exact opposite:
> >
> > "Once these new operators are defined, the original call/cc
> > would have to be retired so as not to interfere with the
> > functioning of the new operators."
>
> You sound as though you believe Scheme's original call/cc is
> the one and only operator that provides full continuations.
Well, no, I don't believe that. It's clearly untrue. But I do believe
(perhaps erroneously) that any operator or set of operators that provide
full continuations are in some sense equivalent to each other, and that
if one such (set of) operator(s) (namely standard Scheme
call-with-current-continuation) is incompatible with unwind-protect (as
Dorai states it is) then all such (sets of) operators would likewise be
incompatible, and that any (set of) operators that are compatible would
in some sense be less powerful. But I admit I did not think deeply
about this.
> I suspect that Kent Pitman was
> the primary source of the disinformation that has been
> repeated in this thread by Rahul Jain, Ron Garret, and
> Pascal Costanza. Would you guys care to confirm that?
No. It's true that Kent's position influenced my thinking, but what I
wrote was based entirely on what I read (cursorily I admit) in Dorai's
tutorial.
rg
Pascal Costanza
> If you change my code [1] so that the "full?" variable is
> initialized to true, change the code for call/cc-escaping
> to set it to false and then back to true, and change the
> code for call/cc-full to leave that variable completely
> alone, then call/cc-full will behave exactly like Scheme's
> standard call-with-current-continuation (modulo multiple
> return values, which would obscure the code without adding
> any new insight), and both will do the right thing with
> respect to unwind-protect.
>
> In other words, call/cc-full is entirely unnecessary.
> Scheme's standard call-with-current-continuation procedure
> should be used instead. Then there is no problem with
> legacy code.
I think there is still a problem with legacy code. Continuations created
in other code can either all be treated as escaping or all as full
continuations, but I don't see a chance to automagically determine what
they were supposed to be in order to make a finer distinction.
In other words, adding explicit escaping continuations + unwind-protect
to Scheme would require existing Scheme code to be updated to reflect
the new features, right?
That's actually good news for Common Lisp: We already have means to
communicate that a continuation is an escaping one, so it would be no
problem to add full continuations to Common Lisp, should the need ever
arise. ;)
> I apologize to anyone whose
> understanding of my code was hindered by the joke
I have found the buggy error reporting more misleading: The call to
errorproc in unwind-protect-proc should report that a full continuation
tries to enter an unwind-protect form again that was already exited,
while it actually reports that an escaping continuation is called twice.
It took me a while to understand that your code actually does something
else.
Pascal
Rahul Jain
Because it's true... Unless you've managed to figure out how to define
UNWIND-PROTECT's semantics usefully in the presence of functions that
return multiple times. If you have, I'd be very interested in the
solution. :)
Rahul Jain
> It is a fact that Scheme's standard DYNAMIC-WIND procedure
> can be used to define UNWIND-PROTECT, without redefining
> any of Scheme's standard procedures.
DYNAMIC-WIND-like functionality isn't what I expect out of an
UNWIND-PROTECT operator. I don't want the file to be rewritten because
some function returned twice. I want the file to be opened once and then
closed once. WITH-OPEN-FILE isn't WITH-FILE-BEING-REWRITTEN-REPEATEDLY.
:)
Rahul Jain
> That's true. IIRC, the first implementation that I showed Kent
> would not have allowed you to jump back into an unwind-protect
> under any circumstances, and he objected to that, arguing for
> the semantics that Dorai and I implemented.
But you implement that by providing two options:
1. Don't use full continuations
2. Cause the code that uses whatever resource is being created to not
work when you try to use a full continuation
I don't see how that's an effective integration of the two operators. I
view what you're trying to do as akin to making an exact garbage
collector for C. It's just a huge impedance mismatch.
> I still think it makes more sense to disallow jumps back into an
> unwind-protect, because that's the semantics that Common Lisp
> programmers would expect.
We would expect that because we don't have full continuations. :)
You can't give us full continuations and then not let us use them with a
commonly-used operator because we didn't have them before...
Jens Axel Søgaard
> "William D Clinger" <cesur...@verizon.net> writes:
>>It is a fact that Scheme's standard DYNAMIC-WIND procedure
>>can be used to define UNWIND-PROTECT, without redefining
>>any of Scheme's standard procedures.
> DYNAMIC-WIND-like functionality isn't what I expect out of an
> UNWIND-PROTECT operator. I don't want the file to be rewritten because
> some function returned twice. I want the file to be opened once and then
> closed once. WITH-OPEN-FILE isn't WITH-FILE-BEING-REWRITTEN-REPEATEDLY.
> :)
Using dynamic-wind to implement it, doesn't mean that that it
behaves like dynamic-wind. With a simple flag, one just checks in
the after-thunk of the dynamic-wind whether or not one should
execute the body (again).
See the comments in:
<http://www.ccs.neu.edu/home/will/UWESC/uwesc.sch>
--
Jens Axel Søgaard
Rahul Jain
> Using dynamic-wind to implement it, doesn't mean that that it
> behaves like dynamic-wind. With a simple flag, one just checks in
> the after-thunk of the dynamic-wind whether or not one should
> execute the body (again).
How do I get it to not execute the body until the continuation is no
longer accessible to the system? Sounds like a finalizer to me, and you
basically end up losing all the usefulness of UNWIND-PROTECT. Might as
well just attach a finalizer to the stream.
Jens Axel Søgaard
> Jens Axel Søgaard <use...@soegaard.net> writes:
>>Using dynamic-wind to implement it, doesn't mean that that it
>>behaves like dynamic-wind. With a simple flag, one just checks in
>>the after-thunk of the dynamic-wind whether or not one should
>>execute the body (again).
> How do I get it to not execute the body until the continuation is no
> longer accessible to the system? Sounds like a finalizer to me, and you
> basically end up losing all the usefulness of UNWIND-PROTECT. Might as
> well just attach a finalizer to the stream.
Perhaps I am confused - are we now talking about how the to implement
the unwind-protect with the specified semantics, or how the semantics
ought to have been?
The puzzle was how to implement unwind-protect with this semantics:
; (unwind-protect <expr1> <expr2>) evaluates <expr1> to obtain its
; single value v, passing v to a continuation that normally (but
; not always; see below) evaluates <expr2> for effect, and returns v
; in any case. If the evaluation of <expr1> results in a throw past
; the natural continuation of <expr1>, and <expr2> has not yet been
; evaluated, and the throw is to a continuation that was created by
; call/cc-escaping, then <expr2> is evaluated for effect during the
; throw, and will not be evaluated again if and when <expr1> returns
; a result to its continuation. If the throw is to a continuation
; created by call-with-current-continuation, however, then <expr2>
; is not evaluated.
; Furthermore no continuation that was created
; by call-with-current-continuation is allowed to continue the
; evaluation of <expr1> once <expr2> has been evaluated.
From the last three lines, one can see that it is possible to
disguish between full and escaping continuations, and have
let them behave differently.
When an continuation c captured in the body of
(unwind-protect <body> <cleanup>) is invoked, the before thunk
of the dynamic-wind expression used to implement the unwind-protect
is called:
(lambda () (if (and exited? full?) (errorproc)))
It checks two flags:
exited?, which if true means that the <cleanup> expressions
already has been evaluated for effect
full?, which if true means that the invoked continuation c,
was captured by the original call-with-current-contination
and not call/cc-escaping
Thus if c was captured by call-with-current-contination, the
before-thunk of dynamic-wind calls errorproc, and the computation
of <body> never resumes.
If on the other hand the contination c was captured by
call/cc-escaping and <cleanup> hasn't been evaluated for effect yet,
then the computation of the <body> can resume -- provided
that the escaping continuation in question hasn't been called before.
The code for call/cc-escaping is
(set! call/cc-escaping
(lambda (f)
(let ((v (call-with-current-continuation
(lambda (escape)
(f (lambda (v)
(let ((k escape))
(set! escape errorproc) ; <- (*)
(set! full? #f)
(k v))))))))
(set! full? #t)
v)))
The line marked with (*) makes sure that an escaping continution
on a second invocation calls errorproc.
To summarize:
One can't resume a computation in <body> using continuations
captured by call-with-current-continuations.
One can resume a computation in <body> using escaping
continuations captured by call/cc-escaping -- but these
can only be resumed once.
The answer to
> How do I get it to not execute the body until the continuation is no
> longer accessible to the system?
must therefore be "nothing".
--
Jens Axel Søgaard
Rahul Jain
> Perhaps I am confused - are we now talking about how the to implement
> the unwind-protect with the specified semantics, or how the semantics
> ought to have been?
I never mentioned the semantics of some scheme programmer's
unwind-protect, nor do I much care about them (no offense). It just has
no relevance to the discussion at hand. Will claimed that I was
spreading misinformation by claiming that unwind-protect and full
continuations could not coexist. Will's solution to allowing them to
coexist is to either make the second use of a full continuation within
an unwind-protect an error or to break code that assumes that the
unwind-protect's cleanup clause will not have been run when code within
the unwind-protect is being run. I contested his solution as being an
admission that the two cannot coexist, and you claim that it works "as
specified". Eh? I don't see anything in the spec that is like what you
quoted later.
> The answer to
>
> > How do I get it to not execute the body until the continuation is no
> > longer accessible to the system?
>
> must therefore be "nothing".
Hmm... not sure what you mean there. "Nothing" is a "what", not a "how".
Do you mean "you don't"? Because if that's the case, then you've
admitted that functions that return twice and unwind-protect just don't
make logical sense together. Therefore, such functions are quite useless
to Common Lispers, as they would break the semantics of an operator that
often wraps the bulk of code as it is run in a production situation.
Jens Axel Søgaard
> Jens Axel Søgaard <use...@soegaard.net> writes:
>>Perhaps I am confused - are we now talking about how the to implement
>>the unwind-protect with the specified semantics, or how the semantics
>>ought to have been?
> I never mentioned the semantics of some scheme programmer's
> unwind-protect, nor do I much care about them (no offense). It just has
> no relevance to the discussion at hand.
Sorry, I can't see that. Having both unwind-protect and full
continations necessitates a description of their interactions.
> Will claimed that I was
> spreading misinformation by claiming that unwind-protect and full
> continuations could not coexist. Will's solution to allowing them to
> coexist is to either make the second use of a full continuation within
> an unwind-protect an error or to break code that assumes that the
> unwind-protect's cleanup clause will not have been run when code within
> the unwind-protect is being run.
Previously, under the assumption that you were talking about the
same semantics as Clinger, I interpreted the phrase "Not able to coexist"
as "It is not possible to implement the wanted semantics in plain R5RS
Scheme".
I now see, that you are not saying, that the Clinger's implementation
doesn't fullfill the specified semantics (sorry for the misinterpretation).
You are saying that you don't consider the specified semantics as
of unwind-protect in the presence of full continuations satisfying.
What should such a sematics fulfill to be satisfying?
> I contested his solution as being an
> admission that the two cannot coexist, and you claim that it works "as
> specified". Eh? I don't see anything in the spec that is like what you
> quoted later.
I clearly misunderstood your original question.
>>The answer to
>>
>> > How do I get it to not execute the body until the continuation is no
>> > longer accessible to the system?
>>
>>must therefore be "nothing".
> Hmm... not sure what you mean there.
I meant to write "do nothing" - that is if your question means:
"What should the programmer do, to make sure that a <body> of
an unwind-protect is executed only once, even though a
full continuation or an escaping continuation is invoked
outside the dynamic extent of the unwind-protect form."
I had this
> I want the file to be opened once and then closed once.
> WITH-OPEN-FILE isn't WITH-FILE-BEING-REWRITTEN-REPEATEDLY.
from the previous message in mind, so I gathered that you
were concerned that the <body> could be evaluated twice.
--
Jens Axel Søgaard
William D Clinger
than admit his mistakes. The purpose of this message is
to identify several of Rahul's mistakes, and to point out
how his posts are misleading.
Rahul Jain wrote:
> cesuraS...@verizon.net writes:
> > Rahul Jain wrote:
> >> > In that case what's the implementation burden of making these
> >> > continuations available with indefinite extent to end users?
>
> >> Infinite, assuming UNWIND-PROTECT is a desired operator.
> > .
> > Why do you think that?
>
> Because it's true... Unless you've managed to figure out how
> to define UNWIND-PROTECT's semantics usefully in the presence
> of functions that return multiple times. If you have, I'd be
> very interested in the solution. :)
This is a classic example of changing the subject. Rahul
had claimed that two implementations of Common Lisp use
full continuations under the hood. He then claimed that
"the implementation burden of making these continuations
available with indefinite extent to end users" is infinite.
In reality, the implementation burden is approximately zero.
To cover his tracks, he is now talking about the difficulty
of defining a useful semantics for UNWIND-PROTECT. So far
as I know, nothing in the Common Lisp standard would prevent
implementations from adding full continuations as an extension
to the language. If that is so (and not even Rahul Jain has
alleged otherwise), then the alleged uselessness of UNWIND-PROTECT
in the presence of that extension, were that allegation true
(which it would most certainly be not :), would constitute a
defect in the CL standard. That defect would be understandable
and forgivable, but would have nothing to do with the burden of
implementation.
In short, Rahul Jain's original claim was infinitely false.
He tried to hide his mistake by burying it under a barrage of
further nonsense, to which I now turn.
> DYNAMIC-WIND-like functionality isn't what I expect out of an
> UNWIND-PROTECT operator. I don't want the file to be rewritten
> because some function returned twice. I want the file to be
> opened once and then closed once. WITH-OPEN-FILE isn't
> WITH-FILE-BEING-REWRITTEN-REPEATEDLY.
> :)
Once again, Rahul appended a smiley face, presumably to
acknowledge that his argument cannot be taken seriously.
The obvious and most common definition of UNWIND-PROTECT in
terms of Scheme's DYNAMIC-WIND does exactly what Rahul Jain
claims to want. Kent Pitman's semantics does not do what
Rahul Jain claims to want.
In short, Rahul Jain is arguing here with Kent Pitman, not
with me.
> But you implement that by providing two options:
> 1. Don't use full continuations
> 2. Cause the code that uses whatever resource is being created
> to not work when you try to use a full continuation
>
> I don't see how that's an effective integration of the two operators.
Here again, Rahul Jain is arguing with Kent Pitman. I have
never claimed that Kent Pitman had defined a useful semantics.
Indeed, I stated well over a year ago, in this newsgroup, that
Kent was having trouble defining a useful semantics, but predicted
(correctly) that his semantics would be easy to implement if
he could but state it clearly.
> I never mentioned the semantics of some scheme programmer's
> unwind-protect, nor do I much care about them (no offense).
> It just has no relevance to the discussion at hand.
No offense taken. Kent Pitman is the Scheme programmer of whom
Rahul Jain spoke.
> Will claimed that I was
> spreading misinformation by claiming that unwind-protect and full
> continuations could not coexist.
I claim, correctly, that Rahul Jain was spreading disinformation
when Rahul said the implementation burden of full continuations
is infinite, assuming Common Lisp's UNWIND-PROTECT is a desired
operator.
Rahul Jain's disinformation was not just untrue, it was obviously
untrue. Here, boys and girls, is the obvious implementation of
Common Lisp's UNWIND-PROTECT in Scheme:
(define-syntax unwind-protect
(syntax-rules ()
((unwind-protect body cleanup ...)
(dynamic-wind (lambda () #t)
(lambda () body)
(lambda () cleanup ...)))))
Rahul Jain claims this is useless in languages that provide full
continuations. Nonetheless myriad Scheme programmers have found
it useful enough to implement this very macro, or something
similar, and to use it in their code. Rahul's claim therefore
fails the test of actual experience.
> Will's solution to allowing them to
> coexist is to either make the second use of a full continuation
> within an unwind-protect an error or to break code that assumes
> that the unwind-protect's cleanup clause will not have been run
> when code within the unwind-protect is being run. I contested his
> solution as being an admission that the two cannot coexist, and
> you claim that it works "as specified". Eh? I don't see anything
> in the spec that is like what you quoted later.
In the above, Rahul Jain is describing and criticizing Kent Pitman's
"solution", not mine. On his web site, Kent Pitman claimed that his
semantics could not be implemented in Scheme. That was nothing but
disinformation. Dorai Sitaram and I have shown how Kent's semantics
can be implemented as Kent had specified it. My implementation shows
that Kent's semantics is perfectly compatible with Scheme's standard
CALL-WITH-CURRENT-CONTINUATION, contrary to the disinformation on
Kent's web site.
Has Rahul Jain not bothered to read Kent Pitman's specification?
> Because if that's the case, then you've
> admitted that functions that return twice and unwind-protect just
> don't make logical sense together. Therefore, such functions are
> quite useless to Common Lispers, as they would break the semantics
> of an operator that often wraps the bulk of code as it is run in
> a production situation.
I agree that Kent Pitman's semantics for UNWIND-PROTECT is not
well thought out. On the other hand, Common Lisp's semantics
for UNWIND-PROTECT makes perfect sense in the context of full
continuations: In effect, UNWIND-PROTECT becomes a way for the
programmer to protect *against* the possibility that continuations
may be invoked more than once.
Rahul Jain would probably claim this is useless, but he would
be very wrong.
Will
William D Clinger
Contrary to my previous post, to which I am replying, here is
what I regard as the obvious implementation of Common Lisp's
UNWIND-PROTECT in Scheme:
(define-syntax unwind-protect
(syntax-rules ()
((unwind-protect body cleanup ...)
(dynamic-wind (let ((ok? #t))
(lambda ()
(if ok? (set! ok? #f) (error))))
(lambda () body)
(lambda () cleanup ...)))))
I tested this one.
I apologize for the previous disinformation.
Will
Kent M Pitman
> Like Kent Pitman, Rahul Jain would rather mislead readers
> than admit his mistakes.
Excuse me, Will, but I not only acknowledged that the technology you cite
is technically workable, but I kept the old version of my remarks online
and marked them superseded, permanently documenting the fact that I had
changed my mind.
What I did not do was to say that I think the solution you cite is elegant.
I don't.
It's fine with me if you want to assert that I am wrong on this or any point.
That's what public debate is about. I am sometimes right and sometimes wrong,
but I endeavor to keep debates impersonal, and to debate issues, not people
or motives.
As such, please do not attribute intent to my remarks, whether you
believe me right or wrong. To demonstrate the correctness of your
term "mislead" above, you would have to show two things: (a) that I'm
taking someone in a direction that is not proper [we can right and
properly debate this at length] and (b) that I am doing so with intent
to deceive [this is either true or not, but you certainly have no
window into my intent, and have no reason that I am aware of to infer
such a desire on my part--if you think you do, please raise this with
me privately, don't let me come back to a newsgroup after months and
months of absence to find my good name being sullied in such a way by
someone I had thought I should call my personal friend]. And even you
debate the term "mislead" here, there is the issue of your use of
"rather", which again speaks to preference and choice, which you again
infer.
I wish I had more time to keep my web pages on this issue up to date,
but I don't. Nevertheless, I have to the best of my ability in the
time I have available, made my feelings available even as they change.
I consider UNWIND-PROTECT a more worthy and useful basic programming
primitive than either first class continuations (which, though novel,
I don't think really is as good in programming as it is in teaching) or
dynamic-wind (which I am very disappointed in as a programming construct
generally). You don't have to agree with me. But please don't disparage
me for attempting to explain my reasons for this.
> The obvious and most common definition of UNWIND-PROTECT in
> terms of Scheme's DYNAMIC-WIND does exactly what Rahul Jain
> claims to want. Kent Pitman's semantics does not do what
> Rahul Jain claims to want.
>
> In short, Rahul Jain is arguing here with Kent Pitman, not
> with me.
Why is it necessary to cite me? Why not just address the issues directly?
Except for this message, I'm not knowingly party to the debate you're having.
> > But you implement that by providing two options:
> > 1. Don't use full continuations
> > 2. Cause the code that uses whatever resource is being created
> > to not work when you try to use a full continuation
> >
> > I don't see how that's an effective integration of the two operators.
>
> Here again, Rahul Jain is arguing with Kent Pitman. I have
> never claimed that Kent Pitman had defined a useful semantics.
> Indeed, I stated well over a year ago, in this newsgroup, that
> Kent was having trouble defining a useful semantics, but predicted
> (correctly) that his semantics would be easy to implement if
> he could but state it clearly.
I'm not sure what your point is here, so I don't even know how to answer
this.
Although I certainly questioned the implementation issues, and you have
demonstrated that such can be overcome, I didn't find anything elucidating
about the way in which they were overcome. It's ugly and uninspiring.
I prefer languages that are graceful in how they express things, not
in how they allow implementation of things. It's one reason I like
languages like Lisp that have both WHEN and UNLESS, when only IF or
COND would suffice. UNWIND-PROTECT speaks to me, but is impossible to
implement at all in Scheme absent DYNAMIC-WIND--I never considered
that a good trade. I thought DYNAMIC-WIND had been added for some
other purpose (environment management for multithreading), but I've
been shown it doesn't suffice for that. That makes me not sure why
it's there at all, except to allow me to implement UNWIND-PROTECT
clumsily. I personally would rather have a language with stack-based
CATCH + UNWIND-PROTECT (like CL) than one with full-CATCH +
DYNAMIC-WIND (like Scheme). I find the Scheme way of saying things
Baroque and the CL way elegant. I prefer keywords calling to
positional arguments that are general-purpose closures. I simply
disagree with you on taste. That doesn't make you wrong and me right.
I hope it doesn't make me wrong and you right. We simply value different
things.
But I have for quite some time acknowledged the "implementation" detail.
And I have no intent to mislead anyone over this. I just have an
opinion. It's based on aesthetics, and I have no denotational semantics
of aesthetics to help me describe my preferences. So my words are
more vague than yours, necessarily. But that doesn't, IMO, make them
less right.
> I agree that Kent Pitman's semantics for UNWIND-PROTECT is not
> well thought out. On the other hand, Common Lisp's semantics
> for UNWIND-PROTECT makes perfect sense in the context of full
> continuations: In effect, UNWIND-PROTECT becomes a way for the
> programmer to protect *against* the possibility that continuations
> may be invoked more than once.
And yet Scheme (last I checked--I understand there have been or are or
will be some newer standards than the ones I've worked on) offers no
such expressive construct... If it fits in so neatly, it should be part
of the language definition. Having the ability to do the with-open-file
equivalent in Scheme, but not having the more basic unwind-protect operation
that conceptually implements it, leaving that as an exercise to the
programmer, seems unfair. Those little programs you and Dorai whipped up
may not seem like child's play to everyone.
William D Clinger
for several years, beginning shortly after I became aware of his
original web page [1]. At first, this communication was private.
It became public when readers of this newsgroup began to cite
Kent's web page in support of various nonsense that I have tried
to address in this newsgroup, most recently within the past week.
IIRC, one of my last private communications with Kent encouraged
him to revise his web site, which he did on 7 November 2003.
Now Kent writes:
> I am sometimes right and sometimes wrong, but I endeavor to
> keep debates impersonal, and to debate issues, not people or
> motives.
>
> As such, please do not attribute intent to my remarks, whether
> you believe me right or wrong.
Part of the reason this debate became personal is that your
original critique of continuations *did* attribute motives to
people who knew enough about this subject and its history to
reject your critique. Here are the relevant quotations:
The community of Scheme designers has rejected my repeated
calls over the years to fix this problem....
The designers as a group seem so enamored of the present
continuation semantics and its simplicity that they don't
want to injure that in order to allow some materially useful
functionality....
If one of the indicated changes were made, I think
unwind-protect could be usefully added as a standard part of
Scheme. However, I sense that Scheme designers (and many
advocates as well) so dearly love the simplicity of
call-with-current-continuation and continuations, and the
conciseness of the associated syntax, that they are willing
to sacrifice usefulness for elegance. This seems to me a bad
trade. But I am just one of the dozen and a half Scheme
designers, so I am outvoted. But at least now my views are
on the record.
(The designers to whom Kent referred included three world-class
experts on continuations and UNWIND-PROTECT, plus several more
who know enough about that area to have published significant
papers in it. I am in that second group.)
> ....you certainly have no window into my intent, and have no
> reason that I am aware of to infer such a desire on my part--if
> you think you do, please raise this with me privately, don't
> let me come back to a newsgroup after months and months of
> absence to find my good name being sullied in such a way by
> someone I had thought I should call my personal friend
I have three things to say to that.
First, I certainly owe you an apology for comparing you to another
contributor in this newsgroup, and for saying you would rather
mislead readers than admit your mistakes. I'm sorry about that.
Second, from what we have seen this past week, it is clear that
your updated web site continues to mislead some readers, and that
some who read it do not realize you have repudiated your original
critique. I should never have suggested you actually prefer this
unhappy state of affairs.
Third, I believe that the best way for you to protect your
good name is to correct the errors that remain on your web site,
and to make your repudiation of the original critique somewhat
clearer to casual readers.
> I wish I had more time to keep my web pages on this issue up to
> date, but I don't.
I do understand that you have not had time to read the research
papers we have been urging you to read. What bothers me is that
you allowed disinformation to remain on your web site for so
long after so many credible people had told you of the basic
research and history that flatly contradicted your opinions.
> Nevertheless, I have to the best of my ability in the time I
> have available, made my feelings available even as they change.
Too many readers of your web site think you are expressing facts,
not just feelings.
> You don't have to agree with me. But please don't disparage
> me for attempting to explain my reasons for this.
I am not disparaging you for attempting to explain your reasons.
To the extent that I am disparaging you, I am disparaging you for
poor scholarship. You have written more about this subject than
you have read. That is how you have become one of the world's
primary sources of disinformation about it.
I am not saying you intended to become a source of disinformation.
I acknowledge that your motives were pure, and I apologize for
having suggested they might be otherwise.
> But I have for quite some time acknowledged the "implementation"
> detail.
No, your updated web page still gives the impression that an
implementation of UNWIND-PROTECT in Scheme must redefine Scheme's
standard CALL-WITH-CURRENT-CONTINUATION procedure. This is a
fairly important point, which I have been making to you for
about three years now, and I don't understand why you continue
to deny it.
I respect your right to your own opinions and preferences in
language design. My complaints concern your errors of fact,
and how long it has taken you to acknowledge them.
Robert Marlow
Say we have two libraries. One which defines a macro such as
with-open-file and another which defines a function making use of call/cc.
If I am a typical user of these libraries and combine the two in some
function while having no idea of the manner in which the utilities were
written am I not likely to end up bashing my head against the wall trying
to figure out why I keep getting errors?
Correct me if I'm wrong but it seems to me your implementation means users
must constantly think about whether any reentering call/cc is likely to
appear in any unwind-protect; information which may not always be readily
available when using 3rd party software.
William D Clinger
side effect that deletes a file. Is that a problem with
MAPCAR? I don't think so.
What if a special version of MAPCAR were written so that
any attempt to delete a file would be detected and converted
into an exception? In my opinion, the possibility of such
exceptions would not be a reason to criticize the special
version of MAPCAR, especially since the unrestricted,
original version of MAPCAR would remain available for
programmers who really do want MAPCAR's function argument
to delete files.
Now we substitute UNWIND-PROTECT for MAPCAR...
Robert Marlow wrote:
> I'm still confused about something.
>
> Say we have two libraries. One which defines a macro such as
> with-open-file and another which defines a function making use
> of call/cc. If I am a typical user of these libraries and
> combine the two in some function while having no idea of the
> manner in which the utilities were written am I not likely to
> end up bashing my head against the wall trying to figure out
> why I keep getting errors?
That's right: typical users are not likely to bash their
heads.
The obvious implementation of CL-like UNWIND-PROTECT in
Scheme does not interfere with any normal uses of full
continuations. It protects against a very specific and
unusual usage that we refer to as a control effect, by
analogy with side effects.
Control effects, like side effects, should be documented.
Undocumented control effects, like undocumented side
effects, are properly regarded as bugs or as malicious.
UNWIND-PROTECT protects against a fairly specific control
effect, which I'll describe below.
> Correct me if I'm wrong but it seems to me your implementation
> means users must constantly think about whether any reentering
> call/cc is likely to appear in any unwind-protect; information
> which may not always be readily available when using 3rd party
> software.
There is only one kind of control effect that can cause
a problem: a continuation that is captured within the
dynamic extent of the UNWIND-PROTECT is returned out of
the UNWIND-PROTECT, or is stored within a variable or
data structure that outlives the dynamic extent of the
UNWIND-PROTECT, and is then invoked after control has
left the UNWIND-PROTECT.
The values returned by third party software should be
documented, as should their side effects, so the fact
that third party software allows a captured continuation
to escape the UNWIND-PROTECT should be documented.
And how would the escaping continuation be invoked? The
third party software would have to trick the programmer
into invoking it after the UNWIND-PROTECT has exited, or
would have to invoke the continuation itself when the
third party software is called at some later time. Once
again, these behaviors are properly regarded as malicious
or buggy if undocumented. CL-style UNWIND-PROTECT protects
against those behaviors.
Will
For reference, here is CL-style UNWIND-PROTECT, implemented
in portable Scheme:
Rahul Jain
> Previously, under the assumption that you were talking about the
> same semantics as Clinger, I interpreted the phrase "Not able to coexist"
> as "It is not possible to implement the wanted semantics in plain R5RS
> Scheme".
We were talking about Common Lisp, not Scheme. See also: the subject
line of your post.
Rahul Jain
> Rahul Jain wrote:
[...]
> Like Kent Pitman, Rahul Jain would rather mislead readers
> than admit his mistakes. The purpose of this message is
> to identify several of Rahul's mistakes, and to point out
> how his posts are misleading.
Ludicrous. First you bring an, at the time not even present, third party
into the discussion and then you twist my words into having something to
do with an attack on him. The fact of the matter is that
> This is a classic example of changing the subject. Rahul
> had claimed that two implementations of Common Lisp use
> full continuations under the hood. He then claimed that
> "the implementation burden of making these continuations
> available with indefinite extent to end users" is infinite.
> In reality, the implementation burden is approximately zero.
Ok, maybe it's the burden on the language _users_ that's infinite, as
your solution would make full continuations completely useless.
Actually, the implementations that use CPS transforming in the compiler
wouldn't use them as full continuations, as that would make a stack
discipline much more complex to implement.
> Once again, Rahul appended a smiley face, presumably to
> acknowledge that his argument cannot be taken seriously.
Yes, because it's about a nonsensical combination of operators.
> The obvious and most common definition of UNWIND-PROTECT in
> terms of Scheme's DYNAMIC-WIND does exactly what Rahul Jain
> claims to want. Kent Pitman's semantics does not do what
> Rahul Jain claims to want.
Wrong. Neither yours nor Kent's semantics are acceptable to me. Of
course, unlike how you portray it, I'm not attacking Kent for this. He's
just trying to think of Scheme in terms of CL programmers naturally and
frequently do. Unfortunately, that's not what Scheme programmers care
for.
>> I never mentioned the semantics of some scheme programmer's
>> unwind-protect, nor do I much care about them (no offense).
>> It just has no relevance to the discussion at hand.
>
> No offense taken. Kent Pitman is the Scheme programmer of whom
> Rahul Jain spoke.
So why exactly did you bring them up?
> Rahul Jain's disinformation was not just untrue, it was obviously
> untrue. Here, boys and girls, is the obvious implementation of
> Common Lisp's UNWIND-PROTECT in Scheme:
This (as corrected) is not Common Lisp's UNWIND-PROTECT. Not as a CL
programmer would expect it to behave.
> Rahul Jain claims this is useless in languages that provide full
> continuations. Nonetheless myriad Scheme programmers have found
> it useful enough to implement this very macro, or something
> similar, and to use it in their code. Rahul's claim therefore
> fails the test of actual experience.
I don't know. Maybe Scheme programmers don't care if their code blows up
at runtime. Some people have to write code for non-programmers to run.
They don't care that your language provides full continuations, they're
just pissed off that you used code that used them when they're
guaranteed to cause errors in any application that's designed to be
production-quality... how ironic.
> Has Rahul Jain not bothered to read Kent Pitman's specification?
Interestingly enough, you started this discussion assuming I was Kent's
mouthpiece. "What happens when we assume?" Perfect example.
> I agree that Kent Pitman's semantics for UNWIND-PROTECT is not
> well thought out. On the other hand, Common Lisp's semantics
> for UNWIND-PROTECT makes perfect sense in the context of full
> continuations: In effect, UNWIND-PROTECT becomes a way for the
> programmer to protect *against* the possibility that continuations
> may be invoked more than once.
>
> Rahul Jain would probably claim this is useless, but he would
> be very wrong.
Oh lovely. So full continuations are really useful because we almost
always use an operator that forbids their use. Hmm... interesting
connection to the source of this discussion: the use of continuations in
web app frameworks. Seems like we have our answer as to why not to do
that if you expect any pages to be side-effecting. Of course, we already
kind-of knew that, but this puts a whole new level on it.
Jens Axel Søgaard
> Jens Axel Søgaard <use...@soegaard.net> writes:
>>Previously, under the assumption that you were talking about the
>>same semantics as Clinger, I interpreted the phrase "Not able to coexist"
>>as "It is not possible to implement the wanted semantics in plain R5RS
>>Scheme".
>
> We were talking about Common Lisp, not Scheme. See also: the subject
> line of your post.
Since you and Marlow were talking about the problems of adding full
continuations to a language with unwind-protect it makes a lot of sense
to compare with other languages having full continuations.
--
Jens Axel Søgaard
Kent M Pitman
> Control effects, like side effects, should be documented.
But sometimes they are not.
There are tools that I've seen written in Scheme that do backtracking,
for example, by checkpointing the state at various points in ways that
are not easily explained to users. In fact, the value of the tool is
that it hides the control aspects and just tells you to rely on the magic
of the abstraction.
Explaining to a user how these tools will interact with others that
inhibit such checkpointing (permitting only one exit) is often difficult.
> Undocumented control effects, like undocumented side
> effects, are properly regarded as bugs or as malicious.
> UNWIND-PROTECT protects against a fairly specific control
> effect, which I'll describe below. [...]
>
> There is only one kind of control effect that can cause
> a problem: a continuation that is captured within the
> dynamic extent of the UNWIND-PROTECT is returned out of
> the UNWIND-PROTECT, or is stored within a variable or
> data structure that outlives the dynamic extent of the
> UNWIND-PROTECT, and is then invoked after control has
> left the UNWIND-PROTECT.
Here's the problem:
UNWIND-PROTECT is fundamental to a lot of programming.
As I'm reading it, you SEEM to expect programs containing
UNWIND-PROTECT to be case-marked in a way that says "maybe you don't
want to go here". (Don't let me put words in your mouth, Will. If
you are saying something else, just say it. I'm just telling you what
I'm hearing so you'll know whether you need to respond here.)
My point is that I worry that the end effect will either be "don't use
Kent's code if you like continuations, because it's riddled with
UNWIND-PROTECT". Which sounds a lot to me like saying "UNWIND-PROTECT
is really second-class and should be avoided because it doesn't work
well with its so-called friends." Which sounds a lot like "This
language isn't designed to allow you to comfortably use all of its
tools in a single program without them tripping over each other."
At least, that's the train of thought I tend to go down as I try to
think about the whole "ecology" of the thing.
I'm shooting from the hip here, a bit, since I haven't had the time I
promised to sit down and think in depth. But I do find myself nervous
about this matter. And I am writing this not to assert you're wrong
but to pick your brain on why you think you're right, in hopes it will
inform my later thought on this.
This kind of meta-problem was a big issue for PROG + RETURN in Common
Lisp for a while, and was finally possible to solve with the issue of
TAGBODY because the notion of named returns could be transparently
inserted in macros, and very few macros needed to export a visible tag
(usually NIL) that would be captured by some client environment.
But the problem with UNWIND-PROTECT and CALL/CC seems to me to be that
they are a little like the "unstoppable force" and the "impenetrable barrier".
Fine concepts considered in isolation, but can't play well together in the
real world. One has to be first-class and the other, at best, second-class
if not left out entirely.
My problem is that I have tons of need for UNWIND-PROTECT and little
need for CALL/CC that isn't handled neatly by stack-CATCH like CL has,
which is not troubled by CALL/CC. I have no desire to piss off
programmers who don't like my style, and I assume they have no desire
to piss me off. But it's the job of language designers to create an
arena in which people can mix and match components in a predictable
way WITHOUT coordinating directly. It should be enough to have read
the manual and to see how things are supposed to work. And it seems
to me you either need to say "feel free to use UNWIND-PROTECT" or
"don't use UNWIND-PROTECT" or "if you use UNWIND-PROTECT, you MUST
document the fact even though you may think it's an internal detail"
or "if you use CALL/CC you MUST document the fact" even though you may
think it's an internal detail.
I think you're right that control abstractions need to be documented, but
CALL/CC is a control abstraction. So maybe you're saying that all uses
of CALL/CC should be documented. ;) I wonder if other Schemers agree.
Or I wonder if maybe you didn't mean to say that.
Again, please don't let me speak for you. I'm just offering some issues
for you to opine on.
Brian Downing
> a problem: a continuation that is captured within the
> dynamic extent of the UNWIND-PROTECT is returned out of
> the UNWIND-PROTECT, or is stored within a variable or
> data structure that outlives the dynamic extent of the
> UNWIND-PROTECT, and is then invoked after control has
> left the UNWIND-PROTECT.
>
> The values returned by third party software should be
> documented, as should their side effects, so the fact
> that third party software allows a captured continuation
> to escape the UNWIND-PROTECT should be documented.
>
> And how would the escaping continuation be invoked? The
> third party software would have to trick the programmer
> into invoking it after the UNWIND-PROTECT has exited, or
> would have to invoke the continuation itself when the
> third party software is called at some later time. Once
> again, these behaviors are properly regarded as malicious
> or buggy if undocumented. CL-style UNWIND-PROTECT protects
> against those behaviors.
What about the other way around - you're using your proposed
UNWIND-PROTECT, and the third party software has a cooperative
threading implementation using continuations. You yield in the
middle of WITH-OPEN-FILE and wonder why everything explodes.
I realize the proposed unwind-protect and call/cc-new in the essay
that sparked this whole horrible debate could work together to
allow continuations that expected to be returned to to not fire
unwind-protect cleanup sections.
However, based upon the simple code you presented for unwind-protect
(with an unmodified call-with-current-continuation), my conclusion
would be either:
a) Your simple unwind-protect is not compatible with all theoretical
use cases of call-with-current-continuation, or
b) call-with-current-continuation is not an acceptable construct to
use to make a userland threading library.
If a), then making an unwind-protect that allows the above requires
a modified version of call-with-current-continuation so that the
intent of a continuation can be communicated.
-bcd
--
*** Brian Downing <bdowning at lavos dot net>
Rahul Jain
Unfortunately, that other language doesn't have an UNWIND-PROTECT that
allows full continuations to be used in more than a tiny fraction of toy
code. Definitely not usable in library code.
Jens Axel Søgaard
> Jens Axel Søgaard <use...@soegaard.net> writes:
>>Since you and Marlow were talking about the problems of adding full
>>continuations to a language with unwind-protect it makes a lot of sense
>>to compare with other languages having full continuations.
> Unfortunately, that other language doesn't have an UNWIND-PROTECT that
> allows full continuations to be used in more than a tiny fraction of toy
> code. Definitely not usable in library code.
Let's try to look at this from a new view point.
We now look at the unwind-protect in Common Lisp, which has the
following specification:
(unwind-protect protected-form cleanup-form ...) => result*
unwind-protect evaluates protected-form and guarantees that cleanup-forms
are executed before unwind-protect exits, whether it terminates normally
or is aborted by a control transfer of some kind. unwind-protect is
intended to be used to make sure that certain side effects take place
after the evaluation of protected-form.
If a non-local exit occurs during execution of cleanup-forms, no special
action is taken. The cleanup-forms of unwind-protect are not protected
by that unwind-protect.
unwind-protect protects against all attempts to exit from protected-form,
including go, handler-case, ignore-errors, restart-case, return-from,
throw, and with-simple-restart.
Now we play a game. We want to build a new language Common Lisp+ (i.e.
a language different from Common lisp), which has full continuations and
a form called unwind-protect+. The goal is that the form unwind-protect+
behaves "similar" to the old unwind-protect.
Let us analyze the situation first, before we try to extend unwind-protect
to the new langauge. Since unwind-protect is all about control transfers,
we need to examine how control behaves in CL and CL+.
In CL control obeys the stack discipline. That implies that the transfers
of control mentioned explicitly in the specification of unwind-protect,
such as throw, can be handled by evaluating the cleanup-forms and then
discarding the control stack up to the point specified by the unwind-protect.
When we add full continuations to CL we need to abandoned the stack model
of control. In stead we get we to talk about a control tree. This opens
the possibility of new kinds of transfers of control that weren't present
previously.
To extend unwind-protect to unwind-protect+ it is reasonable to expect that
unwind-protect+ behaves the same as unwind-protect, when the control
transfers are of the same type as in CL. In the tree model of control,
all control transfers originating from throw and friends are from
one node of the control tree to an ancestor node ("upward going").
I.e. (unwind-proctect+ protected-form cleaup-forms*) shall behave as
unwind-protect in the case where attempts to exit protected-forms
in an upwards-going fashion.
In the tree model other types of control transfer are possible. One can
jump from one branch to another ("sidewards"). Since this type of
control is not possible in CL, one can't use the definition of
unwind-protect ad verbatim to define unwind-protect+. One now has
a *choice* on how unwind-protect+ should behave. Should the
cleanup-forms be run or not?
If full continuation imply infinite extent, one also has to consider
the possibility of attempts to reenter the protected-form. Whether
or not to allow this is a *choice*.
If one wants to allow returns to the protected body, it makes
sense not to the cleanup-forms, when leaving sidewards - but
still running them, when leaving them upwards.
If one wants to disallow returns to the protected body, then
it makes sense to always run the cleanup-forms on all types
of exits.
The right choice might depend on the situtation, so perhaps
one should include variations of unwind-protect in CL+,
like unwind-protect/no-reentry, unwind-protect/renetry-allowed.
Since talk is cheap, the crucial question is: can the above
variations be implemented at all, and if they can, is it
costly?
The absolutely astonishing thing is that it is possible to define
*all* the above variations using call/cc alone. The burden on the
implementor is thus alone to implement call/cc - then he
can define the different variations of unwind-protect+ essentially
for free.
--
Jens Axel Søgaard
Peter Seibel
> If one wants to allow returns to the protected body, it makes sense
> not to the cleanup-forms, when leaving sidewards - but still running
> them, when leaving them upwards.
But what about when you return to the protected body after returning
upwards. That still seems like the one where it's hard to make sense
of the combination of unwind protect and call/cc. Hmmmm. How about
this, define a new operator, REWIND-PROTECT that runs a protection
form *before* the protected form *and* whenever the scope of the
REWIND-PROTECT's body is reentered as a result of invoking a
continuation captured with call/cc. Thus this:
(with-open-file (in file)
(do-stuff in))
would expand into something roughly like this:
(unwind-protect
(let ((in nil))
(rewind-protect
(setf in (open file)) ;; rewind form
(progn (do-stuff in))))
(close in))
Is that possible to implement? Is it just stupid? Inquiring minds want
to know.
-Peter
--
Peter Seibel pe...@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
Jens Axel Søgaard
I would say yes. It seems to me that your rewind-protect
corresponds to the before-thunk of dynamic-wind:
<http://schemers.org/Documents/Standards/R5RS/HTML/r5rs-Z-H-9.html#%_idx_576>
--
Jens Axel Søgaard
Rahul Jain
> (unwind-protect
> (let ((in nil))
> (rewind-protect
> (setf in (open file)) ;; rewind form
> (progn (do-stuff in))))
> (close in))
This is more-or-less dynamic-wind. The problem is that the file is
_re_-opened each time some part of the body is re-entered. What if you
have :if-exists :overwrite in the OPEN form? Then the file will keep
getting overwritten with just the tail end of the file instead of
rewinding the actual writing of the file back to where it was before the
continuation was invoked the last time. Your filesystem needs to know
about your continuations, too. The real problem is statefulness vs.
continuations. If you want the combination, the things which keep track
of state have to know about and react to your usage of continuations.
Jens Axel Søgaard
> The absolutely astonishing thing is that it is possible to define
> *all* the above variations using call/cc alone. The burden on the
> implementor is thus alone to implement call/cc - then he
> can define the different variations of unwind-protect+ essentially
> for free.
Make that "using call/cc and dynamic-wind".
--
Jens Axel Søgaard
Jens Axel Søgaard
> Peter Seibel <pe...@javamonkey.com> writes:
>
>
>> (unwind-protect
>> (let ((in nil))
>> (rewind-protect
>> (setf in (open file)) ;; rewind form
>> (progn (do-stuff in))))
>> (close in))
>
>
> This is more-or-less dynamic-wind. The problem is that the file is
> _re_-opened each time some part of the body is re-entered. What if you
> have :if-exists :overwrite in the OPEN form?
Putting a counter in the begin-thunk of dynamic-wind, one
can keep track of how many times one has entered the
dynamic extent of the dynamic-wind. Thus in an implementation
of, say, with-open-file one could simple ignore the :if-exists :overwrite
unless it was the first time the dynamic extent was entered.
Similar in the after-thunk one could remember the current position
in file, and then restore it in the begin-thunk.
--
Jens Axel Søgaard
Rahul Jain
> Putting a counter in the begin-thunk of dynamic-wind, one
> can keep track of how many times one has entered the
> dynamic extent of the dynamic-wind. Thus in an implementation
> of, say, with-open-file one could simple ignore the :if-exists :overwrite
> unless it was the first time the dynamic extent was entered.
>
> Similar in the after-thunk one could remember the current position
> in file, and then restore it in the begin-thunk.
Yes, As I corrected myself, the biggest burden is on the users of the
language, not on the implementors. Although who knows what
WITH-OPEN-FILE should be specified to do in the presence of full
continuations. You'd need various extra keyword options like
:allow-reentry, :if-exists-and-reentered-existing,
:if-not-exists-and-reentered-not-existing,
:if-not-existed-and-reentered-existing,
:if-existed-and-reentered-not-existing.
And you'd need a mode :rolling-back which would need to hook into when
the continuation was saved and roll back to that point in the file.
I just might prefer Intercal. At least it was meant to be a joke.
William D Clinger
> The fact of the matter is that
>
>
> Ok, maybe it's the burden on the language _users_ that's infinite, as
> your solution would make full continuations completely useless.
In previous posts, I predicted this response and also answered it.
> Actually, the implementations that use CPS transforming in the
> compiler wouldn't use them as full continuations, as that would
> make a stack discipline much more complex to implement.
Are you retracting your claim that two specific implementations
of Common Lisp already use full continuations?
> Wrong. Neither yours nor Kent's semantics are acceptable to me.
That's a pretty good endorsement of both.
> This (as corrected) is not Common Lisp's UNWIND-PROTECT. Not as
> a CL programmer would expect it to behave.
Rahul, I hope you don't think that's one of the ten smartest
things you've ever said.
> I don't know. Maybe Scheme programmers don't care if their code
> blows up at runtime. Some people have to write code for
> non-programmers to run.
Maybe the tooth fairy left Santa Claus under a pillow. Some
haystacks just don't have any needles. (I'm just demonstrating
to Rahul that he is not the only one who is proficient in this
mode of logic.)
> Interestingly enough, you started this discussion assuming I was
> Kent's mouthpiece. "What happens when we assume?" Perfect example.
I know Kent well enough to realize you would never be his mouthpiece.
I asked why you believed a thing. You said you believed that
thing because it was true. That didn't answer the question,
unless you're so special that you believe all true things.
Now that you admit the thing you believed is false, I'll repeat
my question: Why did you think that? Because it was false?
Will
William D Clinger
> What about the other way around - you're using your proposed
> UNWIND-PROTECT, and the third party software has a cooperative
> threading implementation using continuations.
That works fine, provided the cooperative threading is confined
within the dynamic extent of the UNWIND-PROTECT, or provided no
yield is performed within the UNWIND-PROTECT.
If neither of these conditions is true, then:
> You yield in the
> middle of WITH-OPEN-FILE and wonder why everything explodes.
But then you read the error message and realize that an attempt
had been made to throw back into an UNWIND-PROTECT that had been
exited. At that point, you either slap your head in consternation
over having failed to heed the warnings in the documentation for
the third-party software or, if there was no such warning, you go
looking for the buggy/malicious code that caused the problem.
> my conclusion would be either:
>
> a) Your simple unwind-protect is not compatible with all theoretical
> use cases of call-with-current-continuation, or
> b) call-with-current-continuation is not an acceptable construct to
> use to make a userland threading library.
or c) each userland threading library should redefine
CALL-WITH-CURRENT-CONTINUATION and/or DYNAMIC-WIND so that
my simple definition of UNWIND-PROTECT works correctly, and
the documentation for each threading library should explain
that this is being done and that the threading library cannot
be loaded dynamically into systems that have already captured
continuations or have set up wind/unwind forms using Scheme's
built-in definitions of those library procedures.
c) is a burden, but it should be compared with the burdens imposed
by other userland threading libraries that are not based on full
continuations.
> If a), then making an unwind-protect that allows the above requires
> a modified version of call-with-current-continuation so that the
> intent of a continuation can be communicated.
Agreed. In my view, the responsibility for installing the modified
version lies with the threading library.
Continuation-based userland threading or coroutining is about the
only "normal" use of full continuations that requires redefinition
of CALL-WITH-CURRENT-CONTINUATION and/or DYNAMIC-WIND, and that is
one of the main reasons it has gone out of vogue. Programmers,
with good reason, prefer to leave that kind of hacking to the
implementors of the language.
Will
William D Clinger
> > Control effects, like side effects, should be documented.
>
> But sometimes they are not.
We all know it is possible to write buggy software. The
documentation is part of the software, and can be buggy
by commission or omission.
The question is not whether it is possible to write buggy
software, but whether it is possible and practical to write
software that reliably performs UNWIND-PROTECT-like operations
in a language that also has full continuations.
Many people, over many years, have demonstrated that the answer
is "yes". My CL-style definition of UNWIND-PROTECT reliably
performs all UNWIND-PROTECT-like operations that can be
expressed in Common Lisp, while signalling a runtime error
when it detects a class of control effects that cannot occur
in Common Lisp.
If you wish to allow the control effects that my CL-style
definition of UNWIND-PROTECT protects against, then you can
use a Pitman-style definition of UNWIND-PROTECT. This style
is also implementable in portable Scheme, without having to
redefine any of Scheme's standard procedures.
If you wish to allow some but not all control effects that my
CL-style definition protects against, then you can design a
Pitman-style definition for the specific needs of your specific
program. This is seldom necessary.
> There are tools that I've seen written in Scheme that do
> backtracking, for example, by checkpointing the state at
> various points in ways that are not easily explained to users.
> In fact, the value of the tool is that it hides the control
> aspects and just tells you to rely on the magic of the
> abstraction.
Yes. For this to work, the tool may have to redefine one or two
of Scheme's standard procedures. The Scheme standards explicitly
allow this. In my view, any tool that purports to hide the
control aspects in some magic way should also assume the burden
of implementing that magic. The specific magic you mentioned
can be and has been implemented in portable Scheme.
> As I'm reading it, you SEEM to expect programs containing
> UNWIND-PROTECT to be case-marked in a way that says "maybe you
> don't want to go here".
No. I expect one specific control effect, namely escaping
continuations, to be documented. As I explain below, the
ordinary standards of documentation that are expected in
languages like Java already imply that escaping continuations
should be documented.
In my view, the purpose of UNWIND-PROTECT is to prevent certain
bad things from happening. In Common Lisp, UNWIND-PROTECT
prevents the body from performing a throw that bypasses the
unwind forms. My CL-style definition of UNWIND-PROTECT does
that, and also prevents the body from being continued after
the unwind forms have been executed.
Programmers who want to prevent other things as well, or to
allow some subset of the things that my CL-style definition
prevents, can implement whatever semantics they desire. If
that kind of power is too frightening, then the programmer
should use someone else's abstraction.
Scheme can and should be criticized for providing too few
predefined abstractions, but in this case it does provide
predefined abstractions that are powerful enough to build
most of the control abstractions that programmers would want,
and far more than could be implemented in most other languages.
> But I do find myself nervous about this matter.
Relax. We now have twenty years of experience with DYNAMIC-WIND
and its interactions with full continuations. We have learned,
sometimes the hard way, what works and what doesn't. Most of
the problems that you and others here have dreamed up are not
real. A few, mainly continuation-based multithreading,
coroutining, or backtracking, are real but can be, should be,
and are solved by the implementors of the abstraction, so
ordinary programmers don't have to worry about the control
aspects.
They still have to worry about the side effects, but that's
true in any such system, independent of full continuations.
For example, any backtracking system is going to require
programmers to guard against closing a file and then
backtracking into a computation that expects the file to
be open.
> I think you're right that control abstractions need to be
> documented, but CALL/CC is a control abstraction. So maybe
> you're saying that all uses of CALL/CC should be documented. ;)
I see the smiley wink. CALL-WITH-CURRENT-CONTINUATION and
DYNAMIC-WIND are control abstractions; they are documented
in the Scheme standards. Not all uses of control abstractions
are control effects that require documentation, just as not
all assignments are externally visible side effects that
require documentation.
Most uses of CALL-WITH-CURRENT-CONTINUATION are for escapes
and other things that cause no more trouble in Scheme than
in Common Lisp. The only uses that could possibly cause a
problem that does not arise in Common Lisp are uses in which
a continuation escapes its dynamic extent. Such escapes can
happen only when a continuation is (1) returned, (2) passed
to a procedure, (3) stored in a variable or structure whose
lifetime exceeds the continuation's dynamic extent, or (4)
becomes a component of some structure (e.g. list or closure)
that escapes.
Every externally visible procedure's arguments and result
should be documented, as should all externally visible side
effects. That already implies that all escaping continuations
should be documented.
> I wonder if other Schemers agree.
Some of the freshmen don't, but it's our job to teach them.
Will
Peter Seibel
> Yes. For this to work, the tool may have to redefine one or two of
> Scheme's standard procedures. The Scheme standards explicitly allow
> this. In my view, any tool that purports to hide the control aspects
> in some magic way should also assume the burden of implementing that
> magic. The specific magic you mentioned can be and has been
> implemented in portable Scheme.
Can you summarize roughly what the implementation of such magic would
entail? Which standard procedures would a library that, say,
implements user-land threads with continuations, have to reimplement
and how would the reimplemented versions work compared to the standard
ones.
Also, what happens if I have two such libraries. For instance, suppose
I'm writing an application using user-land threads and some of the
work I do in those threads involves calling into a library that uses
continuations for backtracking. Is that likely to be possible?
Rahul Jain
> Rahul Jain wrote more of the same, only better:
>> The fact of the matter is that
>>
>> > This is a classic example of changing the subject....
>>
>> Ok, maybe it's the burden on the language _users_ that's infinite, as
>> your solution would make full continuations completely useless.
>
> In previous posts, I predicted this response and also answered it.
What? That the point of a foundational operator in CL is to forbid the
use of full continuations? Did I miss something in all this
back-and-forth?
>> Actually, the implementations that use CPS transforming in the
>> compiler wouldn't use them as full continuations, as that would
>> make a stack discipline much more complex to implement.
>
> Are you retracting your claim that two specific implementations
> of Common Lisp already use full continuations?
They use full continuations in the same way that disallowing the use of
full continuations with UNWIND-PROTECT is an integration of the two. :)
(Yes, they don't use continuations fully because no control structure in
CL requires a function to return multiple times.)
>> Wrong. Neither yours nor Kent's semantics are acceptable to me.
>
> That's a pretty good endorsement of both.
I see. So now it's me who is the problem, not the incompatible semantics
you're trying to smash together.
>> This (as corrected) is not Common Lisp's UNWIND-PROTECT. Not as
>> a CL programmer would expect it to behave.
>
> Rahul, I hope you don't think that's one of the ten smartest
> things you've ever said.
Why? Do you really think that it's useful to have full continuations if
they result in an error in most real-world situations?
>> I don't know. Maybe Scheme programmers don't care if their code
>> blows up at runtime. Some people have to write code for
>> non-programmers to run.
>
> Maybe the tooth fairy left Santa Claus under a pillow. Some
> haystacks just don't have any needles. (I'm just demonstrating
> to Rahul that he is not the only one who is proficient in this
> mode of logic.)
OK, maybe you're just so obsessed with the perfection of full
continuations that you don't care if other people want simple, reliable,
resource-friendly code. Or maybe you could address the point at hand.
HOW will you actually make such semantics USEFUL? Or is the point that
you provide such semantics and then hit the programmer with a hammer
each time they use them? And then we have to say "If it hurts..."
>> Interestingly enough, you started this discussion assuming I was
>> Kent's mouthpiece. "What happens when we assume?" Perfect example.
>
> I know Kent well enough to realize you would never be his mouthpiece.
>
> I asked why you believed a thing. You said you believed that
> thing because it was true. That didn't answer the question,
> unless you're so special that you believe all true things.
>
> Now that you admit the thing you believed is false, I'll repeat
> my question: Why did you think that? Because it was false?
I've already said that the burden is on the programmer to come up with
useful semantics of all their code in the presence of re-entry into it.
But then all operators that define dynamically-allocated resources,
data, etc, will need to be modified to have options as to how they're
supposed to work in these cases, so that once the programmer figures out
semantics that work for their application, they can actually ask for
them. We'd need keyword args for LET bindings...
(let ((*foo* (compute-a-foo) :on-re-entry :re-evaluate)
(*bar* (compute-a-bar) :on-re-entry :re-bind)
(*baz* (compute-a-baz) :on-re-entry :error))
)
will be needed, off the top of my head. Of course, the :on-re-entry arg
won't make sense for lexically bound variables. Another example is the
one I gave for WITH-OPEN-FILE in another post.
What exactly would full continuations add to CL other than an explosion
of complexity in various operators designed to connect a program to the
real world? You've already said that using continuations directly by
programmers to implement multiprocessing isn't accepted as wise these
days. What other uses are there for functions that return multiple times
that CL doesn't already cover?
This will be hardest to deal with for novice programmers, exactly the
kind who will have the need for those options the most, as they'll be
using plenty of third party libraries. Do you expect to teach them how
full continuations work first and then teach them how to actually get
work done? CL is a language designed for programming, not for the
teaching of the various nooks and crannies of computer science to
students.
Robert Marlow
lines as you're talking now; incompatibility of unwind-protect and full
continuations is something that can be left to the user to watch out for.
I'm less certain now...
Your comparison with documenting side effects falls short of being a good
analogy with control effects. When a function in one library causes a side
effect on a dynamic variable for example, generally speaking that dynamic
variable is quarantined by the package system so it's fairly safe and
unlikely to conflict with dynamic variables in other libraries. I can see
more the analogy with file system side effects but usually naming
conventions of files modified without being undocumented are chosen to
make it pretty difficult for two separate libraries to conflict in their
use of files.
Control effects are a completely different ball game. With control effects
if a function makes an escape it can't be protected by packaging or
complex naming conventions, it necessarily effects everything between its
escape point and continuation whether it was expected or not. Documenting
it consequently isn't going to be much use; while documenting a
side-effect allows users to ensure they choose different names, the only
way to avoid the control effect's impact on your software is to avoid the
library. This makes any library using continuations in a way that
reentries and escapes cross user code vastly useless for any domain of
applications which will conflict with reentrant escapes. Since a library
writer generally wants to ensure qis library is useful in all situations a
library user feels appropriate it follows that these control effects would
necessarily need to be avoided in libraries to ensure users of unwind
protect and such won't be surprised or otherwise disadvantaged.
Am I right or wrong in my assumption that some classes of problems which
full continuations are useful for are most elegantly solved when crossing
user code with continuations is permitted? I'm thinking along the lines of
non-deterministic operators and such.
(qis: pron. gender neutral form of his or her)
Rahul Jain
> Your comparison with documenting side effects falls short of being a good
> analogy with control effects. When a function in one library causes a side
> effect on a dynamic variable for example, generally speaking that dynamic
> variable is quarantined by the package system so it's fairly safe and
> unlikely to conflict with dynamic variables in other libraries. I can see
> more the analogy with file system side effects but usually naming
> conventions of files modified without being undocumented are chosen to
> make it pretty difficult for two separate libraries to conflict in their
> use of files.
I think you're barking up the wrong tree here. These are side-effects
that _don't_ need to be documented. Ones that do need to be documented
are ones that affect shared state, such as the binding of
*standard-input* or the modification of a parameter passed in (e.g.,
SORT).
That said, the rest of your post addresses the problem at hand
accurately. You don't look at the documentation to find out if you need
to copy some data and then pass it or contain the operator with some
dynamic binding, etc. You simply look for uses of full continuations and
throw the library away if you need to use it interleaved with doing
anything needing unwind-protecting, such as file or network I/O or
anything that uses system-wide limited resources. How exactly do you
manage rollback of a network socket? You'd need to modify HTTP to allow
deletion of part of the document while it's being sent. :)
William D Clinger
> > Yes. For this to work, the tool may have to redefine one or two
> > of Scheme's standard procedures. The Scheme standards explicitly
> > allow this. In my view, any tool that purports to hide the control
> > aspects in some magic way should also assume the burden of
> > implementing that magic. The specific magic you mentioned can be
> > and has been implemented in portable Scheme.
>
> Can you summarize roughly what the implementation of such magic
> would entail?
Yes, but you'd get a more complete answer from
http://repository.readscheme.org/ftp/papers/sw2003/Threads.pdf
For even more complete answers, see the rather extensive bibliography
on the listed topics at
http://library.readscheme.org/page6.html
Continuations and Continuation Passing Style
http://library.readscheme.org/page8.html
Basic Techniques
http://library.readscheme.org/page9.html
Extension of Scheme for Parallel and Concurrent Programming
Threading and Parallel Programming with Continuations
Most everyone agrees that it's better for the implementors of
a Scheme system to implement the thread library at a more primitive
level, but it is still possible to implement a continuation-based
thread library in Scheme. To do this right, the implementors of
the thread library should think of themselves as the implementors
of a new Scheme system, while treating the underlying Scheme system
as the implementation language for that new system.
> Which standard procedures would a library that, say,
> implements user-land threads with continuations, have to
> reimplement and how would the reimplemented versions work compared
> to the standard ones.
CALL-WITH-CURRENT-CONTINUATION and DYNAMIC-WIND usually have to
be reimplemented. The implementor of the thread library would
capture and use the standard versions of these procedures to
implement the context-switching and other abstractions of the
thread system, but would hide the standard versions from clients.
For clients, the thread system would provide redefined versions
of these procedures. For example, the redefined version of
DYNAMIC-WIND would not run the wind/unwind forms on a thread
switch, because we don't want thread switches to interfere with
the use of DYNAMIC-WIND for UNWIND-PROTECT-like behavior. The
wind/unwind forms would run only when client code throws to a
continuation that was captured by the redefined version of
CALL-WITH-CURRENT-CONTINUATION.
> Also, what happens if I have two such libraries.
You probably don't want to run two different thread libraries in
the same system, but...
> For instance, suppose
> I'm writing an application using user-land threads and some of the
> work I do in those threads involves calling into a library that
> uses continuations for backtracking. Is that likely to be possible?
Yes. If the implementors of the thread library did their job,
then the redefined versions of CALL-WITH-CURRENT-CONTINUATION
and DYNAMIC-WIND should work as specified in the R5RS, so the
backtracking library can use them. The backtracking library
may also redefine those procedures, but that should work too.
Some caveats: The thread library will probably have to be
loaded/initialized first, in a relatively pristine Scheme system.
Then the backtracking library can be loaded/initialized in the
thread-aware system. All of the redefinitions of standard
procedures should be completed before client code begins to
execute.
Will
William D Clinger
> Am I right or wrong in my assumption that some classes of problems
> which full continuations are useful for are most elegantly solved
> when crossing user code with continuations is permitted? I'm
> thinking along the lines of non-deterministic operators and such.
I think you're right. Non-deterministic operators are a good
example. If implemented via backtracking, then programmers
are going to have to avoid closing a file and backtracking
into code that expects the file to be open. If implemented
via time-sliced threads, then programmers are going to have
to avoid race conditions and other complexities.
Fortunately, the implementors of backtracking and thread
systems can arrange for their control effects to occur at
a lower level than is visible using DYNAMIC-WIND. Please
see my response to Peter Seibel.
Will
William D Clinger
> What? That the point of a foundational operator in CL is to
> forbid the use of full continuations? Did I miss something
> in all this back-and-forth?
Evidently.
> Do you really think that it's useful to have full continuations
> if they result in an error in most real-world situations?
Do you really think it's useful to have newsgroups if most of
your posts are filled with errors?
(My question, like yours, is hypothetical.)
Will
Ron Garret
"William D Clinger" <cesur...@verizon.net> wrote:
> For clients, the thread system would provide redefined versions
> of these procedures. For example, the redefined version of
> DYNAMIC-WIND would not run the wind/unwind forms on a thread
> switch, because we don't want thread switches to interfere with
> the use of DYNAMIC-WIND for UNWIND-PROTECT-like behavior. The
> wind/unwind forms would run only when client code throws to a
> continuation that was captured by the redefined version of
> CALL-WITH-CURRENT-CONTINUATION.
What happens (in terms of winding/unwinding) if one thread invokes a
continuation that was created in a different thread?
rg
Robert Marlow
I can't pick which response to Peter you're talking about. If you're
talking about his suggestion of using unwind-protect in a dynamic-wind
sort of way to ensure files are opened and closed as continuations are
entered and escaped then this is trivially an incorrect solution as
evidenced by Rahul's network socket example.
If you're talking about redefinition of standard scheme procedures to
cater for control structures, how can you redefine such that users can
happily use libraries using unwind-protect and full continuations without
ever having to worry about errors relating to their use together?
If you're talking about another response altogether, please clarify.
If unwind-protect solutions in portable scheme are so elegant, why isn't
there a with-open-file equivalent in scheme? Instead there's
call-with-input-file which doesn't guarantee the file is closed and
with-input-from-file which looks like it may or may not succeed in
intelligently closing the file depending on the implementation (despite
solid solutions allegedly being available portably).
William D Clinger
> What happens (in terms of winding/unwinding) if one thread invokes a
> continuation that was created in a different thread?
That should work fine, but the responsibility for making
it work lies with the implementors of the thread system.
Not all implementors have gotten this right. There is
an excellent discussion of this in section 3 of the
Gasbichler et al paper I cited in a previous post:
http://repository.readscheme.org/ftp/papers/sw2003/Threads.pdf
Will
William D Clinger
> I can't pick which response to Peter you're talking about.
The one with the URLs for papers at http://readscheme.org/
> If you're talking about redefinition of standard scheme procedures
> to cater for control structures, how can you redefine such that
> users can happily use libraries using unwind-protect and full
> continuations without ever having to worry about errors relating
> to their use together?
I said a few words about this in my response to Peter.
I could go into more detail, but I think it makes more
sense to refer you to the paper by Gasbichler et al
that I cited in my response to Peter:
http://repository.readscheme.org/ftp/papers/sw2003/Threads.pdf
That paper is an excellent summary of the state of the
art. For more information on specific issues, you can
track down the references cited in the relevant sections
of that paper.
> If unwind-protect solutions in portable scheme are so elegant,
> why isn't there a with-open-file equivalent in scheme? Instead
> there's call-with-input-file which doesn't guarantee the file
> is closed and with-input-from-file which looks like it may or
> may not succeed in intelligently closing the file depending on
> the implementation (despite solid solutions allegedly being
> available portably).
No one is happy with Scheme's standard i/o procedures.
Gary Brooks and I designed them in 1984, long before
DYNAMIC-WIND was added to the language. The problems
you identify came about because several MIT people
argued for the MIT semantics, in which a file would be
closed during the unwind, and reopened at the same place
on the wind. Gary and I were skeptical, but the MIT
people had more implementation experience and a much
larger user base at that time, so we compromised with
a spec that allowed the MIT semantics (for the WITH-
constructs) without requiring it (for either set).
The MIT people later concluded they were wrong, but by
then it was very difficult for the Scheme authors to
approve any changes to the language. Change required
consensus, which was interpreted as unanimity. Imagine
trying to get Kent Pitman and me to agree on a proposed
change, and then imagine having to get the agreement
from twenty other strong-minded and opinionated people.
More language problems went unfixed because of this
political problem than went unfixed because of technical
issues.
In that political climate, this particular i/o problem
was regarded as relatively low priority because the more
sensible behavior could be synthesized using DYNAMIC-WIND
and the lowest-level OPEN-{INPUT,OUTPUT}-FILE and
CLOSE-{INPUT,OUTPUT}-PORT. The fact that there was no
portable way to delete a file, and therefore no reliable
way to open an output file that might already exist, was
an even more serious problem, and we were never able to
fix even that.
These are the kinds of political problems that killed the
"Scheme authors" process, and darn near killed Scheme as
well.
Will
Jens Axel Søgaard
As a concrete example of what happens in an implementation
with threads and full continuations, one can see what
happens in Gambit:
<http://www.iro.umontreal.ca/~gambit/doc/gambit-c_12.html>
(The section in the manual explaining the interaction between
threads, the dynamic environment and continuations)
--
Jens Axel Søgaard
From votaru
Rahul Jain <rj...@nyct.net> writes:
> You don't look at the documentation to find out if you need
> to copy some data and then pass it or contain the operator with some
> dynamic binding, etc. You simply look for uses of full continuations and
> throw the library away if you need to use it interleaved with doing
> anything needing unwind-protecting, such as file or network I/O or
> anything that uses system-wide limited resources. How exactly do you
> manage rollback of a network socket? You'd need to modify HTTP to allow
> deletion of part of the document while it's being sent. :)
I guess there, may be a way for continuations to integrate smoothly with
unwind-protect, but this one still lies in the path of "infinite
burden".
I'll give an example. I was playing recently with gtk-server (from Lisp)
(www.gtk-server.org). You start the server, and send him strings like
this, "gtk_window_new 0", "gtk_label_new \"Hello\"" "gtk_widget_show
32038402", where 32038402 is wigdet-id and so on. If something goes
wrong, you may need to kill the server manually. A with-gtk-server macro
using unwind-protect will do this for you.
But how will this interact with continuations? If you leave the
protected form with the intent of returning there later, there is no
point of returning if the cleanup form has already run. There is no more
gtk-server running. (And the kludge propossed for with-open-file,
re-open the file is not feasible here). As soon as the cleanup form has
run, the state you were hoping to preserve is lost.
IMRHO, the solution is (deceiptively) simple. If you have multiple exits from
the protected form, the cleanup form should be run only for the *last*
exit. Be it normal or exceptional. Well, this would by really *cool*.
But how would the /poor/ implementation :) know that this is last jump
out, and there would not be another jump in?
It may be doable along the lines of whole program analysis and runtime
checks. Not trivial, at the least. But being neither a Lisp implementor
not call/cc hacker I'll rather shup up :).
P.S.
(with-brainstorming
Is there anything like runtime dead code elimination? In this case the
cleanup forms may be run when the "unwind-protect"-s will be garbage
collected.)
--
An ideal Lisp is left as an exersize to the reader.
William D Clinger
> P.S.
> (with-brainstorming
> Is there anything like runtime dead code elimination? In this case
> the cleanup forms may be run when the "unwind-protect"-s will be
> garbage collected.)
#t
You're asking for reliable finalization. There is a fairly
large body of literature on this subject. One of the best
algorithms was developed by the implementors of Chez Scheme:
# R. Kent Dybvig, David Eby and Carl Bruggeman.
"Guardians in a Generation-based Collector".
ACM SIGPLAN 1993 Conference on Programming
Language Design and Implementation. June 1993.
Available online:
http://www.cs.indiana.edu/~dyb/papers/guardians-pldi93.ps.gz
See also the documentation of MAKE-WILL, WILL?, WILL-TESTATOR,
and WILL-EXECUTE! in Gambit:
http://www.iro.umontreal.ca/~gambit/doc/gambit-c_6.html
I am told that the names of these procedures were in part a
pun on my first name.
Will
Marcin 'Qrczak' Kowalczyk
>> (with-brainstorming
>> Is there anything like runtime dead code elimination? In this case
>> the cleanup forms may be run when the "unwind-protect"-s will be
>> garbage collected.)
>
> #t
>
> You're asking for reliable finalization.
IMHO it's not appropriate. unwind-protect is supposed to be run
as soon as the code using the resource finishes, not when the GC
discovers that something is unreachable.
It's not possible to detect whether the given use of a continuation is
the last time. And it will never be, because in perverse cases it can
*depend* on whether the unwind-protect has been run, such that
unwind-protect was always wrong.
The programmer should explicitly specify the intent.
> # R. Kent Dybvig, David Eby and Carl Bruggeman.
> "Guardians in a Generation-based Collector".
> ACM SIGPLAN 1993 Conference on Programming
> Language Design and Implementation. June 1993.
> Available online:
>
> http://www.cs.indiana.edu/~dyb/papers/guardians-pldi93.ps.gz
This is not what I would expect from finalizers, because guardians
must be explicitly invoked from time to time to check which objects
are finalizable.
Perhaps they are often enough for managing scarce resources, because
you can insert a cleaning code in each place before such resource is
obtained. But this requires finding and augmenting all such places.
And it delays releasing the resource if the program doesn't allocate
a different instance of such resource soon.
> See also the documentation of MAKE-WILL, WILL?, WILL-TESTATOR,
> and WILL-EXECUTE! in Gambit:
>
> http://www.iro.umontreal.ca/~gambit/doc/gambit-c_6.html
This fails to notice that reliable asynchronous finalizers must be run
in a separate thread.
http://www.hpl.hp.com/techreports/2002/HPL-2002-335.html
--
__("< Marcin Kowalczyk
\__/ qrc...@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/
William D Clinger
newsgroup for the restraint and respect they have shown here.
Lispers used to take pride in their concern for facts over
personalities, and I was pleased to see that spirit prevail
among most who contributed to this thread.
Marcin 'Qrczak' Kowalczyk quoting me addressing someone else:
> > You're asking for reliable finalization.
>
> IMHO it's not appropriate. unwind-protect is supposed to be run
> as soon as the code using the resource finishes, not when the GC
> discovers that something is unreachable.
I agree that finalization and UNWIND-PROTECT are quite different
things. My interpretation of the post to which I was responding
is that the person really wanted finalization, not UNWIND-PROTECT.
Perhaps I was wrong.
> This is not what I would expect from finalizers, because guardians
> must be explicitly invoked from time to time to check which objects
> are finalizable.
In modern systems, that is not much of a burden. A low-priority
thread takes care of it.
> > See also the documentation of MAKE-WILL, WILL?, WILL-TESTATOR,
> > and WILL-EXECUTE! in Gambit:
> >
> > http://www.iro.umontreal.ca/~gambit/doc/gambit-c_6.html
>
> This fails to notice that reliable asynchronous finalizers must be
> run in a separate thread.
I discovered that the hard way in 1986, with MacScheme+Toolsmith.
Fortunately it was easy to arrange for finalizers to run in a
separate thread.
Will
Hannah Schroeter
John Thingstad <john.th...@chello.no> wrote:
>[...]
>Seems to me that depends on the threading model.
>Lightweight threads (sharing process and memory) shouldn't have
>much of a overhead compared to continuations.
OS threads under e.g. Linux cost at least 1 page of thread-owned
userlevel stack and at least 1 page (dunno about how many Linux
actually needs, but 1 is the theoretical minimum) of kernel level
stack, in addition to the kernel level thread/process descriptor.
That's much compared to e.g. the memory overhead (3-figure measured
in *bytes*) of threads in Erlang, or SML/NJ (which uses continuations
internally for the implementation of threads, but they're preemptive
still, in a way that the compiler ensures that a possible yield point
is reached once every *short* while, and using things like setitimer
to set a flag for the thread scheduler to be called from the next
such yield point reached, at least if this is still valid, I remember
a description like that from Compiling With Continuations by Andrew
Appel).
>[...]
Kind regards,
Hannah.
Peter Seibel
Can threads in Erland and/or SML/NJ run concurrently on multiple CPUs?
Hannah Schroeter
Peter Seibel <pe...@javamonkey.com> wrote:
>[...]
>Can threads in Erland and/or SML/NJ run concurrently on multiple CPUs?
IIRC no, but many uses for threads in Erlang aren't so much CPU bound,
but to multiplex highly concurrent I/O activities in a natural fashion.
And I'd think it'd be a possibility to use a n:m implementation model
rather than the usually (alas) favoured 1:1 ones, so you could create
one (more costly) OS thread per CPU, but schedule your programming
language level threads inside them using some mechanism of really
cheap concurrency.
Kind regards,
Hannah.
Adrian Kubala
>>Can threads in Erlang and/or SML/NJ run concurrently on multiple CPUs?
> IIRC no, but many uses for threads in Erlang aren't so much CPU bound,
> but to multiplex highly concurrent I/O activities in a natural fashion.
Since Erlang was designed for distributed concurrency, you can just run
one Erlang VM per CPU and distribute your processes among VMs as you
like. But of course this isn't quite as efficient as a VM able to do SMP
on its own.
Robert Marlow
I've been thinking about the alleged incompatibility between full
continuations and unwind-protect and I think I've figured it out
(apologies if I'm the last one here to grasp this concept, just posting
this in case others haven't yet figured it out).
The key to compatibility between unwind-protect and full continuations is
that CL style unwind-protect only expects to protect against a specific
class of continuations - those which will never reenter. Luckily these
continuations are clearly defined: go, throw, return-from, errors and any
others I've missed. So the solution is simple - simply define these
operators and unwind-protect such that they play nicely with each other
and perform as expected, and leave full continuations out of the picture
altogether. In other words, treat unwind-protect and friends as they are -
a special type of escape which never reenters.
I imagine when a CL user or scheme user or any other user uses full
continuations from within unwind-protect qe doesn't expect to reenter and
find the cleanup forms have already been run or an error thrown at them.
In general a user of full continuations wants to decide for themselves
whether such events happen, so they use dynamic-wind (as evidenced by the
fact that this is precisely what scheme users do in the absence of
unwind-protect). So unwind-protect should never affect use of full
continuations, and should only affect the specific subdomain of
continuations above. A form such as with-cleanup could also be defined
which handles all the bookkeeping associated with ensuring unwind-protect
forms are cleaned up to make it easy for users to define their own
variations of throws and return-froms.
Does my perspective make these incompatibilities disappear or have I
missed something?
I wrote some code to test my thoughts and as an example:
(define *clean-up* #f)
(define-macro (unwind-protect thunk . cleanup)
`(dynamic-wind
(lambda () #f)
,thunk
(lambda () (if *clean-up*
(begin (set! *clean-up* #f)
,@cleanup)))))
(define-macro (with-cleanup . body)
`(begin
(set! *clean-up* #t)
,@body))
(define *throw-tags* '())
(define (new-catch tag thunk)
(call-with-current-continuation
(lambda (k)
(set! *throw-tags* (cons (cons tag k) *throw-tags*))
(thunk))))
(define (new-throw tag . args)
(with-cleanup
(let ((catcher (assoc tag *throw-tags*)))
(if catcher
(begin
(set! *throw-tags* (delete! catcher *throw-tags*))
(apply (cdr catcher) args))
(error "No such tag")))))
;;; Here we use throw so the unwind-protect cleanup form should run
(define (foo)
(unwind-protect
(lambda ()
(new-throw 'bar 'bar))
(display 'foo)
(newline)))
(define (bar thunk)
(display
(new-catch 'bar (lambda () (thunk))))
(newline))
;;; here we're escaping without using throw or with-cleanup
;;; so the cleanup form should be left alone
(define (baz)
(unwind-protect
(lambda ()
((cdr (assoc 'bar *throw-tags*)) 'bar))
(display 'baz)))
> (bar foo)
foo
bar
>(bar baz)
bar
vrotaru
> I've been thinking about the alleged incompatibility between full
> continuations and unwind-protect and I think I've figured it out
> (apologies if I'm the last one here to grasp this concept, just posting
> this in case others haven't yet figured it out).
Me too :)
> The key to compatibility between unwind-protect and full continuations is
> that CL style unwind-protect only expects to protect against a specific
> class of continuations - those which will never reenter. Luckily these
> continuations are clearly defined: go, throw, return-from, errors and any
> others I've missed. So the solution is simple - simply define these
> operators and unwind-protect such that they play nicely with each other
> and perform as expected, and leave full continuations out of the picture
> altogether. In other words, treat unwind-protect and friends as they are -
> a special type of escape which never reenters.
I'm afraid that's not that easy. Pseudocode follows
(with-log-file (..)
(log-coroutine ..))
(define-coroutine log-coroutine (..)
(loop
(write-sensor-data ..)
(transfer-control sensor-coroutine)))
and somewhere in the sensor-coroutine you have
(if (not (full-phase-of-moon ...))
(transfer-control log-coroutine)
(happily-go-to-dance-elsewhere ..))
Obviously the cleanup forms for with-log-file should only be run if
the second posibility is chosen, before calling
happily-go-to-dance-elsewhere. To make things worse, these cleanup forms
should be run only if the control has been transferred to
sensor-coroutine from within a with-log-file. Tricky.
To put it shortly. As soon you call call/cc from within a protected
form, the continuation of this form escapes from unwind-protect, and no
one can know beforehand whether it would be ever called again, and if so
how many times.
(with-usual-disclaimer
This is how I undestand it.)
--
If in doubt, argue.
William D Clinger
> I'm afraid that's not that easy....
> --
> If in doubt, argue.
I suggest you go to http://scholar.google.com/ and run
a search for delimited continuations. Several of the
more pragmatic papers on this subject are listed at the
readscheme.org site, which I cited a while ago.
Delimited continuations can be implemented in Scheme
using the techniques described previously in this thread.
Will
vrotaru
> vrotaru wrote:
> > I'm afraid that's not that easy....
> > --
> > If in doubt, argue.
I knew I was asking for trouble... :)
>
> I suggest you go to http://scholar.google.com/ and run
> a search for delimited continuations. Several of the
> more pragmatic papers on this subject are listed at the
> readscheme.org site, which I cited a while ago.
>
I will do that search, but... Does anyone here have an implementation of
a with-48-hours-day macro available?
> Delimited continuations can be implemented in Scheme
^^^^^^^^^
> using the techniques described previously in this thread.
>
> Will
>
It just me, or is this a general sentiment that mixing full
^^^^
continuations and unwind-protect is bad idea?
William D Clinger
> > Delimited continuations can be implemented in Scheme
> ^^^^^^^^^
> > using the techniques described previously in this thread.
>
> > Will
>
> It just me, or is this a general sentiment that mixing full
> ^^^^
> continuations and unwind-protect is bad idea?
Despite the names, delimited continuations have strictly more
expressive power than full continuations. That theorem applies
to purely functional languages, however. Filinski showed that
delimited continuations are equivalent to full continuations
plus one mutable cell; hence Scheme's full continuations also
provide the power associated with delimited continuations.
Don't worry, I wouldn't have expected you to know any of this.
Will
Robert Marlow
> (with-log-file (..)
> (log-coroutine ..))
>
> (define-coroutine log-coroutine (..)
> (loop
> (write-sensor-data ..)
> (transfer-control sensor-coroutine)))
>
>
> and somewhere in the sensor-coroutine you have
>
> (if (not (full-phase-of-moon ...))
> (transfer-control log-coroutine)
> (happily-go-to-dance-elsewhere ..))
>
> Obviously the cleanup forms for with-log-file should only be run if
> the second posibility is chosen, before calling
> happily-go-to-dance-elsewhere. To make things worse, these cleanup forms
> should be run only if the control has been transferred to
> sensor-coroutine from within a with-log-file. Tricky.
>
> To put it shortly. As soon you call call/cc from within a protected
> form, the continuation of this form escapes from unwind-protect, and no
> one can know beforehand whether it would be ever called again, and if so
> how many times.
>
> (with-usual-disclaimer
> This is how I undestand it.)
So why not use unwind-protect to cleanup on non-reentering escapes in
with-log-file and then call the cleanup calls as a special case prior to
running happily-go-to-dance-elsewhere? It seems to me that your example is
more a problem for using full continuations than any compatibility with
unwind-protect - users of scheme trying to clean up would be faced with
the same sorts of questions. That's kind of my point; if you use
continuations you'll have to think about the same questions a schemer
would in such situations, if you use unwind protect you'll have to think
about the same questions a common-lisper would in such situations. Using
my suggestions there's no point (that I've yet thought of) where using
unwind-protect and continuations together create additional questions. So
there's no incompatibilities.
Rahul Jain
> But how would the /poor/ implementation :) know that this is last jump
> out, and there would not be another jump in?
>
> It may be doable along the lines of whole program analysis and runtime
> checks. Not trivial, at the least. But being neither a Lisp implementor
> not call/cc hacker I'll rather shup up :).
>
> P.S.
> (with-brainstorming
> Is there anything like runtime dead code elimination? In this case the
> cleanup forms may be run when the "unwind-protect"-s will be garbage
> collected.)
You're talking about finalizers. These have nothing to do with
unwind-protect, and are intended to be used mostly for finding bugs as
well as making sure that such bugs do not compromise the usability of
the application permanently or possibly the OS (maybe just for that
user) for the duration of the application's execution.
Rahul Jain
> I agree that finalization and UNWIND-PROTECT are quite different
> things. My interpretation of the post to which I was responding
> is that the person really wanted finalization, not UNWIND-PROTECT.
> Perhaps I was wrong.
I interpreted it as having a similar tone to one of my posts (at least
it resonated with me) in indicating that finalization is the reliable
way to ensure that _some_ uses of unwind-protect result in the
programmer's intended semantics. Unfortunately, this is sometimes not
what the programmer intends for some I/O-related uses and sometimes
needs quite a bit more functionality around it for other cases.
I remember a post a while ago where all sockets, fds, etc were closed
only when garbage collected, but the GC would run when trying and
failing to acquire such a resource. Unfortunately, OS-wide limits would
cause such a system to make other applications to fail.
Ray Dillinger
> Marcin 'Qrczak' Kowalczyk quoting me addressing someone else:
>
>>>You're asking for reliable finalization.
>>
>>IMHO it's not appropriate. unwind-protect is supposed to be run
>>as soon as the code using the resource finishes, not when the GC
>>discovers that something is unreachable.
>
>
> I agree that finalization and UNWIND-PROTECT are quite different
> things. My interpretation of the post to which I was responding
> is that the person really wanted finalization, not UNWIND-PROTECT.
> Perhaps I was wrong.
That's pretty much the conclusion I reached as well. Finalization
is what the system does when it proves it can't reenter the context
again. Unwind-protect is just what it does when it's leaving.
They are different contracts, and finalizing actions shouldn't be
in an unwind-protect, nor expected to run instantly unless you
explicitly *tell* the system that a particular exit is the last
exit.
>>This is not what I would expect from finalizers, because guardians
>>must be explicitly invoked from time to time to check which objects
>>are finalizable.
Not necessarily. I think the garbage collector can be made to
handle finalizers correctly (although not immediately)
>>> http://www.iro.umontreal.ca/~gambit/doc/gambit-c_6.html
>>
>>This fails to notice that reliable asynchronous finalizers must be
>>run in a separate thread.
>
>
> I discovered that the hard way in 1986, with MacScheme+Toolsmith.
> Fortunately it was easy to arrange for finalizers to run in a
> separate thread.
Could you elaborate on this need please? I would think that if
your finalizer does something that requires it to be in a separate
thread, that would tend to indicate a design problem with what you
are using finalization for. What am I missing?
Bear
Ray Dillinger
> I remember a post a while ago where all sockets, fds, etc were closed
> only when garbage collected, but the GC would run when trying and
> failing to acquire such a resource. Unfortunately, OS-wide limits would
> cause such a system to make other applications to fail.
>
Not necessarily - or at least, no more so than memory itself. Think
about A GC'd application that allocates memory (including virtual
memory) until there is NO MORE, before running the garbage collector.
It's rude in exactly the same way you're thinking about sockets,
file handles, etc, and other applications crash because they're
unable to allocate.
But if your GC'd application does what most, in practice, do, and
limits its arena to some "reasonable" size, other applications can
run just fine.
The same basic rule applies; If you're going to manage these other
resources by a GC, you have to limit your allocation appropriately
so that other applications can get what they need.
With disk file handles, at any rate, it's frequently possible
to save enough information to reopen and seek the same place.
Then you can close the file, freeing up a handle for other
use, and reopen it next time you actually need to read/write
something in that file. Thus, an application using scores of
disk files simultaneously might get by with an aware GC and an
"allocation arena" of just a half-dozen file handles.
Bear
William D Clinger
> Could you elaborate on this need please? I would think that if
> your finalizer does something that requires it to be in a separate
> thread, that would tend to indicate a design problem with what you
> are using finalization for.
I won't argue with that!
Here's an example from my experience: a GUI object x
acquires some OS resources and spawns a low-priority
thread that polls those resources. When x becomes
unreachable, its finalizer needs to kill the thread
and release the resources, in that order. If the
finalizer happens to run in the thread it kills,
then it won't get around to releasing the resources.
More generally, the correctness of many threaded apps
depends upon invariants of the form "no code executed
by this thread ever does such-and-such". If finalizers
run in arbitrary threads, then those invariants cannot
be established without global knowledge of all finalizers.
Will
Marcin 'Qrczak' Kowalczyk
>>>This fails to notice that reliable asynchronous finalizers must be
>>>run in a separate thread.
>>
>> I discovered that the hard way in 1986, with MacScheme+Toolsmith.
>> Fortunately it was easy to arrange for finalizers to run in a
>> separate thread.
>
> Could you elaborate on this need please?
http://www.hpl.hp.com/techreports/2002/HPL-2002-335.html
Section 3.5
> I would think that if your finalizer does something that requires
> it to be in a separate thread, that would tend to indicate a design
> problem with what you are using finalization for.
On the contrary. If the finalizer does anything useful besides I/O,
it means that it changes objects reachable from the main program
(if it changes only unreachable objects, the changes are not
observable and thus useless). GC may happen at virtually any point,
including in the middle of a non-atomic access of these objects.
The finalizer is asynchronous with respect to the code, so they need
locks for mutual exclusion and they can't be run from the same thread
(otherwise we have an error, deadlock or data corruption, depending
on the semantics of recursive mutexes).
Rahul Jain
> With disk file handles, at any rate, it's frequently possible
> to save enough information to reopen and seek the same place.
> Then you can close the file, freeing up a handle for other
> use, and reopen it next time you actually need to read/write
> something in that file. Thus, an application using scores of
> disk files simultaneously might get by with an aware GC and an
> "allocation arena" of just a half-dozen file handles.
Hmm, interesting idea. You could grow the file handle arena as needed,
as well, in the same way as the heap is grown as needed.
However, there is the issue of someone else writing to the file between
different "windings" of your code path. (Of course, this is irrelevant
on some OSes, but may be important to some applications on OSes that
support file locking.)
I still don't like the idea, right now, because OSes are ill-equipped to
handle such problems (OOM killers exist, at least in concept, in the
minds of OS writers -- they still haven't figured out what criteria are
actually salient to the problem they are trying to solve, last I
checked).
Also, the space of memory pages is far larger than the space of file
handles or sockets on the typical OS configuration. Especially file
handles can be exhausted unintentionally if all applications used a GC
scheme to manage them. Of course, the ideal would be that the GCs in all
applications could be triggered by scarcity or pressure on the resources
(be they memory, file handles, sockets, whatever) managed by the GC. Too
bad "modern" OS writers don't care much for providing that kind of
information to applications.
Ray Dillinger
> Ray Dillinger <be...@sonic.net> writes:
>
>
>>With disk file handles, at any rate, it's frequently possible
>>to save enough information to reopen and seek the same place.
>>Then you can close the file, freeing up a handle for other
>>use, and reopen it next time you actually need to read/write
>>something in that file. Thus, an application using scores of
>>disk files simultaneously might get by with an aware GC and an
>>"allocation arena" of just a half-dozen file handles.
>
>
> Hmm, interesting idea. You could grow the file handle arena as needed,
> as well, in the same way as the heap is grown as needed.
>
> However, there is the issue of someone else writing to the file between
> different "windings" of your code path. (Of course, this is irrelevant
> on some OSes, but may be important to some applications on OSes that
> support file locking.)
Yup. I would consider that profoundly rude, but we're
supposed to assume every other program on the system is
crafted by the devil himself for the express purpose of
causing yours to crash, so it's a reasonable concern.
Hmmm. I suppose you could detect it if someone has
written the file since you last had it open, but then
what exactly should you do? Halting with an error is
better than silently proceeding to do something wrong,
but not by much.
> Also, the space of memory pages is far larger than the space of file
> handles or sockets on the typical OS configuration. Especially file
> handles can be exhausted unintentionally if all applications used a GC
> scheme to manage them. Of course, the ideal would be that the GCs in all
> applications could be triggered by scarcity or pressure on the resources
> (be they memory, file handles, sockets, whatever) managed by the GC. Too
> bad "modern" OS writers don't care much for providing that kind of
> information to applications.
We can trigger GC on resource exhaustion - when attempting
to open a file fails for lack of handles - but you're right
that that isn't really adequate for a bunch of applications
each doing their own independent handle management.
And I think that this is a strong case for Garbage
Collection as an OS service rather than a subthread
of individual programs. Done properly as an OS service
(in a Lispy OS) it could be a lot easier to keep track
of issues such as "pressure" on resources short of
exhaustion.
Bear
Ray Dillinger
> Ray Dillinger <be...@sonic.net> writes:
>>I would think that if your finalizer does something that requires
>>it to be in a separate thread, that would tend to indicate a design
>>problem with what you are using finalization for.
>
>
> On the contrary. If the finalizer does anything useful besides I/O,
> it means that it changes objects reachable from the main program
> (if it changes only unreachable objects, the changes are not
> observable and thus useless). GC may happen at virtually any point,
> including in the middle of a non-atomic access of these objects.
>
> The finalizer is asynchronous with respect to the code, so they need
> locks for mutual exclusion and they can't be run from the same thread
> (otherwise we have an error, deadlock or data corruption, depending
> on the semantics of recursive mutexes).
Hmmm. Okay... in the first place, I don't have a problem with
a single (OS) threaded application (which may have multiple
cooperative threads inside itself) invoking finalizers from the
garbage collector. There is no need for a separate thread in
this case since the finalizer will be the only thread active
at that instant, anyway. In cooperative multithreading, you
get to pick the moments at which you trade off from one to the
other, and you can trade off when the GC gets back. And
since most machines are *STILL* single-processor, there's no
performance gain from increasing the number of OS threads
anyway on most machines.
In the second place, if you use a Lisp with general continuations
(which I do), cooperative multithreading works and OS-level
multithreading creates hairy problems with your environment.
So I do tend to assume that threads are cooperative and serially
scheduled rather than preemptive and potentially simultaneous. I
realize this is not your assumption, and given that, I understand
how the issue works. Tomayto, tomawto, etc.
In the third place, there is lots of useful stuff for
finalizers to do that doesn't involve either I/O or reachable
objects. My canonical use of finalizers is to release operating
system resources (icons, handles, etc) bound up with the
objects being reaped. If you ensure that the GC runs when
threads are blocked waiting for an OS resource, this is a
way to have access to a lot of stuff that otherwise would
trip up your lisp system and get it in trouble with the OS.
And finally, I still can't think of a use for finalizers
that touches non-garbage objects and doesn't reflect either
a design problem or a workaround for a problem that arises
because of non-cooperative multithreading.
I guess it's my considered opinion that preemptive multitasking
is for allocating time between programs, and when allocating
time between threads within a program, cooperative is, at
least until SMP becomes common on the desktop, better.
Bear
Marcin 'Qrczak' Kowalczyk
> Hmmm. Okay... in the first place, I don't have a problem with
> a single (OS) threaded application (which may have multiple
> cooperative threads inside itself) invoking finalizers from the
> garbage collector. There is no need for a separate thread in
> this case since the finalizer will be the only thread active
> at that instant, anyway.
Yes, cooperative multithreading is enough for finalizers, as long as
from time to time something nevertheless triggers a switch to the
finalizing thread even without the main thread being blocked.
> In the second place, if you use a Lisp with general continuations
> (which I do), cooperative multithreading works and OS-level
> multithreading creates hairy problems with your environment.
People working on Haskell have designed and implemented an interesting
hybrid between green threads and OS threads:
Simon Marlow, Simon Peyton Jones, Wolfgang Thaller
"Extending the Haskell Foreign Function Interface with Concurrency"
I took their design and in recent days implemented it for my language
Kogut. I believe it gives benefits over pure cooperative multithreading
or green threads, and over using OS threads directly.
The main problem with living in a single OS thread is poor integration
with certain kinds of libraries implemented in other languages, which
can only be partially solved by non-portable hackery of stack switching.
If a function written in a foreign language takes long to complete,
or perhaps waits for something indefinitely, other green threads
aren't running.
If the implementation attempts to solve this by introducing OS threads
somewhere, the programmer must be able to specify that certain related
calls are performed by the same OS thread, because for some libraries
it matters from which OS threads their functions are called.
And if a single OS thread with a single system stack is used for
multiplexing threads of our language, it is possible that the program
will make calls to C functions which call back to our language and
ask to return from the C functions out of the LIFO order. This is
unimplementable portably. Forcing the threads to wait until the right
callback returns may help or may cause a deadlock.
Of course pure OS threads also have disadvantages:
They are expensive. The program doesn't scale to a large number of
threads.
Implementing the runtime and the garbage collector in a reentrant way
is a hard challenge (without excessive synchronization that is). Many
runtimes which map their threads 1-1 to OS threads actually serialize
execution of the high level language, and allow only some C calls
which block to be parallelized (e.g. in OCaml and Python).
Some used libraries written in foreign languages might not be thread-safe.
The mentioned hybrid design runs some threads on a smaller pool of OS
threads, and binds other threads to fixed OS threads. Both the Haskell
implementation and mine serialize access to the runtime, but context
switching is costly only when execution actually changes the OS thread.
If the program spawns 10000 threads which only do I/O and don't call C,
they will be handled by a single OS thread; but when they do call C,
it will be parallelized.
(In my case not all C calls are parallelized because I don't yet have
a sufficiently high level FFI. This must be marked manually, after
ensuring that the runtime is not accessed in the foreign region where
other threads are unblocked.)
> And finally, I still can't think of a use for finalizers
> that touches non-garbage objects and doesn't reflect either
> a design problem or a workaround for a problem that arises
> because of non-cooperative multithreading.
When implementing a dictionary with weak keys, entries should be
removed when keys are garbage collected. Unless the implementation
provides this as a whole as a primitive, with equivalent magic placed
in the low level. (In typical runtimes which don't allow concurrent
access from multiple OS threads, implementing a feature in C is enough
to make it atomic, without explicit locks.)
> I guess it's my considered opinion that preemptive multitasking
> is for allocating time between programs, and when allocating
> time between threads within a program, cooperative is, at
> least until SMP becomes common on the desktop, better.
Even if the time is sliced and split between threads, OS threads
give benefits when using libraries written in foreign languages.