Re: to-be-closed: the case for terminating exceptions

[John Belmonte]

I've revisited suppression of errors via to-be-closed vars and have a working prototype based on Lua 5.4.  Though the change is bytecode compatible, it's not trivial.  I think it's at least interesting as an experiment, and I could use it to continue the "Structured concurrency and Lua" series to give a concrete example of what it enables.

Synopsis:

do

  local x <close> = setmetatable({}, {

    __close = function(self, err)

      print('__close()')

      return true

    end

  })

  error('oops')

  print('not reached')

end

print('goodbye')

-- expected output:

--     __close()

--     goodbye

A difficult subtlety is that execution resumes after the enclosing block-- i.e. blocks behave as if they had an implicit pcall.  And of course it supports yielding __close methods (for asynchronous cleanup).

https://github.com/belm0/lua/pull/2

On Wed, Oct 12, 2022 at 5:30 PM John Belmonte <jo...@neggie.net> wrote:

The ability to have __close() suppress an exception is useful -- more so than I thought.

I'll leave it to Roberto to judge whether this is practical, but as far as API, I think it would be:

"If __close() returns the specific value `false`, then the exception is terminated, and execution continues normally, following the exited scope (after __close of remaining to-be-closed variables)."  Since existing __close() instances are likely to return nothing (`nil`), this behavior is backwards-compatible.

Some example use cases follow.  Of course, pcall() is always a crude replacement, but the presumption is that maintaining return/break/goto within the code block, and avoiding multiple levels of nested lambdas, is highly desirable.

1. ignoring an exception

do

  local scope <close> = open_move_on_if_error()

  -- .. some code that may raise an error ..

end

-- .. continue here if there was an exception ..

If an application has some exception hierarchy, then `open_move_on_if_error()` can optionally accept a base exception type.

2. asserting an error in unit testing

do

  local scope <close> = open_assert_error()

  -- .. some code that is expected to raise an error ..

end

An assert error is raised out of the do/end block if no exception was encountered.  Here, too, there's the option to pass a base exception type into `open_assert_error()`.

3. cancel scopes

It's very useful to declare a scope around arbitrary sequences of asynchronous code, such that execution of the block can be cancelled at any coroutine resume point.  There are numerous uses:

3.a  timeouts

do

  local scope <close> = open_move_on_after_seconds(5)

  -- .. any code that includes transitive yields ..

end

-- .. continue here if timeout reached

At each yield, the scheduler will check if the deadline has been exceeded, and raise an exception, which propagates a Cancelled exception up through the task hierarchy, finally terminated in __close().  (Assumes the recently proposed structured concurrency framework.)

3.b  cancellation by event

do

  local scope <close> = open_move_on_when(event.await)

  -- .. any code that includes transitive yields ..

end

-- .. continue here if event.await() returns

3.c  manual cancellation

do

  local scope <close> = open_cancel_scope()

  -- .. any code that includes transitive yields ..

  if (some_condition) then

     scope.cancel() -- request cancel of this scope

  end

end

Of course, the cancel() call may be encountered transitively within the scope of the block (nested function or task), as well as being invoked by some task outside the hierarchy, which happens to have a handle to the scope object.

Actually, the recently proposed nurseries fall into this use case.  When a nursery is cancelled, the body of the nursery itself needs to exit promptly, despite being blocked on sleep or I/O.

As to-be-closed stands, with no way to terminate exceptions, the next installment of  "Structured concurrency and Lua" may be the end of the road for now.  It will introduce cancel scopes, but explain that the nursery implementation is incomplete, and cancel scopes aren't implemented at all -- pending some solution to terminating exceptions.

On Wed, Oct 12, 2022 at 5:30 PM John Belmonte <jo...@neggie.net> wrote:

The ability to have __close() suppress an exception is useful -- more so than I thought.

I'll leave it to Roberto to judge whether this is practical, but as far as API, I think it would be:

"If __close() returns the specific value `false`, then the exception is terminated, and execution continues normally, following the exited scope (after __close of remaining to-be-closed variables)."  Since existing __close() instances are likely to return nothing (`nil`), this behavior is backwards-compatible.

Some example use cases follow.  Of course, pcall() is always a crude replacement, but the presumption is that maintaining return/break/goto within the code block, and avoiding multiple levels of nested lambdas, is highly desirable.

1. ignoring an exception

do

  local scope <close> = open_move_on_if_error()

  -- .. some code that may raise an error ..

end

-- .. continue here if there was an exception ..

If an application has some exception hierarchy, then `open_move_on_if_error()` can optionally accept a base exception type.

2. asserting an error in unit testing

do

  local scope <close> = open_assert_error()

  -- .. some code that is expected to raise an error ..

end

An assert error is raised out of the do/end block if no exception was encountered.  Here, too, there's the option to pass a base exception type into `open_assert_error()`.

3. cancel scopes

It's very useful to declare a scope around arbitrary sequences of asynchronous code, such that execution of the block can be cancelled at any coroutine resume point.  There are numerous uses:

3.a  timeouts

do

  local scope <close> = open_move_on_after_seconds(5)

  -- .. any code that includes transitive yields ..

end

-- .. continue here if timeout reached

At each yield, the scheduler will check if the deadline has been exceeded, and raise an exception, which propagates a Cancelled exception up through the task hierarchy, finally terminated in __close().  (Assumes the recently proposed structured concurrency framework.)

3.b  cancellation by event

do

  local scope <close> = open_move_on_when(event.await)

  -- .. any code that includes transitive yields ..

end

-- .. continue here if event.await() returns

3.c  manual cancellation

do

  local scope <close> = open_cancel_scope()

  -- .. any code that includes transitive yields ..

  if (some_condition) then

     scope.cancel() -- request cancel of this scope

  end

end

Of course, the cancel() call may be encountered transitively within the scope of the block (nested function or task), as well as being invoked by some task outside the hierarchy, which happens to have a handle to the scope object.

Actually, the recently proposed nurseries fall into this use case.  When a nursery is cancelled, the body of the nursery itself needs to exit promptly, despite being blocked on sleep or I/O.

As to-be-closed stands, with no way to terminate exceptions, the next installment of  "Structured concurrency and Lua" may be the end of the road for now.  It will introduce cancel scopes, but explain that the nursery implementation is incomplete, and cancel scopes aren't implemented at all -- pending some solution to terminating exceptions.