[Python-ideas] ExitStack: Allow exiting individual context managers

[Ram Rachum]

Hi guys,

I'm using `contextlib.ExitStack` today, and pushing context managers into it. I find myself wanting to exit specific context managers that I've pushed into it, while still inside the `with` suite of the `ExitStack`. In other words, I want to exit one of the context managers but still keep the `ExitStack`, and all other context managers, acquired. This isn't currently possible, right? What do you think about implementing this? 

Thanks,

Ram.

[Steven D'Aprano]

I'm not entirely sure what you mean. Can you give an example?

Some of the examples given here:

https://docs.python.org/3/library/contextlib.html#examples-and-recipes

sound like they might be related to what you are trying to do.

Otherwise, if I have understood your requirement correctly, I think you
might have a good case for *not* using ExitStack. If you have one
context manager that you want to treat differently from the others,
perhaps you should write it differently from the others:

with ExitStack() as stack:
files = [stack.enter_context(open(fname)) for fname in filenames]
do_things()
with open(special_file) as sf:
do_other_things()
do_more_things()



--
Steven
_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Nick Coghlan]

On 7 December 2015 at 10:32, Steven D'Aprano <st...@pearwood.info> wrote:
> On Sun, Dec 06, 2015 at 11:48:41PM +0200, Ram Rachum wrote:
>> Hi guys,
>>
>> I'm using `contextlib.ExitStack` today, and pushing context managers into
>> it. I find myself wanting to exit specific context managers that I've
>> pushed into it, while still inside the `with` suite of the `ExitStack`. In
>> other words, I want to exit one of the context managers but still keep the
>> `ExitStack`, and all other context managers, acquired. This isn't currently
>> possible, right? What do you think about implementing this?
>
> I'm not entirely sure what you mean. Can you give an example?

It's a concept I considered implementing, but decided to hold off on
it because there are a few different design options here and I didn't
have any use cases to guide the API design, nor the capacity to do
usability testing to see if the additional API complexity actually
made ExitStack easier to use overall.

The first design option is the status quo: using multiple with
statement blocks, potentially in conjunction with multiple ExitStack
instances. The virtue of this approach is that it means that once a
context manager is added to an ExitStack instance, that's it - its
lifecycle is now coupled to that of the other context managers in the
stack. You can postpone cleaning up all of them with "pop_all()"
(transferring responsibility for the cleanup to a fresh ExitStack
instance), but you can't selectively clean them up from the end. The
usage guidelines are thus relatively simple: if you don't want to
couple the lifecycles of two context managers together, then don't add
them to the same ExitStack instance.

However, there is also that "Stack" in the name, so it's natural for
users to expect to be able to both push() *and* pop() individual
context managers on the stack.

The (on the surface) simplest design option to *add* to the API would
be a single "pop()" operation that returned a new ExitStack instance
(for return type consistency with pop_all()) that contained the last
context manager pushed onto the stack. However, this API is
problematic, as you've now decoupled the nesting of the context
manager stack - the popped context manager may now survive beyond the
closure of the original ExitStack. Since this kind a pop_all()
inspired selective clean-up API would result in two different
ExitStack instances anyway, the status quo seems cleaner to me than
this option, as it's obvious from the start that there are seperate
ExitStack instances with potentially distinct lifecycles.

The next option would then be to offer a separate
"exit_last_context()" method, that exited the last context manager
pushed onto the stack. This would be a single-stepping counterpart to
the existing close() method, that allowed you to dynamically descend
and ascend the context management stack during normal operation, while
still preserving the property that the entire stack will be cleaned up
when encountering an exception.

Assuming we went with that simpler in-place API, there would still be
a number of further design questions to be answered:

* Do we need to try to manage the reported exception context the way
ExitStack.__exit__ does?
* Does "exit_last_context()" need to accept exception details like
__exit__ does?
* Does "exit_last_context()" need to support the ability to suppress exceptions?
* What, if anything, should the return value be?
* What happens if there are no contexts on the stack to pop?
* Should it become possible to query the number of registered callbacks?

Taking them in order, as a matter of implementation feasibility, the
answer to the first question likely needs to be "No". For consistency
with calling __exit__ methods directly, the answers to the next three
questions likely need to be "support the same thing __exit__
supports".

For the second last question, while it's reasonable to call close(),
pop_all() or __exit__() on an empty stack and have it silently do
nothing, if someone has taken it on themselves to manage the stack
depth manually, then it's likely more helpful to complain that the
stack is empty than it is to silently do nothing. Since
exit_last_context() may behave differently depending on whether or not
there are items on the stack, and the number of items on the stack
would be useful for diagnostic purposes, then it likely also makes
sense to implement a __len__ method that delegated to
"len(self._exit_callbacks)".

That all suggests a possible implementation along the lines of the following:

def exit_last_context(self, *exc_details):
if not self._exit_callbacks:
raise RuntimeError("Attempted to exit last context on
empty ExitStack instance")
cb = self._exit_callbacks.pop()
return cb(*exc_details)

def __len__(self):
return len(self._exit_callbacks)

What I don't know is whether or not that's actually a useful enough
improvement over the status quo to justify the additional cognitive
burden when learning the ExitStack API - the current API was designed
around the ExitStack recipes in the documentation, which were all
fairly common code patterns, but most cases where I might consider
using an "exit_last_context()" method, I'd be more inclined to follow
Steven's advice and use a separate context manager for the resource
with an independent lifecycle.

Cheers,
Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

[Ram Rachum]

I would actually want a method that exits not just the last context manager, but any context manager in the stack according to my choosing. Maybe it clashes with the fact that you're using `deque`, but I'm not sure that you have a compelling reason to use `deque`.

If you're asking about my use case: It's pretty boring. I have a sysadmin script with a long function that does remote actions on a few servers. I wrapped it all in an `ExitStack` since I use file-based locks and I want to ensure they get released eventually. Now, at some point I want to release the file-based lock manually, but I can't use a with statement, because there's a condition around the place where I acquire the lock. It's something like this: 

if condition:

    exit_stack.enter_context(get_lock_1())

else:

    exit_stack.enter_context(get_lock_2())

So ideally I would want a method that takes a context manager and just exits it. Maybe even add an optional argument `context_manager` to the existing `close` method. Personally I don't care about exception-handling in this case, and while I think it would be nice to include exception-handling, I see that the existing close method doesn't provide exception-handling either, so I wouldn't feel bad about it.

So maybe something like this: 

def close(self, context_manager=None):

    """Immediately unwind the context stack"""

    if context_manager is None:

        self.__exit__(None, None, None)

    else:

        for _exit_wrapper in reversed(self._exit_callbacks):

            if _exit_wrapper.__self__ is context_manager:

                _exit_wrapper(None, None, None)

                self._exit_callbacks.remove(_exit_wrapper)

Maybe also support accepting a tuple of context managers.

[Michael Selik]

On Mon, Dec 7, 2015 at 3:37 PM Ram Rachum <r...@rachum.com> wrote:

I can't use a with statement, because there's a condition around the place where I acquire the lock. It's something like this: 

if condition:

    exit_stack.enter_context(get_lock_1())

else:

    exit_stack.enter_context(get_lock_2())

You can't put the condition in a context manager's __init__ or __enter__?

    class CM:

        def __init__(self, condition):

            self.lock = lock1 if condition else lock2

        def __enter__(self):

            self.lock.acquire()

        delf __exit__(self, *info):

            self.lock.release()

 

[Nick Coghlan]

On 8 December 2015 at 06:37, Ram Rachum <r...@rachum.com> wrote:
> I would actually want a method that exits not just the last context manager,
> but any context manager in the stack according to my choosing. Maybe it
> clashes with the fact that you're using `deque`, but I'm not sure that you
> have a compelling reason to use `deque`.

deque itself is an implementation detail, but from a design
perspective, ExitStack is intended to recreate the semantics of
lexical context management using with statements, without having to
actually use that layout in your code. In other words, if the exit
semantics can't be expressed in terms of with statements, then I'm not
interested in allowing it in ExitStack specifically (see the ExitPool
discussion below for more on that qualifier).

That means the semantic structures I'm open to ExitStack supporting are:

* a nested context stack (which it already does)
* a tree structure (which exit_last_context() would allow)

The first structure corresponds to passing multiple contexts to the
with statement:

with cm1(), cm2(), cm3():
...

Which in turn corresponds to nested with statements:

with cm1():
with cm2():
with cm3():
...

The ExitStack equivalent is:

with ExitStack() as stack:
stack.enter_context(cm1())
stack.enter_context(cm2())
stack.enter_context(cm3())
...

Adding an exit_last_context() method would make it possible to
replicate the following kind of structure:

with cm1():
with cm2():
...
with cm3():
...

Given exit_last_context(), replicating that dynamically would look like:

with ExitStack() as stack:
stack.enter_context(cm1())
stack.enter_context(cm2())
...
stack.exit_last_context()
stack.enter_context(cm3())
...

I'm not aware of any specific use cases for the latter behaviour
though, which is why that feature doesn't exist yet.

> If you're asking about my use case: It's pretty boring. I have a sysadmin
> script with a long function that does remote actions on a few servers. I
> wrapped it all in an `ExitStack` since I use file-based locks and I want to
> ensure they get released eventually. Now, at some point I want to release
> the file-based lock manually, but I can't use a with statement, because
> there's a condition around the place where I acquire the lock. It's
> something like this:
>
> if condition:
>
> exit_stack.enter_context(get_lock_1())
>
> else:
>
> exit_stack.enter_context(get_lock_2())
>
> So ideally I would want a method that takes a context manager and just exits
> it. Maybe even add an optional argument `context_manager` to the existing
> `close` method. Personally I don't care about exception-handling in this
> case, and while I think it would be nice to include exception-handling, I
> see that the existing close method doesn't provide exception-handling
> either, so I wouldn't feel bad about it.

OK, thanks for the clarification. The additional details show that
this is a slightly different use case from those that ExitStack is
designed to support, as ExitStack aims to precisely replicate the
semantics of nested with statements (as described above). That
includes both the order in which the __exit__ methods get called, and
which context managers can suppress exceptions from which other
context managers.

That's not the only way to manage cleanup logic though, and one
relevant alternative is the way the atexit module works:
https://docs.python.org/3/library/atexit.html

In that model, the cleanup handlers are all considered peer
operations, and while they're defined to be run in last-in-first-out
order, the assumption is that there aren't any direct dependencies
between them the way there can be with lexically nested context
managers. That then makes it reasonable to offer the ability to
unregister arbitrary callbacks without worrying about the potential
impact on other callbacks that were registered later.

While I'm not open to adding atexit style logic to ExitStack, I'm *am*
amenable to the idea of adding a separate ExitPool context manager
that doesn't try to replicate with statement semantics the way
ExitStack does, and instead offers atexit style logic where each exit
function receives the original exception state passed in to
ExitPool.__exit__. One key difference from atexit would be that if any
of the exit methods raised an exception, then I'd have ExitPool raise
a subclass of RuntimeError (PoolExitError perhaps?) containing a list
of all of the cleanup operations that failed.

The API for that would probably look something like:

class ExitPool:
def enter_context(cm):
# Call cm.__enter__ and register cm
def register(cm):
# Register cm.__exit__ to be called on pool exit
def callback(func, *args, **kwds):
# Register func to be called on pool exit
def unregister(cm_or_func):
# Unregister a registered CM or callback function
def unregister_all():
# Empty the pool without calling anything
def close():
# Empty the pool, calling all registered callbacks in LIFO
order (via self.__exit__)

Internally, the main data structure would be an OrderedDict instance
mapping from cm's or functions to their registered callbacks (for ease
of unregistration).

At this point, if you're open to filing one, we should probably move
further discussion over to a new RFE on the contextlib2 issue tracker:
https://bitbucket.org/ncoghlan/contextlib2/

That's still pending a rebase on Python 3.5 standard library version
of contextlib though...

[Ram Rachum]

Thanks for the detailed reply Nick.

I would like an `ExitPool` class but I'm not invested enough in this feature to champion it, so I'll let it go at this point. If you'll open the ticket and want any feedback from me about the API I'll be happy to give it, just email me. I think I can solve my personal problem with a context manager wrapper that doesn't complain when you try to close the context manager twice.

Thanks,

Ram.

[Nick Coghlan]

On 9 December 2015 at 03:21, Ram Rachum <r...@rachum.com> wrote:
> Thanks for the detailed reply Nick.
>
> I would like an `ExitPool` class but I'm not invested enough in this feature
> to champion it, so I'll let it go at this point. If you'll open the ticket
> and want any feedback from me about the API I'll be happy to give it, just
> email me.

Unfortunately, I haven't even found the time to bring contextlib2 up
to date with the current standard library version, let alone consider
using it to evaluate new APIs like ExitPool :(

Regards,

Nick.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia