[Python-Dev] PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

Hi all,

We would like to request feedback on PEP 654 -- Exception Groups and except*.

https://www.python.org/dev/peps/pep-0654/

It proposes language extensions that allow programs to raise and handle multiple unrelated

exceptions simultaneously, motivated by the needs of asyncio and other concurrency libraries,

but with other use cases as well.

* A new standard exception type,  ExceptionGroup, to represent multiple exceptions with

  shared traceback.

* Updates to the traceback printing code to display (possibly nested) ExceptionGroups.
* A new syntax except* for handling ExceptionGroups.

A reference implementation (unreviewed) can be found at:

https://github.com/iritkatriel/cpython/pull/10

Thank you for your help

Kind regards

Irit, Yury & Guido

[Damian Shaw]

Apologies I'm not a core dev so this might be the wrong place to ask but I have 2 small clarifying questions about the PEP.

Firstly, if I have a library which supports multiple versions of Python and I need to catch all standard exceptions, what is considered the best practise after this PEP is introduced?

Currently I might have code like this right now:

try:

    ... # Code

except Exception as e:

    ... # Logic to handle exception

Reading through the PEP, particularly on backwards compatibility it looks like I would now need to do this:

try:

    ExceptionGroup

except NameError:

    standard_exceptions = Exception

else:

    standard_exceptions = (Exception, ExceptionGroup)

try:

    ... # Code

except standard_exceptions as e:

    ... # Logic to handle exception

 

Secondly, once I only support versions of Python that support Exception groups, what replaces the best practise for generally catching standard exceptions? Is it this:

try:

    ... # Code

except *Exception as e:

    ... # Logic to handle exception

Thanks,

Damian (He/Him)

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/L5Q27DVKOKZCDNCAWRIQVOZ5DZCZHLRM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Ivan Pozdeev via Python-Dev]

• In https://www.python.org/dev/peps/pep-0654/#programming-without-except, the natural way isn't shown:

try:
    <smth>
except (MultiError, ValueError) as e:
    def _handle(e):
        if isinstance(e, ValueError):
            return None
        else:
            return exc
    MultiError.filter(_handle,e)

So a statement that the current handling is "cumbersome" and "unintuitive" is unconvincing.

If this results in lots of boilerplate code with isinstance(), filter() can be changed to e.g. accept a dict of exception types and handlers. Actually, I wonder why you didn't do that already if handling specific exception types and reraising the rest is the standard procedure!
Then the code would be reduced to:

try:
    <smth>
except (MultiError, ValueError) as e:
    MultiError.filter({ValueError: lambda _: None},
                      e)

• https://github.com/python-trio/trio/issues/611 says that it somehow causes problems with 3rd-party libraries -- but there's no explanation how -- either there or in the PEP.
  

If some code doesn't know about MultiError's, it should handle one like any other unknown exception that it cannot do anything intelligent about.
If it wishes to handle them, you need to split MultiError into a separate library that anyone could use without having to pull the entire `trio`.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/L5Q27DVKOKZCDNCAWRIQVOZ5DZCZHLRM/
Code of Conduct: http://python.org/psf/codeofconduct/

-- 
Regards,
Ivan

[Damian Shaw]

Hi Irit,

Catching exceptions like this is an extremely common pattern in the real world, e.g. this pattern has over 5 million GitHub matches: https://github.com/search?l=&q=%22except+Exception%22+language%3APython&type=code

How common it is aside there are also many valid use cases for this pattern, e.g. logging a final unhandled exception of a script, catching exceptions inside an orchestration framework, exploring code where the list of exceptions is unknown, etc.

A description from the PEP on how to handle this kind of pattern, especially for code that must support multiple versions of Python, would be extremely helpful.

Damian

On Tue, Feb 23, 2021 at 2:35 PM Irit Katriel <iritk...@googlemail.com> wrote:

On Tue, Feb 23, 2021 at 3:49 PM Damian Shaw <damian.p...@gmail.com> wrote:

Firstly, if I have a library which supports multiple versions of Python and I need to catch all standard exceptions, what is considered the best practise after this PEP is introduced?

Currently I might have code like this right now:

try:

    ... # Code

except Exception as e:

    ... # Logic to handle exception

Hi Damian, 

Catching all exceptions in this way is not a common pattern. Typically you have an idea what kind of exceptions you expect to get from an operation, and which of those exceptions you are interested in handling. Most of your try/except blocks will be targeted, like handling KeyError from d[k], an operation that will never raise an ExceptionGroup.

ExceptionGroups will only be raised by specific APIs which will advertise themselves as such. We don't expect that there will be a need for blanket migrations of "except T" to "except (T, ExceptionGroup)" to "except *T".

Irit

[Irit Katriel via Python-Dev]

On Tue, Feb 23, 2021 at 3:49 PM Damian Shaw <damian.p...@gmail.com> wrote:

Firstly, if I have a library which supports multiple versions of Python and I need to catch all standard exceptions, what is considered the best practise after this PEP is introduced?

Currently I might have code like this right now:

try:

    ... # Code

except Exception as e:

    ... # Logic to handle exception

[Irit Katriel via Python-Dev]

Hi Damian, 

While I agree that there are a handful of use cases for catching all Exceptions, if you look at some of the 5 million hits you found in github, the vast majority are not such cases. They are mostly try/except blocks that wrap a particular operation (a dict access, a raw_input call, etc). You don't want to automatically translate "except Exception" to something that involves ExceptionGroups, because most of the time it is not needed -- the code in the body can't raise ExceptionGroups.

Your first email on this thread showed that you were able to quite quickly, just from what's already in the PEP, think of how to do what you need for your valid "except Exception" use case. My feeling is that people who need it will be able to do it quite easily. I'm more worried about people assuming that they need it when they don't.

Irit

[Stestagg]

This pep sounded kinda scary to me, so I wanted to try it out.

The reference implementation appears to have some bugs in it (around reraise star) , so it's not entirely clear what the expected behavior is supposed to be, but by checking out ffc493b5 I got some OK results.

I had a look at the tempfile example given under the Motivations section, and wanted to see how this would work (as it's the area of code I'm most familiar with).

Would the proposed tempfile change look something like this? (functionally, if not stylistically):

--- a/Lib/tempfile.py
+++ b/Lib/tempfile.py
@@ -819,8 +819,14 @@ def __repr__(self):
     def __enter__(self):
         return self.name
 
-    def __exit__(self, exc, value, tb):
-        self.cleanup()
+    def __exit__(self, exc_cls, exc_value, tb):
+        try:
+            self.cleanup()
+        except Exception as clean_exc:
+            if exc_value is not None:
+                raise ExceptionGroup('Exception occurred during cleanup', [exc_value, clean_exc])
+            else:
+                raise
 
     def cleanup(self):
         if self._finalizer.detach():

If so, then the following code fails to catch the ZeroDivisionError (there is an uncaught exception raised):

    import tempfile, os, pathlib

    def do_some_stuff():
        with tempfile.TemporaryDirectory() as td:
            os.rmdir(td)
            pathlib.Path(td).write_text("Surprise!")
            1/0

    if __name__ == '__main__':
        try:
            do_some_stuff()
        except Exception:
            print("Something went wrong")
        else:
            print("No error")

The output I get:

"""

Traceback (most recent call last):
  File "/home/sstagg/tmp/fuzztest/cpython/td.py", line 7, in do_some_stuff
    1/0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sstagg/tmp/fuzztest/cpython/Lib/tempfile.py", line 824, in __exit__
    self.cleanup()
  File "/home/sstagg/tmp/fuzztest/cpython/Lib/tempfile.py", line 833, in cleanup
    self._rmtree(self.name)
  File "/home/sstagg/tmp/fuzztest/cpython/Lib/tempfile.py", line 809, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "/home/sstagg/tmp/fuzztest/cpython/Lib/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/home/sstagg/tmp/fuzztest/cpython/Lib/shutil.py", line 631, in _rmtree_safe_fd
    onerror(os.scandir, path, sys.exc_info())
  File "/home/sstagg/tmp/fuzztest/cpython/Lib/shutil.py", line 627, in _rmtree_safe_fd
    with os.scandir(topfd) as scandir_it:
NotADirectoryError: [Errno 20] Not a directory: '/tmp/tmpeedn6r0n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sstagg/tmp/fuzztest/cpython/td.py", line 12, in <module>
    do_some_stuff()
  File "/home/sstagg/tmp/fuzztest/cpython/td.py", line 7, in do_some_stuff
    1/0
  File "/home/sstagg/tmp/fuzztest/cpython/Lib/tempfile.py", line 827, in __exit__
    raise ExceptionGroup('Exception occurred during cleanup', [exc_value, clean_exc])
ExceptionGroup: Exception occurred during cleanup

"""

===

So, to catch and handle errors raised from TemporaryDirectory safely, the try-except has to be wrapped in a try-*except block?:

    if __name__ == '__main__':
        try:
            try:
                do_some_stuff()
            except Exception:
                print("Fail Site 1")
        except *NotADirectoryError:
            print("Fail Site 2")
        except *Exception:

            print("Fail Site 3")

In this situation, Sites 2 and 3 are called, but if there is no problem during cleanup then Site 1 is called?

If, instead, the ExceptionGroup is explicitly handled:

    if __name__ == '__main__':
        try:
            do_some_stuff()
        except (Exception, ExceptionGroup):
            print("Fail Site 1")

Then this actually works quite nicely for the 'except Exception' scenario, but is much more complex if you want to catch and handle specific types of exceptions:

    if __name__ == '__main__':
        try:
            do_some_stuff()
        except ExceptionGroup as exc_group:
            zd_errors, others = exc_group(lambda e: isinstance(e, ZeroDivisionError))
            if zd_errors:
                print("Fail Site 1")
            if others:
                raise others
        except ZeroDivisionError:
            print("Fail Site 2")

If the idea is that tempfile.TemporaryDirectory *always* raises an ExceptionGroup (even if there was no cleanup exception) then any code that calls anything that might eventually call TemporaryDirectory will have to be aware that a different type of exception could appear that normal handling doesn't catch (or for /every/ usage of TemporaryDirectory be immediately wrapped in try-*except handling code).

It feels like ever letting an ExceptionGroup unwind more than 1 or two stack frames is super dangerous, as it directly requires all code up the stack to consider this situation and handle it separately from 'normal' exceptions.

It's quite possible that I've completely mis-understood, or skipped over something critical here.  If so, apologies!

Steve

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/XAPO475TVFICD77M3VRLC7OKA5O7WTOT/

[Petr Viktorin]

On 2/23/21 1:24 AM, Irit Katriel via Python-Dev wrote:
>
> Hi all,
>
> We would like to request feedback on PEP 654 -- Exception Groups and
> except*.
>
> https://www.python.org/dev/peps/pep-0654/

> <https://www.python.org/dev/peps/pep-0654/>


Thank you for this PEP!

Also, thank you Nathaniel (and possibly other author(s) of Trio – sadly,
I'm not following closely enough to track contributions) for doing
things right even though it's hard (and slow), and documenting your work
beautifully. I'm glad to see the ideas assimilated into Python & asyncio!

The PEP reminds me of PEP 380 (yield from): it looks like syntax sugar
for code you could already write, but once you look closer, it turns out
that there are so many details and corner cases to keep track of,
getting it correct is very hard.


I kept notes as I read the PEP, then deleted most as I went through
Rejected Ideas. These remained:


> The `ExceptionGroup` class is final, i.e., it cannot be subclassed.

What's the rationale for this?


> It is possible to catch the ExceptionGroup type with except, but not
with except* because the latter is ambiguous

What about `except *(TypeError, ExceptionGroup):`?


> Motivation: Errors in wrapper code

This use case sticks out a bit: it's the only one where ExceptionGroup
doesn't represent joining equivalent tasks.
Consider code similar to bpo-40857:

try:
with TemporaryDirectory() as tempdir:
os.rmdir(tempdir)
n = 1 / 0
except ArithmeticError:
# that error can be safely ignored!
pass

Instead of a FileNotFoundError with ArithmeticError for context you'd
now get an ExceptionGroup. Neither is handled by `except
ArithmeticError`. Where is the win?


> Motivation: Multiple failures when retrying an operation

This is somewhat similar to the TemporaryDirectory, except there's no
`with` block that feels like it should be "transparent" w.r.t. user errors.
If I currently have:

try:
create_connection(*addresses)
except (Timeout, NetworkNotConnected):
# that's OK, let's try later
pass

what should happen after Python 3.10? Apart from adding a new function,
I can see two possible changes:
- create_connection() starts always raising ExceptionGroup on error,
breaking backwards compatibility in the error case
- create_connection() starts only raising ExceptionGroup only for 2+
errors, breaking backwards compatibility in the 2+ errors case

Both look like heisenbug magnets. IMO, the second one is worse; "code
which is now *potentially* raising ExceptionGroup" (as mentioned in the
Backwards Compatibility section; emphasis mine) should be discouraged.

Arguably, this here is a problem with the create_connection function:
the PEP adds a better way how it could have been designed, and that is
virtuous. Still, having it in Motivation might be misleading.



> long term plan to replace `except` by `catch`

Whoa! Is that a real plan?



--

Also, one of the examples has such a missed opportunity to use
print(f'{e1 = }')!

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/LFGOEAX4A46BVQHBX6IECOAW63KQAFGU/

[Damian Shaw]

Hi Irit,

A common example I see from the GitHub search is a catch all exception on some third party API. If the user wants to catch all exceptions from calling a third party API why would they want to let an ExceptionGroup go unhandled in the future? Once ExceptionGroups are introduced then libraries will surely implement them where appropriate and users would want to continue having their board exception handling working, would they not?

I don't actually know if the code I wrote in the first email is correct though, would it catch all standard Exception / ExceptionGroups across Python 3.x if this PEP was implemented in the future? And would it be best practice for this pattern? If so it seems it would require many users to rewrite their code as it adds boilerplate and nuance for a pattern that is a relatively simple and very commonly used right now.

I guess summarized my commentary is please don't dismiss this pattern out of hand, which this PEP currently disrupts, it is used widely in Python right now and has many valid use cases. I don't have anything further to add so I won't continue this discussion and I assume anything further you have to add to be considered and more informed than my own opinion.

Thanks,

Damian

[Irit Katriel via Python-Dev]

Hi Steve, 

Thank you for trying out the implementation. Please do let me know about bugs you find.

This part of your code looks good:

--- a/Lib/tempfile.py

+++ b/Lib/tempfile.py
@@ -819,8 +819,14 @@ def __repr__(self):
     def __enter__(self):
         return self.name
 
-    def __exit__(self, exc, value, tb):
-        self.cleanup()
+    def __exit__(self, exc_cls, exc_value, tb):
+        try:
+            self.cleanup()
+        except Exception as clean_exc:
+            if exc_value is not None:
+                raise ExceptionGroup('Exception occurred during cleanup', [exc_value, clean_exc])
+            else:
+                raise
 
     def cleanup(self):
         if self._finalizer.detach():

def do_some_stuff():

    with tempfile.TemporaryDirectory() as td:
        os.rmdir(td)
        pathlib.Path(td).write_text("Surprise!")
        1/0

Then, use it like this:

if __name__ == '__main__':
    try:
        do_some_stuff()

    except *NotADirectoryError as e:
        print("Fail Site 2", repr(e))
    except *Exception as e:
        print("Fail Site 3", repr(e))

    else:
        print("No error")

Output:

   Fail Site 2 ExceptionGroup('Exception occurred during cleanup', [NotADirectoryError(20, 'The directory name is invalid')])
   Fail Site 3 ExceptionGroup('Exception occurred during cleanup', [ZeroDivisionError('division by zero')])

This works whether do_some_stuff raises naked exceptions or ExceptionGroups, because "except *T"  catches a naked T (and wraps it in an ExceptionGroup).

So if I comment out the 1/0 I get:

    Fail Site 2 ExceptionGroup('', (NotADirectoryError(20, 'The directory name is invalid'),))

There is no need for it to always raise ExceptionGroup. It can propagate the user's exception as is if it has no exceptions to add. But the caller needs to assume that it may raise an ExceptionGroup, and use except*.

Irit

 

[Caleb Donovick]

What is the motivation for returning `None` on empty splits?  I feel like this creates an unnecessary asymmetry.  I don't personally have a use case for the this feature so I may be missing something but it seems like it would force an annoying pattern:

```

try:
    foo()
except ExceptionGroup as eg:
    g1, g2 = eg.split(predicate)
    for e in g1:
        handle_true(e)
    for e in g2:
        handle_false(e)
```
Would need to be written:
```

try:
    foo()
except ExceptionGroup as eg:
    g1, g2 = eg.split(predicate)
    if g1 is not None:
        for e in g1:
            handle_true(e)
    if g2 is not None:
        for e in g2:
            handle_false(e)

``` 

Also this creates an subtle difference with subgroup:

```
g1, g2 = eg.split(predicate)
h1, h2  = eg.subgroup(predicate), eg.subgroup(lambda e: not predicate(e))
assert g1 == h1 and g2 == h2 # only true if `None not in {g1, g2}`
```

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/L5Q27DVKOKZCDNCAWRIQVOZ5DZCZHLRM/

[MRAB]

.subgroup returns an ExceptionGroup and .split returns 2 of them.

ExceptionGroup isn't iterable (it's mentioned in the "Rejected Ideas"
section), so your code wouldn't work anyway.

> On Mon, Feb 22, 2021 at 4:47 PM Irit Katriel via Python-Dev
> <pytho...@python.org <mailto:pytho...@python.org>> wrote:
>
>
> Hi all,
>
> We would like to request feedback on PEP 654 -- Exception Groups and
> except*.
>
> https://www.python.org/dev/peps/pep-0654/
> <https://www.python.org/dev/peps/pep-0654/>
>
> It proposes language extensions that allow programs to raise and
> handle multiple unrelated
> exceptions simultaneously, motivated by the needs of asyncio and
> other concurrency libraries,
> but with other use cases as well.
>
> * A new standard exception type, ExceptionGroup, to represent
> multiple exceptions with
>   shared traceback.
> * Updates to the traceback printing code to display (possibly
> nested) ExceptionGroups.
> * A new syntax except* for handling ExceptionGroups.
>
> A reference implementation (unreviewed) can be found at:
> https://github.com/iritkatriel/cpython/pull/10
> <https://github.com/iritkatriel/cpython/pull/10>
>
> Thank you for your help
>

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/U7A4PZE7SSDFNKAFZZ3BZBLL7VGTOWVL/

[Irit Katriel via Python-Dev]

Hi Caleb, 

On Tue, Feb 23, 2021 at 11:05 PM Caleb Donovick <dono...@cs.stanford.edu> wrote:

What is the motivation for returning `None` on empty splits?  I feel like this creates an unnecessary asymmetry.  I don't personally have a use case for the this feature so I may be missing something but it seems like it would force an annoying pattern:

Split is used by the interpreter to implement except*. It needs to check whether the first value of the split is empty or not (this determines whether the except* clause executes). 

This is more efficiently done with None. And then there is creating an object that you don't need. 

The intention is that you use except* rather than do this:

 

```

try:
    foo()
except ExceptionGroup as eg:
    g1, g2 = eg.split(predicate)
    for e in g1:
        handle_true(e)
    for e in g2:
        handle_false(e)
```

(as an aside - exception groups ended up not being iterable - that is one of the rejected ideas, see explanation there).

 

Also this creates an subtle difference with subgroup:

```
g1, g2 = eg.split(predicate)
h1, h2  = eg.subgroup(predicate), eg.subgroup(lambda e: not predicate(e))
assert g1 == h1 and g2 == h2 # only true if `None not in {g1, g2}`
```

An empty subgroup should also return None. I see this is not mentioned in the PEP, so I will add it. Thanks.

Irit

[Irit Katriel via Python-Dev]

Hi Petr, 

Thank you for your careful reading and encouragement.

 > The `ExceptionGroup` class is final, i.e., it cannot be subclassed.

What's the rationale for this?

ExceptionGroup.subgroup()/split() need to create new instances, and subclassing

would make that complicated if we want the split results to have the same type.

 > It is possible to catch the ExceptionGroup type with except, but not
with except* because the latter is ambiguous

What about `except *(TypeError, ExceptionGroup):`?

Good question. We should block that as well. 

 

 > Motivation: Errors in wrapper code

This use case sticks out a bit: it's the only one where ExceptionGroup
doesn't represent joining equivalent tasks.
Consider code similar to bpo-40857:

   try:
       with TemporaryDirectory() as tempdir:
           os.rmdir(tempdir)
           n = 1 / 0
   except ArithmeticError:
       # that error can be safely ignored!
       pass

Instead of a FileNotFoundError with ArithmeticError for context you'd
now get an ExceptionGroup. Neither is handled by `except
ArithmeticError`. Where is the win?

I agree, if TemporaryDirectory() were ever to adopt ExceptionGroups then it will probably

be through a new API (like an opt-in parameter) to avoid breaking current code. Users of 

such a TemporaryDirectory would need to wrap the calls with try-except* (see the example

in my previous reply to Steve).

 

 > Motivation: Multiple failures when retrying an operation

This is somewhat similar to the TemporaryDirectory, except there's no
`with` block that feels like it should be "transparent" w.r.t. user errors.
If I currently have:

     try:
         create_connection(*addresses)
     except (Timeout, NetworkNotConnected):
         # that's OK, let's try later
         pass

what should happen after Python 3.10? Apart from adding a new function,
I can see two possible changes:
- create_connection() starts always raising ExceptionGroup on error,
breaking backwards compatibility in the error case
- create_connection() starts only raising ExceptionGroup only for 2+
errors, breaking backwards compatibility in the 2+ errors case 

Both look like heisenbug magnets. IMO, the second one is worse; "code
which is now *potentially* raising ExceptionGroup" (as mentioned in the
Backwards Compatibility section; emphasis mine) should be discouraged.

Arguably, this here is a problem with the create_connection function:
the PEP adds a better way how it could have been designed, and that is
virtuous. Still, having it in Motivation might be misleading.

I agree. Raising ExceptionGroups is an API breaking change, for any library. 

No function should just start raising ExceptionGroups.

 > long term plan to replace `except` by `catch`

Whoa! Is that a real plan?

In the rejected ideas section, anything goes! 

Irit

[Ethan Furman]

On 2/22/21 4:24 PM, Irit Katriel via Python-Dev wrote:

> We would like to request feedback on PEP 654 -- Exception Groups and
> except*.
>
> https://www.python.org/dev/peps/pep-0654/
>
> It proposes language extensions that allow programs to raise and

> handle multiple unrelatedexceptions simultaneously, motivated by the
> needs of asyncio and other concurrency libraries,but with other use
> cases as well.

It sounds like the long-term goal is to move away from `except` and
replace it with `except *` -- is that correct?

--
~Ethan~

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/IT56OKHUKVUNJHAF2UYNDXTK5MYPDNZ6/

[Guido van Rossum]

On Tue, Feb 23, 2021 at 7:37 PM Ethan Furman <et...@stoneleaf.us> wrote:

On 2/22/21 4:24 PM, Irit Katriel via Python-Dev wrote:

 > We would like to request feedback on PEP 654 -- Exception Groups and
 > except*.
 >
 > https://www.python.org/dev/peps/pep-0654/
 >
 > It proposes language extensions that allow programs to raise and

 > handle multiple unrelated exceptions simultaneously, motivated by the
 > needs of asyncio and other concurrency libraries, but with other use

 > cases as well.

It sounds like the long-term goal is to move away from `except` and
replace it with `except *` -- is that correct?

I don't think so -- if we expected that to happen the extra '*' in the syntax would be a nuisance. The premise of the PEP is rather that raising and catching multiple exceptions at once is always going to be an esoteric hobby. The most common case by far would be in async frameworks, but we don't expect 'async def' to eventually become the standard function definition either (nor 'await f()' the standard call :-).

And even in an async framework, I don't expect that every async function would be "upgraded" to raise ExceptionGroup -- only specific APIs like gather() or create_connection(). (Even for those, backwards compatibility concerns will probably mean that we'll have to introduce *new* APIs that can raise ExceptionGroup.) The PEP lists a handful of other use cases, none of which seem to suggest to me that this is going to be a common thing.

--

--Guido van Rossum (python.org/~guido)

Pronouns: he/him (why is my pronoun here?)

[Guido van Rossum]

On Tue, Feb 23, 2021 at 2:27 PM Damian Shaw <damian.p...@gmail.com> wrote:

A common example I see from the GitHub search is a catch all exception on some third party API. If the user wants to catch all exceptions from calling a third party API why would they want to let an ExceptionGroup go unhandled in the future? Once ExceptionGroups are introduced then libraries will surely implement them where appropriate and users would want to continue having their board exception handling working, would they not?

See my response to Ethan -- I think libraries should think twice before raising ExceptionGroup.

Note that ExceptionGroup is a subclass of BaseException, so if you're catching the latter you don't have to do anything special.

OTOH we might reconsider deriving ExceptionGroup from BaseException -- maybe it's better to inherit from Exception? I don't think the PEP currently has a clear reason why it must derive from BaseException. I believe it has to do with the possibility that ExceptionGroup might wrap a BaseException instance (e.g. KeyboardInterrupt or SystemExit). But I believe that this was more important in an earlier revision of the design. Note that there is a section labeled "Should MultiError inherit from BaseException, or from Exception? (Or something else?)" in Trio's MultiError v2 issue (https://github.com/python-trio/trio/issues/611). I think the key argument here is "However, if MultiError is an Exception, then it can only contain Exception subclasses, because otherwise a MultiError(KeyboardInterrupt) could get caught by an except Exception handler, which would be bad." But those notes are from back in 2018, not from the 2020 core dev sprint when we got much closer to the current design. So I'm not sure the reasoning still applies. (Yury? Can you help us out here?)

 

I don't actually know if the code I wrote in the first email is correct though, would it catch all standard Exception / ExceptionGroups across Python 3.x if this PEP was implemented in the future? And would it be best practice for this pattern? If so it seems it would require many users to rewrite their code as it adds boilerplate and nuance for a pattern that is a relatively simple and very commonly used right now.

I guess summarized my commentary is please don't dismiss this pattern out of hand, which this PEP currently disrupts, it is used widely in Python right now and has many valid use cases. I don't have anything further to add so I won't continue this discussion and I assume anything further you have to add to be considered and more informed than my own opinion.

I expect that if you want to do a good job of emulating `except *ValueError` in Python versions that don't have ExceptionGroup you're going to have to write more code. It's relatively straightforward if you just want to ignore or log the errors collectively, but it'll be more complex if you want to handle each suberror separately, because the PEP-654-aware code will have to iterate over the errors, recursing into ExceptionGroup instances. (Irit, maybe the PEP could use a worked-out example of how to do the latter -- even with `except *` available it seems non-trivial.)

Thanks,

Damian

Thanks Damian for getting us to think more about 'except Exception:'

[Ethan Furman]

On 2/23/21 7:56 PM, Guido van Rossum wrote:

> On Tue, Feb 23, 2021 at 7:37 PM Ethan Furman wrote:

>> It sounds like the long-term goal is to move away from `except` and
>> replace it with `except *` -- is that correct?
>
> I don't think so -- if we expected that to happen the extra '*' in the
> syntax would be a nuisance. The premise of the PEP is rather that
> raising and catching multiple exceptions at once is always going
> to be an esoteric hobby. The most common case by far would be in async
> frameworks, but we don't expect 'async def' to eventually become the
> standard function definition either (nor 'await f()' the standard call :-).
>
> And even in an async framework, I don't expect that every async function
> would be "upgraded" to raise ExceptionGroup -- only specific APIs like
> gather() or create_connection(). (Even for those, backwards compatibility
> concerns will probably mean that we'll have to introduce *new* APIs that
> can raise ExceptionGroup.) The PEP lists a handful of other use cases,
> none of which seem to suggest to me that this is going to be a common thing.

I can see the value in `except *` for concurrent code; my concern is how it will operate with the standard `try/except` framework. With the new non-Exception-derived ExceptionGroup class, existing try-excepts that catch Exception will fail for every ExceptionGroup that leaks through; the fix for that is a doubly-nested try-except/try-except* -- what do we get for that code churn? What is the advantage in not deriving from Exception? If I recall correctly, we have only three exception types that derive directly from BaseException: Exception, SystemExit, and KeyboardInterrupt -- with SystemExit and KeyboardInterrupt both being concerned with making sure an application can quit.

--
~Ethan~
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/445RAHP7XXLQFFGFKCXITPQPGAB55MDY/

[Guido van Rossum]

On Tue, Feb 23, 2021 at 8:00 AM Ivan Pozdeev via Python-Dev <pytho...@python.org> wrote:

• In https://www.python.org/dev/peps/pep-0654/#programming-without-except, the natural way isn't shown:

try:
    <smth>
except (MultiError, ValueError) as e:
    def _handle(e):
        if isinstance(e, ValueError):
            return None
        else:
            return exc
    MultiError.filter(_handle,e)

So a statement that the current handling is "cumbersome" and "unintuitive" is unconvincing.

If this results in lots of boilerplate code with isinstance(), filter() can be changed to e.g. accept a dict of exception types and handlers. Actually, I wonder why you didn't do that already if handling specific exception types and reraising the rest is the standard procedure!
Then the code would be reduced to:

try:
    <smth>
except (MultiError, ValueError) as e:
    MultiError.filter({ValueError: lambda _: None},
                      e)

• https://github.com/python-trio/trio/issues/611 says that it somehow causes problems with 3rd-party libraries -- but there's no explanation how -- either there or in the PEP.
  

If some code doesn't know about MultiError's, it should handle one like any other unknown exception that it cannot do anything intelligent about.
If it wishes to handle them, you need to split MultiError into a separate library that anyone could use without having to pull the entire `trio`.

I think the main point our PEP tries to make is that having to define a helper function to handle exceptions (and then having to call a utility to call the helper) feels clumsy.

However there may be an omission in the PEP -- what if we want to do something special for each suberror? If we just iterate over `eg.errors` we would have to do something recursive for exceptions wrapped inside multiple nested groups. We could add a helper method to ExceptionGroup that iterates over suberrors, flattening nested groups. But we'd need to do something special to recover the tracebacks there (groups share the common part of the traceback). Or we could say "if you need the traceback you're going to have to write something recursive" -- like we have to do in traceback.py. But we might as well generalize whatever we're doing there. (Irit: I suspect there's a piece of API design we missed here.)

[Stestagg]

On Tue, Feb 23, 2021 at 11:00 PM Irit Katriel <iritk...@googlemail.com> wrote:

Hi Steve, 

Thank you for trying out the implementation. Please do let me know about bugs you find.

Thanks you for your response!  It seems I had missed the (now) obvious naked exception wrapping logic.  Apologies for adding noise to the conversation.

I've added a comment to your fork's PR with the error detail.

Best of luck with the PEP

Steve

 

[Jim J. Jewett]

Petr: What about except *(TypeError, ExceptionGroup):?

Irit: Good question. We should block that as well.

But https://www.python.org/dev/peps/pep-0654/#backwards-compatibility
seems to explicitly recommend the very similar:

except (Exception, ExceptionGroup):

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/N22HZBABDVM2GCEIWHOCTPGJDRSRVGS5/

[Jim J. Jewett]

The "no subclasses" seems pretty severe, particularly if you can't even use marker attributes because it isn't clear when a different instance of container ExceptionGroup will be substituted.

The justification for this restriction was that .split() had to be be able to instantiate ... wouldn't it be enough to just say that subclasses need a compatible __init__ and __new__ for that reason, and then leave it to consenting adults?

-jJ

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/5OLFDKJCNKLIJJQMNYYLQGV53Z7ZTRC5/

[Jim J. Jewett]

Please include an example for:

except *ExceptionGroup: pass

I *think* it swallows all exceptions, instead of re-raising.
I *think* it also catches (and then swallows) bare exception that were not initially part of an ExceptionGroup.
I *think* it catches anything that gets raised, not just subclasses of Exception (or even BaseException).

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/NTQ22LUIKNRNA2YHKOLVPR5GI2KW6TLW/

[Irit Katriel via Python-Dev]

On Wed, Feb 24, 2021 at 4:55 AM Guido van Rossum <gu...@python.org> wrote:

However there may be an omission in the PEP -- what if we want to do something special for each suberror? If we just iterate over `eg.errors` we would have to do something recursive for exceptions wrapped inside multiple nested groups. We could add a helper method to ExceptionGroup that iterates over suberrors, flattening nested groups. But we'd need to do something special to recover the tracebacks there (groups share the common part of the traceback). Or we could say "if you need the traceback you're going to have to write something recursive" -- like we have to do in traceback.py. But we might as well generalize whatever we're doing there. (Irit: I suspect there's a piece of API design we missed here.)

This is mentioned briefly where we talk about why ExceptionGroup is not iterable: https://www.python.org/dev/peps/pep-0654/#the-exceptiongroup-api

We left it out pending a use case.

Rather than iterator, I think we should add a visitor that calls a function for each leaf exception, plus a utility that returns the traceback of a leaf

exception (as a single list, copying frames from the tracebacks of ExceptionGroups so that it's not destructive.) This way the expensive traceback

construction is happening explicitly when it is needed.

I think accessing one exception at a time to do something other than formatting it is more likely to be overused than actually needed.

We did that in an earlier version of the PEP for the "filter OSErrors by type" example, which we did with iteration and now do with subgroup:

try:
    low_level_os_operation()
except *OSerror as errors:
    raise errors.subgroup(lambda e: e.errno != errno.EPIPE) from None

[Irit Katriel via Python-Dev]

On Wed, Feb 24, 2021 at 4:39 AM Guido van Rossum <gu...@python.org> wrote:

OTOH we might reconsider deriving ExceptionGroup from BaseException -- maybe it's better to inherit from Exception? I don't think the PEP currently has a clear reason why it must derive from BaseException. I believe it has to do with the possibility that ExceptionGroup might wrap a BaseException instance (e.g. KeyboardInterrupt or SystemExit).

That was the reason, and we also said that an ExceptionGroup escaping something that isn't supposed to be raising ExceptionGroups is a bug, so if you call an API that raises ExceptionGroups it is your responsibility to handle them.

We could make ExceptionGroup be an Exception, and refuse to wrap anything which is not an Exception. So asyncio either raises KeyboardInterrupt or an ExceptionGroup of user exceptions.

Those are quite different directions. 

[Irit Katriel via Python-Dev]

Hi Jim,

On Wed, Feb 24, 2021 at 2:16 PM Jim J. Jewett <jimjj...@gmail.com> wrote:

Petr:  What about except *(TypeError, ExceptionGroup):?

Irit:  Good question. We should block that as well.

But https://www.python.org/dev/peps/pep-0654/#backwards-compatibility
seems to explicitly recommend the very similar:

    except (Exception, ExceptionGroup):

That's except, not except*. 

Please include an example for:

    except *ExceptionGroup: pass

There is one here:  https://www.python.org/dev/peps/pep-0654/#forbidden-combinations

It raises a Runtime Error.

The "no subclasses" seems pretty severe, particularly if you can't even use marker attributes because it isn't clear when a different instance of container ExceptionGroup will be substituted.

The justification for this restriction was that .split() had to be be able to instantiate ... wouldn't it be enough to just say that subclasses need a compatible __init__ and __new__ for that reason, and then leave it to consenting adults?

What do you mean by compatible?  How would one's marker attributes be copied to the new instances?

split() currently copies (context, cause, traceback) from the original ExceptionGroup to the new ones. We could have some protocol, but I'm not sure what it would look like. I'm pretty sure there will be a cost, and I'm not sure I understand why no subclassing is a serious enough limitation to justify that.

Irit

[Guido van Rossum]

Hi Jim,

Let me echo Irit's response here -- what's your use case for subclassing ExceptionGroup?

Requiring that subclasses have a compatible __init__ or __new__ sounds like a bug magnet, since this can't be checked when the subclass is being defined, and it goes against conventional rules for those signatures.

--Guido

[Guido van Rossum]

Hi Ethan,

See Irit's response to my about why ExceptionGroup doesn't subclass Exception -- I think this is still a little bit of an open issue.

Another response (which I also read somewhere in one of Irit's messages) is that a library API shouldn't go from raising subclasses of Exception only to raising ExceptionGroups. That would be a backwards incompatible change to an API.

--Guido

[Jim J. Jewett]

By the time a KeyboardInterrupt or SystemExit is being grouped into an ExceptionGroup, it already isn't being treated as an immediate interruption ... it has (presumably) already killed its own execution path, but it should not kill the program as a whole. Whether ExceptionGroup inherits from Exception or BaseException shouldn't matter for deciding what it can group.

-jJ

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/XSEL22SDNEX2BWT7GS7OUPDAA2UVMH42/

[Jim J. Jewett]

Are you saying that

except *ValueError as e:

will catch

ValueError
and
ExceptionGroup(ValueError)
but miss
ExceptionGroup(ExceptionGroup(ValueError))
?

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/VLPYAILCWTGG25WYJXUA47LIQPULQ7ZM/

[Jim J. Jewett]

If it (compatible __new__ and __init__) needs to be checked at definition time, just try create an instance passing the same arguments you would pass to the base class. If the creation it doesn't raise an exception, that is good enough.

This isn't about theoretical type safety against malice; it is about defining the minimal protocol for an ExceptionGrouping that has to be supported by someone who wants something other than the default flavor.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/JW4BFAR7SN23NF7S6TRGOROJIMPDGQEJ/

[Jim J. Jewett]

Ideally, (at least) trivial subclasses could be declared, and the class itself would serve as the marker. I would prefer regular subclasses, so that they could offer methods as well. Alternatively, at least copy the instance __dict__ to the new ExceptionGroup instance.

By compatible __init__ and __new__, I mean "whatever you do in ExceptionGroup.subgroup to create a new instance with fewer contained Exceptions ... do the same with MyExceptionGroup." I'm assuming that "whatever" is to call __new__ and __init__ with "a message string and a sequence of the nested exceptions, for example: ExceptionGroup('issues', [ValueError('bad value'), TypeError('bad type')])."

-jJ

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/SFE46EGTGLXLCJDXZ4EI6HOEATYREOL4/

[Emily Bowman]

After reading through the PEP and skimming the code (but I didn't build it), something I didn't see: What happens to a currently conforming except check?

try:
    async with trio.open_nursery() as nursery:
        # Make two concurrent calls to child()
        nursery.start_soon(child)
        nursery.start_soon(child)
except ValueError:
    pass

I've removed the * from the example: Say the interface was built for 3.7, but the "trio" module has been upgraded to use ExceptionGroups which can't fall back to a standalone exception.

Silently hand back the first exception, or the first matching exception? Deprecation warning? Silently skip? Context-specific error? Run the default error handler?

I think that deserves a statement in the PEP.

-Em

On Mon, Feb 22, 2021 at 4:48 PM Irit Katriel via Python-Dev <pytho...@python.org> wrote:

Hi all,

We would like to request feedback on PEP 654 -- Exception Groups and except*.

https://www.python.org/dev/peps/pep-0654/

It proposes language extensions that allow programs to raise and handle multiple unrelated

exceptions simultaneously, motivated by the needs of asyncio and other concurrency libraries,

but with other use cases as well.

* A new standard exception type,  ExceptionGroup, to represent multiple exceptions with

  shared traceback.

* Updates to the traceback printing code to display (possibly nested) ExceptionGroups.
* A new syntax except* for handling ExceptionGroups.

A reference implementation (unreviewed) can be found at:

https://github.com/iritkatriel/cpython/pull/10

Thank you for your help

[Guido van Rossum]

[Subthread: handling individual errors]

> Rather than iterator, I think we should add a visitor that calls a function for each leaf exception,

I agree that we shouldn't add an `__iter__` method. But what if we added a separate method that returns an iterator? Then you could write

```

for e in eg.something():

    print("caught", e)

```

This looks cleaner to me than having to write

```

def handler(e):

    print("caught", e)

eg.something(handler)

```

(Does my bias against callback functions shine through? :-)

> plus a utility that returns the traceback of a leaf exception

> (as a single list, copying frames from the tracebacks of ExceptionGroups so that it's not destructive.)

Hm, what would the arguments for that utility be? Since ExceptionGroups can be nested, if you pass in the root EG and the leaf exception, it would require a full tree search to find the nodes that have all the needed tracebacks.

Maybe the API for the iterator-returning method should be that it returns an iterator of (exception, list-of-tracebacks) pairs? Or maybe there should be two iterator-returning methods, one that yields just exceptions and one that yields such pairs, so that if you don't need the tracebacks we don't have to construct them. (Separate methods so the static type doesn't depend on the value of a flag.) If in the end we were to go for something with a handler callback we could do a similar thing.

[Guido van Rossum]

On Wed, Feb 24, 2021 at 5:51 PM Jim J. Jewett <jimjj...@gmail.com> wrote:

Are you saying that

    except *ValueError as e:

will catch

    ValueError
and
    ExceptionGroup(ValueError)
but miss
    ExceptionGroup(ExceptionGroup(ValueError))
?

No, it will catch it regardless of how many levels of ExceptionGroup wrap it (zero or more, in fact).

[Guido van Rossum]

On Wed, Feb 24, 2021 at 5:58 PM Jim J. Jewett <jimjj...@gmail.com> wrote:

If it (compatible __new__ and __init__) needs to be checked at definition time, just try create an instance passing the same arguments you would pass to the base class.  If the creation it doesn't raise an exception, that is good enough.

User code can do that, the interpreter should not use such heuristics.

 

This isn't about theoretical type safety against malice; it is about defining the minimal protocol for an ExceptionGrouping that has to be supported by someone who wants something other than the default flavor.

You still haven't shown a use case for wanting to subclass ExceptionGroup. It's easy to allow this later if there's a valid use case (it wouldn't require a PEP, just a well-reasoned use case). But if we allow it now and in the future we discover it causes subtle bugs, it would be difficult to roll back (requiring deprecation over several releases). Why would you want a different flavor of ExceptionGroup?

[Guido van Rossum]

On Wed, Feb 24, 2021 at 7:09 PM Emily Bowman <silver...@gmail.com> wrote:

After reading through the PEP and skimming the code (but I didn't build it), something I didn't see: What happens to a currently conforming except check?

try:
    async with trio.open_nursery() as nursery:
        # Make two concurrent calls to child()
        nursery.start_soon(child)
        nursery.start_soon(child)
except ValueError:
    pass

I've removed the * from the example: Say the interface was built for 3.7, but the "trio" module has been upgraded to use ExceptionGroups which can't fall back to a standalone exception.

Silently hand back the first exception, or the first matching exception? Deprecation warning? Silently skip? Context-specific error? Run the default error handler?

I think that deserves a statement in the PEP.

The ExceptionGroup would bubble up. IIRC this Trio behavior is considered problematic by the Trio developers, because it means that if *two* children fail with ValueError, it will raise MultiError and the except clause will not trigger (see the "MultiError v2" issue linked from the PEP).

But changing trio.open_nursery() to raise ExceptionGroup would be considered backwards compatible. I would recommend introducing a new API instead. This is discussed (though maybe not in enough detail?) under Backwards Compatibility in the PEP.

 

[Guido van Rossum]

Indeed. I don't think we require that all wrapped exceptions derive from Exception; an important special case in asyncio is CancelledError, which doesn't derive from Exception. (Ditto for trio.Cancelled.)

It is really worth carefully reading the section I mentioned in Trio's MultiError v2 issue. But its "plan" seems complex, as it would require asyncio to define `class TaskGroupError(ExceptionGroup, Exception)` -- that was fine when we considered this *only* for asyncio, but now that we've found several other use cases it really doesn't feel right.

Here's a potentially alternative plan, which is also complex, but doesn't require asyncio or other use cases to define special classes. Let's define two exceptions, BaseExceptionGroup which wraps BaseException instances, and ExceptionGroup which only wraps Exception instances. (Names to be bikeshedded.) They could share a constructor (always invoked via BaseExceptionGroup) which chooses the right class depending on whether there are any non-Exception instances being wrapped -- this would do the right thing for split() and subgroup() and re-raising unhandled exceptions.

Then 'except Exception:' would catch ExceptionGroup but not BaseExceptionGroup, so if a group wraps e.g. KeyboardError it wouldn't be caught (even if there's also e.g. a ValueError among the wrapped errors).

Pseudo-code:

class BaseExceptionGroup(BaseException):

    def __new__(cls, msg, errors):

        if cls is BaseExceptionGroup and all(isinstance(e, Exception) for e in errors):

            cls = ExceptionGroup

        return BaseException.__new__(cls, msg, errors)

class ExceptionGroup(Exception, BaseExceptionGroup):

    pass

[Irit Katriel via Python-Dev]

On Thu, Feb 25, 2021 at 5:59 AM Guido van Rossum <gu...@python.org> wrote:

Here's a potentially alternative plan, which is also complex, but doesn't require asyncio or other use cases to define special classes. Let's define two exceptions, BaseExceptionGroup which wraps BaseException instances, and ExceptionGroup which only wraps Exception instances. (Names to be bikeshedded.) They could share a constructor (always invoked via BaseExceptionGroup) which chooses the right class depending on whether there are any non-Exception instances being wrapped -- this would do the right thing for split() and subgroup() and re-raising unhandled exceptions.

Then 'except Exception:' would catch ExceptionGroup but not BaseExceptionGroup, so if a group wraps e.g. KeyboardError it wouldn't be caught (even if there's also e.g. a ValueError among the wrapped errors).

Pseudo-code:

class BaseExceptionGroup(BaseException):

    def __new__(cls, msg, errors):

        if cls is BaseExceptionGroup and all(isinstance(e, Exception) for e in errors):

            cls = ExceptionGroup

        return BaseException.__new__(cls, msg, errors)

class ExceptionGroup(Exception, BaseExceptionGroup):

    pass

This could be a valid compromise.

We keep the ability to wrap any exception, while we lose the "fail-fast if you forget to handle an ExceptionGroup" feature, which was intended as a kindness towards those who abuse "except Exception".

If we adopt this solution then letting an ExceptionGroup escape from code that is not supposed to raise it, is not a fatal error, it's just some exception like any other.

So there is no longer a distinction between code that raises ExceptionGroups and code that doesn't. Any code can propagate them, like any code can raise any other exception. 

Does this mean that more code needs to be aware of the possibility of them showing up?  Is that a problem?  Maybe this a simpler state of affairs overall.

What would we have done here if we were building Python from scratch?

Irit

[Irit Katriel via Python-Dev]

On Thu, Feb 25, 2021 at 5:19 AM Guido van Rossum <gu...@python.org> wrote:

[Subthread: handling individual errors]

> Rather than iterator, I think we should add a visitor that calls a function for each leaf exception,

I agree that we shouldn't add an `__iter__` method. But what if we added a separate method that returns an iterator? Then you could write

```

for e in eg.something():

    print("caught", e)

```

This looks cleaner to me than having to write

```

def handler(e):

    print("caught", e)

eg.something(handler)

```

(Does my bias against callback functions shine through? :-)

I hear you. I suggested a visitor to make it a bit awkward to use because I'm not sure why people should iterate over individual exceptions in an ExceptionGroup, and I think that by providing an iteration utility we are implying that this is what you should do.

So let me be more direct instead of proposing passive-aggressive APIs. 

Can we take a step back and talk about how we think people would want to handle ExceptionGroups?

In the rejected ideas section we discuss this a bit, with a reference to Yury's writeup: https://github.com/python/exceptiongroups/issues/3#issuecomment-716203284

TL;DR:  when you get a group of exceptions from asyncio or the like, you may want to query it for exception types is contains (with subgroup()), you may want to format it into a log (with traceback.* methods), but you are unlikely to care whether there are 1, 2 or 300 ValueErrors. Your program will probably do the same thing regardless. If you allowed your ValueError get collected into an ExceptionGroup you already lost the context in which it happened to it's unlikely that you can make a targeted recovery which is relevant to this particular exception.

So, what is the use case for iterating over single exceptions?

 

[Guido van Rossum]

Good question. The main use case I see for iterating over individual exceptions is logging frameworks that want to format their own exception tracebacks for whatever reason. I know that the traceback.py module has extensive utilities for exactly that, but I betcha there are plenty of people who still want to roll their own, e.g. for compatibility with other tooling on their platform.

Presumably the code for traversing a tree of exceptions exists in traceback.py. I honestly don't think it matters much whether we do it as an iterator or using callbacks, as long as it visits all the leaves in the tree.

Hm, a different idea: maybe it's simple enough that we can just add an example showing how to do this? Then people can tailor that e.g. to use various traversal orders. (We could just link to the code in traceback.py, but it probably is full of distractions.)

[Guido van Rossum]

So is "fail-fast if you forget to handle an ExceptionGroup" really a feature? (Do we call this out in the PEP?)

We may believe that "except Exception" is an abuse, but it is too common to dismiss out of hand. I think if some app has e.g. a main loop where they repeatedly do something that may fail in many ways (e.g. handle a web request), catch all errors and then just log the error and continue from the top, it's a better experience if it logs "ExceptionGroup: <message> [<list of subexceptions>]" than if it crashes.

There's also code that catches Exception, logs the error and some relevant data, and then re-raises it. The logged error may go to a more specialized destination than the generic traceback, and the log message may contain additional data that's only available in that stack frame.

So I think there are enough serious use cases that we should do what's best for those use cases, and not try to lecture users about abuse of the idiom.

I don't know what we would have done if we were building Python from scratch. Possibly we would not have BaseException at all, and the whole mess would go away. (But there are some good reasons why we introduced BaseException, so I don't know that that would really be a better overall experience.)

--Guido

[Irit Katriel via Python-Dev]

We don't call it out but we talked about this at the sprint.  I think the reason we didn't come up with this solution then is that at the time ExceptionGroups in our plans were not immutable, so this would not have been possible (you don't know at construction time what it is going to contain).

[Guido van Rossum]

Then we should get some more opinions on this. I think it's the best idea so far for kindness towards code using `except Exception:`. I even think that the names I came up with are reasonable. The pseudo-code needs work: when instantiating ExceptionGroup, all errors should inherit from Exception, and I can't decide which base class for ExceptionGroup should come first: Exception or BaseExceptionGroup. I'm currently leaning slightly towards putting BaseExceptionGroup first. This is my current attempt:

```

class BaseExceptionGroup(BaseException):

    def __new__(cls, msg, errors):

        if cls is BaseExceptionGroup and all(isinstance(e, Exception) for e in errors):

            cls = ExceptionGroup

        else:

            assert all(isinstance(e, Exception) for e in errors)

        return super().__new__(cls, msg, errors)

    def __init__(self, msg, errors):

        self.msg = msg

        self.errors = errors

class ExceptionGroup(BaseExceptionGroup, Exception):

    pass

```

OT: Is ExceptionGroup *really* immutable in the current implementation? As long as the 'errors' field is a list, I think one could mutate the list directly. Which brings me to the question, do you have a branch that matches the PEP yet?

--Guido

[Irit Katriel via Python-Dev]

On Thu, Feb 25, 2021 at 9:06 PM Guido van Rossum <gu...@python.org> wrote:

Hm, a different idea: maybe it's simple enough that we can just add an example showing how to do this? Then people can tailor that e.g. to use various traversal orders. (We could just link to the code in traceback.py, but it probably is full of distractions.)

I've added it here: https://github.com/python/peps/pull/1841 

I'd rather do this for now, and add it to the standard library only when we have a better idea about common patterns // best practices for handling ExceptionGroups.

[Guido van Rossum]

Great. Let’s wait for others to catch up with the discussion then.

On Thu, Feb 25, 2021 at 18:44 Irit Katriel <iritk...@googlemail.com> wrote:

On Fri, Feb 26, 2021 at 2:00 AM Guido van Rossum <gu...@python.org> wrote:

Then we should get some more opinions on this. I think it's the best idea so far for kindness towards code using `except Exception:`.

I agree that it's the best idea so far.

If someone says 'except Exception' they better mean it because we'll believe them, and if someone forgets to handle an ExceptionGroup then they have a bug and that's how it is.

OT: Is ExceptionGroup *really* immutable in the current implementation? As long as the 'errors' field is a list, I think one could mutate the list directly.

It's not, but we were going to make it an immutable tuple. 

 

Which brings me to the question, do you have a branch that matches the PEP yet?

The implementation matches the PEP less and less every day because the PEP is developing faster than the implementation.

But these aren't differences that are more than a technicality to fix (rename things, move an error from runtime to the parser, things like that).

The except* implementation is probably pretty close to the PEP because it's the most recent bit.

--

--Guido (mobile)

[Irit Katriel via Python-Dev]

On Fri, Feb 26, 2021 at 2:00 AM Guido van Rossum <gu...@python.org> wrote:

Then we should get some more opinions on this. I think it's the best idea so far for kindness towards code using `except Exception:`.

I agree that it's the best idea so far.

If someone says 'except Exception' they better mean it because we'll believe them, and if someone forgets to handle an ExceptionGroup then they have a bug and that's how it is.

OT: Is ExceptionGroup *really* immutable in the current implementation? As long as the 'errors' field is a list, I think one could mutate the list directly.

It's not, but we were going to make it an immutable tuple. 

 

Which brings me to the question, do you have a branch that matches the PEP yet?

[Nathaniel Smith]

On Thu, Feb 25, 2021 at 2:13 PM Guido van Rossum <gu...@python.org> wrote:
>
> So is "fail-fast if you forget to handle an ExceptionGroup" really a feature? (Do we call this out in the PEP?)
>
> We may believe that "except Exception" is an abuse, but it is too common to dismiss out of hand. I think if some app has e.g. a main loop where they repeatedly do something that may fail in many ways (e.g. handle a web request), catch all errors and then just log the error and continue from the top, it's a better experience if it logs "ExceptionGroup: <message> [<list of subexceptions>]" than if it crashes.

Yeah, 'except Exception' happens a lot in the wild, and what to do
about that has been a major sticking point in the ExceptionGroup
debates all along. I wouldn't say that 'except Exception' is an abuse
even -- what do you want gunicorn to do if your buggy flask app raises
some random exception? Crash your entire web server, or log it and
attempt to keep going? (This is almost your example, but adding in the
part where gunicorn is reliable and well-respected, and that its whole
job is to invoke arbitrarily flaky code written by random users.)
Yury/I/others did discuss the idea of a
BaseExceptionGroup/ExceptionGroup split a lot, and I think the general
feeling is that it could potentially work, but feels like a
complicated and awkward hack, so no-one was super excited about it.
For a while we also had a compromise design where only
BaseExceptionGroup was built-in, but we left it non-final specifically
so asyncio could define an ExceptionsOnlyExceptionGroup.

Another somewhat-related awkward part of the API is how ExceptionGroup
and plain-old 'except' should interact *in general*. The intuition is
that if you have 'except ValueError' and you get an
'ExceptionGroup(ValueError)', then the user's code has some kind of
problem and we should probably do.... something? to let them know? One
idea I had was that we should raise a RuntimeError if this happens,
sort of similar to PEP 479. But I could never quite figure out how
this would help (gunicorn crashing with a RuntimeError isn't obviously
better than gunicorn crashing with an ExceptionGroup).

== NEW IDEA THAT MAYBE SOLVES BOTH PROBLEMS ==

Proposal:

- any time an unwinding ExceptionGroup encounters a traditional
try/except, then it gets replaced with a RuntimeError whose __cause__
is set to the original ExceptionGroup and whose first traceback entry
points to the offending try/except block

- CUTE BIT I ONLY JUST THOUGHT OF: this substitution happens right
*before* we start evaluating 'except' clauses for this try/except

So for example:

If an ExceptionGroup hits an 'except Exception': The ExceptionGroup is
replaced by a RuntimeError. RuntimeError is an Exception, so the
'except Exception' clause catches it. And presumably logs it or
something. This way your log contains both a notification that you
might want to switch to except* (from the RuntimeError), *along with*
the full original exception details (from the __cause__ attribute). If
it was an ExceptionGroup(KeyboardInterrupt), then it still gets caught
and that's not so great, but at least you get the RuntimeError to
point out that something has gone wrong and tell you where?

If an ExceptionGroup(ValueError) hits an 'except ValueError': it
doesn't get caught, *but* a RuntimeError keeps propagating out to tell
you you have a problem. And when that RuntimeError eventually hits the
top of your program or ends up in your webserver logs or whatever,
then the RuntimeError's traceback will point you to the 'except
ValueError' that needs to be fixed.

If you write 'except ExceptionGroup': this clause is a no-op that will
never execute, because it's impossible to still have an ExceptionGroup
when we start matching 'except' clauses. (We could additionally emit a
diagnostic if we want.)

If you write bare 'except:', or 'except BaseException': the clause
always executes (as before), but they get the RuntimeError instead of
the ExceptionGroup. If you really *wanted* the ExceptionGroup, you can
retrieve it from the __cause__ attribute. (The only case I can think
of where this would be useful is if you're writing code that has to
straddle both old and new Python versions *and* wants to do something
clever with ExceptionGroups. I think this would happen if you're
implementing Trio, or implementing a higher-level backport library for
catching ExceptionGroups, something like that. So this only applies to
like half a dozen users total, but they are important users :-).)

-n

--
Nathaniel J. Smith -- https://vorpus.org

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/JYRWNHRBPFQQPT44TZQRTVFNXPAY27UL/

[Glenn Linderman]

I wondered why an ExceptionGroup couldn't just be an Exception. This
effectively makes it so (via wrapping).

So then why do you need  except*  at all?  Only to catch unwrapped
ExceptionGroup before it gets wrapped?

So why not write except ExceptionGroup, and let it catch unwrapped
ExceptionGroup?

That "CUTE BIT" could be done only when hitting an except chain that
doesn't include an except ExceptionGroup.

Nope, I didn't read the PEP, and don't understand the motivation, but
the discussion sure sounded confusing. This is starting to sound almost
reasonable.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/PQ352LVVPKL5DJAPN73MRTHD5RBXIXIH/

[Nathaniel Smith]

On Thu, Feb 25, 2021 at 10:23 PM Glenn Linderman <v+py...@g.nevcal.com> wrote:
> So then why do you need except* at all? Only to catch unwrapped
> ExceptionGroup before it gets wrapped?
>
> So why not write except ExceptionGroup, and let it catch unwrapped
> ExceptionGroup?
>
> That "CUTE BIT" could be done only when hitting an except chain that
> doesn't include an except ExceptionGroup.
>
> Nope, I didn't read the PEP, and don't understand the motivation, but
> the discussion sure sounded confusing. This is starting to sound almost
> reasonable.

I'm not sure what to make of the complaint that not reading the
motivation makes the motivation confusing :-).

But very briefly: the core reason we need ExceptionGroup is because in
concurrent programs, multiple things can go wrong at the same time, so
our error handling system needs to have some way to represent and cope
with that. This means that in concurrent programs (e.g. anything using
asyncio, trio, etc.), it's safest to assume that *any* exception could
be wrapped in an ExceptionGroup and use except* for everything. The
question is how to make error handling in those programs as ergonomic
as Python exceptions are currently in non-concurrent programs, without
creating too many issues for existing code.

So in programs like this, you have to assume that 'except
ExceptionGroup' catches *all* exceptions, or worse, a
random/unpredictable subset. Saying that that would be good enough is
like saying that bare 'except:' is good enough for regular
non-concurrent Python (after all, you can always do an isinstance
check inside the 'except' block and re-raise the ones you don't want,
right?), and that 'except <some type>' is a pointless frivolity. I
think most people would disagree with that :-).

-n

--
Nathaniel J. Smith -- https://vorpus.org

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/BYCDFS5FVIF4FYNKUPVEVWRSCCEIU6CB/

[Irit Katriel via Python-Dev]

FTR: nobody on this long thread so far has suggested that there are no valid use cases for `except Exception`. 

Thank you for turning to what happens with 'except ValueError' when an ExceptionGroup[ValueError] is raised, this is important. 

I'm not sure it's safe to assume that it is necessarily a programming error, and that the interpreter can essentially break the program in this case.

Is this not allowed?

try:

    try:

        obj.func()    # function that raises ExceptionGroups

    except AttributeError:

        logger.info("obj doesn't have a func")

except *(AttributeError, SyntaxError):

    logger.info("func had some problems")

[Nathaniel Smith]

On Fri, Feb 26, 2021 at 5:05 AM Irit Katriel <iritk...@googlemail.com> wrote:
> I'm not sure it's safe to assume that it is necessarily a programming error, and that the interpreter can essentially break the program in this case.
> Is this not allowed?
>
> try:
> try:
> obj.func() # function that raises ExceptionGroups
> except AttributeError:
> logger.info("obj doesn't have a func")
> except *(AttributeError, SyntaxError):
> logger.info("func had some problems")

I'd be fine with disallowing that. The intuition is that things will
be simplest if ExceptionGroup is kept as transparent and meaningless
as possible, i.e. ExceptionGroup(ValueError) and ValueError mean
exactly the same thing -- "some code inside this block raised
ValueError" -- and ideally should be processed in exactly the same
way. (Of course we can't quite achieve that due to backcompat issues,
but the closer we can get, the better, I think?)

If you need to distinguish between the AttributeError from
'obj.__getattr__("func")' vs the AttributeError from the call to
func(), then there's already an obvious way to do that, that works for
all functions, not just ones that happen to raise ExceptionGroups:

try:
f = obj.func
except AttributeError:
...
try:
f()
except ...: # or except *...:
...

-n

--
Nathaniel J. Smith -- https://vorpus.org

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/VSIGKUNU2GNLMKKH4GLMRSICTQYVLVCZ/

[Emily Bowman]

A side benefit is that if an Exception somehow propagates up where only ExceptionGroup is defined, except *() could just work anyway, though it might take a little magic to make sure they act the same. Like Guido said, I don't think it can be retrofitted into existing *-less APIs, and it'll either need a new API and deprecation, or dumping the old for the new hotness if that's the maintenance strategy, but as long as docs call out that this is now returning collective exceptions, I don't see a problem.

I don't think it's been mentioned, but it's a nice synergy with pattern matching, even if it's wholly unrelated, and having both in one release will actually make sense.

-Em

[Irit Katriel via Python-Dev]

On Fri, Feb 26, 2021 at 2:42 PM Nathaniel Smith <n...@pobox.com> wrote:

On Fri, Feb 26, 2021 at 5:05 AM Irit Katriel <iritk...@googlemail.com> wrote:
> I'm not sure it's safe to assume that it is necessarily a programming error, and that the interpreter can essentially break the program in this case.
> Is this not allowed?
>
> try:
>     try:
>         obj.func()    # function that raises ExceptionGroups
>     except AttributeError:
>         logger.info("obj doesn't have a func")
> except *(AttributeError, SyntaxError):
>     logger.info("func had some problems")

I'd be fine with disallowing that. The intuition is that things will
be simplest if ExceptionGroup is kept as transparent and meaningless
as possible, i.e. ExceptionGroup(ValueError) and ValueError mean
exactly the same thing -- "some code inside this block raised
ValueError" -- and ideally should be processed in exactly the same
way.

But they don't mean the same thing if this prints hello:

try:

    raise ValueError()

except ValueError:

   print('hello') 

and this raises a RuntimeError:

try:

    raise ExceptionGroup("", [ValueError()])

except ValueError:

   print('hello')

What am I missing?

[Marco Sulla]

Excuse me if I post here. Maybe is a stupid question: why, instead of
introducing except*, Python can't extend the functionality of except,
so it can do what except* would do?

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/34E3TMVLHRT2UQSH5RWZXOPT6AQ6DTYZ/

[Jim J. Jewett]

> We keep the ability to wrap any exception, while we lose the "fail-fast if
> you forget to handle an ExceptionGroup" feature, which was intended as a
> kindness towards those who abuse "except Exception".

How is this a kindness?

Whenever I've used except Exception or stronger, it was a sanitary barrier around code that might well do unpredictable or even stupid things. Adding a new kind of exception that I hadn't predicted -- including ExceptionGroup -- would certainly fit this description, and I want my driver loop to do what I told it. (Probably log an unexpected exception, and continue with the next record. I honestly don't even want a page, let alone a crash, because data from outside that barrier ... is often bad, and almost never in ways the on-call person can safely fix. And if they don't have time to find it in the logs, then it isn't a priority that week.)

> If we adopt this solution then letting an ExceptionGroup escape from code
> that is not supposed to raise it, is not a fatal error, it's just some
> exception like any other.

Good! If we're not coordinating so closely that I already know to handle the ExceptionGroup in advance, then that is exactly what should happen (and except Exception with a log and log analysis is the right way to deal with it).

> So there is no longer a distinction between code that raises
> ExceptionGroups and code that doesn't. Any code can propagate them, like
> any code can raise any other exception.

Good; special cases and all.

> Does this mean that more code needs to be aware of the possibility of them
> showing up? Is that a problem?

Honestly, no. You seem to be assuming a very well-controlled environment where any potential problems would be caught long before production. My experience is that such optimism is never warranted, *particularly* at places that claim to have a heavy process to ensure such early (or in-time) bug-catches.

> What would we have done here if we were building Python from scratch?

Raise anything you want. Or maybe only strings and exceptions. Or maybe only stuff inheriting from a marker class named BaseException ... but we probably wouldn't add a parallel base marker that catch-all code *also* needs to be aware of. (And since we would be starting from scratch, the catch-all wrapper code would certainly not have to be deployed on a conservatively managed server, before lightweight exploratory less-centralized client code could start using it.)

-jJ

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/TEBYSGMMT3SJRVTPCZQFMHJQPBOV6Q74/

[Guido van Rossum]

On Thu, Feb 25, 2021 at 9:40 PM Nathaniel Smith <n...@pobox.com> wrote:

[...]

Yury/I/others did discuss the idea of a
BaseExceptionGroup/ExceptionGroup split a lot, and I think the general
feeling is that it could potentially work, but feels like a
complicated and awkward hack, so no-one was super excited about it.

Oh, I didn't realize (or recall) the idea was considered before. Given the pseudo code I showed I don't think it's particularly complicated or awkward, although it's of course more complicated than the design that's currently in our PEP.

 

For a while we also had a compromise design where only
BaseExceptionGroup was built-in, but we left it non-final specifically
so asyncio could define an ExceptionsOnlyExceptionGroup.

I did notice that. Given how many distinct use cases we've discovered by now I think this is a dead end.

 

Another somewhat-related awkward part of the API is how ExceptionGroup
and plain-old 'except' should interact *in general*. The intuition is
that if you have 'except ValueError' and you get an
'ExceptionGroup(ValueError)', then the user's code has some kind of
problem and we should probably do.... something? to let them know? One
idea I had was that we should raise a RuntimeError if this happens,
sort of similar to PEP 479. But I could never quite figure out how
this would help (gunicorn crashing with a RuntimeError isn't obviously
better than gunicorn crashing with an ExceptionGroup).

I'm not sure I agree with the intuition that such user code definitely has a problem. Irit tried to give an example using AttributeError, and while that particular example is easily countered, I think it's not as straightforward as you state. For example there could be a framework exception that we know statically will not be wrapped in ExceptionGroup. Basically, if we have some code that can either raise some exceptions of category A wrapped in ExceptionGroup, or a bare exception of category B (without overlap between the categories), writing code that catches category B using `except B:` should not affect the propagation of ExceptionGroup.

Or what about code that uses this idiom?

```

try:

    . . .

except Exception:

    . . .

    raise

```

There should be no need to switch this to `except *` just to prevent the ExceptionGroup from becoming a RuntimeError.

This is all very clever, but something feels off with it. Perhaps it improves this:

```

try:

    raise ExceptionGroup(ValueError)  # Hidden in more complex code

except ValueError:

    . . .

```

(Though how? The RuntimeError wouldn't be caught.)

But I feel strongly that if the ExceptionGroup doesn't wrap any exceptions that would be caught by the except clauses of a given try statement, the ExceptionGroup should bubble through unchanged. As you said yourself, "gunicorn crashing with a RuntimeError isn't obviously better than gunicorn crashing with an ExceptionGroup."

In fact, if you're happy with RuntimeError here, making ExceptionGroup inherit from Exception (instead of BaseException) would do just as well -- after all RuntimeError is pretty arbitrary (in fact, it's wrong, because the problem is a static bug in the code, not something that went wrong at run time).

Let's see how this compares to the alternatives. First let's define three key examples.

Example 1:

```

try:

    raise ExceptionGroup(ValueError)

except Exception:

    . . .

```

Example 2:

```

try:

    raise ExceptionGroup(ValueError)

except ValueError:

    . . .

```

Example 3:

```

try:

    raise ExceptionGroup(asyncio.CancelledError)

except Exception:

    . . .

```

Now let's tabulate the outcome for each of the proposals. Does the except clause get executed?

```

                        Example 1       Example 2       Example 3

Current PEP                No              No              No

BaseExceptionGroup +

ExceptionGroup             Yes             No              No

Inherit from Exception     Yes             No              Yes

Change to RuntimeError     Yes             No(*)           Yes

```

(*) Spoils things for outer frames that use `except *` correctly.

So example 2 mostly doesn't help discriminating between examples. What we arguably want there is `Yes`, but that would require adding more logic to the exception matching code (and there are other reasons why we don't want that).

I feel that the intended result for Example 3 is No (because CancelledError inherits directly from BaseException), so if we want a Yes for Example 1, BaseExceptionGroup + ExceptionGroup still looks more attractive than any other proposal.

[Guido van Rossum]

On Fri, Feb 26, 2021 at 3:18 PM Marco Sulla <Marco.Sul...@gmail.com> wrote:

Excuse me if I post here. Maybe is a stupid question: why, instead of
introducing except*, Python can't extend the functionality of except,
so it can do what except* would do?

Good question. Here's an example:

```

try:

    . . .

except OSError as err:

    if err.errno != ENOENT:

        raise

    . . .

```

If this would catch ExceptionGroup(OSError), the `err` variable would be an ExceptionGroup instance, which does not have an `errno` attribute.

(Irit: Does the PEP answer this question? I couldn't quickly find it in the rejected ideas. I think it's a reasonable question and we should answer it, either in the Rationale or in Rejected Ideas.)

[Jim J. Jewett]

FWIW, the only situation I can think of where you would care that the enclosed exception instances are BaseException but not regular Exception is interactive debugging, and even then there are enough other ways to kill the whole process that I think most people would use one of them instead of wanting a different BaseExceptionGroup that breaks the server instead of just the servlet.

I suppose you could argue that the distinction encourages the "good practice" of defensively wrapping "except Exception" in an "except BaseException" that is itself wrapped in a bare except. I suspect it would actually just push people to replace that "except Exception" with the bare except and give up on the logging, because that is a quicker adjustment.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/IDVNUNBAMFKZVT64TFV5QYLB3BJTABBD/

[Jim J. Jewett]

You still need except* for the (unusual?) case where the ExceptionGroup contains multiple individual Exceptions, and you want them all to be processed. (This possibility is the justification for the PEP, but the difficulty of associating an exception with the specific task that raised it suggests that exceptions are still intended to be rare, rather than a back-channel communications band.)

As written, you also need except* to unwrap, so that ExceptionGroup(ValueError) can be handled by "except ValueError" instead of "except RuntimeError" (or except ExceptionGroup) followed by cause-unwrapping.

-jJ

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/B6JKV2JIFA76CMDOK2IBVBA2MIQKYDVR/

[Greg Ewing]

While I don't particularly mind if we get ExceptionGroup, giving
it special magical semantics, and especially new syntax, seems like
bringing a massively overpowered weapon to bear on something that
will only be used rarely.

Handling multiple exceptions from an ExceptionGroup could be done
using a loop with pattern matching, so is except* really justified?

--
Greg

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/BRQDP4KNRYU6UCKQBHRAUAHWWCENNIVM/

[Jim J. Jewett]

> Thank you for turning to what happens with 'except ValueError' when an
> ExceptionGroup[ValueError] is raised, this is important.

> I'm not sure it's safe to assume that it is necessarily a > programming
>error, and that the interpreter can essentially break the program in this
> case.

I'm betting it means a ValueError was raised, but then something (probably an asynchronous framework) aggregated it. I won't swear you would never want to distinguish the two cases, or to distinguish them both from ExceptionGroup[ExceptionGroup[ExceptionGroup[ValueError]]], but ... normally you wouldn't.

> Is this not allowed?

>try:
> try:
> obj.func() # function that raises ExceptionGroups
> except AttributeError:
> logger.info("obj doesn't have a func")
>except *(AttributeError, SyntaxError):
> logger.info("func had some problems")

Allowed, but probably in error ... no AttributeError will get through to the except * unless it happened inside the except AttributeError handler. Did you mean:

try:
try:
obj.func # function that raises ExceptionGroups

except AttributeError:
logger.info("obj doesn't have a func")

obj.func()

except *(AttributeError, SyntaxError):
logger.info("func had some problems")

I see this as an argument that the except/except* split is tricky, but I don't think it says anything about whether except* clauses should be able to see into nested ExceptionGroups ... nor am I at all confident that I understood your intent.

-jJ

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/XOAB7IJNXOHRL3HRVZ5VZON6MVHOPXB3/

[Irit Katriel via Python-Dev]

On Sat, Feb 27, 2021 at 12:47 AM Jim J. Jewett <jimjj...@gmail.com> wrote:

> Is this not allowed?

>try:
>    try:
>        obj.func()    # function that raises ExceptionGroups
>    except AttributeError:
>        logger.info("obj doesn't have a func")
>except *(AttributeError, SyntaxError):
>    logger.info("func had some problems")

Allowed, but probably in error ... no AttributeError will get through to the except * unless it happened inside the except AttributeError handler.  Did you mean:

If obj.func() raises an ExceptionGroup that contains AttributeError then "except AttributeError" doesn't catch it. So it will get through.

[Irit Katriel via Python-Dev]

On Fri, Feb 26, 2021 at 11:43 PM Guido van Rossum <gu...@python.org> wrote:

On Fri, Feb 26, 2021 at 3:18 PM Marco Sulla <Marco.Sul...@gmail.com> wrote:

Excuse me if I post here. Maybe is a stupid question: why, instead of
introducing except*, Python can't extend the functionality of except,
so it can do what except* would do?

Good question. Here's an example:

```

try:

    . . .

except OSError as err:

    if err.errno != ENOENT:

        raise

    . . .

```

If this would catch ExceptionGroup(OSError), the `err` variable would be an ExceptionGroup instance, which does not have an `errno` attribute.

(Irit: Does the PEP answer this question? I couldn't quickly find it in the rejected ideas. I think it's a reasonable question and we should answer it, either in the Rationale or in Rejected Ideas.)

Added here:  https://github.com/python/peps/pull/1846

[Irit Katriel via Python-Dev]

On Sat, Feb 27, 2021 at 12:35 AM Greg Ewing <greg....@canterbury.ac.nz> wrote:

While I don't particularly mind if we get ExceptionGroup, giving
it special magical semantics, and especially new syntax, seems like
bringing a massively overpowered weapon to bear on something that
will only be used rarely.

Handling multiple exceptions from an ExceptionGroup could be done
using a loop with pattern matching, so is except* really justified?

It is of course an option to split the PEP into two, add ExceptionGroup first and then ask ourselves if we want except*. 

We do have some experience with this from the Trio experiments with MultiError though, so we are not starting from scratch.  

Can you spell out how you see ExceptionGroup handling work with pattern matching?

[Jim J. Jewett]

ValueError("Partner: 3 File: 127 record: 93 is missing field: currency") tells the production support people who to contact and what to say.

I'm not sure what additional context would be helpful, let alone how it might have been available "at the time", but lost now that the ValueAttribute is collected into an ExceptionGroup.

-jJ
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/GXUJTE25TWR5AKAYCTETKL4YWXQC2FR5/

[Jim J. Jewett]

> Why would you want a different flavor of ExceptionGroup?

The easiest example you took care of by saving the exceptions in a list instead of a set. Next would be the ExceptionGroup vs BaseExceptionGroup that I *don't* like, but someone might. There are also libraries that promise to raise only exceptions derived (possibly through multiple inheritance) from a particular marker exception.

But honestly, my biggest concern is that it just seems wrong for any class to be final, and it has sometimes been an irritant even for such obvious cases cases as Boolean. So why is this so special?

-jJ
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/2AJJIOLL4RB2TKLYDA2Y7HGDRWEBUD5J/

[Paul Moore]

On Fri, 26 Feb 2021 at 23:36, Jim J. Jewett <jimjj...@gmail.com> wrote:
>
> Whenever I've used except Exception or stronger, it was a sanitary barrier around code that might well do unpredictable or even stupid things. Adding a new kind of exception that I hadn't predicted -- including ExceptionGroup -- would certainly fit this description, and I want my driver loop to do what I told it. (Probably log an unexpected exception, and continue with the next record. I honestly don't even want a page, let alone a crash, because data from outside that barrier ... is often bad, and almost never in ways the on-call person can safely fix. And if they don't have time to find it in the logs, then it isn't a priority that week.)

This is my biggest concern. Disclaimer: I've yet to read the PEP,
because async makes my head hurt, but I am aware of some of the
background with Trio. Please take this as the perspective of someone
thinking "I don't use async/await in my code, can I assume this
doesn't affect me?"

For me, the biggest requirement I would have is that if I have code like:

def safe_fn():
try:
do_something_funky()
return True
except Exception:
print("Well, that was a shame...")
return False

then I am intending to guarantee that calling safe_fn() will never
raise an exception. Obviously, the example is a toy (in reality, I'd
log the error, skip a record, whatever) but that's not the point here
- the point is that I consider it reasonable to expect `except
Exception` to be a hard barrier that allows me to be sure I've covered
everything.

My worry is that if ExceptionGroup exceptions are *not* trapped by
`except Exception`, then code like this is no longer safe. And by
making ExceptionGroup a BaseException subclass, that's what will
happen.

Ultimately, there's a genuine use case for code that says "whatever
happens in the code I call, this is my last line of defense and I want
to be as near to 100% sure as I can be that I regain control at this
point". At the moment, that is spelled `except Exception`. If you're
proposing to change that, even if it's just to require that it be
spelled differently, you're going to break quite a lot of code - and
worse, the code you're breaking will be code that has taken care to
ensure it's written safely, so you're penalising people who *tried to
get stuff right*, which IMO is the worst type of breakage.

I will at some point read the PEP fully, to better understand the
async side of the story, but for now please consider this as the
perspective of someone who doesn't expect to care about async, and
therefore wants to feel safe in assuming this PEP won't affect them.

Paul

PS If you're thinking "but if you were using async, you'd know about
it", consider that I could be using a PyPI library that, for example,
returned stock values. That library switches to using async, and has a
bug that lets an exception group escape. My `except Exception` code is
designed *precisely* to protect me against such bugs, without me
needing to know anything about how the library works...

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/SKU4RIRPRRGESKRMP55ZPZNYL4XWFFUR/

[Jim J. Jewett]

> In fact, if you're happy with RuntimeError here, making
> ExceptionGroup inherit from Exception (instead of
> BaseException) would do just as well -- after all RuntimeError
> is pretty arbitrary

Agreed, and I think it should inherit from Exception.

> (in fact, it's wrong, because the problem is a static bug in the
> code, not something that went wrong at run time).

Something went wrong deep inside an individual task. The only clear static bug is that the async library changed how it reported that, and my except clause hasn't done the make-work to keep in step.

> Let's see how this compares to the alternatives. First let's define
> three key examples.

> Example 1:

> try:
> raise ExceptionGroup(ValueError)
> except Exception:

> Example 2:

> try:
> raise ExceptionGroup(ValueError)
> except ValueError:

Example 2(a) :

try:
raise ExceptionGroup(ValueError,OtherError)
except ValueError:

> Example 3:

> try:
> raise ExceptionGroup(asyncio.CancelledError)
> except Exception:

I would prefer that the except clause be executed in all of the examples, but I admit that 2a is a problem, because it could end up silently losing OtherError. And I admit getting example 2 to do the right thing if 2a doesn't might be too much of a contortion.

For example 3, I may be missing a subtle point, but I feel that by the time you get to code which doesn't expect asyncio.CancelledError, then you have already cancelled as far as you should. Cancelling may want to bubble up through several other tasks, but it shouldn't kill the whole server, just because the trampoline got sloppy.

-jJ

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/RHL3E2IBCIW2SP5L5W3BHWIWB4DV6U55/

[Guido van Rossum]

On Sat, Feb 27, 2021 at 2:39 AM Paul Moore <p.f....@gmail.com> wrote:

On Fri, 26 Feb 2021 at 23:36, Jim J. Jewett <jimjj...@gmail.com> wrote:
>
> Whenever I've used except Exception or stronger, it was a sanitary barrier around code that might well do unpredictable or even stupid things.  Adding a new kind of exception that I hadn't predicted -- including ExceptionGroup -- would certainly fit this description, and I want my driver loop to do what I told it.  (Probably log an unexpected exception, and continue with the next record.  I honestly don't even want a page, let alone a crash, because data from outside that barrier ... is often bad, and almost never in ways the on-call person can safely fix.  And if they don't have time to find it in the logs, then it isn't a priority that week.)

This is my biggest concern. Disclaimer: I've yet to read the PEP,
because async makes my head hurt, but I am aware of some of the
background with Trio. Please take this as the perspective of someone
thinking "I don't use async/await in my code, can I assume this
doesn't affect me?"

The PEP doesn't actually do anything with async -- it just uses asyncio as one of the motivating use cases. The spec itself does not use async or asyncio. At some point in the future we might consider adding new APIs to asyncio that can raise ExceptionGroup, but other uses of 'async def' will definitely not be affected.

Our goal for the PEP is that *unless* you're going to use APIs that are documented to raise ExceptionGroup, you won't have to use `except *` nor will you have to deal with ExceptionGroup otherwise.

 

For me, the biggest requirement I would have is that if I have code like:

def safe_fn():
    try:
        do_something_funky()
        return True
    except Exception:
        print("Well, that was a shame...")
        return False

then I am intending to guarantee that calling safe_fn() will never
raise an exception. Obviously, the example is a toy (in reality, I'd
log the error, skip a record, whatever) but that's not the point here
- the point is that I consider it reasonable to expect `except
Exception` to be a hard barrier that allows me to be sure I've covered
everything.

The modification I proposed where we have both BaseExceptionGroup and ExceptionGroup will satisfy this need. It basically means that *unless* you are explicitly using an API that is documented to raise [Base]ExceptionGroup (such as a future variant of asyncio.gather()), you don't have to care about it. The other exception is if you're writing a library for formatting exceptions (like the stdlib traceback.py module).

 

My worry is that if ExceptionGroup exceptions are *not* trapped by
`except Exception`, then code like this is no longer safe. And by
making ExceptionGroup a BaseException subclass, that's what will
happen.

Right, and that's why I am proposing to change the PEP so that your code will remain safe.

Note that the class headings are like this:

```

class BaseExceptionGroup(BaseException): ...

class ExceptionGroup(BaseExceptionGroup, Exception): ...

```

which produces the following MROs:

```

ExceptionGroup -> BaseExceptionGroup -> Exception -> BaseException -> object

                  BaseExceptionGroup ->              BaseException -> object

```

Ultimately, there's a genuine use case for code that says "whatever
happens in the code I call, this is my last line of defense and I want
to be as near to 100% sure as I can be that I regain control at this
point". At the moment, that is spelled `except Exception`. If you're
proposing to change that, even if it's just to require that it be
spelled differently, you're going to break quite a lot of code - and
worse, the code you're breaking will be code that has taken care to
ensure it's written safely, so you're penalising people who *tried to
get stuff right*, which IMO is the worst type of breakage.

I got you.

 

I will at some point read the PEP fully, to better understand the
async side of the story, but for now please consider this as the
perspective of someone who doesn't expect to care about async, and
therefore wants to feel safe in assuming this PEP won't affect them.

There really is no async side. And you will be safe.

 

Paul

PS If you're thinking "but if you were using async, you'd know about
it", consider that I could be using a PyPI library that, for example,
returned stock values. That library switches to using async, and has a
bug that lets an exception group escape. My `except Exception` code is
designed *precisely* to protect me against such bugs, without me
needing to know anything about how the library works...

And it will.

[Guido van Rossum]

If you're using 'except Exception:' and an asyncio.CancelledError  (or trio.Cancelled) exception escapes, you're already dead, because those don't inherit from Exception, only from BaseException. And everything inherits from BaseException, including [Base]ExceptionGroup.

[Larry Hastings]

On 2/27/21 2:37 AM, Paul Moore wrote:

On Fri, 26 Feb 2021 at 23:36, Jim J. Jewett <jimjj...@gmail.com> wrote:

Whenever I've used except Exception or stronger, it was a sanitary barrier around code that might well do unpredictable or even stupid things.  Adding a new kind of exception that I hadn't predicted -- including ExceptionGroup -- would certainly fit this description, and I want my driver loop to do what I told it.  (Probably log an unexpected exception, and continue with the next record.  I honestly don't even want a page, let alone a crash, because data from outside that barrier ... is often bad, and almost never in ways the on-call person can safely fix.  And if they don't have time to find it in the logs, then it isn't a priority that week.)

This is my biggest concern. Disclaimer: I've yet to read the PEP,
because async makes my head hurt, but I am aware of some of the
background with Trio. Please take this as the perspective of someone
thinking "I don't use async/await in my code, can I assume this
doesn't affect me?"

I haven't read the PEP either.  But I assume it could (should?) affect anyone managing multiple simultaneous things in Python:

• async code, "fibers", "greenlets", Stackless "microthreads", "cooperative multitasking", or any other userspace mechanism where you manage multiple "threads" of execution with multiple stacks
  
• code managing multiple OS-level threads
• code managing multiple processes
  

It seems to me that any of those could raise multiple heterogeneous exceptions, and Python doesn't currently provide a mechanism to manage this situation.  My dim understanding is that ExceptionGroup proposes a mechanism to handle exactly this thing.

Cheers,

/arry

[Cameron Simpson]

On Fri, 26 Feb 2021 at 23:36, Jim J. Jewett <jimjj...@gmail.com> wrote:
> Whenever I've used except Exception or stronger, it was a sanitary
> barrier around code that might well do unpredictable or even stupid
> things. Adding a new kind of exception that I hadn't predicted --
> including ExceptionGroup -- would certainly fit this description, and
> I want my driver loop to do what I told it.

This is my concern as well. I've got plenty of "except Exception" and
IMO none consititutes "abuse" - they're there for the same reason Jim
cites above: a barrier around _arbitrary_ callback code - so that the
outer control facility continues to roll on as expected.

All of these are handler frameworks for use by other code - they
inherently call an unknown and arbitrary thing at some point and expect
to catch an Exception raised and suitably handle it (log it, record it
for the caller to see later, whatever).

Now _all_ those handlers will need a special except* handler for
ExceptionGroup, because they DO NOT KNOW whether the called code might
raise it. And "except Exception", the longstanding recommended way to
catch "everything (well almost everything)" no longer works.

Now, I've long wanted something like MultiError or ExceptionGroup to
gather miltiple failures for raising together at a suitable
slightly-later point.

[...]

On 27Feb2021 19:06, Guido van Rossum <gu...@python.org> wrote:
>Our goal for the PEP is that *unless* you're going to use APIs that are
>documented to raise ExceptionGroup, you won't have to use `except *` nor
>will you have to deal with ExceptionGroup otherwise.

But.. this isn't Java, where the (hah!) type annotations document the
exceptions it raises.

Jim again, catching Exception to protect calling code:

>> def safe_fn():
>> try:
>> do_something_funky()
>> return True
>> except Exception:
>> print("Well, that was a shame...")
>> return False
>>
>> then I am intending to guarantee that calling safe_fn() will never

>> raise an exception. [...]

Guido:

>The modification I proposed where we have both BaseExceptionGroup and
>ExceptionGroup will satisfy this need. It basically means that *unless* you
>are explicitly using an API that is documented to raise
>[Base]ExceptionGroup (such as a future variant of asyncio.gather()), you

>don't have to care about it. [...] that's why I am proposing to change the PEP so that your
>code will remain safe.

That would be welcome to me, too.

Cheers,
Cameron Simpson <c...@cskk.id.au>

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/IKGK5YEPUPPWCAQ6NQEVGMKA63KPFUJQ/

[Cameron Simpson]

On 26Feb2021 02:44, Irit Katriel <iritk...@googlemail.com> wrote:
>On Fri, Feb 26, 2021 at 2:00 AM Guido van Rossum <gu...@python.org> wrote:
>> OT: Is ExceptionGroup *really* immutable in the current
>> implementation? As
>> long as the 'errors' field is a list, I think one could mutate the list
>> directly.
>
>It's not, but we were going to make it an immutable tuple.

Could you say why? Other than wanting to discourage mutation happy code
getting out there?

The reason I ask is that the scenario which comes to my mind is
something like:

except *OSError as e:

AIUI "e" is an ExceptionGroup containing only OSErrors. So, one common
thing in my own code is this:

try:
do something with a file
except OSError as e:
if e.errno == ENOENT:
# file is missing, but that is ok
# because we treat it like an empty file
elif ... some other ok situation ...
else:
raise

My natural inclination with an ExceptionGroup would be to winnow the
OSErrors I'm handed, and push the _unhandled_ errors back into the
original ExceptionGroup. That way, after the various except* clauses, a
nonempty ExceptionGroup would remain with the unhandled errors, and it
might perhaps be reraised then.

Cheers,
Cameron Simpson <c...@cskk.id.au>
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/FZI5JRQKTSMF7BYSIM2PJTO4E6IHSPDA/

[Guido van Rossum]

We really don’t want users pushing non-exceptions into the list, nor do we want e.g. KeyboardInterrupt to be added to a (non-Base-) ExceptionGroup.

There’s a pattern for what you propose using the split() method and a lambda, or you can keep the exceptions in a list and re-wrap them at the end.

--

--Guido (mobile)

[Irit Katriel via Python-Dev]

In earlier versions of the PEP ExceptionGroup was not immutable and split actually removed matching exceptions from it. 

It was also iterable so you could get a flat list of all the contained leaf exceptions. Then we changed it.

ExceptionGroup is a tree of exceptions, and the internal nodes of the tree (which are ExceptionGroups) have metadata on them -  context, cause, traceback.

If you want the full traceback of a leaf exception you need to concatenate the tracebacks of all the exceptions on the path from the root to the leaf.

If you flatten an ExceptionGroup and create a new list ExceptionGroup(list(eg)) you don't lose metadata (unless eg's tree has depth 1).

Similarly, if you add an exception it needs to have matching metadata to the ExceptionGroup you are adding it to. It's too easy to get that wrong.

split() and subgroup() take care to preserve the correct metadata on all the internal nodes, and if you just use them you only make safe operations.

This is why I am hesitating to add iteration utilities to the API. Like we did, people will naturally try that first, and it's not the safest API.

We actually have the  OSErrors example in the PEP, just above https://www.python.org/dev/peps/pep-0654/#caught-exception-objects:

try:
    low_level_os_operation()
except *OSerror as errors:
    raise errors.subgroup(lambda e: e.errno != errno.EPIPE) from None

[Guido van Rossum]

On Sun, Feb 28, 2021 at 2:40 AM Irit Katriel <iritk...@googlemail.com> wrote:

In earlier versions of the PEP ExceptionGroup was not immutable and split actually removed matching exceptions from it. 

It was also iterable so you could get a flat list of all the contained leaf exceptions. Then we changed it.

ExceptionGroup is a tree of exceptions, and the internal nodes of the tree (which are ExceptionGroups) have metadata on them -  context, cause, traceback.

If you want the full traceback of a leaf exception you need to concatenate the tracebacks of all the exceptions on the path from the root to the leaf.

If you flatten an ExceptionGroup and create a new list ExceptionGroup(list(eg)) you don't lose metadata (unless eg's tree has depth 1).

Is this a typo? Did you mean "If you flatten [it] and create a new list... you *do* lose metadata (unless ... depth 1)"?

 

Similarly, if you add an exception it needs to have matching metadata to the ExceptionGroup you are adding it to. It's too easy to get that wrong.

split() and subgroup() take care to preserve the correct metadata on all the internal nodes, and if you just use them you only make safe operations.

This is why I am hesitating to add iteration utilities to the API. Like we did, people will naturally try that first, and it's not the safest API.

We actually have the  OSErrors example in the PEP, just above https://www.python.org/dev/peps/pep-0654/#caught-exception-objects:

try:
    low_level_os_operation()
except *OSerror as errors:
    raise errors.subgroup(lambda e: e.errno != errno.EPIPE) from None

[Irit Katriel via Python-Dev]

On Sun, Feb 28, 2021 at 6:17 PM Guido van Rossum <gu...@python.org> wrote:

On Sun, Feb 28, 2021 at 2:40 AM Irit Katriel <iritk...@googlemail.com> wrote:

In earlier versions of the PEP ExceptionGroup was not immutable and split actually removed matching exceptions from it. 

It was also iterable so you could get a flat list of all the contained leaf exceptions. Then we changed it.

ExceptionGroup is a tree of exceptions, and the internal nodes of the tree (which are ExceptionGroups) have metadata on them -  context, cause, traceback.

If you want the full traceback of a leaf exception you need to concatenate the tracebacks of all the exceptions on the path from the root to the leaf.

If you flatten an ExceptionGroup and create a new list ExceptionGroup(list(eg)) you don't lose metadata (unless eg's tree has depth 1).

Is this a typo? Did you mean "If you flatten [it] and create a new list... you *do* lose metadata (unless ... depth 1)"?

It is a typo, thanks.

[Jim J. Jewett]

Looking at the following PEP example, I'm still not sure what he should do to handle some but not all OSError instances:

... raise ExceptionGroup(
... "eg",
... [
... ValueError(1),
... TypeError(2),
... OSError(3),
... ExceptionGroup(
... "nested",
... [OSError(4), TypeError(5), ValueError(6)])
... ]

except *OSError seems to get him an ExceptionGroup that he still has to manually tree-walk to handle the OSErrors that he can actually handle.

Once he has manually tree-walked, he has to manually re-raise the other OSErrors that he hasn't filtered out, and they become a sibling of the original ExceptionGroup, because they're (now) being raised at a different location. (Or, not *really* the original ExceptionGroup, but one that looks like it without any OSErrors.) This new sibling relationship is there to show that a failed or incomplete attempt was made at resolving the OSErrors, but no such attempt was made at the ValueErrors or TypeErrors.

Alternatively, he *might* be able to just grab Exception, then use a complicated callback to subgroup only the OSErrors that he will be able to handle, and raise the complement of that. It will still add another layer of ExceptionGroup, but won't split the remainders.

This seems like an awful lot of work to preserve and elaborate structural relations between different exceptions that were originally accidental. (I am assuming that the original relation was "these all happened, probably in different execution threads" and the new relations are just accidents of what got partially handled along the way, or maybe the order in which different execution threads got merged.)

I can't imagine ever needing to care about the full tree structure unless I'm re-implementing asycio/trio/curio. Even if that structural information is important to my own framework, that just means I should create more intermediate tasks to handle it more quickly, before the next level of aggregation.

I would much prefer an API that let me pattern-match (even just by class of the exception instances, though preferably also by attributes), process each exception (as though it were the only one) within the code block that match selects, and indicate whether that one exception should be raised further or not.

-jJ

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/WTZUGBFWFXMGAFHNTTBMUYFGMQCPV5N6/

[Guido van Rossum]

You can pass an arbitrary function to eg.split() and eg.subgroup(), and that function can have a side effect. Suppose you want to log and ignore OSError with errno==ENOENT but re-raise the rest, you can do this:

```

def log_and_ignore(err):

    if err.errno == ENOENT:

        log(err)

        return False

    else:

        return True

try:

    . . .

except *OSError as eg:

    eg = eg.subgroup(log_and_ignore)

    if eg is not None:

        raise eg

```

[Cameron Simpson]

This message is longer than I had anticipated.

To aid comprehension, I'm:

- accepting that .split and .subgroup help my "handle some excpetions
but not others" situation, barely

- arguing for ExceptionGroups acting like other containers: truthy if
nonempty, falsey if empty; iterable; .subgroup and .split _not_
returning None for an empty subgroup, so that the container-like
aspects can be used directly

On 28Feb2021 10:40, Irit Katriel <iritk...@googlemail.com> wrote:
>split() and subgroup() take care to preserve the correct metadata on
>all
>the internal nodes, and if you just use them you only make safe operations.
>This is why I am hesitating to add iteration utilities to the API. Like we
>did, people will naturally try that first, and it's not the safest API.

Wouldn't it be safe if the ExceptionGroup were immutable, as you plan?
Or have you another hazard in mind?

>We actually have the OSErrors example in the PEP, just above
>https://www.python.org/dev/peps/pep-0654/#caught-exception-objects:
>
>try:
> low_level_os_operation()
>except *OSerror as errors:
> raise errors.subgroup(lambda e: e.errno != errno.EPIPE) from None

Indeed. That basicly addresses my pattern. Along with:

>On Sun, Feb 28, 2021 at 6:30 AM Guido van Rossum <gu...@python.org>
>wrote:

>> There’s a pattern for what you propose using the split() method and a
>> lambda, or you can keep the exceptions in a list and re-wrap them at the
>> end.

The keep-a-list approach was my fallback, absent a way to push an
unhandled exception back in some sense.

Guido:

>> We really don’t want users pushing non-exceptions into the list, nor
>> do we
>> want e.g. KeyboardInterrupt to be added to a (non-Base-) ExceptionGroup.

I was only imagining pushing exceptions from the original group back in.
Enforcing that would be tedious and confusing though, so I was imagining
some way of marking specific subexeceptions as handled or not handled.

But I had not understood the subgroup method.

I think my handled/unhandled concerns are (barely) sufficient addressed
above. If I wanted to sweep the group for handled exceptions and then
reraise the unhandled ones in their own ExceptionGroup at the end, that
seems tractable.

But all that said, being able to iterable the subexceptions seems a
natural way to manage that:

unhandled = []
try:
.........
except *OSError as eg:
for e in eg:
if an ok exception:
handle it
else:
unhandled.append(e)
if unhandled:
raise ExceptionGroup("unhandled", unhandled)

There are some immediate shortcomings above. In particular, I have no
way of referencing the original ExceptionGroup without surprisingly
cumbersome:

try:
.........
except ExceptionGroup as eg0:
unhandled = []
eg, os_eg = eg0.split(OSError)
if os_eg:
for e in os_eg:
if an ok exception:
handle it
else:
unhandled.append(e)
if eg:
eg, value_eg = eg.split(ValueError)
if value_eg:
for e in value_eg:
if some_ValueError_we_understand:
handle it
else:
unhandled.append(e)
if eg:
unhandled.append(eg)
if unhandled:
raise ExceptionGroup("unhandled", unhandled) from eg0

I have the following concerns with the pattern above:

There's no way to make a _new_ ExceptionGroup with the same __cause__
and __context__ and message as the original group: not that I can't
assign to these, but I'd need to enuerate them; what if the exceptions
grew new attributes I wanted to replicate?

This cries out for another factory method like .subgroup but which makes
a new ExceptionGroup from an original group, containing a new sequence
of exceptions but the same message and coontext. Example:

unhandled_eg = eg0.with_exceptions(unhandled)

I don't see a documented way to access the group's message.

I'm quite unhappy about .subgroup (and presumably .split) returning None
when the group is empty. The requires me to have the gratuitous "if eg:"
and "if value_eg:" if-statements in the example above.

If, instead, ExceptionGroups were like any other container I could just
test if they were empty:

if eg:

_and_ even if they were empty, iterate over them. Who cares if the loop
iterates zero times? The example code would become:

try:
.........
except ExceptionGroup as eg0:
unhandled = []
eg, os_eg = eg0.split(OSError)
for e in os_eg:
if an ok exception:
handle it
else:
unhandled.append(e)
eg, value_eg = eg.split(ValueError)
for e in value_eg:
if some_ValueError_we_understand:
handle it
else:
unhandled.append(e)
if eg:
unhandled.append(eg)
if unhandled:
raise eg0.with_exceptions(unhandled)

You'll note that "except*" is not useful in this pattern. However...

If a subgroup had a reference to its parent this gets cleaner again:

unhandled = []
eg0 = None
try:
.........
except* OSError as os_eg:
eg0 = os_eg.__original__ # or __parent__ or something
for e in os_eg:
if an ok exception:
handle it
else:
except* ValueError as value_eg:
eg0 = os_eg.__original__ # or __parent__ or something
for e in value_eg:
if some_ValueError_we_understand:
handle it
else:
unhandled.append(e)
except* Exception as eg:
eg0 = os_eg.__original__ # or __parent__ or something
unhandled.extend(eg)
if unhandled:
raise eg0.with_exceptions(unhandled)

Except that here I have no way to get "eg0", the original
ExceptionGroup, for the final raise without the additional .__original__
attribute.

Anyway, I'm strongly of the opinion that ExceptionGroups should look
like containers, be iterable, be truthy/falsey based on empty/nonempty
and that .split and .subgroup should return empty subgroups instead of
None.

Cheers,
Cameron Simpson <c...@cskk.id.au>

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/XLJCHKZMFMIHBGVSVMFPRQQG2SZVVHEJ/

[Cameron Simpson]

On 27Feb2021 00:54, Irit Katriel <iritk...@googlemail.com> wrote:
>On Sat, Feb 27, 2021 at 12:47 AM Jim J. Jewett <jimjj...@gmail.com> wrote:
>> > Is this not allowed?
>>
>> >try:
>> > try:
>> > obj.func() # function that raises ExceptionGroups
>> > except AttributeError:
>> > logger.info("obj doesn't have a func")
>> >except *(AttributeError, SyntaxError):
>> > logger.info("func had some problems")
>>
>> Allowed, but probably in error ... no AttributeError will get through to
>> the except * unless it happened inside the except AttributeError handler.
>> Did you mean:
>
>If obj.func() raises an ExceptionGroup that contains AttributeError then
>"except AttributeError" doesn't catch it. So it will get through.

And I, for one, would expect that. And _want_ that: I want the code to
do what I said, not have some magic which silently/invisibly intercepts
ExceptionGroups which contain something buried deep in their subgroup
tree.

We already allow "deep" exceptions out, to be caught at an arbitrary
outer call stack level. I don't see why ExceptionGroups should be any
different.

I certainly do not want ExceptionGroup([AttributeError]) conflated with
AttributeError. That fills me with horror.

Cheers,
Cameron Simpson <c...@cskk.id.au>

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/CSJJSEEW2EY56JKMBCSD6R5SEJOF2ZVU/

[Marco Sulla]

On Sat, 27 Feb 2021 at 00:35, Guido van Rossum <gu...@python.org> wrote:
> On Fri, Feb 26, 2021 at 3:18 PM Marco Sulla <Marco.Sul...@gmail.com> wrote:
>>
>> Excuse me if I post here. Maybe is a stupid question: why, instead of
>> introducing except*, Python can't extend the functionality of except,
>> so it can do what except* would do?
>
>
> Good question. Here's an example:
> ```
> try:
> . . .
> except OSError as err:
> if err.errno != ENOENT:
> raise
> . . .
> ```
> If this would catch ExceptionGroup(OSError), the `err` variable would be an ExceptionGroup instance, which does not have an `errno` attribute.

Thank you for the clarification :)

I must admit I read the PEP quickly, so I thought that the
subexception will be raised by except*, not the exception group. But
obviously this can't work.

The fact is I am really used to think that

except OsError as e

means "except for any OsError, which we name `e` from this moment on"

that I thought it applied also for except*. But, if I understood well

except* OsError as eg

means "except for any OsError contained in an ExceptionGroup. If the
OsError is not contained in an ExceptionGroup, Python wraps it in a
new ExceptionGroup. The ExceptionGroup is named `eg` from this moment
on"

I have to say that it's not obvious for me reading except* this way.
Anyway, I can't find a more readable semantic that does not use a new
keyword and it's short ^^'

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/XZRM57YQU3NRU5JLAHDULVR7TKUF5FT3/

[Irit Katriel via Python-Dev]

Hi Cameron, 

If you go long, I go longer :)

On Sun, Feb 28, 2021 at 10:51 PM Cameron Simpson <c...@cskk.id.au> wrote:

On 28Feb2021 10:40, Irit Katriel <iritk...@googlemail.com> wrote:
>split() and subgroup() take care to preserve the correct metadata on
>all
>the internal nodes, and if you just use them you only make safe operations.
>This is why I am hesitating to add iteration utilities to the API. Like we
>did, people will naturally try that first, and it's not the safest API.

Wouldn't it be safe if the ExceptionGroup were immutable, as you plan? 
Or have you another hazard in mind?

Making them immutable won't help the metadata issue. split() and subgroup()

copy the (context, cause traceback) from the original ExceptionGroups (root

and internal nodes of the tree) to the result trees.   If you DIY creating new

ExceptionGroups you need to take care of that.

But all that said, being able to iterable the subexceptions seems a
natural way to manage that:

    unhandled = []
    try:
        .........
    except *OSError as eg:
        for e in eg:
            if an ok exception:
                handle it
            else:
                unhandled.append(e)
    if unhandled:
        raise ExceptionGroup("unhandled", unhandled)

You just lost the metadata of eg. It has no context, cause and its traceback begins here. And the exceptions contained in it, if they came from a deeper tree that you flattened into the list, now look like their traceback jumps straight to here from the place they were actually first inserted into an ExceptionGroup. This may well be an impossible code path.  Here's an example:

>>> import traceback
>>>
>>> def flatten(exc):
...     if isinstance(exc, ExceptionGroup):
...         for e in exc.errors:
...            yield from flatten(e)
...     else:
...         yield exc
...
>>> def f():
...     try:
...         raise ValueError(42)
...     except ValueError as e:
...         return e
...
>>> def g():
...     try:
...         raise ExceptionGroup("g", [f()])
...     except ExceptionGroup as e:
...         return e
...
>>> def h():
...     try:
...         raise ExceptionGroup("h", [g()])
...     except ExceptionGroup as e:
...         return e
...
>>> def flat_h():
...     try:
...         raise ExceptionGroup("flat_h", list(flatten(h())))
...     except ExceptionGroup as e:
...         return e
...
>>>
>>> traceback.print_exception(h())
Traceback (most recent call last):
  File "<stdin>", line 3, in h
ExceptionGroup: h
   ------------------------------------------------------------
   Traceback (most recent call last):
     File "<stdin>", line 3, in g
   ExceptionGroup: g
      ------------------------------------------------------------
      Traceback (most recent call last):
        File "<stdin>", line 3, in f
      ValueError: 42

>>> traceback.print_exception(flat_h())
Traceback (most recent call last):
  File "<stdin>", line 3, in flat_h
ExceptionGroup: flat_h
   ------------------------------------------------------------
   Traceback (most recent call last):
     File "<stdin>", line 3, in f
   ValueError: 42
>>>

traceback.print_exception(h()) prints a reasonable traceback - h() called g() called f().

But according to  traceback.print_exception(flat_h()),   flat_h() called f().

You can preserve the metadata (and the nested structure with all its metadata) if you replace the last line with:

raise eg.subgroup(lambda e: e in  unhandled)

And for the part before that, iteration, Guido's pattern showed that you can roll it into the subgroup callback.

This is where except* can help:

  try:
        .........
    except except *OSError as eg:

        unhandled = []

        handled, unhandled = eg.split(lambda e: e is an ok exception)  # with side effect to handle e

        if unhandled:

            raise unhandled

    except *ValueError as eg:

        handled, unhandled = eg.split(lambda e: e is a value error we understand)  # with side effect to handle e

        if  unhandled:

 

I have the following concerns with the pattern above:

There's no way to make a _new_ ExceptionGroup with the same __cause__
and __context__ and message as the original group: not that I can't
assign to these, but I'd need to enuerate them; what if the exceptions
grew new attributes I wanted to replicate?

As I said, split() and subgroup() do that for you.

 

This cries out for another factory method like .subgroup but which makes
a new ExceptionGroup from an original group, containing a new sequence
of exceptions but the same message and coontext. Example:

    unhandled_eg = eg0.with_exceptions(unhandled)

Why same message and context but not same cause and traceback?

 

I don't see a documented way to access the group's message.

In the PEP: "The ExceptionGroup class exposes these parameters in the fields message and errors".

(In my implementation it's still msg and not message, I need to change that).

I think it is useful, see above.

This would be true if iteration was a useful pattern for working with ExceptionGroup, but I still think subgroup/split is a better tool in most cases. 

 

Cheers,
Cameron Simpson <c...@cskk.id.au>

[Cameron Simpson]

On 28Feb2021 23:56, Irit Katriel <iritk...@googlemail.com> wrote:
>If you go long, I go longer :)

:-)

>On Sun, Feb 28, 2021 at 10:51 PM Cameron Simpson <c...@cskk.id.au> wrote:
>> On 28Feb2021 10:40, Irit Katriel <iritk...@googlemail.com> wrote:
>> >split() and subgroup() take care to preserve the correct metadata on
>> >all
>> >the internal nodes, and if you just use them you only make safe
>> operations.
>> >This is why I am hesitating to add iteration utilities to the API. Like we
>> >did, people will naturally try that first, and it's not the safest API.
>>
>> Wouldn't it be safe if the ExceptionGroup were immutable, as you plan?
>> Or have you another hazard in mind?
>
>Making them immutable won't help the metadata issue. split() and subgroup()
>copy the (context, cause traceback) from the original ExceptionGroups (root
>and internal nodes of the tree) to the result trees. If you DIY creating
>new ExceptionGroups you need to take care of that.

Ah, right. Yes.

The overflows into my request for a factory method to construct a "like"
ExceptionGroup with a new exception tree lower down.

>> But all that said, being able to iterable the subexceptions seems a
>> natural way to manage that:
>>
>> unhandled = []
>> try:
>> .........
>> except *OSError as eg:
>> for e in eg:
>> if an ok exception:
>> handle it
>> else:
>> unhandled.append(e)
>> if unhandled:
>> raise ExceptionGroup("unhandled", unhandled)
>
>
>You just lost the metadata of eg. It has no context, cause and its
>traceback begins here.

Aye. Hence a wish, again lower down, for some reference to the source
ExceptionGroup and therefore a handy factory for making a new group with
the right metadata.

>And the exceptions contained in it, if they came
>from a deeper tree that you flattened into the list, now look like their
>traceback jumps straight to here from the place they were actually first
>inserted into an ExceptionGroup. This may well be an impossible code path.

Perhaps so. But it doesn't detract from how useful it is to iterate over
the inner exceptions. I see this as an argument for making it possible
to obtain the correct metadata, not against iteration itself. Even if
the iteration yielded some proxy or wrapper for the inner exception
instead of the naked exception itself.

>Here's an example:
[... flattened ExceptionGroup with uninformative tracebacks ...]

>>>> import traceback
>>>> def flatten(exc):
>... if isinstance(exc, ExceptionGroup):
>... for e in exc.errors:
>... yield from flatten(e)
>... else:
>... yield exc

[...]

>>>> traceback.print_exception(flat_h())
>Traceback (most recent call last):
> File "<stdin>", line 3, in flat_h
>ExceptionGroup: flat_h
> ------------------------------------------------------------
> Traceback (most recent call last):
> File "<stdin>", line 3, in f
> ValueError: 42
>
>traceback.print_exception(h()) prints a reasonable traceback - h() called
>g() called f().
>
>But according to traceback.print_exception(flat_h()), flat_h() called
>f().
>
>You can preserve the metadata (and the nested structure with all its
>metadata) if you replace the last line with:
> raise eg.subgroup(lambda e: e in unhandled)

Ok. That works for me as my desired factory. Verbose maybe, but
workable. Um. It presumes exception equality will do - that feels
slightly error prone (including some similar but not unhandled
exceptions). But I can write something more pointed based on id().

>And for the part before that, iteration, Guido's pattern showed that you
>can roll it into the subgroup callback.

Aye.

Alas, that raises within each branch. I want to gather all the unhandled
exceptions together into single ExceptionGroup. Using split gets me a
bunch of groups, were I to defer the raise to after all the checking (as
I want to in my pattern).

So I've have to combine multiple groups back together.

unhandled_groups = []

try:
.........
except except *OSError as eg:

handled, unhandled = eg.split(lambda e: e is an ok exception) # with side effect to handle e
if unhandled:

unhandled_groups.append(unhandled)

except *ValueError as eg:
handled, unhandled = eg.split(lambda e: e is a value error we understand) # with side effect to handle e
if unhandled:

unhandled_groups.append(unhandled)
if unhandled_groups:
# combine them here?
new_eg = eg0.subgroup(lambda e: e in all-the-es-from-all-the-unhandled-groups)

That last line needs eg0, the original ExceptionGroup (unavailable
AFAICT), _and_ something to test for a naked exception being one of the
unhandled ones.

I _could_ gather the latter as a side effect of my split() lambda, but
that makes things even more elaborate. A closure, even!

If I could just iterate over the nested naked exceptions _none_ of this
complexity would be required.

>I have the following concerns with the pattern above:
>>
>> There's no way to make a _new_ ExceptionGroup with the same __cause__
>> and __context__ and message as the original group: not that I can't
>> assign to these, but I'd need to enuerate them; what if the exceptions
>> grew new attributes I wanted to replicate?
>
>As I said, split() and subgroup() do that for you.

They do if I'm only concerned with the particular exception subclass (eg
OSError) i.e. my work is complete within the single except* clause. If I
use except*, but later want to build an overarching "unhandled
exceptions group", I have nothing to build on.

>> This cries out for another factory method like .subgroup but which
>> makes
>> a new ExceptionGroup from an original group, containing a new sequence
>> of exceptions but the same message and coontext. Example:
>>
>> unhandled_eg = eg0.with_exceptions(unhandled)
>
>Why same message and context but not same cause and traceback?

Insufficient verbiage. I meant same message and context and cause. The
point being that it is exceptly such a missing out of some attribute
that I'd like to avoid by not having to enumerate the preserved
attributes - the factory should do that, knowing the ExceptionGroup
internals.

Anyway, as you point out, .subgroup does this.

>> I don't see a documented way to access the group's message.
>
>In the PEP: "The ExceptionGroup class exposes these parameters in the
>fields message and errors".

Hmm. Missed that. Thanks.

I did find the PEP hard to read in some places. I think it could well do
with saying "eg" instead of "e" throughout where "e" is an
ExceptionGroup - I had a distinct tendency to want to read "e" as one of
the naked exceptions and not the group.

>> I'm quite unhappy about .subgroup (and presumably .split) returning
>> None
>> when the group is empty. The requires me to have the gratuitous "if eg:"
>> and "if value_eg:" if-statements in the example above.
>>
>> If, instead, ExceptionGroups were like any other container I could
>> just test if they were empty:
>>
>> if eg:
>>
>> _and_ even if they were empty, iterate over them. Who cares if the loop
>> iterates zero times?

>> [...]

>> Anyway, I'm strongly of the opinion that ExceptionGroups should look
>> like containers, be iterable, be truthy/falsey based on empty/nonempty
>> and that .split and .subgroup should return empty subgroups instead of
>> None.
>
>This would be true if iteration was a useful pattern for working with
>ExceptionGroup, but I still think subgroup/split is a better tool in most
>cases.

I think I want to iterate over these things - it is useful to me. I want
an ExceptionGroup to look like a container. I accept that subgroup makes
a new group with metadata intact, but often that is not what _I_ care
about, particularly when I'm just winnowing some special known cases.

I cannot see that returning empty groups instead of None does any harm
at all, and brings benefits in streamlining testing and processing.

I guess if we don't get iteration and containerness I'll just have to
subclass ExceptionGroup for my own use, giving it iterations and
truthiness. It is hard to override subgroup and split to return empty
ExceptionGroups though, without hacking with internals.

Let's turn this on its head:

- what specific harm comes from giving EGs container truthiness for size
testing?

- what specific harm comes from returning an empty EG from split on no
match instead of None?

- what specific harm comes from supporting iteration with a caveat about
metadata in the docstring, and maybe a recommendation of subgroup?

- if I wanted to override subgroup and split to not return None, is that
even possible with the current ? i.e. can I make a clean metadata
preserved EG from an empty list? For example:

eg2 = eg.subgroup(lambda e: False)

Does that get me None, or an empty group? If the latter, I can roll my
own subclass with my desired features. If not, I can't AFAICT.

EGs _have_ a .errors attribute which has all these aspects, why not
expand it to the class as a whole?

You seem very happen to implement 80% of what I want using callbacks
(lambda e: ...), but I find explicit iteration much easier to read. I
rarely use filter() for example, and often prefer a generator expression
of list comprehension.

Cheers,
Cameron Simpson <c...@cskk.id.au>

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/46SOJ67QFCC4KEWJOKK53RSV3ND2YBBX/

[Guido van Rossum]

I'm trying to shorten this again...

On Sun, Feb 28, 2021 at 5:54 PM Cameron Simpson <c...@cskk.id.au> wrote:

Let's turn this on its head:

- what specific harm comes from giving EGs container truthiness for size
  testing?

For something that's not a pure container, this is an anti-pattern. No other subclass of BaseException (that I know of) can ever be falsey, so existing code would be forgiven to write "if e:" where "if e is not None:" was meant.

 

- what specific harm comes from returning an empty EG from split on no
  match instead of None?

It would be an exception. What is raising an empty EG supposed to do?

 

- what specific harm comes from supporting iteration with a caveat about
  metadata in the docstring, and maybe a recommendation of subgroup?

Iteration is such an easy-to-use API that it would become an attractive nuisance -- people would use it regardless of whether it's the right tool for the job.

 

- if I wanted to override subgroup and split to not return None, is that
  even possible with the current ? i.e. can I make a clean metadata
  preserved EG from an empty list? For example:

    eg2 = eg.subgroup(lambda e: False)

Does that get me None, or an empty group? If the latter, I can roll my
own subclass with my desired features. If not, I can't AFAICT.

It returns None. Writing a subclass that behaves differently than the base class will just confuse users who aren't aware of your overrides.

 

EGs _have_ a .errors attribute which has all these aspects, why not
expand it to the class as a whole?

They are different. The `errors` attribute may contain other EGs. But presumably iteration would recurse into those so as to flatten the tree and yield only leaf nodes. The attribute is necessary so it is possible to write traversal functions (including that iterator you so desperately want -- as a helper function).

 

You seem very happen to implement 80% of what I want using callbacks
(lambda e: ...), but I find explicit iteration much easier to read. I
rarely use filter() for example, and often prefer a generator expression
of list comprehension.

Our (the PEP authors') belief is that in most cases users are better off not handling the leaf exceptions. If you want to handle e.g. KeyError, the place to do that is in the try/except clause immediately surrounding the offending `__getitem__` operation (which will never raise an EG), not several stack frames below, where you're potentially dealing with an aggregate of many KeyErrors, or perhaps a KeyError and a TypeError. At that point the most common option is to ignore or log some or all exception types.

[Cameron Simpson]

On 28Feb2021 20:05, Guido van Rossum <gu...@python.org> wrote:
>I'm trying to shorten this again...
>
>On Sun, Feb 28, 2021 at 5:54 PM Cameron Simpson <c...@cskk.id.au> wrote:
>> Let's turn this on its head:
>> - what specific harm comes from giving EGs container truthiness for
>> size
>> testing?
>
>For something that's not a pure container, this is an anti-pattern. No
>other subclass of BaseException (that I know of) can ever be falsey, so
>existing code would be forgiven to write "if e:" where "if e is not None:"
>was meant.

I that's how you feel, I'll withdraw my advocacy. I agree it's a
difference.

I'll outline my view below the objections below, but agree to disagree
here - I'm not seeking to convince you any more.

>> - what specific harm comes from returning an empty EG from split on
>> no
>> match instead of None?
>
>It would be an exception. What is raising an empty EG supposed to do?

It'd be rather pointless. It assumes a little care on the person making
it (well, using it where I'd asked subgroup to make it). I guess if
there's a fall through in the except* which reraises, its a burden on
the clause author to check such a contingency. But wouldn't such a
check normally be written:

if eg:
raise

on the presumption that None (falsey) means nothing to raise? And
wouldn't the same code work if eg looks like a container in truthiness?

>> - what specific harm comes from supporting iteration with a caveat
>> about
>> metadata in the docstring, and maybe a recommendation of subgroup?
>
>Iteration is such an easy-to-use API that it would become an attractive
>nuisance -- people would use it regardless of whether it's the right tool
>for the job.

Perhaps. But an easy to use API is normally what one wants. For you,
this is a downside because you hold that it is also usually a poor
choice for this facility. Maybe my use case and habits are unusual.

>> - if I wanted to override subgroup and split to not return None, is
>> that
>> even possible with the current ? i.e. can I make a clean metadata
>> preserved EG from an empty list? For example:
>>
>> eg2 = eg.subgroup(lambda e: False)
>>
>> Does that get me None, or an empty group? If the latter, I can roll my
>> own subclass with my desired features. If not, I can't AFAICT.
>
>It returns None. Writing a subclass that behaves differently than the base
>class will just confuse users who aren't aware of your overrides.

Yeah, a subclass might be misleading. Maybe all I need is a help
function or class.

Cheers,
Cameron Simpson <c...@cskk.id.au>
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/Z4BM5JR7XQOIXCLW6GCE5N6V5Q72XE7W/

[Sven R. Kunze]

Just to be on the safe side here, I would want to asked a little bit further.

On 02.03.21 12:22, Irit Katriel wrote:

1) What is the right "except pattern" when ExceptionGroup is introduced for the use cases described above (remote or concurrent python processes)?

 

If you want to catch the whole ExceptionGroup and format its traceback, then you can just do "except ExceptionGroup as eg" and then traceback.print_exception(eg).

The except* syntax is for when you want to handle only certain types of exceptions from the group, selectively.

Just to make sure, I understand this completely.

In order to make it more tangible here is an example from the stdlib:

https://github.com/python/cpython/blob/bf2e7e55d7306b1e2fce7dce767e8df5ff42cf1c/Lib/concurrent/futures/process.py#L215

As you can see, BaseException is caught multiple times as the only except-clause. _sendback_result(...) then used to transfer the return_val/exception back to parent process.

How is the supposed way of handling a raised ExceptionGroup?

a) will the existing code simply work as it?
b) if not what are the changes to this lib specifically as a blueprint example for others

Reading from the other subthread for this PEP, I am not 100% sure now that "except BaseException" will suffice if ExceptionGroup inherits from Exception instead of BaseException because it seems that ExceptionGroup somehow is handled specially. So, why I try to approach this very issue with existing code. Once that is clear, it should be easy for everybody to apply the same pattern to their code. Here is the copy from github:

try:
    r = call_item.fn(*call_item.args, **call_item.kwargs)
except BaseException as e:
    exc = _ExceptionWithTraceback(e, e.__traceback__)
    _sendback_result(result_queue, call_item.work_id, exception=exc)
else:
    _sendback_result(result_queue, call_item.work_id, result=r)
    del r

Maybe even _sendback_result could benefit from using ExceptionGroup itself. You can see there another "except BaseException" in case of an error. But that's maybe a different topic.

 

2) What is the recommended approach of printing the traceback potentially incorporating multiple tracebacks - I couldn't find it in the PEP and tracebacks are a really neat tool for error hunting.

Part of the proposal of the PEP is that we teach the builtin traceback formatting code to display ExceptionGroups with all that information.

As long as there's the possibility to receive the usual (two-line) entry-based list, I guess that is enough for every one to work with it (e.g. hiding unnecessary tb entries).

The reference implementation has this, and the examples in the PEP were produced with it. Some of the examples (the early ones) show exceptions that were never raised so there is no traceback. But if you scroll down to the later examples, you see the output for exceptions with tracebacks, cause, context etc.

Ah I see them now. Thank you. :-)

We didn't put too much effort into making the traceback output pretty, so at the moment it's a draft. If there is an ascii artist out there who has suggestions on improving this, that would be great.

Well, yes. It's not really pretty. I would recommend a tree-based solution such as the following:

Traceback (most recent call last):

  File "<stdin>", line 3, in <module>
ExceptionGroup
|
+---+   Traceback (most recent call last):
|   |     File "<stdin>", line 3, in <module>
|   |   ExceptionGroup: eg
|   |   |
|   |   +---+ ValueError: a
|   |
|   |   During handling of the above exception, another exception occurred:

|   | 
|   |   Traceback (most recent call last):

|   |     File "<stdin>", line 5, in <module>
|   |   KeyError: 'x'
|
|
+---+   Traceback (most recent call last):
|   |     File "<stdin>", line 3, in <module>
|   |   ExceptionGroup: eg
|   |   |
|   |   +---+ TypeError: b

I used a similar pattern for the remote-execution lib and received good user feedback (even requesting color encoding for each layer of the tree (not the text), so people know what belongs together). Besides colors, I used https://en.wikipedia.org/wiki/Box-drawing_character but I guess pipes, dashes and pluses could suffice for CPython.

One other remark from my side here: in the example of the PEP there seem to be missing a space before 'File "<stdin>"'. Comparing outer tbs with inner tbs, you can see that the "F" of "File" is not underneath the a of "Traceback" but underneath the "r" of it.

Regards,
Sven

[Irit Katriel via Python-Dev]

Hi Sven,

On Mon, Mar 1, 2021 at 8:59 PM Sven R. Kunze <srk...@mail.de> wrote:

Hey Irit,

cool proposal.

Thank you.

 

I just have two questions regarding "except Exception" or "except BaseException" as it is used e.g. by concurrent.futures.process (def _process_worker) from the stdlib.

Almost similarly, I maintain a library using this pattern to wrap/unwrap exceptions from remote Python processes to create nicely formatted tracebacks (also recursively of course if needed) at the calling python process.

Usually these exceptions are wrapped, transferred to the parent process, there is the current call stack added, then reraised as a different exception, and so on and so forth if a chain of parents exist. The outermost parent process takes care of printing the tb.

My two questions regarding PEP 654:

1) What is the right "except pattern" when ExceptionGroup is introduced for the use cases described above (remote or concurrent python processes)?

 

If you want to catch the whole ExceptionGroup and format its traceback, then you can just do "except ExceptionGroup as eg" and then traceback.print_exception(eg).

The except* syntax is for when you want to handle only certain types of exceptions from the group, selectively.

 

2) What is the recommended approach of printing the traceback potentially incorporating multiple tracebacks - I couldn't find it in the PEP and tracebacks are a really neat tool for error hunting.

Part of the proposal of the PEP is that we teach the builtin traceback formatting code to display ExceptionGroups with all that information. The reference implementation has this, and the examples in the PEP were produced with it. Some of the examples (the early ones) show exceptions that were never raised so there is no traceback. But if you scroll down to the later examples, you see the output for exceptions with tracebacks, cause, context etc.

We didn't put too much effort into making the traceback output pretty, so at the moment it's a draft. If there is an ascii artist out there who has suggestions on improving this, that would be great.

Irit

[Barry Scott]

On 23 Feb 2021, at 00:24, Irit Katriel via Python-Dev <pytho...@python.org> wrote:

Hi all,

We would like to request feedback on PEP 654 -- Exception Groups and except*.

https://www.python.org/dev/peps/pep-0654/

It proposes language extensions that allow programs to raise and handle multiple unrelated

exceptions simultaneously, motivated by the needs of asyncio and other concurrency libraries,

but with other use cases as well.

* A new standard exception type,  ExceptionGroup, to represent multiple exceptions with

  shared traceback.

* Updates to the traceback printing code to display (possibly nested) ExceptionGroups.
* A new syntax except* for handling ExceptionGroups.

A reference implementation (unreviewed) can be found at:

https://github.com/iritkatriel/cpython/pull/10

Have you considered alternative syntax to the "except*"?

I feel that expect* is too easy to confuse with plain except.

As others have said needing to change all except Exception:

handlers would be a breaking change for code that I maintain.

I think its accepted that that idiom should continue to be as

reliable as it currently is. Can you confirm that I have understood

the conversation on this point?

I have been thinking about why I would be getting an ExceptionGroup

with, for example, a number of OSError for a handler to deal with.

Would it not be a difficult piece of code to write without the context of

each OSError?

What I'm think is that the lower level pieces would be catching the OSError

closer to the source of the problem and raising a App/Lib specific exception

that provides the context. For example AppFailedToSaveConfigError and

AppFailedToSaveDataError as oppose to a two permission OSError's.

With context I can imagine that handling the ExceptionGroup would be

a lot easier for the App designer.

If that the pattern that emerges then is the complexity of nested

ExceptionGroups going to only rarely used?

Barry

Thank you for your help

Kind regards

Irit, Yury & Guido

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/L5Q27DVKOKZCDNCAWRIQVOZ5DZCZHLRM/

[Sven R. Kunze]

Hey Irit,

cool proposal.

I just have two questions regarding "except Exception" or "except BaseException" as it is used e.g. by concurrent.futures.process (def _process_worker) from the stdlib.

Almost similarly, I maintain a library using this pattern to wrap/unwrap exceptions from remote Python processes to create nicely formatted tracebacks (also recursively of course if needed) at the calling python process.

Usually these exceptions are wrapped, transferred to the parent process, there is the current call stack added, then reraised as a different exception, and so on and so forth if a chain of parents exist. The outermost parent process takes care of printing the tb.

My two questions regarding PEP 654:

1) What is the right "except pattern" when ExceptionGroup is introduced for the use cases described above (remote or concurrent python processes)?

2) What is the recommended approach of printing the traceback potentially incorporating multiple tracebacks - I couldn't find it in the PEP and tracebacks are a really neat tool for error hunting.

Best Regards,
Sven

On 23.02.21 01:24, Irit Katriel via Python-Dev wrote:

Hi all,

We would like to request feedback on PEP 654 -- Exception Groups and except*.

https://www.python.org/dev/peps/pep-0654/

It proposes language extensions that allow programs to raise and handle multiple unrelated

exceptions simultaneously, motivated by the needs of asyncio and other concurrency libraries,

but with other use cases as well.

* A new standard exception type,  ExceptionGroup, to represent multiple exceptions with

  shared traceback.

* Updates to the traceback printing code to display (possibly nested) ExceptionGroups.
* A new syntax except* for handling ExceptionGroups.

A reference implementation (unreviewed) can be found at:

https://github.com/iritkatriel/cpython/pull/10

Thank you for your help

Kind regards

Irit, Yury & Guido

[Irit Katriel via Python-Dev]

Hi Barry, 

On Mon, Mar 1, 2021 at 6:12 PM Barry Scott <ba...@barrys-emacs.org> wrote:

Have you considered alternative syntax to the "except*"?

I feel that expect* is too easy to confuse with plain except.

We briefly considered things like "except Group[ValueError, TypeError]".  Do you (or anyone else) have any suggestions on syntax?

 

As others have said needing to change all except Exception:

handlers would be a breaking change for code that I maintain. 

I think its accepted that that idiom should continue to be as

reliable as it currently is. Can you confirm that I have understood

the conversation on this point?

That's not happening. The PEP has been updated regarding this.  It was never going to be all "except Exception", most of them would not be affected. But we have a better plan now so this discussion is moot.

As an aside - I found it interesting that the option to wrap BaseException instances by an Exception, which came up a couple of times in this thread, didn't generate any anxiety. In my day job I maintain a proprietary DSL that ~4000 developers use, and my experience has been that a bug that swallows an exception you need is worse than a bug that makes you get an exception you did not expect. The latter can hurt you once (and then you fix it). But the former is (1) harder to detect (2) longer to debug (3) once you know about it there is nothing you can do to work around it. Problems like that cost us years and years of work.

 

I have been thinking about why I would be getting an ExceptionGroup

with, for example, a number of OSError for a handler to deal with.

Would it not be a difficult piece of code to write without the context of

each OSError?

What I'm think is that the lower level pieces would be catching the OSError

closer to the source of the problem and raising a App/Lib specific exception

that provides the context. For example AppFailedToSaveConfigError and

AppFailedToSaveDataError as oppose to a two permission OSError's.

I think you're right about this. OSError is an example of an exception with a data field that is often inspected for its value, so we picked on it to talk about how you would do something like this. Most of the time this won't be needed, but we still need to know that it's possible and reasonably ergonomic to do it.

 

With context I can imagine that handling the ExceptionGroup would be

a lot easier for the App designer.

If that the pattern that emerges then is the complexity of nested

ExceptionGroups going to only rarely used?

I can't imagine people building deep trees of exceptions in practice (at least not on purpose).  But some nesting may show up naturally, and we need to support it because otherwise it can get awkward if, for example, asyncio.gather() needs to wrap an exception group that came from a TemporaryDirectory.__exit__().

 

Irit

[Guido van Rossum]

On Tue, Mar 2, 2021 at 1:53 PM Irit Katriel via Python-Dev <pytho...@python.org> wrote:

[...]

I can't imagine people building deep trees of exceptions in practice (at least not on purpose).  But some nesting may show up naturally, and we need to support it because otherwise it can get awkward if, for example, asyncio.gather() needs to wrap an exception group that came from a TemporaryDirectory.__exit__().

I assume that in serious asyncio apps it is not uncommon to see some operation use gather() to wrap several tasks, and then to see that operation itself be passed to an outer gather() call together with some other tasks. If several things go wrong you could easily end up with a nested EG. Same in Trio -- you could have multiple nested "nurseries".

If you're catching such exceptions it may not be so important to be aware of the nesting, but if you're logging tracebacks (or if the exception is never caught and the program exits with a traceback) it is pretty important to be able to show everything that went wrong without unnecessary duplication of stack traces.

[Paul Moore]

On Tue, 2 Mar 2021 at 21:46, Irit Katriel via Python-Dev
<pytho...@python.org> wrote:
> As an aside - I found it interesting that the option to wrap BaseException instances by an Exception, which came up a couple of times in this thread, didn't generate any anxiety.

Probably because it wasn't clear that was ever being proposed... (or
at least the implication wasn't obvious - presumably this is somehow
related to BaseExceptions being accumulated in ExceptionGroups?) :-(

I would consider it essential that if someone hits Ctrl-C and that
generates a KeyboardInterrupt, then:

1. That KeyboardInterrupt will *not* get caught by exception handlers
only interested in Exception instances
2. That KeyboardInterrupt *will* get caught by any handler that does
an explicit `except KeyboardInterrupt` or an `except BaseException`.

To me, that's pretty much essential to correct Ctrl-C handling in any
app (never ignore a user's Ctrl-C and always exit cleanly if one is
raised).

That might mean that BaseException instances shouldn't be "groupable",
but I don't want to comment on that until I've properly read the PEP
(I've skimmed it now, but only superficially). At a minimum, I'd
consider it a bug for library code to manually wrap a
KeyboardInterrupt in an exception group (just like code that catches
KeyboardInterrupt and re-raises it as a ValueError would be today).

Paul

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/S5SPIFHGWASEGKBITWPDDRXOU33NLG46/

[Irit Katriel via Python-Dev]

Hi Sven, 

I like your formatting suggestion, thanks. I will do something like that.

I'm not sure I understand your question. ExceptionGroup is a subclass of Exception (which is a subclass of BaseException). So ExceptionGroup is caught by "except Exception" or "except BaseException". BaseExceptionGroup is a subclass only of BaseException so it is caught by "except BaseException" but not "except Exception". And ExceptionGroup is allowed to wrap only Exceptions while BaseException can wrap Exceptions and and BaseExceptions. Makes sense?

Irit

[Irit Katriel via Python-Dev]

Hi Paul, 

I agree that your condition (1) is essential, while condition (2) is desirable.   (I explained in the email you replied to why I think 2 is less critical than 1).

The current state of the PEP is that ExceptionGroup does not wrap BaseExceptions and is caught by "except Exception", while BaseExceptionGroup wraps BaseException and is only caught by "except BaseException" but not "except Exception". 

This is covered the PEP, but TL;DR:

If we could make "except KeyboardInterrupt" catch BaseExceptionGroup([KeyboardInterrupt]) in a reasonably backwards compatible way then we wouldn't need except*. 

For example, suppose that this:

try:

    raise BaseExceptionGroup([KeyboardInterrupt()])

except  KeyboardInterrupt:

   email_the_boss()

worked as you suggest. Then what should this do?  Send two emails to the boss?

try:

    raise BaseExceptionGroup([KeyboardInterrupt(), KeyboardInterrupt() ])

except  KeyboardInterrupt:

   email_the_boss()

As you noted, no library is under any obligation to wrap KeyboardInterrupts into the exception groups it raises. You may decide it's a bad idea and not do it.  What we are discussing here is what the language should make possible. We agree that wrapping a BaseException by an Exception is something we should definitely block. When it's wrapping a BaseException by another, new BaseException type, in my view that's ok. You may have a bug where you don't catch an exception you want to catch, because you are using a new API incorrectly. But you won't have bugs where you swallow an exception that you didn't swallow before.

Irit

[Paul Moore]

On Wed, 3 Mar 2021 at 12:37, Irit Katriel <iritk...@googlemail.com> wrote:
> This is covered the PEP, but TL;DR:
> If we could make "except KeyboardInterrupt" catch BaseExceptionGroup([KeyboardInterrupt]) in a reasonably backwards compatible way then we wouldn't need except*.

[...]

> As you noted, no library is under any obligation to wrap KeyboardInterrupts into the exception groups it raises. You may decide it's a bad idea and not do it. What we are discussing here is what the language should make possible. We agree that wrapping a BaseException by an Exception is something we should definitely block. When it's wrapping a BaseException by another, new BaseException type, in my view that's ok. You may have a bug where you don't catch an exception you want to catch, because you are using a new API incorrectly. But you won't have bugs where you swallow an exception that you didn't swallow before.

Thanks for the explanation. I understand your point here, and I see
what you're saying. But I have a couple of questions still:

1. Having now read the PEP, I don't actually see a use case for
grouping BaseExceptions. Why would anyone catch KeyboardInterrupt or
SystemExit and wrap them in a BaseExceptionGroup anyway? It seems to
me that the right thing to do, even in async or parallel code, is to
just propogate the KeyboardInterrupt/SystemExit up to the main
program. Losing a ValueError that happened at the exact same time as
the user pressed Ctrl-C seems like it's not going to be a problem in
practice...
2. Given the above, why even have a means of grouping BaseExceptions
at all? Why not just have ExceptionGroup that can only catch instances
of Exception?

If there really *was* a low-level case where some code absolutely had
to (temporarily) group KeyboardInterrupt, for example, it could be
temporarily wrapped:

class IGotInterrupted(Exception):
def __init__(self, exc):
self.exc = exc

def might_be_interrupted():
try:
critical_stuff()
except KeyboardInterrupt as exc:
raise IGotInterrupted(exc)

def funky_parallel_stuff():
try:
do_in_parallel(might_be_interrupted)
except *IGotInterrupted as e:
# Please excuse the fact that I don't know the
# correct way to re-raise here. Might need raise from.
raise e.exc
# Do we also need to consider an *ungrouped* IGotInterrupted?
# That's for the library to decide, and is why not letting it
escape is a good thing...

That would be appropriate for really low-level code, and it would be a
fairly obvious anti-pattern for an exception class like
IGotInterrupted to "escape" so that user code would ever see it.

I guess the argument for grouping BaseExceptions is "why make it
necessary to do complicated wrapping like this?" To which my answer is
"because it should almost never be something you do, and so why add
support for it when it can be done *without* needing explicit support
in the PEP?" And by not having stdlib support for wrapping
BaseExceptions, we're signalling that we don't think people should be
doing it casually...

Paul
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/5LVHBCIWOLYVUYAMFYIZAPEDV7JUVBYS/

[Irit Katriel via Python-Dev]

Hi Paul,

On Wed, Mar 3, 2021 at 2:20 PM Paul Moore <p.f....@gmail.com> wrote:

1. Having now read the PEP, I don't actually see a use case for
grouping BaseExceptions. Why would anyone catch KeyboardInterrupt or
SystemExit and wrap them in a BaseExceptionGroup anyway? It seems to
me that the right thing to do, even in async or parallel code, is to
just propogate the KeyboardInterrupt/SystemExit up to the main
program. Losing a ValueError that happened at the exact same time as
the user pressed Ctrl-C seems like it's not going to be a problem in
practice...

It is possible that your program wants to do something other than just terminate when it gets a KeyboardInterrupt. Otherwise why does the interpreter bother propagating the KeyboardInterrupt rather than just terminate the program there and then? 

And the ValueError you lost may be some other error type that involves cleanup.

If you do this: 

try:

    async.gather(...)      # raises BaseExceptionGroup( DoCleanup(), KeyboardInterrupt())

except * DoCleanup:

    do_cleanup() # log stuff, send notifications, close sockets, whatever

That will do the cleanup in addition to propagating a BaseExceptionGroup(KeyboardInterrupt())

Also, KeyboardInterrupt/SystemExit are not the only BaseExceptions.

 

2. Given the above, why even have a means of grouping BaseExceptions
at all? Why not just have ExceptionGroup that can only catch instances
of Exception?

Because the IGotInterrupted alternative involves wrapping a BaseException by an Exception, which is not something we should push people into doing (it's not that different from allowing ExceptionGroup to wrap BaseExceptions directly).

What's the harm/complication in offering a BaseExceptionGroup(BaseException) in addition to ExceptionGroup(BaseExceptionGroup, Exception)?

I think the only reason you're comfortable with having to select between the exceptions that were raised and discard some of them is because that's where we are today. The PEP lists several standard library and other APIs that discard errors because they need to pick one. That's what we're trying to fix.

Irit

[Paul Moore]

On Wed, 3 Mar 2021 at 14:59, Irit Katriel <iritk...@googlemail.com> wrote:
>> 2. Given the above, why even have a means of grouping BaseExceptions
>> at all? Why not just have ExceptionGroup that can only catch instances
>> of Exception?
>
> Because the IGotInterrupted alternative involves wrapping a BaseException by an Exception, which is not something we should push people into doing (it's not that different from allowing ExceptionGroup to wrap BaseExceptions directly).

That's a fair point.

> What's the harm/complication in offering a BaseExceptionGroup(BaseException) in addition to ExceptionGroup(BaseExceptionGroup, Exception)?

Similar to the argument for "except Exception". Applications that trap
KeyboardInterrupt so that they can exit cleanly without an ugly
traceback will no longer trap *all* keyboard interrupts, as they could
miss grouped ones.

If we accept that grouped exceptions should never escape out of a
well-defined context, then this wouldn't be such an issue. But there's
nothing in the PEP that enforces that, and there *is* code that needs
to be prepared for "any sort of result". It's the except Exception
argument again.

So code that wants to exit cleanly in the face of Ctrl-C will need to
be rewritten from:

try:
main()
except KeyboardInterrupt:
print("User interrupted the program. Exiting")
sys.exit(1)

to:

try:
try:
main()
except KeyboardInterrupt:
print("User interrupted the program. Exiting")
sys.exit(1)
except *KeyboardInterrupt:
print("User interrupted the program. Exiting")
sys.exit(1)

Did I miss an easier way of writing this code? And worse, how would I
write it so that it was portable between Python 3.9 and later versions
(which is a common requirement for library code - admittedly library
code wouldn't normally be doing this sort of top-level trap, but it
could just as easily be "catch Ctrl-C and do a bit of tidy-up and
re-raise").

> I think the only reason you're comfortable with having to select between the exceptions that were raised and discard some of them is because that's where we are today. The PEP lists several standard library and other APIs that discard errors because they need to pick one. That's what we're trying to fix.

Maybe. But I'm not looking at it as being "comfortable" with the
current situation, but rather as "I don't use any of these new
features, why am I having to change my code to accommodate stuff I
don't use?" If I own the full stack, that's not an issue, but
frameworks and libraries typically have to interact with other users'
code, and there the contract has changed from "do what you like in
your code and I'll cope" to "do what you like in your code as long as
you don't let an exception group escape, and I'll cope"... And I have
to change *my* code to get the old contract back.

But it's a small point in the wider scheme of things, and I'm not
going to labour the point any more. Thanks for listening and taking
the time to reply.

Paul

Paul
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/WSUEOGDCBBOZ7PCQGUCXKIZEZ7RK34LK/

[Joao S. O. Bueno]

Just to add +1 for Paul's concerns.

 
Even though ExceptionGroups "are not supposed" to not

leak into caller code, don't mean they "won't". Making  "except Exception"

catch them would make this part a non issue, and the

feature looks great otherwise.