[Python-Dev] PEP 654: Exception Groups and except* [REPOST]
215 views
Skip to first unread message
Irit Katriel via Python-Dev
Mar 20, 2021, 1:11:36 PM3/20/21
to python-dev, Yury Selivanov
We would like to present for feedback a new version of PEP 654, which incorporates the feedback we received in the discussions so far:
The reference implementation has also been updated along with the PEP.
The changes we made since the first post are:
1. Instead of ExceptionGroup(BaseException), we will have two new builtin types: BaseExceptionGroup(BaseException) and ExceptionGroup(BaseExceptionGroup, Exception).
This is so that "except Exception" catches ExceptionGroups (but not BaseExceptionGroups). BaseExceptionGroup.__new__ inspects the wrapped exceptions, and if they are all Exception subclasses, it creates an ExceptionGroup instead of a BaseExceptionGroup.
2. The exception group classes are not final - they can be subclassed and split()/subgroup() work correctly if the subclass overrides the derive() instance method as described here: https://www.python.org/dev/peps/pep-0654/#subclassing-exception-groups
3. We had some good suggestions on formatting exception groups, which we have implemented as you can see in the output shown for the examples in the PEP.
4. We expanded the section on handling Exception Groups, to show how subgroup can be used (with side effects) to do something for each leaf exception, and how to iterate correctly when the tracebacks of leaf exceptions are needed: https://www.python.org/dev/peps/pep-0654/#handling-exception-groups
5. We expanded the sections on rationale and backwards compatibility to explain our premise and expectations regarding how exception groups will be used and how the transition to using them will be managed.
6. We added several items to the rejected ideas section.
We did not receive any comments (or make any changes) to the proposed semantics of except*, hopefully this is because everyone thought they are sensible.
Irit, Yury and Guido
Guido van Rossum
Mar 26, 2021, 7:25:00 PM3/26/21
to Irit Katriel, python-dev, Yury Selivanov
Everyone,
Given the resounding silence I'm inclined to submit this to the Steering Council. While I'm technically a co-author, Irit has done almost all the work, and she's done a great job. If there are no further issues I'll send this SC-wards on Monday.
--Guido
--Guido van Rossum (python.org/~guido)
Pronouns: he/him (why is my pronoun here?)
Terry Reedy
Mar 26, 2021, 9:47:28 PM3/26/21
to pytho...@python.org
On 3/26/2021 7:19 PM, Guido van Rossum wrote:
> Everyone,
>
> Given the resounding silence I'm inclined to submit this to the Steering
> Council. While I'm technically a co-author, Irit has done almost all the
> work, and she's done a great job. If there are no further issues I'll
> send this SC-wards on Monday.
The current version looks very carefully done. I leave it to the SC to
> Everyone,
>
> Given the resounding silence I'm inclined to submit this to the Steering
> Council. While I'm technically a co-author, Irit has done almost all the
> work, and she's done a great job. If there are no further issues I'll
> send this SC-wards on Monday.
review the details. But two thoughts on other possible uses.
1. unittests already uses the idea of exception groups by catching
test_xyz exceptions and adding them to a group, to be printed it when
all those for a file are run. If the PEP is accepted, someone might
request an option to have them packaged as an ExceptionGroup.
2. At least some compilers for other languages can report multiple
syntax exceptions, but at least for C, the usefulness of more than 2 or
3 is crippled by the synchronization problem. Python's compile() only
reports the first, but I imagine that the usually dependable parsing
into statements might make multiple reports for Python useful. I am
thinking of a 'multiple' mode that repeatedly and recursively did
'single' compiles. (There would still be only one report per statement
or compound statement header.) I have seen newbie code posted on
Stackoverflow that needed a multi-error report.
--
Terry Jan Reedy
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/CT2SLYRZLNEDCPOSNGFZGTUHRRNJUUBE/
Code of Conduct: http://python.org/psf/codeofconduct/
Nick Coghlan
Mar 27, 2021, 1:04:19 AM3/27/21
to Guido van Rossum, Irit Katriel, python-dev, Yury Selivanov
On Sat, 27 Mar 2021, 9:24 am Guido van Rossum, <gu...@python.org> wrote:
Everyone,Given the resounding silence I'm inclined to submit this to the Steering Council. While I'm technically a co-author, Irit has done almost all the work, and she's done a great job. If there are no further issues I'll send this SC-wards on Monday.
+1 from me. I remember talking to Nathaniel (and Yury IIRC) about early iterations of the MultiError handling in trio at PyCon years ago, so it's great to see the way the concept has matured in the intervening time.
The one thing I did think was that it might be good to mention concurrent.futures and contextlib in the PEP as modules that could potentially take advantage of exception groups. I'm not sure it's really necessary though, as the presentation is already compelling as it is.
Cheers,
Nick.
P.S. The potential use case I see in contextlib: the way ExitStack nests dynamic resource handling *works*, but the tracebacks when multiple clean up steps fail can be spectacularly cryptic. With exception groups available, it should be possible to define a new "context lib.ExitGroup" API that flattens out the resource clean up and collects any exceptions that occur into a more comprehensible exception group, rather than building a nested exception context tree the way ExitStack does.
--GuidoOn Sat, Mar 20, 2021 at 10:05 AM Irit Katriel <iritk...@googlemail.com> wrote:We would like to present for feedback a new version of PEP 654, which incorporates the feedback we received in the discussions so far:The reference implementation has also been updated along with the PEP.The changes we made since the first post are:1. Instead of ExceptionGroup(BaseException), we will have two new builtin types: BaseExceptionGroup(BaseException) and ExceptionGroup(BaseExceptionGroup, Exception).This is so that "except Exception" catches ExceptionGroups (but not BaseExceptionGroups). BaseExceptionGroup.__new__ inspects the wrapped exceptions, and if they are all Exception subclasses, it creates an ExceptionGroup instead of a BaseExceptionGroup.2. The exception group classes are not final - they can be subclassed and split()/subgroup() work correctly if the subclass overrides the derive() instance method as described here: https://www.python.org/dev/peps/pep-0654/#subclassing-exception-groups3. We had some good suggestions on formatting exception groups, which we have implemented as you can see in the output shown for the examples in the PEP.4. We expanded the section on handling Exception Groups, to show how subgroup can be used (with side effects) to do something for each leaf exception, and how to iterate correctly when the tracebacks of leaf exceptions are needed: https://www.python.org/dev/peps/pep-0654/#handling-exception-groups5. We expanded the sections on rationale and backwards compatibility to explain our premise and expectations regarding how exception groups will be used and how the transition to using them will be managed.6. We added several items to the rejected ideas section.We did not receive any comments (or make any changes) to the proposed semantics of except*, hopefully this is because everyone thought they are sensible.Irit, Yury and Guido
----Guido van Rossum (python.org/~guido)Pronouns: he/him (why is my pronoun here?)
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Paul Sokolovsky
Mar 27, 2021, 6:36:29 AM3/27/21
to Guido van Rossum, Irit Katriel, python-dev, Yury Selivanov
Hello,
On Fri, 26 Mar 2021 16:19:26 -0700
Guido van Rossum <gu...@python.org> wrote:
> Everyone,
>
> Given the resounding silence I'm inclined to submit this to the
> Steering Council. While I'm technically a co-author, Irit has done
> almost all the work, and she's done a great job. If there are no
> further issues I'll send this SC-wards on Monday.
One issue with PEP654 is that it introduces pretty adhoc and
complex-semantics concept (ExceptionGroup) on the language level.
Here's an idea (maybe duplicate) on how to introduce a much simpler,
and more generic concept on the language level, and let particular
frameworks to introduce (and elaborate without further changing the
language) adhoc concept they need.
So, let's look how the usual "except MyExc as e" works: it performs
"isinstance(e0, MyExc)" operation, where e0 is incoming exception
(roughly, sys.exc_info[1]), and if it returns True, then assigns e0 to
the "e" variable and executes handler body. "isinstance(e0, MyExc)" is
formally known as an "exception filter".
As we see, currently Python hardcodes isinstance() as exception filter.
The idea is to allow to use an explicit exception filter. Let's reuse
the same "except *" syntax to specify it. Also, it's more flexible
instead of returning True/False from filter, to return either None
(filter didn't match), or an exception object to make available to
handler (which in general may be different than passed to the filter).
With this, ExceptionGroup usecases should be covered.
Examples:
1. Current implicit exception filter is equivalent to:
def implicit(e0, excs): # one or tuple, as usual
if isinstance(e0, excs):
return e0
return None
try:
1/0
except *implicit(ZeroDivisionError) as e:
print(e)
2. Allow to catch main or chained exception (context manager example
from PEP)
def chained(e, excs):
while e:
if isinstance(e, excs):
return e
e = e.__cause__ # Simplified, should consider __context__ too
try:
tempfile.TemporaryDirectory(...)
except *chained(OSError) as e:
print(e)
3. Rough example of ExceptionGroup functionality (now not a language
builtin, just implemented by framework(s) which need it, or as a
separate module):
class ExceptionGroup:
...
@staticmethod
def match(e0, excs):
cur, rest = e0.split_by_types(excs)
# That's how we allow an exception handler to re-raise either an
# original group in full or just "unhandled" exception in the
# group (or anything) - everything should be passed via
# exception attributes (or computed by methods).
cur.org = e0
cur.rest = rest
return cur
try:
...
except *ExceptionGroup.match((TypeError, ValueError)) as e:
# try to handle a subgroup with (TypeError, ValueError) here
...
# now reraise a subgroup with unhandled exceptions from the
# original group
raise e.rest
> *Pronouns: he/him **(why is my pronoun here?)*
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
--
Best regards,
Paul mailto:pmi...@gmail.com
On Fri, 26 Mar 2021 16:19:26 -0700
Guido van Rossum <gu...@python.org> wrote:
> Everyone,
>
> Given the resounding silence I'm inclined to submit this to the
> Steering Council. While I'm technically a co-author, Irit has done
> almost all the work, and she's done a great job. If there are no
> further issues I'll send this SC-wards on Monday.
complex-semantics concept (ExceptionGroup) on the language level.
Here's an idea (maybe duplicate) on how to introduce a much simpler,
and more generic concept on the language level, and let particular
frameworks to introduce (and elaborate without further changing the
language) adhoc concept they need.
So, let's look how the usual "except MyExc as e" works: it performs
"isinstance(e0, MyExc)" operation, where e0 is incoming exception
(roughly, sys.exc_info[1]), and if it returns True, then assigns e0 to
the "e" variable and executes handler body. "isinstance(e0, MyExc)" is
formally known as an "exception filter".
As we see, currently Python hardcodes isinstance() as exception filter.
The idea is to allow to use an explicit exception filter. Let's reuse
the same "except *" syntax to specify it. Also, it's more flexible
instead of returning True/False from filter, to return either None
(filter didn't match), or an exception object to make available to
handler (which in general may be different than passed to the filter).
With this, ExceptionGroup usecases should be covered.
Examples:
1. Current implicit exception filter is equivalent to:
def implicit(e0, excs): # one or tuple, as usual
if isinstance(e0, excs):
return e0
return None
try:
1/0
except *implicit(ZeroDivisionError) as e:
print(e)
2. Allow to catch main or chained exception (context manager example
from PEP)
def chained(e, excs):
while e:
if isinstance(e, excs):
return e
e = e.__cause__ # Simplified, should consider __context__ too
try:
tempfile.TemporaryDirectory(...)
except *chained(OSError) as e:
print(e)
3. Rough example of ExceptionGroup functionality (now not a language
builtin, just implemented by framework(s) which need it, or as a
separate module):
class ExceptionGroup:
...
@staticmethod
def match(e0, excs):
cur, rest = e0.split_by_types(excs)
# That's how we allow an exception handler to re-raise either an
# original group in full or just "unhandled" exception in the
# group (or anything) - everything should be passed via
# exception attributes (or computed by methods).
cur.org = e0
cur.rest = rest
return cur
try:
...
except *ExceptionGroup.match((TypeError, ValueError)) as e:
# try to handle a subgroup with (TypeError, ValueError) here
...
# now reraise a subgroup with unhandled exceptions from the
# original group
raise e.rest
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
--
Best regards,
Paul mailto:pmi...@gmail.com
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/HVSD3OETIBE3NKFNFLK2GDGXCGFB2CQ2/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Irit Katriel via Python-Dev
Mar 27, 2021, 7:00:48 AM3/27/21
to Paul Sokolovsky, python-dev, Yury Selivanov
One of the motivations for introducing ExceptionGroup as a builtin is so that we won't have a different custom version in each library that needs it. So if you are writing a library the needs to raise multiple exceptions, and then you decide to call Trio, you don't need to translate Trio's MultiError into your own exception group type, because everybody uses the builtin. And your users don't need to learn how your particular exception group works because they know that you are using the builtin one.
I see the aesthetic value of your suggestion, but does it have practical advantages in light of the above?
Paul Sokolovsky
Mar 27, 2021, 8:47:40 AM3/27/21
to Irit Katriel, python-dev, Yury Selivanov
Hello,
On Sat, 27 Mar 2021 10:55:40 +0000
Irit Katriel <iritk...@googlemail.com> wrote:
> One of the motivations for introducing ExceptionGroup as a builtin is
> so that we won't have a different custom version in each library that
> needs it. So if you are writing a library the needs to raise multiple
> exceptions, and then you decide to call Trio, you don't need to
> translate Trio's MultiError into your own exception group type,
> because everybody uses the builtin.
Looking from a different angle shows a different perspective:
1. Trio devised an interesting concept of "nurseries" to deal with
multiple tasks in asyncio-like programming.
2. It was all nice and beautiful until ... it came to error handling.
Note that it's a rather typical situation - one can write nice, clean,
beautiful code, which is not adequate in real-world scenarios,
particularly because of error handling concerns. Oftentimes, such
initial code/concepts are discarded, and more robust (though maybe not
as beautiful) concepts/code is used.
3. But that's not what happened in the Trio case. The concept of
nurseries got pushed forward, until it became clear that it requires
changes on the programming language level.
4. That's how PEP654 was born, which, besides async scheduling of
multiple tasks, bring *somewhat similar* usecases of e.g. raising
exceptions thru context managers' own exceptions.
Note that where errors and exceptions lead us is in questions on how to
handle them. And beyond a couple of well known patterns ("dump and
crash" or "dump and continue with next iteration"), error handling is
very adhoc to a particular application and particular error(s). Seen
from that angle, Trio wants to vendor-lock the entire language into its
particular (experimental) way of handling multiple errors.
So yes, maybe being well aware that you're handling exactly Trio's (or
asyncio's, or context manager's) error isn't bad thing actually. And if
it's clear that multiple asyncio frameworks are interested in sharing
common exception base class for such usecases, then it could be
introduced on the "asyncio" package level, or maybe even better, as a
module similar to "concurrent.futures".
> And your users don't need to
> learn how your particular exception group works because they know
> that you are using the builtin one.
>
> I see the aesthetic value of your suggestion, but does it have
> practical advantages in light of the above?
The concern is that it codifies pretty complex and special-purpose
things on the language level. And it seems that the whole concept is
rather experimental and "original design". We can compare that with
another sufficiently complex feature which landed recently: pattern
matching. At all phases of the design and discussion of that feature,
one of the guiding principles was: "Many other languages implement it,
so we know it's generally useful, and have design space more or less
charted, and can vary things to find local optimum for Python".
Contrary to that, the original PEP654 didn't refer to (cross-language,
cross-framework) prior art in handling multiple errors, and I don't see
that changed in the latest version. So I'm sorry, but it seems like NIH
feature of a specific 3rd-party framework being promoted to a whole
language's way of doing things.
Under such circumstance, I guess it would be good idea to try to
decouple behavior of that feature from the languages core, and make
aspects of behavior more explicit (following "explicit is better than
implicit" principle), and allow to vary/evolve it without changing the
core language.
I tried to draft a scheme aspiring to allow that. (Which would
definitely need more work to achieve parity with functionality in
PEP654, because again, it tries to codify rather complex and "magic"
behavior. Where complex and magic behavior in exception handling is
itself of concern, so making it more explicit may be a good idea. (A
good "how other languages deal with it") review would mention that Go,
Rust, Zig, etc. don't have and condemn exception handling at all (normal
simple classy exceptions, not magic we discuss here!)).
Thanks,
Paul
On Sat, 27 Mar 2021 10:55:40 +0000
Irit Katriel <iritk...@googlemail.com> wrote:
> One of the motivations for introducing ExceptionGroup as a builtin is
> so that we won't have a different custom version in each library that
> needs it. So if you are writing a library the needs to raise multiple
> exceptions, and then you decide to call Trio, you don't need to
> translate Trio's MultiError into your own exception group type,
> because everybody uses the builtin.
1. Trio devised an interesting concept of "nurseries" to deal with
multiple tasks in asyncio-like programming.
2. It was all nice and beautiful until ... it came to error handling.
Note that it's a rather typical situation - one can write nice, clean,
beautiful code, which is not adequate in real-world scenarios,
particularly because of error handling concerns. Oftentimes, such
initial code/concepts are discarded, and more robust (though maybe not
as beautiful) concepts/code is used.
3. But that's not what happened in the Trio case. The concept of
nurseries got pushed forward, until it became clear that it requires
changes on the programming language level.
4. That's how PEP654 was born, which, besides async scheduling of
multiple tasks, bring *somewhat similar* usecases of e.g. raising
exceptions thru context managers' own exceptions.
Note that where errors and exceptions lead us is in questions on how to
handle them. And beyond a couple of well known patterns ("dump and
crash" or "dump and continue with next iteration"), error handling is
very adhoc to a particular application and particular error(s). Seen
from that angle, Trio wants to vendor-lock the entire language into its
particular (experimental) way of handling multiple errors.
So yes, maybe being well aware that you're handling exactly Trio's (or
asyncio's, or context manager's) error isn't bad thing actually. And if
it's clear that multiple asyncio frameworks are interested in sharing
common exception base class for such usecases, then it could be
introduced on the "asyncio" package level, or maybe even better, as a
module similar to "concurrent.futures".
> And your users don't need to
> learn how your particular exception group works because they know
> that you are using the builtin one.
>
> I see the aesthetic value of your suggestion, but does it have
> practical advantages in light of the above?
things on the language level. And it seems that the whole concept is
rather experimental and "original design". We can compare that with
another sufficiently complex feature which landed recently: pattern
matching. At all phases of the design and discussion of that feature,
one of the guiding principles was: "Many other languages implement it,
so we know it's generally useful, and have design space more or less
charted, and can vary things to find local optimum for Python".
Contrary to that, the original PEP654 didn't refer to (cross-language,
cross-framework) prior art in handling multiple errors, and I don't see
that changed in the latest version. So I'm sorry, but it seems like NIH
feature of a specific 3rd-party framework being promoted to a whole
language's way of doing things.
Under such circumstance, I guess it would be good idea to try to
decouple behavior of that feature from the languages core, and make
aspects of behavior more explicit (following "explicit is better than
implicit" principle), and allow to vary/evolve it without changing the
core language.
I tried to draft a scheme aspiring to allow that. (Which would
definitely need more work to achieve parity with functionality in
PEP654, because again, it tries to codify rather complex and "magic"
behavior. Where complex and magic behavior in exception handling is
itself of concern, so making it more explicit may be a good idea. (A
good "how other languages deal with it") review would mention that Go,
Rust, Zig, etc. don't have and condemn exception handling at all (normal
simple classy exceptions, not magic we discuss here!)).
Thanks,
Paul
--
Best regards,
Paul mailto:pmi...@gmail.com
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/FN5IH3ICXSYSLGXWRBZLNWCC6E4SYPZL/
Best regards,
Paul mailto:pmi...@gmail.com
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Irit Katriel via Python-Dev
Mar 27, 2021, 10:35:01 AM3/27/21
to Paul Sokolovsky, python-dev, Yury Selivanov
Hi Paul,
IIUC, you are saying that exception group should not be a builtin type because it is (1) complex (2) special-purposed. Instead, you propose that we make exception handling pluggable.
(1) I agree, it is somewhat complex - it took us several iterations to get from the idea of "a container of exceptions" to something that works correctly with split()/subgroup(), maintaining the integrity of __traceback__/__cause__/__context__ information, allowing subclassing and having a relatively simple API. To me this is a reason to provide it as a builtin rather than a reason not to (I think we should add this point to the PEP- thanks).
You wrote:
> So yes, maybe being well aware that you're handling exactly Trio's (or asyncio's, or context manager's) error isn't bad thing actually.
Maybe, in which case you can create a subclass of ExceptionGroup to mark the groups coming from your library. But also maybe not, in which case you don't have to.
(2) special-purposed? How so? When I asked for a practical advantage of your pluggable solution I meant an example of a use case that our proposal doesn't accommodate well and where a pluggable one does better.
The only example you suggested so far was the TemporaryDirectory one, which I don't find compelling:
> > def chained(e, excs):
> > while e:
> > if isinstance(e, excs):
> > return e
> > e = e.__cause__ # Simplified, should consider __context__
> > too
> >
> > try:
> > tempfile.TemporaryDirectory(...)
> > except *chained(OSError) as e:
> > print(e)
> > while e:
> > if isinstance(e, excs):
> > return e
> > e = e.__cause__ # Simplified, should consider __context__
> > too
> >
> > try:
> > tempfile.TemporaryDirectory(...)
> > except *chained(OSError) as e:
> > print(e)
What if the user code raised an OSError too, so now that exception group has more than one? Even without that problem, why should I prefer this to our proposal?
I am interested to know (1) what limitations you see in the ExceptionGroup type we are proposing (in what sense is it special-purposed? what purpose is it unsuitable for?) and (2) an example where the pluggable solution works better.
Irit
On Sat, Mar 27, 2021 at 12:42 PM Paul Sokolovsky <pmi...@gmail.com> wrote:
Hello,
Nick Coghlan
Mar 27, 2021, 12:02:40 PM3/27/21
to Irit Katriel, python-dev, Yury Selivanov
On Sun, 28 Mar 2021, 12:34 am Irit Katriel via Python-Dev, <pytho...@python.org> wrote:
Hi Paul,IIUC, you are saying that exception group should not be a builtin type because it is (1) complex (2) special-purposed. Instead, you propose that we make exception handling pluggable.
Note that (2) would be the *exact opposite* of the direction that we have taken exception handling in general: it used to be far more permissive than it is, and then got more restrictive over time to make structured exception handling more consistent.
In particular, features like exception chaining and use case independent trace back printing only work because exception consumers can rely on all exceptions being instances of BaseException and having particular double-underscore attributes like "cause", "context", and "traceback"
The lesson we've learned from Trio's experience is:
* regular exceptions are still fine for most purposes
* when they're not fine, they're not fine in a consistent way across a variety of use cases where the existing exception machinery currently loses or hides information, the problem isn't specific to Trio's nursery API
I understand the temptation to generalise, but a general proposal would fail for the same reason the "user defined infix operators" PEPs failed: despite decades of experience, there just weren't compelling use cases for a general purpose feature. Instead, the proposal that succeeded was narrow: it targeted the feature that people actually wanted (an infix operator for matrix multiplication), rather than proposing more general machinery that could be used to build that feature.
Cheers,
Nick.
Paul Sokolovsky
Mar 27, 2021, 2:06:11 PM3/27/21
to Irit Katriel, python-dev, Yury Selivanov
Hello,
On Sat, 27 Mar 2021 14:29:21 +0000
Irit Katriel <iritk...@googlemail.com> wrote:
> Hi Paul,
>
> IIUC, you are saying that exception group should not be a builtin type
> because it is (1) complex (2) special-purposed. Instead, you
> propose that we make exception handling pluggable.
Yes, I wanted to mention that alternative possibility. I have to admit
that I haven't read the entire discussion of the PEP, but the messages
I read oftentimes were over-optimistic about the feature, rather than
considering its issues/possible alternatives.
> (1) I agree, it is somewhat complex - it took us several iterations
> to get from the idea of "a container of exceptions" to something that
> works correctly with split()/subgroup(), maintaining the integrity of
> __traceback__/__cause__/__context__ information, allowing subclassing
> and having a relatively simple API. To me this is a reason to provide
> it as a builtin rather than a reason not to (I think we should add
> this point to the PEP- thanks).
It definitely feels like a lot of effort went into devising and
polishing ExceptionGroup's and except*, thanks. But I'm not sure if you
gentlemen come up with the "ultimate" way to deal with multiple errors,
which deserves being codified on the language level (vs be added as
pluggable means indeed). That's why I asked if there's any prior art of
a similar solution(s) in other languages and/or systems - that would
definitely sooth concerns and add credibility.
(And I'm not saying that Python can't lead there - just that more
sustainable way would be to try and make it more generic/explicit rather
than fix on a particular novel solution right away.)
> You wrote:
> > So yes, maybe being well aware that you're handling exactly Trio's
> > (or
> asyncio's, or context manager's) error isn't bad thing actually.
>
> Maybe, in which case you can create a subclass of ExceptionGroup to
> mark the groups coming from your library. But also maybe not, in
> which case you don't have to.
>
> (2) special-purposed? How so? When I asked for a practical
> advantage of your pluggable solution I meant an example of a use case
> that our proposal doesn't accommodate well and where a pluggable one
> does better. The only example you suggested so far was the
> TemporaryDirectory one, which I don't find compelling:
That's the impression I've got from reading the "Motivation" section.
It cites Trio and multiple errors from async tasks as the guiding
usecase. It also mentions a couple of cases in the stdlib, where my
personal impression is that a simpler mechanism would suffice. Then it
also cites some 3rd-party libs and their maintainers, who say such
thing *might* be useful. But it's going to take practice to see if the
actual thing added is useful. What if other projects turn out
to want something slightly different (e.g. simpler?). Oh, they won't
have much choice (to complain), as it's already shipped to them, not
even in the stdlib, but in the language!
Bottom line: this seems like a Trio's special-purpose feature, with
good wishes of becoming the de facto standard. I find that somewhat
ironical, as Trio's original motivation was to explore alternative ways
of doing async framework in Python. And now it hardcodes its way of
dealing with multiple errors in the language (no more exploration and
alternatives).
> > > def chained(e, excs):
> > > while e:
> > > if isinstance(e, excs):
> > > return e
> > > e = e.__cause__ # Simplified, should consider __context__
> > > too
> > >
> > > try:
> > > tempfile.TemporaryDirectory(...)
> > > except *chained(OSError) as e:
> > > print(e)
>
> What if the user code raised an OSError too, so now that exception
> group has more than one?
I don't see how that's different from PEP654's ExceptionGroup's case.
It's just in ExceptionGroup, there would be 2 (unrelated?) OSError's,
while using normal chaining rules, there's at least context info which
is preserved.
> Even without that problem, why should I
> prefer this to our proposal?
My idea was to show an alternative not mentioned in PEP654's "Rejected
Ideas" section.
> I am interested to know (1) what limitations you see in the
> ExceptionGroup type we are proposing (in what sense is it
> special-purposed?
Just as a specific example, what if my application indeed needs
ExceptionGroup's (and not just more flexible matching of existing
exception chains), but I want it to be a bit simpler: a) to have a flat
structure of contained exceptions; b) want to actually emphasize the
users that they should not rely on the ordering, so want it to be a set.
My understanding is that ExceptionGroup is carefully designed to allow
subclassing it and override behavior, and the above should be possible
by subclassing. But usually, a simpler base class is subclassed to have
more advanced/complex behavior. So again, current ExceptionGroup feels
kind of specialized with its careful attention to hierarchy and
multi-story "3D" tracebacks. It will be good if you actually found the
sweet spot. But what if not, and it will be a UX chore (Python's chore,
nor a particular framework's chore)?
> what purpose is it unsuitable for?) and (2) an
> example where the pluggable solution works better.
From my PoV, a solution which doesn't add particular complex behavior
into the language core, but allows to plug it in, and keep whole thing
more explicit, is something that "works better".
Again, I wanted to mention that point, as between the initial PEP
version and the latest one, I don't see that alternative to have been
mentioned in the PEP (which might mean that it wasn't mentioned enough
by other reviewers).
On Sat, 27 Mar 2021 14:29:21 +0000
Irit Katriel <iritk...@googlemail.com> wrote:
> Hi Paul,
>
> IIUC, you are saying that exception group should not be a builtin type
> because it is (1) complex (2) special-purposed. Instead, you
> propose that we make exception handling pluggable.
that I haven't read the entire discussion of the PEP, but the messages
I read oftentimes were over-optimistic about the feature, rather than
considering its issues/possible alternatives.
> (1) I agree, it is somewhat complex - it took us several iterations
> to get from the idea of "a container of exceptions" to something that
> works correctly with split()/subgroup(), maintaining the integrity of
> __traceback__/__cause__/__context__ information, allowing subclassing
> and having a relatively simple API. To me this is a reason to provide
> it as a builtin rather than a reason not to (I think we should add
> this point to the PEP- thanks).
polishing ExceptionGroup's and except*, thanks. But I'm not sure if you
gentlemen come up with the "ultimate" way to deal with multiple errors,
which deserves being codified on the language level (vs be added as
pluggable means indeed). That's why I asked if there's any prior art of
a similar solution(s) in other languages and/or systems - that would
definitely sooth concerns and add credibility.
(And I'm not saying that Python can't lead there - just that more
sustainable way would be to try and make it more generic/explicit rather
than fix on a particular novel solution right away.)
> You wrote:
> > So yes, maybe being well aware that you're handling exactly Trio's
> > (or
> asyncio's, or context manager's) error isn't bad thing actually.
>
> Maybe, in which case you can create a subclass of ExceptionGroup to
> mark the groups coming from your library. But also maybe not, in
> which case you don't have to.
>
> (2) special-purposed? How so? When I asked for a practical
> advantage of your pluggable solution I meant an example of a use case
> that our proposal doesn't accommodate well and where a pluggable one
> does better. The only example you suggested so far was the
> TemporaryDirectory one, which I don't find compelling:
It cites Trio and multiple errors from async tasks as the guiding
usecase. It also mentions a couple of cases in the stdlib, where my
personal impression is that a simpler mechanism would suffice. Then it
also cites some 3rd-party libs and their maintainers, who say such
thing *might* be useful. But it's going to take practice to see if the
actual thing added is useful. What if other projects turn out
to want something slightly different (e.g. simpler?). Oh, they won't
have much choice (to complain), as it's already shipped to them, not
even in the stdlib, but in the language!
Bottom line: this seems like a Trio's special-purpose feature, with
good wishes of becoming the de facto standard. I find that somewhat
ironical, as Trio's original motivation was to explore alternative ways
of doing async framework in Python. And now it hardcodes its way of
dealing with multiple errors in the language (no more exploration and
alternatives).
> > > def chained(e, excs):
> > > while e:
> > > if isinstance(e, excs):
> > > return e
> > > e = e.__cause__ # Simplified, should consider __context__
> > > too
> > >
> > > try:
> > > tempfile.TemporaryDirectory(...)
> > > except *chained(OSError) as e:
> > > print(e)
>
> What if the user code raised an OSError too, so now that exception
> group has more than one?
It's just in ExceptionGroup, there would be 2 (unrelated?) OSError's,
while using normal chaining rules, there's at least context info which
is preserved.
> Even without that problem, why should I
> prefer this to our proposal?
Ideas" section.
> I am interested to know (1) what limitations you see in the
> ExceptionGroup type we are proposing (in what sense is it
> special-purposed?
ExceptionGroup's (and not just more flexible matching of existing
exception chains), but I want it to be a bit simpler: a) to have a flat
structure of contained exceptions; b) want to actually emphasize the
users that they should not rely on the ordering, so want it to be a set.
My understanding is that ExceptionGroup is carefully designed to allow
subclassing it and override behavior, and the above should be possible
by subclassing. But usually, a simpler base class is subclassed to have
more advanced/complex behavior. So again, current ExceptionGroup feels
kind of specialized with its careful attention to hierarchy and
multi-story "3D" tracebacks. It will be good if you actually found the
sweet spot. But what if not, and it will be a UX chore (Python's chore,
nor a particular framework's chore)?
> what purpose is it unsuitable for?) and (2) an
> example where the pluggable solution works better.
into the language core, but allows to plug it in, and keep whole thing
more explicit, is something that "works better".
Again, I wanted to mention that point, as between the initial PEP
version and the latest one, I don't see that alternative to have been
mentioned in the PEP (which might mean that it wasn't mentioned enough
by other reviewers).
--
Best regards,
Paul mailto:pmi...@gmail.com
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/SAL5DF3PUJ2DHDI5YUV6T4LPCNN4JY3C/
Best regards,
Paul mailto:pmi...@gmail.com
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Irit Katriel via Python-Dev
Mar 27, 2021, 4:07:05 PM3/27/21
to Paul Sokolovsky, python-dev, Yury Selivanov
Hi Paul,
On Sat, Mar 27, 2021 at 6:00 PM Paul Sokolovsky <pmi...@gmail.com> wrote:
It definitely feels like a lot of effort went into devising and
polishing ExceptionGroup's and except*, thanks. But I'm not sure if you
gentlemen come up with the "ultimate" way to deal with multiple errors,
I've been mistaken for a man before, but no-one has ever confused me for gentle. I'll take that as a compliment.
which deserves being codified on the language level (vs be added as
pluggable means indeed).
Pluggable is not without its problems. I'm all in favor of you developing this idea and proposing an alternative.
As I said before, you just need to answer two questions:
1. show a limitation of our approach (the contrived flat-set is not one - see below)
2. describe how a pluggable approach works for that case
> > > def chained(e, excs):
> > > while e:
> > > if isinstance(e, excs):
> > > return e
> > > e = e.__cause__ # Simplified, should consider __context__
> > > too
> > >
> > > try:
> > > tempfile.TemporaryDirectory(...)
> > > except *chained(OSError) as e:
> > > print(e)
>
> What if the user code raised an OSError too, so now that exception
> group has more than one?
I don't see how that's different from PEP654's ExceptionGroup's case.
It's just in ExceptionGroup, there would be 2 (unrelated?) OSError's,
while using normal chaining rules, there's at least context info which
is preserved.
It's different because your "match" function returns a single exception (the first one that is of OSError type). Any further OSErrors will be reraised. The PEP's except* knows how to match multiple exceptions of the relevant type.
> I am interested to know (1) what limitations you see in the
> ExceptionGroup type we are proposing (in what sense is it
> special-purposed?
Just as a specific example, what if my application indeed needs
ExceptionGroup's (and not just more flexible matching of existing
exception chains), but I want it to be a bit simpler: a) to have a flat
structure of contained exceptions; b) want to actually emphasize the
users that they should not rely on the ordering, so want it to be a set.
My understanding is that ExceptionGroup is carefully designed to allow
subclassing it and override behavior, and the above should be possible
by subclassing.
Indeed:
class PaulsExceptionGroup(ExceptionGroup):
def __new__(cls, message, excs):
for e in excs:
if isinstance(e, BaseExceptionGroup):
raise TypeError("me is a flat exception group")
return super().__new__(cls, message, excs)
@property
def exceptions(self):
return set(super().exceptions)
def derive(self, excs):
return PaulsExceptionGroup(self.message, excs)
eg1 = PaulsExceptionGroup("Paul's group", [ValueError(12), TypeError(42)])
print(repr(eg1))
print(eg1.exceptions)
match, rest = eg1.split(ValueError)
print(f'{match=!r}\n{rest=!r}')
print(f'{match.exceptions=}\n{rest.exceptions=}')
Output:
PaulsExceptionGroup("Paul's group", [ValueError(12), TypeError(42)])
{TypeError(42), ValueError(12)}
match=PaulsExceptionGroup("Paul's group", [ValueError(12)])
rest=PaulsExceptionGroup("Paul's group", [TypeError(42)])
match.exceptions={ValueError(12)}
rest.exceptions={TypeError(42)}
See - split() just works for you out of the box.
But usually, a simpler base class is subclassed to have
more advanced/complex behavior. So again, current ExceptionGroup feels
kind of specialized with its careful attention to hierarchy and
multi-story "3D" tracebacks.
So your concern with our design is that ExceptionGroup implements a generic split() that handles tracebacks and nested structure correctly, and you might not need that because maybe you don't nest or you don't care about tracebacks?
If that is the case then I think you are confusing "generic" with "special-purpose". ExceptionGroup is generic, it works for general nested groups with arbitrary tracebacks and cause/context. That's the opposite of special-purpose.
Irit
Greg Ewing
Mar 27, 2021, 7:00:38 PM3/27/21
to pytho...@python.org
While we're talking about compelling use cases, does anyone have an
actual, concrete use case for the proposed "except *" feature that's
strong enough to justify new syntax?
I'm fine with having ExceptionGroup as a built-in type. I'm not fine
with adding new syntax that will apparently be used only in rare
circumstances.
Can code that's aware of the possibility of getting an ExceptionGroup
not simply catch it as a normal exception and then pick it apart? Do
we really need a whole new piece of machinery for this?
--
Greg
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/EIW6JGXRWKEEIY72CGZ7BMKU3LHJIZQK/
actual, concrete use case for the proposed "except *" feature that's
strong enough to justify new syntax?
I'm fine with having ExceptionGroup as a built-in type. I'm not fine
with adding new syntax that will apparently be used only in rare
circumstances.
Can code that's aware of the possibility of getting an ExceptionGroup
not simply catch it as a normal exception and then pick it apart? Do
we really need a whole new piece of machinery for this?
--
Greg
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Irit Katriel via Python-Dev
Mar 28, 2021, 8:54:01 AM3/28/21
to Greg Ewing, python-dev
Hi Greg,
If all you want is to catch an exception group and process it, then except* does look like overkill.
It gets more interesting if you want to handle only some of the exceptions and reraise the rest (without adding the current frame to the traceback), or when the exception handler raises/reraises exceptions (with 'raise e' or bare 'raise').
We can look to Trio to see what exception handling would be like if we add ExceptionGroup without except*.
Trio's exception group is called MultiError, and this part of the tutorial explains how they can be handled:
So asyncio (or some new module in the stdlib) would probably end up exposing something along the lines of MultiError.catch, a context manager where you register the exception handler as a callback and the unhandled exceptions are automatically reraised.
You can see the trio code for that here: https://github.com/python-trio/trio/blob/master/trio/_core/_multierror.py#L129
The trick it does to reset __context__ in the finally block won't work for the __traceback__ (I tried). So it always adds the current frame (if it doesn't return before line 145).
Also, it doesn't seem to support raising exceptions from the handler: https://github.com/python-trio/trio/blob/6754c74eacfad9cc5c92d5c24727a2f3b620624e/trio/_core/_multierror.py#L79
So yes, we can catch exception groups and handle them like normal exceptions, but I don't see a clean way to catch parts of them selectively or to raise exceptions from exception handling code.
Irit
Guido van Rossum
Mar 28, 2021, 11:57:18 AM3/28/21
to Greg Ewing, Python-Dev
On Sat, Mar 27, 2021 at 3:56 PM Greg Ewing <greg....@canterbury.ac.nz> wrote:
While we're talking about compelling use cases, does anyone have an
actual, concrete use case for the proposed "except *" feature that's
strong enough to justify new syntax?
I'm fine with having ExceptionGroup as a built-in type. I'm not fine
with adding new syntax that will apparently be used only in rare
circumstances.
Can code that's aware of the possibility of getting an ExceptionGroup
not simply catch it as a normal exception and then pick it apart? Do
we really need a whole new piece of machinery for this?
As Irit already wrote, the code to do this correctly is increasingly subtle and without special syntax it becomes too easy to do it wrong. Also note that multiple `except *` clauses can all run, since they handle (disjunct) different subsets of the group.
--
--Guido van Rossum (python.org/~guido)
Yury Selivanov
Mar 29, 2021, 3:46:15 PM3/29/21
to Paul Sokolovsky, Irit Katriel, python-dev, Yury Selivanov
Just a few comments to add to Irit's response.
On Sat, Mar 27, 2021 at 11:03 AM Paul Sokolovsky <pmi...@gmail.com> wrote:
[..]
and intuitive primitive. But the error handling API around them is
unintuitive and hard to use. Now that we want to add an equivalent of
nurseries to asyncio (we'll likely call them Task Groups) we need to
sort out the error handling mechanism, finally.
> From my PoV, a solution which doesn't add particular complex behavior
> into the language core, but allows to plug it in, and keep whole thing
> more explicit, is something that "works better".
I understand your PoV, but there's a point where building explicit
complex APIs becomes so detrimental to the overall usability that
there's a warrant for solving the problem in syntax and with builtin
types.
Yury
On Sat, Mar 27, 2021 at 11:03 AM Paul Sokolovsky <pmi...@gmail.com> wrote:
[..]
> Bottom line: this seems like a Trio's special-purpose feature, with
> good wishes of becoming the de facto standard.
The bottom line is that Trio nurseries were proven to be a very useful
> good wishes of becoming the de facto standard.
and intuitive primitive. But the error handling API around them is
unintuitive and hard to use. Now that we want to add an equivalent of
nurseries to asyncio (we'll likely call them Task Groups) we need to
sort out the error handling mechanism, finally.
> From my PoV, a solution which doesn't add particular complex behavior
> into the language core, but allows to plug it in, and keep whole thing
> more explicit, is something that "works better".
complex APIs becomes so detrimental to the overall usability that
there's a warrant for solving the problem in syntax and with builtin
types.
Yury
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/YIOMBHTEF4VHXSOW5CKSXAK5ELFYQMFO/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Paul Sokolovsky
Mar 30, 2021, 6:15:43 AM3/30/21
to Irit Katriel, python-dev, Yury Selivanov
Hello,
On Sat, 27 Mar 2021 20:01:27 +0000
Irit Katriel <iritk...@googlemail.com> wrote:
[]
> > you gentlemen come up with the "ultimate" way to deal with multiple
> > errors,
>
> I've been mistaken for a man before, but no-one has ever confused me
> for gentle. I'll take that as a compliment.
Sorry, was just a figure of speech ;-).
[]
> Pluggable is not without its problems. I'm all in favor of you
> developing this idea and proposing an alternative.
> As I said before, you just need to answer two questions:
> 1. show a limitation of our approach (the contrived flat-set is not
> one - see below)
> 2. describe how a pluggable approach works for that case
As I mentioned, I wanted to share concern about adding complex,
arguably special-purpose semantics to the core language, and back that
by an example on how to address some (not all) points of PEP 645 in a
more lightweight (on the language more manner).
It's hard to show a technical limitation of your approach - because
again, it's already complex and special-purpose, and I can't say I've
faced cases where regular application of it would be guaranteed and
omissions would be visible.
I vice versa can point to a limitation in "generic exception
filters" approach - it handles "how to match exceptions more flexibly"
issue, but doesn't cover "how to handle automagic reraising of
multiple (unhandled) exceptions" part. In my list, that's good - in
normal exception handling, exception are (re)raised explicitly. It's
pretty big conceptual jump to make handling of that implicit (and
exception handling in general is condemned as too-implicit control flow
construct by some parties).
Overall, my idea wasn't to come up with an alternative proposal so late
in PEP654 lifecycle, nor even delay its submission to SC. I just
decided to sounds these concern in the end (had them in my notes since
initial posting), and ask for clarifications like if there's a
cross-language prior art regarding it.
[]
> > > > def chained(e, excs):
> > > > while e:
> > > > if isinstance(e, excs):
> > > > return e
> > > > e = e.__cause__ # Simplified, should consider __context__
concerned only with catching OSError anywhere in the exception chain.
As mentioned in the other comment, if there's a need to reraise (parts)
of original exception, in must be stored as an attribute of an
exception returned from filter. All that's pretty clear and explicit,
though I agree if it (reraising) needs to be done often, it may be
cumbersome.
If it needs to be done. We here touch the question of how exactly such
(multiple) exceptions would be handled. And PEP654 is pretty bare on
that too - it offers machinery to make that possible, but barely talks
of actual usecases (where it starts talk about that is in "Rejected
ideas" section, not in the main narrative).
> > My understanding is that ExceptionGroup is carefully designed to
> > allow subclassing it and override behavior, and the above should be
> > possible by subclassing.
>
> Indeed:
>
> class PaulsExceptionGroup(ExceptionGroup):
Thanks for confirming.
[]
> So your concern with our design is that ExceptionGroup implements a
> generic split() that handles tracebacks and nested structure
> correctly, and you might not need that because maybe you don't nest
> or you don't care about tracebacks?
My concern is still that it adds quite a big chunk of complexity to the
core semantics of language. PEP654's ExceptionGroup is clearly
carefully designed to (try) to be general, there's no questions about
that.
> If that is the case then I think you are confusing "generic" with
> "special-purpose". ExceptionGroup is generic, it works for general
> nested groups with arbitrary tracebacks and cause/context. That's
> the opposite of special-purpose.
Sadly, to me it still looks that all its generic complexity is rooted
in Trio and friends, and for other usecases simpler approach would
suffice. Again, that's just personal opinion. I'm just worried to see
"except*" in code where it's really needed (and be able to easily grasp
what it does), and dread seeing it where it's not really needed (in
that regard, PEP654 tries to delineate it clearly from the rest of
Python, thanks).
> Irit
--
Best regards,
Paul mailto:pmi...@gmail.com
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/IMLSCMN74RJBFYGATQS7PK3GKBR6MLWG/
On Sat, 27 Mar 2021 20:01:27 +0000
Irit Katriel <iritk...@googlemail.com> wrote:
[]
> > you gentlemen come up with the "ultimate" way to deal with multiple
> > errors,
>
> I've been mistaken for a man before, but no-one has ever confused me
> for gentle. I'll take that as a compliment.
[]
> Pluggable is not without its problems. I'm all in favor of you
> developing this idea and proposing an alternative.
> As I said before, you just need to answer two questions:
> 1. show a limitation of our approach (the contrived flat-set is not
> one - see below)
> 2. describe how a pluggable approach works for that case
arguably special-purpose semantics to the core language, and back that
by an example on how to address some (not all) points of PEP 645 in a
more lightweight (on the language more manner).
It's hard to show a technical limitation of your approach - because
again, it's already complex and special-purpose, and I can't say I've
faced cases where regular application of it would be guaranteed and
omissions would be visible.
I vice versa can point to a limitation in "generic exception
filters" approach - it handles "how to match exceptions more flexibly"
issue, but doesn't cover "how to handle automagic reraising of
multiple (unhandled) exceptions" part. In my list, that's good - in
normal exception handling, exception are (re)raised explicitly. It's
pretty big conceptual jump to make handling of that implicit (and
exception handling in general is condemned as too-implicit control flow
construct by some parties).
Overall, my idea wasn't to come up with an alternative proposal so late
in PEP654 lifecycle, nor even delay its submission to SC. I just
decided to sounds these concern in the end (had them in my notes since
initial posting), and ask for clarifications like if there's a
cross-language prior art regarding it.
[]
> > > > def chained(e, excs):
> > > > while e:
> > > > if isinstance(e, excs):
> > > > return e
> > > > e = e.__cause__ # Simplified, should consider __context__
> It's different because your "match" function returns a single
> exception (the first one that is of OSError type). Any further
> OSErrors will be reraised. The PEP's except* knows how to match
> multiple exceptions of the relevant type.
For this case, it was "chained" function (exception filter), which was
> exception (the first one that is of OSError type). Any further
> OSErrors will be reraised. The PEP's except* knows how to match
> multiple exceptions of the relevant type.
concerned only with catching OSError anywhere in the exception chain.
As mentioned in the other comment, if there's a need to reraise (parts)
of original exception, in must be stored as an attribute of an
exception returned from filter. All that's pretty clear and explicit,
though I agree if it (reraising) needs to be done often, it may be
cumbersome.
If it needs to be done. We here touch the question of how exactly such
(multiple) exceptions would be handled. And PEP654 is pretty bare on
that too - it offers machinery to make that possible, but barely talks
of actual usecases (where it starts talk about that is in "Rejected
ideas" section, not in the main narrative).
> > My understanding is that ExceptionGroup is carefully designed to
> > allow subclassing it and override behavior, and the above should be
> > possible by subclassing.
>
> Indeed:
>
> class PaulsExceptionGroup(ExceptionGroup):
[]
> So your concern with our design is that ExceptionGroup implements a
> generic split() that handles tracebacks and nested structure
> correctly, and you might not need that because maybe you don't nest
> or you don't care about tracebacks?
core semantics of language. PEP654's ExceptionGroup is clearly
carefully designed to (try) to be general, there's no questions about
that.
> If that is the case then I think you are confusing "generic" with
> "special-purpose". ExceptionGroup is generic, it works for general
> nested groups with arbitrary tracebacks and cause/context. That's
> the opposite of special-purpose.
in Trio and friends, and for other usecases simpler approach would
suffice. Again, that's just personal opinion. I'm just worried to see
"except*" in code where it's really needed (and be able to easily grasp
what it does), and dread seeing it where it's not really needed (in
that regard, PEP654 tries to delineate it clearly from the rest of
Python, thanks).
> Irit
--
Best regards,
Paul mailto:pmi...@gmail.com
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Nathaniel Smith
Apr 5, 2021, 6:03:21 AM4/5/21
to Irit Katriel, python-dev, Yury Selivanov
OK, better late than never... here's a much-delayed review of the PEP. Thank you Irit and Guido for carrying this forward while I've been AWOL! It's fantastic to see my old design sketches turned into something like, actually real.
== Overall feelings ==
Honestly, I have somewhat mixed feelings ExceptionGroups. I don't see any way around adding ExceptionGroups in some form, because it's just a fact of life that in a concurrent program, multiple things can go wrong at once, and we want Python to be usable for writing concurrent programs. Right now the state of the art is "exceptions in background threads/tasks get dropped on the floor", and almost anything is better than that. The current PEP is definitely better than that. But at the same time, there are a lot of compromises needed to retrofit this onto Python's existing system, and the current proposal feels like a bunch of awkward hacks with hacks on top. That's largely my fault for not thinking of something better, and maybe there is nothing better. But I still wish we could come up with something more elegant, and I do see why this proposal has made people uncomfortable. For example:
- I'm uncomfortable with how in some contexts we treat EG's as placeholders for the contained exceptions, and other places we treat them like a single first-class exceptions. (Witness all the feedback about "why not just catch the ExceptionGroup and handle it by hand?", and imagine writing the docs explaining all the situations where that is or isn't a good idea and the pitfalls involved...) If we could somehow pick one and stick to it then I think that would make it easier for users to grasp. (My gut feeling is that making them pure containers is better, which to me is why it makes sense for them to be @final and why I keep hoping we can figure out some better way for plain 'except' and EGs to interact.)
- If a function wants to start using concurrency internally, then now *all* its exceptions have to get wrapped in EGs and callers have to change *all* their exception handling code to use except* or similar. You would think this was an internal implementation detail that the caller shouldn't have to care about, but instead it forces a major change on the function's public API. And this is because regular 'except' can't do anything useful with EGs.
- We have a special-case hack to keep 'except Exception' working, but it has tricky edge cases (Exceptions can still sneak past if they're paired up with a BaseException), and it really is specific to 'except Exception'; it doesn't work for any other 'except SomeError' code. This smells funny.
Anyway, that's just abstract context to give an idea where I'm coming from. Maybe we just have to accept these trade-offs, but if anyone has any ideas, speak up...
== Most important comment ==
Flat ExceptionGroups: there were two basic design approaches we discussed last year, which I'll call "flat" vs "nested". The current PEP uses the nested design, where ExceptionGroups form a tree, and traceback information is distributed in pieces over this tree. This is the source of a lot of the complexity in the current PEP: for example, it's why EG's don't have one obvious iteration semantics, and it's why once an exception is wrapped in an EG, it can never be unwrapped again (because it would lose traceback information).
The idea of the "flat" design is to instead store all the traceback info directly on the leaf exceptions, so the EG itself can be just a pure container holding a list of exceptions, that's it, with no nesting. The downside is that it requires changes to the interpreter's code for updating __traceback__ attributes, which is currently hard-coded to only update one __traceback__ at a time.
For a third-party library like Trio, changing the interpreter is obviously impossible, so we never considered it seriously. But in a PEP, changing the interpreter is possible. And now I'm worried that we ruled out a better solution early on for reasons that no longer apply. The more I think about it, the more I suspect that flat EGs would end up being substantially simpler all around? So I think we should at least think through what that would look like (and Irit, I'd love your thoughts here now that you're the expert on the CPython details!), and document an explicit decision one way or another. (Maybe we should do a call or something to go over the details? I'm trying to keep this email from ballooning out of control...)
== Smaller points ==
- In my original proposal, EGs didn't just hold a list of exceptions, but also a list of "origins" for each exception. The idea being that if, say, you attempt to connect to a host with an IPv4 address and an IPv6 address, and they raised two different OSErrors that got bundled together into one EG, then it would be nice to know which OSError came from which attempt. Or in asyncio/trio it would be nice if tracebacks could show which task each exception came from. It seems like this got dropped at some point?
On further consideration, I think this might be better handled as a special kind of traceback entry that we can attach to each exception, that just holds some arbitrary text that's inserted into the traceback at the appropriate place? But either way, I think it would be good to be able to attach this kind of information to tracebacks somehow.
- Recording pre-empted exceptions: This is another type of metadata that would be useful to print along with the traceback. It's non-obvious and a bit hard to explain, but multiple trio users have complained about this, so I assume it will bite asyncio users too as soon as TaskGroups are added. The situation is, you have a parent task P and two child tasks C1 and C2:
P
/ \
C1 C2
C1 terminates with an unhandled exception E1, so in order to continue unwinding, the nursery/taskgroup in P cancels C2. But, C2 was itself in the middle of unwinding another, different exception E2 (so e.g. the cancellation arrived during a `finally` block). E2 gets replaced with a `Cancelled` exception whose __context__=E2, and that exception unwinds out of C2 and the nursery/taskgroup in P catches the `Cancelled` and discards it, then re-raises E1 so it can continue unwinding.
The problem here is that E2 gets "lost" -- there's no record of it in the final output. Basically E1 replaced it. And that can be bad: for example, if the two children are interacting with each other, then E2 might be the actual error that broke the program, and E1 is some exception complaining that the connection to C2 was lost. If you have two exceptions that are triggered from the same underlying event, it's a race which one survives.
This is conceptually similar to the way an exception in an 'except' block used to cause exceptions to be lost, so we added __context__ to avoid that. And just like for __context__, it would be nice if we could attach some info to E1 recording that E2 had happened and then got preempted. But I don't see how we can reuse __context__ itself for this, because it's a somewhat different relationship: __context__ means that an exception happened in the handler for another exception, while in this case you might have multiple preempted exceptions, and they're associated with particular points in the stack trace where the preemption occurred.
This is a complex issue and maybe we should call it out-of-scope for the first version of ExceptionGroups. But I mention it because it's a second place where adding some extra annotations to the traceback info would be useful, and maybe we can keep it simple by adding some minimal hooks in the core traceback machinery and let libraries like trio/asyncio handle the complicated parts?
- There are a number of places where the Python VM itself catches exceptions and has hard-coded handling for certain exception types. For example:
- Unhandled exceptions that reach the top of the main thread generally cause a traceback to be printed, but if the exception is SystemExit then the interpreter instead exits silently with status exc.args[0].
- 'for' loops call iter.__next__, and catch StopIteration while allowing other exceptions to escape.
- Generators catch StopIteration from their bodies and replace it with RuntimeError (PEP 479)
With this PEP, it's now possible for the main thread to terminate with ExceptionGroup(SystemExit), __next__ to raise ExceptionGroup(StopIteration), a generator to raise ExceptionGroup(StopIteration), either alone or mixed with other exceptions. How should the VM handle these new cases? Should they be using except* or except?
I don't think there's an obvious answer here, and possibly the answer is just "don't do that". But I feel like the PEP should say what the language semantics are in these cases, one way or another.
- traceback module API changes: The PEP notes that traceback.print_tb and traceback.print_exception will be updated to handle ExceptionGroups. The traceback module also has some special data structures for representing "pre-processed" stack traces, via the traceback.StackSummary type. This is used to capture tracebacks in a structured way but without holding onto the full frame objects. Maybe this API should also be extended somehow so it can also represent traceback trees?
== Overall feelings ==
Honestly, I have somewhat mixed feelings ExceptionGroups. I don't see any way around adding ExceptionGroups in some form, because it's just a fact of life that in a concurrent program, multiple things can go wrong at once, and we want Python to be usable for writing concurrent programs. Right now the state of the art is "exceptions in background threads/tasks get dropped on the floor", and almost anything is better than that. The current PEP is definitely better than that. But at the same time, there are a lot of compromises needed to retrofit this onto Python's existing system, and the current proposal feels like a bunch of awkward hacks with hacks on top. That's largely my fault for not thinking of something better, and maybe there is nothing better. But I still wish we could come up with something more elegant, and I do see why this proposal has made people uncomfortable. For example:
- I'm uncomfortable with how in some contexts we treat EG's as placeholders for the contained exceptions, and other places we treat them like a single first-class exceptions. (Witness all the feedback about "why not just catch the ExceptionGroup and handle it by hand?", and imagine writing the docs explaining all the situations where that is or isn't a good idea and the pitfalls involved...) If we could somehow pick one and stick to it then I think that would make it easier for users to grasp. (My gut feeling is that making them pure containers is better, which to me is why it makes sense for them to be @final and why I keep hoping we can figure out some better way for plain 'except' and EGs to interact.)
- If a function wants to start using concurrency internally, then now *all* its exceptions have to get wrapped in EGs and callers have to change *all* their exception handling code to use except* or similar. You would think this was an internal implementation detail that the caller shouldn't have to care about, but instead it forces a major change on the function's public API. And this is because regular 'except' can't do anything useful with EGs.
- We have a special-case hack to keep 'except Exception' working, but it has tricky edge cases (Exceptions can still sneak past if they're paired up with a BaseException), and it really is specific to 'except Exception'; it doesn't work for any other 'except SomeError' code. This smells funny.
Anyway, that's just abstract context to give an idea where I'm coming from. Maybe we just have to accept these trade-offs, but if anyone has any ideas, speak up...
== Most important comment ==
Flat ExceptionGroups: there were two basic design approaches we discussed last year, which I'll call "flat" vs "nested". The current PEP uses the nested design, where ExceptionGroups form a tree, and traceback information is distributed in pieces over this tree. This is the source of a lot of the complexity in the current PEP: for example, it's why EG's don't have one obvious iteration semantics, and it's why once an exception is wrapped in an EG, it can never be unwrapped again (because it would lose traceback information).
The idea of the "flat" design is to instead store all the traceback info directly on the leaf exceptions, so the EG itself can be just a pure container holding a list of exceptions, that's it, with no nesting. The downside is that it requires changes to the interpreter's code for updating __traceback__ attributes, which is currently hard-coded to only update one __traceback__ at a time.
For a third-party library like Trio, changing the interpreter is obviously impossible, so we never considered it seriously. But in a PEP, changing the interpreter is possible. And now I'm worried that we ruled out a better solution early on for reasons that no longer apply. The more I think about it, the more I suspect that flat EGs would end up being substantially simpler all around? So I think we should at least think through what that would look like (and Irit, I'd love your thoughts here now that you're the expert on the CPython details!), and document an explicit decision one way or another. (Maybe we should do a call or something to go over the details? I'm trying to keep this email from ballooning out of control...)
== Smaller points ==
- In my original proposal, EGs didn't just hold a list of exceptions, but also a list of "origins" for each exception. The idea being that if, say, you attempt to connect to a host with an IPv4 address and an IPv6 address, and they raised two different OSErrors that got bundled together into one EG, then it would be nice to know which OSError came from which attempt. Or in asyncio/trio it would be nice if tracebacks could show which task each exception came from. It seems like this got dropped at some point?
On further consideration, I think this might be better handled as a special kind of traceback entry that we can attach to each exception, that just holds some arbitrary text that's inserted into the traceback at the appropriate place? But either way, I think it would be good to be able to attach this kind of information to tracebacks somehow.
- Recording pre-empted exceptions: This is another type of metadata that would be useful to print along with the traceback. It's non-obvious and a bit hard to explain, but multiple trio users have complained about this, so I assume it will bite asyncio users too as soon as TaskGroups are added. The situation is, you have a parent task P and two child tasks C1 and C2:
P
/ \
C1 C2
C1 terminates with an unhandled exception E1, so in order to continue unwinding, the nursery/taskgroup in P cancels C2. But, C2 was itself in the middle of unwinding another, different exception E2 (so e.g. the cancellation arrived during a `finally` block). E2 gets replaced with a `Cancelled` exception whose __context__=E2, and that exception unwinds out of C2 and the nursery/taskgroup in P catches the `Cancelled` and discards it, then re-raises E1 so it can continue unwinding.
The problem here is that E2 gets "lost" -- there's no record of it in the final output. Basically E1 replaced it. And that can be bad: for example, if the two children are interacting with each other, then E2 might be the actual error that broke the program, and E1 is some exception complaining that the connection to C2 was lost. If you have two exceptions that are triggered from the same underlying event, it's a race which one survives.
This is conceptually similar to the way an exception in an 'except' block used to cause exceptions to be lost, so we added __context__ to avoid that. And just like for __context__, it would be nice if we could attach some info to E1 recording that E2 had happened and then got preempted. But I don't see how we can reuse __context__ itself for this, because it's a somewhat different relationship: __context__ means that an exception happened in the handler for another exception, while in this case you might have multiple preempted exceptions, and they're associated with particular points in the stack trace where the preemption occurred.
This is a complex issue and maybe we should call it out-of-scope for the first version of ExceptionGroups. But I mention it because it's a second place where adding some extra annotations to the traceback info would be useful, and maybe we can keep it simple by adding some minimal hooks in the core traceback machinery and let libraries like trio/asyncio handle the complicated parts?
- There are a number of places where the Python VM itself catches exceptions and has hard-coded handling for certain exception types. For example:
- Unhandled exceptions that reach the top of the main thread generally cause a traceback to be printed, but if the exception is SystemExit then the interpreter instead exits silently with status exc.args[0].
- 'for' loops call iter.__next__, and catch StopIteration while allowing other exceptions to escape.
- Generators catch StopIteration from their bodies and replace it with RuntimeError (PEP 479)
With this PEP, it's now possible for the main thread to terminate with ExceptionGroup(SystemExit), __next__ to raise ExceptionGroup(StopIteration), a generator to raise ExceptionGroup(StopIteration), either alone or mixed with other exceptions. How should the VM handle these new cases? Should they be using except* or except?
I don't think there's an obvious answer here, and possibly the answer is just "don't do that". But I feel like the PEP should say what the language semantics are in these cases, one way or another.
- traceback module API changes: The PEP notes that traceback.print_tb and traceback.print_exception will be updated to handle ExceptionGroups. The traceback module also has some special data structures for representing "pre-processed" stack traces, via the traceback.StackSummary type. This is used to capture tracebacks in a structured way but without holding onto the full frame objects. Maybe this API should also be extended somehow so it can also represent traceback trees?
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/MQ2UCSQ2ZC4FIGT7KSVI6BJA4FCXSOCL/
Code of Conduct: http://python.org/psf/codeofconduct/
--
Nathaniel J. Smith -- https://vorpus.org
Chris Jerdonek
Apr 5, 2021, 10:00:31 AM4/5/21
to Nathaniel Smith, Irit Katriel, python-dev, Yury Selivanov
On Mon, Apr 5, 2021 at 3:07 AM Nathaniel Smith <n...@pobox.com> wrote:
- Recording pre-empted exceptions: This is another type of metadata that would be useful to print along with the traceback. It's non-obvious and a bit hard to explain, but multiple trio users have complained about this, so I assume it will bite asyncio users too as soon as TaskGroups are added. The situation is, you have a parent task P and two child tasks C1 and C2:
P
/ \
C1 C2
C1 terminates with an unhandled exception E1, so in order to continue unwinding, the nursery/taskgroup in P cancels C2. But, C2 was itself in the middle of unwinding another, different exception E2 (so e.g. the cancellation arrived during a `finally` block). E2 gets replaced with a `Cancelled` exception whose __context__=E2, and that exception unwinds out of C2 and the nursery/taskgroup in P catches the `Cancelled` and discards it, then re-raises E1 so it can continue unwinding.
The problem here is that E2 gets "lost" -- there's no record of it in the final output. Basically E1 replaced it. And that can be bad: for example, if the two children are interacting with each other, then E2 might be the actual error that broke the program, and E1 is some exception complaining that the connection to C2 was lost. If you have two exceptions that are triggered from the same underlying event, it's a race which one survives.
This point reminded me again of this issue in the tracker ("Problems with recursive automatic exception chaining" from 2013): https://bugs.python.org/issue18861
I'm not sure if it's exactly the same, but you can see that a couple of the later comments there talk about "exception trees" and other types of annotations.
If that issue were addressed after ExceptionGroups were introduced, does that mean there would then be two types of exception-related trees layered over each other (e.g. groups of trees, trees of groups, etc)? It makes me wonder if there's a more general tree structure that could accommodate both use cases...
--Chris
Irit Katriel via Python-Dev
Apr 5, 2021, 12:49:31 PM4/5/21
to Nathaniel Smith, python-dev, Yury Selivanov
Hi Nathaniel,
Thank you for your feedback. See a few comment below.
OK, better late than never... here's a much-delayed review of the PEP. Thank you Irit and Guido for carrying this forward while I've been AWOL! It's fantastic to see my old design sketches turned into something like, actually real.
== Overall feelings ==
Honestly, I have somewhat mixed feelings ExceptionGroups. I don't see any way around adding ExceptionGroups in some form, because it's just a fact of life that in a concurrent program, multiple things can go wrong at once, and we want Python to be usable for writing concurrent programs. Right now the state of the art is "exceptions in background threads/tasks get dropped on the floor", and almost anything is better than that. The current PEP is definitely better than that. But at the same time, there are a lot of compromises needed to retrofit this onto Python's existing system, and the current proposal feels like a bunch of awkward hacks with hacks on top. That's largely my fault for not thinking of something better, and maybe there is nothing better. But I still wish we could come up with something more elegant, and I do see why this proposal has made people uncomfortable.
People were uncomfortable about three main points:
1. In the first draft "except Exception" would not catch exception groups, this was changed in the second draft.
2. In the first draft exception groups were not subclassable, this also changed in the second draft.
3. Do we really need except*? Why is ExceptionGroup not enough? I think we replied, but this question may still be on some people's minds and if so we should continue discussing it.
Was there anything else?
For example:
- I'm uncomfortable with how in some contexts we treat EG's as placeholders for the contained exceptions, and other places we treat them like a single first-class exceptions. (Witness all the feedback about "why not just catch the ExceptionGroup and handle it by hand?", and imagine writing the docs explaining all the situations where that is or isn't a good idea and the pitfalls involved...) If we could somehow pick one and stick to it then I think that would make it easier for users to grasp. (My gut feeling is that making them pure containers is better, which to me is why it makes sense for them to be @final and why I keep hoping we can figure out some better way for plain 'except' and EGs to interact.)
I'm not sure what you mean by "a placeholder". An exception group is an exception, and except* is a new syntax that helps you manipulate exception groups. I don't think the confusion you mention was due to the fact that except can catch EGs, but rather due to the fact that the simplest examples don't show you why except is not good enough, and why we need except*.
Suppose we did as you suggest and made exception group a container which is not exception. Now suppose that an exception is raised in an except* block. What is the context of this exception? Do we change the requirement that an exception's context is always an exception?
How do you raise an exception group if it's not an Exception? Do we go back to allowing random objects being raised?
- If a function wants to start using concurrency internally, then now *all* its exceptions have to get wrapped in EGs and callers have to change *all* their exception handling code to use except* or similar. You would think this was an internal implementation detail that the caller shouldn't have to care about, but instead it forces a major change on the function's public API. And this is because regular 'except' can't do anything useful with EGs.
I don't understand this point. If you are using concurrency internally and don't want to raise EGs externally, then surely you will catch EGs, select one of the exceptions to raise and throw away all the others like in the old days when there weren't EGs (which is presumably what your caller is expecting). In other words, if you make drastic changes in a function's implementation, it is your responsibility to ensure that the old API is preserved.
- We have a special-case hack to keep 'except Exception' working, but it has tricky edge cases (Exceptions can still sneak past if they're paired up with a BaseException), and it really is specific to 'except Exception'; it doesn't work for any other 'except SomeError' code. This smells funny.
I agree with this point, my personal preference would be to have ExceptionGroup(BaseExceptionGroup) as in the first draft, with a runaway exception group being something you should fix. The current choice is a compromise for backwards compatibility, not so much with the language but with the practice of using "except Exception" to "catch almost everything, log it and move on". I don't think we can go against that at this point.
Anyway, that's just abstract context to give an idea where I'm coming from. Maybe we just have to accept these trade-offs, but if anyone has any ideas, speak up...
== Most important comment ==
Flat ExceptionGroups: there were two basic design approaches we discussed last year, which I'll call "flat" vs "nested". The current PEP uses the nested design, where ExceptionGroups form a tree, and traceback information is distributed in pieces over this tree. This is the source of a lot of the complexity in the current PEP: for example, it's why EG's don't have one obvious iteration semantics, and it's why once an exception is wrapped in an EG, it can never be unwrapped again (because it would lose traceback information).
The idea of the "flat" design is to instead store all the traceback info directly on the leaf exceptions, so the EG itself can be just a pure container holding a list of exceptions, that's it, with no nesting. The downside is that it requires changes to the interpreter's code for updating __traceback__ attributes, which is currently hard-coded to only update one __traceback__ at a time.
For a third-party library like Trio, changing the interpreter is obviously impossible, so we never considered it seriously. But in a PEP, changing the interpreter is possible. And now I'm worried that we ruled out a better solution early on for reasons that no longer apply. The more I think about it, the more I suspect that flat EGs would end up being substantially simpler all around? So I think we should at least think through what that would look like (and Irit, I'd love your thoughts here now that you're the expert on the CPython details!), and document an explicit decision one way or another. (Maybe we should do a call or something to go over the details? I'm trying to keep this email from ballooning out of control...)
A Flat EG is in essence a "denormalized EG" - the tracebacks don't share parts, each leaf exception has its own linked list of frames as its traceback. It is easy enough to write a denormalize() function in traceback.py that constructs this from the current EG structure, if you need it
(use the leaf_generator from the PEP). I'm not sure I see why we should trouble the interpreter with this.
For display purposes, it is probably nicer to look at a normalized traceback where common parts are not repeated.
== Smaller points ==
- In my original proposal, EGs didn't just hold a list of exceptions, but also a list of "origins" for each exception. The idea being that if, say, you attempt to connect to a host with an IPv4 address and an IPv6 address, and they raised two different OSErrors that got bundled together into one EG, then it would be nice to know which OSError came from which attempt. Or in asyncio/trio it would be nice if tracebacks could show which task each exception came from. It seems like this got dropped at some point?
I don't remember seeing this, so it wasn't deliberately dropped. It's not that we copied MultiError, or your sketches for MultiError2, and then modified them into what is now in the PEP. We learned from your experience, particularly the MutliError2 document which spells out the aspects of MultiError that didn't work well. But we designed ExceptionGroup pretty much from first principles
(and then redesigned it several times as we learned our own lessons)
. So you shouldn't assume that anything we didn't take from MultiError or the MultiError2 sketch was deliberately dropped. We may have just not paid attention to all of the details.
On further consideration, I think this might be better handled as a special kind of traceback entry that we can attach to each exception, that just holds some arbitrary text that's inserted into the traceback at the appropriate place? But either way, I think it would be good to be able to attach this kind of information to tracebacks somehow.
It sounds like you want some way to enrich exceptions. This should be optional (we wouldn't want EG to require additional metadata for exceptions) so yeah, I agree it should sit on the leaf exception and not on the group. In that sense it's orthogonal to this PEP.
- Recording pre-empted exceptions: This is another type of metadata that would be useful to print along with the traceback. It's non-obvious and a bit hard to explain, but multiple trio users have complained about this, so I assume it will bite asyncio users too as soon as TaskGroups are added. The situation is, you have a parent task P and two child tasks C1 and C2:
P
/ \
C1 C2
C1 terminates with an unhandled exception E1, so in order to continue unwinding, the nursery/taskgroup in P cancels C2. But, C2 was itself in the middle of unwinding another, different exception E2 (so e.g. the cancellation arrived during a `finally` block). E2 gets replaced with a `Cancelled` exception whose __context__=E2, and that exception unwinds out of C2 and the nursery/taskgroup in P catches the `Cancelled` and discards it, then re-raises E1 so it can continue unwinding.
The problem here is that E2 gets "lost" -- there's no record of it in the final output. Basically E1 replaced it. And that can be bad: for example, if the two children are interacting with each other, then E2 might be the actual error that broke the program, and E1 is some exception complaining that the connection to C2 was lost. If you have two exceptions that are triggered from the same underlying event, it's a race which one survives.
This is conceptually similar to the way an exception in an 'except' block used to cause exceptions to be lost, so we added __context__ to avoid that. And just like for __context__, it would be nice if we could attach some info to E1 recording that E2 had happened and then got preempted. But I don't see how we can reuse __context__ itself for this, because it's a somewhat different relationship: __context__ means that an exception happened in the handler for another exception, while in this case you might have multiple preempted exceptions, and they're associated with particular points in the stack trace where the preemption occurred.
This is a complex issue and maybe we should call it out-of-scope for the first version of ExceptionGroups. But I mention it because it's a second place where adding some extra annotations to the traceback info would be useful, and maybe we can keep it simple by adding some minimal hooks in the core traceback machinery and let libraries like trio/asyncio handle the complicated parts?
Isn't this something that Trio/asyncio should handle, as in: if you catch Cancelled that has a __context__ then you need to put that __context__ into the exception group you are raising so that it's not lost?
- There are a number of places where the Python VM itself catches exceptions and has hard-coded handling for certain exception types. For example:
- Unhandled exceptions that reach the top of the main thread generally cause a traceback to be printed, but if the exception is SystemExit then the interpreter instead exits silently with status exc.args[0].
- 'for' loops call iter.__next__, and catch StopIteration while allowing other exceptions to escape.
- Generators catch StopIteration from their bodies and replace it with RuntimeError (PEP 479)
With this PEP, it's now possible for the main thread to terminate with ExceptionGroup(SystemExit), __next__ to raise ExceptionGroup(StopIteration), a generator to raise ExceptionGroup(StopIteration), either alone or mixed with other exceptions. How should the VM handle these new cases? Should they be using except* or except?
I don't think there's an obvious answer here, and possibly the answer is just "don't do that". But I feel like the PEP should say what the language semantics are in these cases, one way or another.
I agree that there are some edge cases in the interpreter that we will need to make a decision about. I would imagine an ExceptionGroup(StopIteration) is a bug of some sort, so we would probably want to not handle that as a StopIteration - just let the program fail and make the error obvious. ExceptionGroup(SystemExit) -- I don't have an opinion, I think it's the application's responsibility to unpack it, but we can bikeshed that.
- traceback module API changes: The PEP notes that traceback.print_tb and traceback.print_exception will be updated to handle ExceptionGroups. The traceback module also has some special data structures for representing "pre-processed" stack traces, via the traceback.StackSummary type. This is used to capture tracebacks in a structured way but without holding onto the full frame objects. Maybe this API should also be extended somehow so it can also represent traceback trees?
print_tb and print_exception are thin wrappers around these data structures, so this is implied but we can make it explicit in the PEP.
Thanks again
Irit
Yury Selivanov
Apr 5, 2021, 1:05:35 PM4/5/21
to Irit Katriel, python-dev, Yury Selivanov
Just wanted to elaborate a little bit on StopIteration to add to Irit's reply:
There's no need to do anything about ExceptionGroups potentially
wrapping StopIteration or StopAsyncIteration exceptions. The point of
PEP 479 was to solve a problem of one of nested frames raising a
StopIteration (often by mistake) and the outer generator being
stopped. That lead to some really tricky situations to debug. In our
case, a rogue StopIteration wrapped in an EG would not stop a
generator silently, it would do that loud and clear.
As for SystemExit, we'll change "asyncio.run()" to unpack SystemExit
and propagate them unwrapped, potentially allowing to dump the
exception tree into a log file for later debug, if configured. Trio
should do the same. Problem solved.
It's important to understand that the PEP doesn't propose a magical
mechanism to turn all exceptions into EGs automatically, it's up to
the framework/user code how to build them and what to propagate out.
In Python 3.9 you can just as well write `except BaseException: pass`
and silence a SystemExit (and people do that from time to time!)
Yury
wrapping StopIteration or StopAsyncIteration exceptions. The point of
PEP 479 was to solve a problem of one of nested frames raising a
StopIteration (often by mistake) and the outer generator being
stopped. That lead to some really tricky situations to debug. In our
case, a rogue StopIteration wrapped in an EG would not stop a
generator silently, it would do that loud and clear.
As for SystemExit, we'll change "asyncio.run()" to unpack SystemExit
and propagate them unwrapped, potentially allowing to dump the
exception tree into a log file for later debug, if configured. Trio
should do the same. Problem solved.
It's important to understand that the PEP doesn't propose a magical
mechanism to turn all exceptions into EGs automatically, it's up to
the framework/user code how to build them and what to propagate out.
In Python 3.9 you can just as well write `except BaseException: pass`
and silence a SystemExit (and people do that from time to time!)
Yury
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/E5RMN6CSKXSIFXBKRIHYLPK2FRXIM4AE/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Irit Katriel via Python-Dev
Apr 7, 2021, 2:20:01 PM4/7/21
to Chris Jerdonek, python-dev, Yury Selivanov
On Mon, Apr 5, 2021 at 2:59 PM Chris Jerdonek <chris.j...@gmail.com> wrote:
This point reminded me again of this issue in the tracker ("Problems with recursive automatic exception chaining" from 2013): https://bugs.python.org/issue18861I'm not sure if it's exactly the same, but you can see that a couple of the later comments there talk about "exception trees" and other types of annotations.If that issue were addressed after ExceptionGroups were introduced, does that mean there would then be two types of exception-related trees layered over each other (e.g. groups of trees, trees of groups, etc)? It makes me wonder if there's a more general tree structure that could accommodate both use cases...--Chris
Interesting, I commented on that issue - I think we may be able to solve it without adding more trees.
That said, we will have groups-of-trees/trees-of-groups. Already today, an exception plus its chained __cause__s and __context__s is the root of a binary tree of exceptions. The nodes of this tree represent the times that the exceptions were caught.
An exception group is a tree where the nodes represent the times when exceptions were grouped together and raised.
Irit
Thomas Wouters
Apr 15, 2021, 1:51:40 PM4/15/21
to Irit Katriel, Guido van Rossum, Yury Selivanov, python-dev
Thanks for submitting PEP 654 (Exception Groups and except *). The Steering Council isn’t done debating this PEP, but we wanted to give an update so everyone knows what to expect.
We are mostly in favour of the PEP, but we’re worried about the timing -- not just because there is still some discussion going on, but also the time available for us to discuss the proposal, and to validate the new API once it’s accepted. We are rapidly approaching feature freeze for Python 3.10, and while we’re not particularly worried about the implementation of this PEP, we would like to see actual usage of the new API with enough time to tweak it before the official release. That is to say, we would like to see the "analysis of how exception groups will likely be used in asyncio programs" turn into "here is how exception groups are used in asyncio programs" before the final release, like in the asyncio module and the standard library (for example, with TaskGroups). There’s also a presentation scheduled in the PyCon US Language Summit, which may lead to more discussion of the PEP and perhaps evolution of its ideas.
What would you and others feel about postponing the PEP until Python 3.11, getting it in as soon as the branch opens, so that potential users of the API -- asyncio or otherwise -- can provide us with practical experience to validate some of the assumptions made in the proposal, and to discover any warts or pitfalls we missed or misjudged? I don’t mean that as a slight on the PEP, which is thoughtful, thorough and excellently written, but as a practical observation of the complexity involved. Most of the PEP is more of a building block for other APIs than something many users will be directly exposed to, and we want to make sure the end result isn’t affected by unanticipated effects of these design decisions. (This would potentially also give third-party users more time to evaluate the API, although we know adoption of alpha releases isn’t big enough to make this a very strong argument for waiting.)
Like I said, we’re still discussing the PEP, so this hasn’t been decided. If you or others feel strongly that the benefit of postponing is too small (or the cost too high), we’ll take that into consideration. The SC has been somewhat busy of late, and we don’t want to let our schedule be the deciding factor here, but this may come down to there just not being enough time before the feature freeze to make a thoughtful decision.
For the SC,
Thomas.
-- Thomas Wouters <tho...@python.org>
Hi! I'm an email virus! Think twice before sending your email to help me spread!
Hi! I'm an email virus! Think twice before sending your email to help me spread!
Irit Katriel via Python-Dev
Apr 15, 2021, 8:03:05 PM4/15/21
to Thomas Wouters, Yury Selivanov, python-dev
Thomas and the rest of the SC,
Thank you for updating us and for your kind words about the PEP.
We agree that it is safer to include this in 3.11 and not rush it into 3.10. As you say, having time to develop the integration with asyncio before finalizing the API will give us more confidence in the design.
We would like to commit the code into the 3.11 development branch soon, as you suggest. We will be looking forward to the (provisional?) acceptance of the PEP by the SC so that we can do this.
Best regards
Irit, Yury & Guido
Nathaniel Smith
Apr 19, 2021, 9:51:31 PM4/19/21
to Irit Katriel, python-dev, Yury Selivanov
On Mon, Apr 5, 2021 at 9:48 AM Irit Katriel <iritk...@googlemail.com> wrote:
> On Mon, Apr 5, 2021 at 11:01 AM Nathaniel Smith <n...@pobox.com> wrote:
>> - I'm uncomfortable with how in some contexts we treat EG's as placeholders for the contained exceptions, and other places we treat them like a single first-class exceptions. (Witness all the feedback about "why not just catch the ExceptionGroup and handle it by hand?", and imagine writing the docs explaining all the situations where that is or isn't a good idea and the pitfalls involved...) If we could somehow pick one and stick to it then I think that would make it easier for users to grasp. (My gut feeling is that making them pure containers is better, which to me is why it makes sense for them to be @final and why I keep hoping we can figure out some better way for plain 'except' and EGs to interact.)
>
>
> I'm not sure what you mean by "a placeholder". An exception group is an exception, and except* is a new syntax that helps you manipulate exception groups. I don't think the confusion you mention was due to the fact that except can catch EGs, but rather due to the fact that the simplest examples don't show you why except is not good enough, and why we need except*.
>
> Suppose we did as you suggest and made exception group a container which is not exception. Now suppose that an exception is raised in an except* block. What is the context of this exception? Do we change the requirement that an exception's context is always an exception?
>
> How do you raise an exception group if it's not an Exception? Do we go back to allowing random objects being raised?
Ah, right. I'm talking about the conceptual/semantic level, not the
> On Mon, Apr 5, 2021 at 11:01 AM Nathaniel Smith <n...@pobox.com> wrote:
>> - I'm uncomfortable with how in some contexts we treat EG's as placeholders for the contained exceptions, and other places we treat them like a single first-class exceptions. (Witness all the feedback about "why not just catch the ExceptionGroup and handle it by hand?", and imagine writing the docs explaining all the situations where that is or isn't a good idea and the pitfalls involved...) If we could somehow pick one and stick to it then I think that would make it easier for users to grasp. (My gut feeling is that making them pure containers is better, which to me is why it makes sense for them to be @final and why I keep hoping we can figure out some better way for plain 'except' and EGs to interact.)
>
>
> I'm not sure what you mean by "a placeholder". An exception group is an exception, and except* is a new syntax that helps you manipulate exception groups. I don't think the confusion you mention was due to the fact that except can catch EGs, but rather due to the fact that the simplest examples don't show you why except is not good enough, and why we need except*.
>
> Suppose we did as you suggest and made exception group a container which is not exception. Now suppose that an exception is raised in an except* block. What is the context of this exception? Do we change the requirement that an exception's context is always an exception?
>
> How do you raise an exception group if it's not an Exception? Do we go back to allowing random objects being raised?
implementation. For the implementation, backcompat certainly means we
need some single object that can represent the whole group of
exceptions, so that it can be returned by sys.exc_info(), C code can
access it through the tstate, etc. But the more important question is
how we explain this thing to users, and make it easy for them to do
the right thing. Is the EG the exception they're trying to catch or
otherwise work with? or is the EG just a detail that most users should
ignore because they're focused on the leaf exceptions inside?
For the concurrency use case (asyncio/trio/etc.), it's all about the
leaf exceptions. ExceptionGroup([SomeError()]) and SomeError() both
mean exactly the same thing, and treating them differently is
basically always a bug. The ExceptionGroup object doesn't tell you
anything about what went wrong, like normal exceptions do. It just
tells you that there was some frame in between where the exception was
raised and where it was caught where some other exception *could* have
happened, but didn't. You *can* give 'except ExceptionGroup' a
meaning, but that meaning doesn't make sense -- it's like saying "I
want to catch any exceptions that was raised by code that *could* have
raised a different exception instead" or "I want to catch all
exceptions whose traceback contains an entry that's a certain subclass
of TracebackType". Similarly, it doesn't make sense to attach error
strings to an EG, or define custom subclasses, etc.. Ideally, plain
'except' would (somehow) handle 'EG([SomeError()])' and 'SomeError()'
in exactly the same way; and if it doesn't, then using plain 'except'
in concurrent code is going to usually be a bug.
For other use cases, it does make sense to think of EG as a regular
exception. Like, if Hypothesis wants to report that it ran some tests
and there were failures, then modelling that as a single
HypothesisError seems like a nice API. You never need 'except*'
semantics to catch part of a HypothesisError. 'except HypothesisError'
is totally sensible. The only real value you get from the PEP is that
if you *don't* catch the HypothesisError, then the default traceback
machinery can now automatically include info about the individual test
failures.
If we were starting from scratch, I don't think this would be a big
conflict. I think the PEP would focus 100% on the concurrency case.
That's the one that's really difficult to solve without language
support. And, if you solve that, then there's a very natural way to
handle the Hypothesis case too: do 'raise HypothesisError from
EG(individual failures)'. That makes it explicit that HypothesisError
is a single exception that should be handled as a unit, while still
letting you introspect the individual exceptions and including them in
the default traceback output. (In fact Trio uses this trick right now,
in it's TCP connection code [1]. We report a single
OSError("connection failed"), but then include all the details about
each individual failure in its __cause__.)
The actual problem is that we're not starting from scratch; we're
trying to retrofit the concurrency-style semantics onto a system
that's currently completely focused around the Hypothesis-style
semantics. So the PEP ends up with a kind of mish-mash of the two
approaches -- 'except*' gives you the semantics that concurrency use
cases want, 'except' gives you the semantics that Hypothesis-style use
cases want, and users are forced to understand all these subtleties
and figure out which one is appropriate on a case-by-case basis. It's
not very "one obvious way to do it".
And again -- I'm *not* saying the PEP sucks or anything like that.
Backcompat is a real problem, and the PEP might already be the best it
can be given the tradeoffs. I'm just explaining why I want to make
sure we consider all the alternatives.
[1] https://github.com/python-trio/trio/blob/cb06242a78d850baaba4057f7dbd22d2a21561f7/trio/_highlevel_open_tcp_stream.py#L361-L367
>> - If a function wants to start using concurrency internally, then now *all* its exceptions have to get wrapped in EGs and callers have to change *all* their exception handling code to use except* or similar. You would think this was an internal implementation detail that the caller shouldn't have to care about, but instead it forces a major change on the function's public API. And this is because regular 'except' can't do anything useful with EGs.
>
> I don't understand this point. If you are using concurrency internally and don't want to raise EGs externally, then surely you will catch EGs, select one of the exceptions to raise and throw away all the others like in the old days when there weren't EGs (which is presumably what your caller is expecting). In other words, if you make drastic changes in a function's implementation, it is your responsibility to ensure that the old API is preserved.
internally so multiple things *could* go wrong at once, only one thing
actually *does* go wrong. So it's unfortunate if some existing code is
prepared for a specific exception that it knows can be raised, that
exact exception is raised... and the existing code fails to catch it
because now it's wrapped in an EG.
>> == Most important comment ==
>>
>> Flat ExceptionGroups: there were two basic design approaches we discussed last year, which I'll call "flat" vs "nested". The current PEP uses the nested design, where ExceptionGroups form a tree, and traceback information is distributed in pieces over this tree. This is the source of a lot of the complexity in the current PEP: for example, it's why EG's don't have one obvious iteration semantics, and it's why once an exception is wrapped in an EG, it can never be unwrapped again (because it would lose traceback information).
>>
>> The idea of the "flat" design is to instead store all the traceback info directly on the leaf exceptions, so the EG itself can be just a pure container holding a list of exceptions, that's it, with no nesting. The downside is that it requires changes to the interpreter's code for updating __traceback__ attributes, which is currently hard-coded to only update one __traceback__ at a time.
>>
>> For a third-party library like Trio, changing the interpreter is obviously impossible, so we never considered it seriously. But in a PEP, changing the interpreter is possible. And now I'm worried that we ruled out a better solution early on for reasons that no longer apply. The more I think about it, the more I suspect that flat EGs would end up being substantially simpler all around? So I think we should at least think through what that would look like (and Irit, I'd love your thoughts here now that you're the expert on the CPython details!), and document an explicit decision one way or another. (Maybe we should do a call or something to go over the details? I'm trying to keep this email from ballooning out of control...)
>
> A Flat EG is in essence a "denormalized EG" - the tracebacks don't share parts, each leaf exception has its own linked list of frames as its traceback.
> It is easy enough to write a denormalize() function in traceback.py that constructs this from the current EG structure, if you need it (use the leaf_generator from the PEP). I'm not sure I see why we should trouble the interpreter with this.
can never be unwrapped, because its traceback information is spread
across the individual exception + the EG tree around it. This is
confusing to users ("how do I check errno?"), and makes the design
more complicated (the need for topology-preserving .split(), the
inability to define a sensible EG.__iter__, ...). The advantage of
making the denormalized form the native form is that now the leaf
exceptions would be self-contained objects like they are now, so you
don't need EG nesting at all, and users can write intuitive code like:
except OSError as *excs:
remainder = [exc for exc in excs if exc.errno != ...]
if remainder:
raise ExceptionGroup(remainder)
> For display purposes, it is probably nicer to look at a normalized traceback where common parts are not repeated.
printing. But now it's only the display code that needs to care about
figuring out shared parts of the traceback, rather than something that
has to be maintained as an invariant everywhere.
>> - In my original proposal, EGs didn't just hold a list of exceptions, but also a list of "origins" for each exception. The idea being that if, say, you attempt to connect to a host with an IPv4 address and an IPv6 address, and they raised two different OSErrors that got bundled together into one EG, then it would be nice to know which OSError came from which attempt. Or in asyncio/trio it would be nice if tracebacks could show which task each exception came from. It seems like this got dropped at some point?
>
> I don't remember seeing this, so it wasn't deliberately dropped. It's not that we copied MultiError, or your sketches for MultiError2, and then modified them into what is now in the PEP. We learned from your experience, particularly the MutliError2 document which spells out the aspects of MultiError that didn't work well. But we designed ExceptionGroup pretty much from first principles (and then redesigned it several times as we learned our own lessons) . So you shouldn't assume that anything we didn't take from MultiError or the MultiError2 sketch was deliberately dropped. We may have just not paid attention to all of the details.
almost exactly the same design as MultiError v2 :-).
>> On further consideration, I think this might be better handled as a special kind of traceback entry that we can attach to each exception, that just holds some arbitrary text that's inserted into the traceback at the appropriate place? But either way, I think it would be good to be able to attach this kind of information to tracebacks somehow.
>
> It sounds like you want some way to enrich exceptions. This should be optional (we wouldn't want EG to require additional metadata for exceptions) so yeah, I agree it should sit on the leaf exception and not on the group. In that sense it's orthogonal to this PEP.
traceback, which are a thing that this PEP is creating :-). And it's
useful for every user of EGs, since by definition, an EG is
multiplexing exceptions from multiple sources, so it's nice to tell
the user which sources those are.
That said, you're right, if we want to handle this by defining a new
kind of traceback entry that code like Trio/asyncio/hypothesis can
manually attach to exceptions, then that could be written as a
separate-but-complementary PEP. In my original design, instead of
defining a new kind of traceback entry, I was storing this on the EG
itself, so that's why I was thinking about it needing to be part of
this PEP.
>> - Recording pre-empted exceptions: This is another type of metadata that would be useful to print along with the traceback. It's non-obvious and a bit hard to explain, but multiple trio users have complained about this, so I assume it will bite asyncio users too as soon as TaskGroups are added. The situation is, you have a parent task P and two child tasks C1 and C2:
>>
>> P
>> / \
>> C1 C2
>>
>> C1 terminates with an unhandled exception E1, so in order to continue unwinding, the nursery/taskgroup in P cancels C2. But, C2 was itself in the middle of unwinding another, different exception E2 (so e.g. the cancellation arrived during a `finally` block). E2 gets replaced with a `Cancelled` exception whose __context__=E2, and that exception unwinds out of C2 and the nursery/taskgroup in P catches the `Cancelled` and discards it, then re-raises E1 so it can continue unwinding.
>>
>> The problem here is that E2 gets "lost" -- there's no record of it in the final output. Basically E1 replaced it. And that can be bad: for example, if the two children are interacting with each other, then E2 might be the actual error that broke the program, and E1 is some exception complaining that the connection to C2 was lost. If you have two exceptions that are triggered from the same underlying event, it's a race which one survives.
>>
>> This is conceptually similar to the way an exception in an 'except' block used to cause exceptions to be lost, so we added __context__ to avoid that. And just like for __context__, it would be nice if we could attach some info to E1 recording that E2 had happened and then got preempted. But I don't see how we can reuse __context__ itself for this, because it's a somewhat different relationship: __context__ means that an exception happened in the handler for another exception, while in this case you might have multiple preempted exceptions, and they're associated with particular points in the stack trace where the preemption occurred.
>>
>> This is a complex issue and maybe we should call it out-of-scope for the first version of ExceptionGroups. But I mention it because it's a second place where adding some extra annotations to the traceback info would be useful, and maybe we can keep it simple by adding some minimal hooks in the core traceback machinery and let libraries like trio/asyncio handle the complicated parts?
>
> Isn't this something that Trio/asyncio should handle, as in: if you catch Cancelled that has a __context__ then you need to put that __context__ into the exception group you are raising so that it's not lost?
It's something that Trio/asyncio would handle manually, yeah, but
>> - Recording pre-empted exceptions: This is another type of metadata that would be useful to print along with the traceback. It's non-obvious and a bit hard to explain, but multiple trio users have complained about this, so I assume it will bite asyncio users too as soon as TaskGroups are added. The situation is, you have a parent task P and two child tasks C1 and C2:
>>
>> P
>> / \
>> C1 C2
>>
>> C1 terminates with an unhandled exception E1, so in order to continue unwinding, the nursery/taskgroup in P cancels C2. But, C2 was itself in the middle of unwinding another, different exception E2 (so e.g. the cancellation arrived during a `finally` block). E2 gets replaced with a `Cancelled` exception whose __context__=E2, and that exception unwinds out of C2 and the nursery/taskgroup in P catches the `Cancelled` and discards it, then re-raises E1 so it can continue unwinding.
>>
>> The problem here is that E2 gets "lost" -- there's no record of it in the final output. Basically E1 replaced it. And that can be bad: for example, if the two children are interacting with each other, then E2 might be the actual error that broke the program, and E1 is some exception complaining that the connection to C2 was lost. If you have two exceptions that are triggered from the same underlying event, it's a race which one survives.
>>
>> This is conceptually similar to the way an exception in an 'except' block used to cause exceptions to be lost, so we added __context__ to avoid that. And just like for __context__, it would be nice if we could attach some info to E1 recording that E2 had happened and then got preempted. But I don't see how we can reuse __context__ itself for this, because it's a somewhat different relationship: __context__ means that an exception happened in the handler for another exception, while in this case you might have multiple preempted exceptions, and they're associated with particular points in the stack trace where the preemption occurred.
>>
>> This is a complex issue and maybe we should call it out-of-scope for the first version of ExceptionGroups. But I mention it because it's a second place where adding some extra annotations to the traceback info would be useful, and maybe we can keep it simple by adding some minimal hooks in the core traceback machinery and let libraries like trio/asyncio handle the complicated parts?
>
> Isn't this something that Trio/asyncio should handle, as in: if you catch Cancelled that has a __context__ then you need to put that __context__ into the exception group you are raising so that it's not lost?
Trio/asyncio need somewhere to put the metadata, that the traceback
printing code will look at :-). Something like a __preempted__ field
on exceptions, or another kind of extended traceback entry ("when
passing through this frame, this exception preempted another
exception").
-n
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/ZDQPOIP5GS4NGREWUGXWX7ZCE4NKQYSD/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Irit Katriel via Python-Dev
Apr 20, 2021, 6:18:08 AM4/20/21
to Nathaniel Smith, python-dev, Yury Selivanov
On Tue, Apr 20, 2021 at 2:48 AM Nathaniel Smith <n...@pobox.com> wrote:
The problem is that most of the time, even if you're using concurrency
internally so multiple things *could* go wrong at once, only one thing
actually *does* go wrong. So it's unfortunate if some existing code is
prepared for a specific exception that it knows can be raised, that
exact exception is raised... and the existing code fails to catch it
because now it's wrapped in an EG.
Yes, this was discussed at length on this list. Raising an exception group is an API-breaking change. If a function starts raising exception groups its callers need to be prepared for that. Realistically we think exception groups will be raised by new APIs. We tried and were unable to define exception group semantics for except that would be
reasonable and backwards compatible. That's why we added except*.
> It is easy enough to write a denormalize() function in traceback.py that constructs this from the current EG structure, if you need it (use the leaf_generator from the PEP). I'm not sure I see why we should trouble the interpreter with this.
In the current design, once an exception is wrapped in an EG, then it
can never be unwrapped, because its traceback information is spread
across the individual exception + the EG tree around it. This is
confusing to users ("how do I check errno?"), and makes the design
more complicated (the need for topology-preserving .split(), the
inability to define a sensible EG.__iter__, ...). The advantage of
making the denormalized form the native form is that now the leaf
exceptions would be self-contained objects like they are now, so you
don't need EG nesting at all, and users can write intuitive code like:
except OSError as *excs:
remainder = [exc for exc in excs if exc.errno != ...]
if remainder:
raise ExceptionGroup(remainder)
We have this precise example in the PEP:
match, rest = excs.split(lambda e: e.errno != ...)
You use split() instead of iteration for that. split() preserves all __context__, __cause__ and __traceback__ information, on all leaf and non-leaf exceptions.
> For display purposes, it is probably nicer to look at a normalized traceback where common parts are not repeated.
Yeah, I agree; display code would want to re-normalize before
printing. But now it's only the display code that needs to care about
figuring out shared parts of the traceback, rather than something that
has to be maintained as an invariant everywhere.
If you *do* want iteration, we show in the PEP how to write a leaf_generator() that gives you the leaf exceptions with their tracebacks (as a list of chunks). It is easy to copy the chunks into a single flat traceback. We didn't propose to add it to traceback.py yet because the use case is unclear but if people need it we're talking about 10-15 lines in traceback.py.
So for your suggestion:
Pros:
1. Those who want a denormalized traceback (if they exist) won't need to call traceback.denormalize().
Cons:
1. A significant change in the interpreter that will make it less efficient (both time and space).
2. Display code will need to normalize the traceback, which is much more complicated than denormalizing because you need to discover the shared parts.
Am I missing something?
>
> It sounds like you want some way to enrich exceptions. This should be optional (we wouldn't want EG to require additional metadata for exceptions) so yeah, I agree it should sit on the leaf exception and not on the group. In that sense it's orthogonal to this PEP.
Well, the extra metadata would specifically be at "join" points in the
traceback, which are a thing that this PEP is creating :-). And it's
useful for every user of EGs, since by definition, an EG is
multiplexing exceptions from multiple sources, so it's nice to tell
the user which sources those are.
That said, you're right, if we want to handle this by defining a new
kind of traceback entry that code like Trio/asyncio/hypothesis can
manually attach to exceptions, then that could be written as a
separate-but-complementary PEP. In my original design, instead of
defining a new kind of traceback entry, I was storing this on the EG
itself, so that's why I was thinking about it needing to be part of
this PEP.
You can also create an ExceptionGroup subclass with whatever extra data you want to include.
Irit
srk...@mail.de
Apr 20, 2021, 3:44:22 PM4/20/21
to Irit Katriel, python-dev, Yury Selivanov
Hi Irit,
reading this subthread specifically, I just got a wild idea and I couldn‘t find any related information in the PEP:
Why not extending BaseException by __group__ among __cause__ and __context__?
Would this reduce some of the added complexity and thus increase broader acceptance?
Cheers,
Sven
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Irit Katriel via Python-Dev
Apr 20, 2021, 4:08:10 PM4/20/21
to srk...@mail.de, Irit Katriel, python-dev, Yury Selivanov
Hi Sven,
I don’t follow. What would the value of __group__ be and how would it work?
Irit
> On 20 Apr 2021, at 20:44, srk...@mail.de wrote:
>
>
I don’t follow. What would the value of __group__ be and how would it work?
Irit
> On 20 Apr 2021, at 20:44, srk...@mail.de wrote:
>
>
> Hi Irit,
>
> reading this subthread specifically, I just got a wild idea and I couldn‘t find any related information in the PEP:
>
> Why not extending BaseException by __group__ among __cause__ and __context__?
>
> Would this reduce some of the added complexity and thus increase broader acceptance?
>
> Cheers,
> Sven
>
> reading this subthread specifically, I just got a wild idea and I couldn‘t find any related information in the PEP:
>
> Why not extending BaseException by __group__ among __cause__ and __context__?
>
> Would this reduce some of the added complexity and thus increase broader acceptance?
>
> Cheers,
> Sven
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/NYAXQQ7EVXUNX7MTQJDLAXVKGRHAUSEF/Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
srk...@mail.de
Apr 20, 2021, 5:12:44 PM4/20/21
to Irit Katriel, Irit Katriel, python-dev, Yury Selivanov
So, forgive me my relatively simple mental model about ExceptionGroup. I still try to create one for daily use.
As noted in the discussion, an EG provides a way to collect exceptions from different sources and raise them as a bundle. They have no apparent relation up until this point in time (for whatever reason they have been separate and for whatever reason they are bundled now). The result would be a tree graph in any case.
A usual datastructure for a tree is to store all child nodes at the parent node.
That was the idea behind the content of BaseException.__group__: it’s the list of child exceptions bundled at a specific point in time and raise as such a bundle. So all exceptions could become EGs with the additional semantics you‘ve described in the PEP.
Illustrative Example:
>>> bundle_exc.__group__
[IOError(123), RuntimerError(‘issue somewhere’)]
I was wondering what of the PEP could be removed to make it simpler and more acceptable/less confusing (also looking at reactions from Twitter etc.) and I found these additional classes to be a part of it. Additionally, I fail to see how to access these bundled exceptions in an easy manner like __cause__ and __context__. (As the PEP also referring to them). So, I removed the classes and added a regular attribute.
The reason I brought this up what the section “rejected ideas” didn’t showed anything in this direction (or I managed to missed that).
Sven
>
> On 20. Apr 2021, at 22:05, Irit Katriel <iritk...@yahoo.com> wrote:
>
> Hi Sven,
As noted in the discussion, an EG provides a way to collect exceptions from different sources and raise them as a bundle. They have no apparent relation up until this point in time (for whatever reason they have been separate and for whatever reason they are bundled now). The result would be a tree graph in any case.
A usual datastructure for a tree is to store all child nodes at the parent node.
That was the idea behind the content of BaseException.__group__: it’s the list of child exceptions bundled at a specific point in time and raise as such a bundle. So all exceptions could become EGs with the additional semantics you‘ve described in the PEP.
Illustrative Example:
>>> bundle_exc.__group__
[IOError(123), RuntimerError(‘issue somewhere’)]
I was wondering what of the PEP could be removed to make it simpler and more acceptable/less confusing (also looking at reactions from Twitter etc.) and I found these additional classes to be a part of it. Additionally, I fail to see how to access these bundled exceptions in an easy manner like __cause__ and __context__. (As the PEP also referring to them). So, I removed the classes and added a regular attribute.
The reason I brought this up what the section “rejected ideas” didn’t showed anything in this direction (or I managed to missed that).
Sven
>
> On 20. Apr 2021, at 22:05, Irit Katriel <iritk...@yahoo.com> wrote:
>
> Hi Sven,
>
> I don’t follow. What would the value of __group__ be and how would it work?
>
> Irit
>
>> On 20 Apr 2021, at 20:44, srk...@mail.de wrote:
>>
>> Hi Irit,
>> reading this subthread specifically, I just got a wild idea and I couldn‘t find any related information in the PEP:
>> Why not extending BaseException by __group__ among __cause__ and __context__?
>> Would this reduce some of the added complexity and thus increase broader acceptance?
>> Cheers,
>> Sven
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/4NUG6R36HN557AY36SPINNRFRYBW4QM7/> I don’t follow. What would the value of __group__ be and how would it work?
>
> Irit
>
>> On 20 Apr 2021, at 20:44, srk...@mail.de wrote:
>>
>> Hi Irit,
>> reading this subthread specifically, I just got a wild idea and I couldn‘t find any related information in the PEP:
>> Why not extending BaseException by __group__ among __cause__ and __context__?
>> Would this reduce some of the added complexity and thus increase broader acceptance?
>> Cheers,
>> Sven
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Irit Katriel via Python-Dev
Apr 20, 2021, 5:28:56 PM4/20/21
to srk...@mail.de, Irit Katriel, python-dev, Yury Selivanov
I don’t see what this simplifies. We still need to implement split, and to worry about wrapping or not wrapping BaseExceptions and we still need to define exception handling semantics (except/except*).
> On 20 Apr 2021, at 22:12, srk...@mail.de wrote:
>
> So, forgive me my relatively simple mental model about ExceptionGroup. I still try to create one for daily use.
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/OJJXMPBD4ZUZOUZ2ULWT4Q6ULI75HLDE/
> On 20 Apr 2021, at 22:12, srk...@mail.de wrote:
>
> So, forgive me my relatively simple mental model about ExceptionGroup. I still try to create one for daily use.
srk...@mail.de
Apr 21, 2021, 2:52:26 PM4/21/21
to Irit Katriel, Irit Katriel, python-dev, Yury Selivanov
Removing two concepts and preserving semantics simplifies the matter for users. People need less to memorize and less to learn.
Or am I missing something here? Couldn’t we achieve our goal without these two new classes?
Splitting, wrapping and exception handling semantics are perfectly fine and serve their purposes. So, of course this still needs to be implemented.
> On 20. Apr 2021, at 23:26, Irit Katriel <iritk...@googlemail.com> wrote:
>
> I don’t see what this simplifies. We still need to implement split, and to worry about wrapping or not wrapping BaseExceptions and we still need to define exception handling semantics (except/except*).
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/OD6IDT5HGDULGLVVQ2HF7WJXNQM7RA3K/
Or am I missing something here? Couldn’t we achieve our goal without these two new classes?
Splitting, wrapping and exception handling semantics are perfectly fine and serve their purposes. So, of course this still needs to be implemented.
> On 20. Apr 2021, at 23:26, Irit Katriel <iritk...@googlemail.com> wrote:
>
> I don’t see what this simplifies. We still need to implement split, and to worry about wrapping or not wrapping BaseExceptions and we still need to define exception handling semantics (except/except*).
Yury Selivanov
Apr 21, 2021, 3:02:23 PM4/21/21
to srk...@mail.de, Irit Katriel, Irit Katriel, python-dev, Yury Selivanov
On Wed, Apr 21, 2021 at 11:50 AM srk...@mail.de <srk...@mail.de> wrote:
>
> Removing two concepts and preserving semantics simplifies the matter for users. People need less to memorize and less to learn.
>
> Or am I missing something here? Couldn’t we achieve our goal without these two new classes?
No, we can't. What you are proposing would make it very hard for users
>
> Removing two concepts and preserving semantics simplifies the matter for users. People need less to memorize and less to learn.
>
> Or am I missing something here? Couldn’t we achieve our goal without these two new classes?
to understand at a glance if what you have in an innocently looking
`except Exception` is correct or not. In my async/await code I'd have
to always check the `__group__` attribute to make sure it's not an
exception group in disguise.
So while you're "simplifying" the proposal by removing a couple of
types, you're complicating it in all other places. Besides, I don't
think that adding the ExceptionGroup type is a controversial idea that
needs any simplification.
Yury
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/IXHLXQYTEFPRC2GLFGAM6GHCYAAQG4DQ/Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Nathaniel Smith
Apr 21, 2021, 5:47:56 PM4/21/21
to srk...@mail.de, Irit Katriel, Irit Katriel, python-dev, Yury Selivanov
On Tue, Apr 20, 2021 at 2:15 PM srk...@mail.de <srk...@mail.de> wrote:
>
> So, forgive me my relatively simple mental model about ExceptionGroup. I still try to create one for daily use.
>
> As noted in the discussion, an EG provides a way to collect exceptions from different sources and raise them as a bundle. They have no apparent relation up until this point in time (for whatever reason they have been separate and for whatever reason they are bundled now). The result would be a tree graph in any case.
>
> A usual datastructure for a tree is to store all child nodes at the parent node.
>
> That was the idea behind the content of BaseException.__group__: it’s the list of child exceptions bundled at a specific point in time and raise as such a bundle. So all exceptions could become EGs with the additional semantics you‘ve described in the PEP.
>
> Illustrative Example:
> >>> bundle_exc.__group__
> [IOError(123), RuntimerError(‘issue somewhere’)]
>
> I was wondering what of the PEP could be removed to make it simpler and more acceptable/less confusing (also looking at reactions from Twitter etc.) and I found these additional classes to be a part of it. Additionally, I fail to see how to access these bundled exceptions in an easy manner like __cause__ and __context__. (As the PEP also referring to them). So, I removed the classes and added a regular attribute.
This seems more confusing to me too. Instead of having a single
>
> So, forgive me my relatively simple mental model about ExceptionGroup. I still try to create one for daily use.
>
> As noted in the discussion, an EG provides a way to collect exceptions from different sources and raise them as a bundle. They have no apparent relation up until this point in time (for whatever reason they have been separate and for whatever reason they are bundled now). The result would be a tree graph in any case.
>
> A usual datastructure for a tree is to store all child nodes at the parent node.
>
> That was the idea behind the content of BaseException.__group__: it’s the list of child exceptions bundled at a specific point in time and raise as such a bundle. So all exceptions could become EGs with the additional semantics you‘ve described in the PEP.
>
> Illustrative Example:
> >>> bundle_exc.__group__
> [IOError(123), RuntimerError(‘issue somewhere’)]
>
> I was wondering what of the PEP could be removed to make it simpler and more acceptable/less confusing (also looking at reactions from Twitter etc.) and I found these additional classes to be a part of it. Additionally, I fail to see how to access these bundled exceptions in an easy manner like __cause__ and __context__. (As the PEP also referring to them). So, I removed the classes and added a regular attribute.
ExceptionGroup class, you're suggesting that all exceptions should
effectively become ExceptionGroups. I know what
ExceptionGroup([ValueError]) means -- it means there was a ValueError,
and it got put in a group, so it should probably be handled the same
way as a ValueError. I have no idea what a KeyError([ValueError])
would mean. Is that a KeyError or a ValueError? Adding flexibility
doesn't necessarily make things simpler :-)
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/F6LVP7CLQGIAZ2PI2SL6FN3VXF2MVY2M/Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Nathaniel Smith
Apr 21, 2021, 6:29:28 PM4/21/21
to Irit Katriel, python-dev, Yury Selivanov
On Tue, Apr 20, 2021 at 3:15 AM Irit Katriel <iritk...@googlemail.com> wrote:
> On Tue, Apr 20, 2021 at 2:48 AM Nathaniel Smith <n...@pobox.com> wrote:
>>
>>
>> The problem is that most of the time, even if you're using concurrency
>> internally so multiple things *could* go wrong at once, only one thing
>> actually *does* go wrong. So it's unfortunate if some existing code is
>> prepared for a specific exception that it knows can be raised, that
>> exact exception is raised... and the existing code fails to catch it
>> because now it's wrapped in an EG.
>
> Yes, this was discussed at length on this list. Raising an exception group is an API-breaking change. If a function starts raising exception groups its callers need to be prepared for that. Realistically we think exception groups will be raised by new APIs. We tried and were unable to define exception group semantics for except that would be reasonable and backwards compatible. That's why we added except*.
Sure. This was in my list of reasons why the backwards compatibility
> On Tue, Apr 20, 2021 at 2:48 AM Nathaniel Smith <n...@pobox.com> wrote:
>>
>>
>> The problem is that most of the time, even if you're using concurrency
>> internally so multiple things *could* go wrong at once, only one thing
>> actually *does* go wrong. So it's unfortunate if some existing code is
>> prepared for a specific exception that it knows can be raised, that
>> exact exception is raised... and the existing code fails to catch it
>> because now it's wrapped in an EG.
>
> Yes, this was discussed at length on this list. Raising an exception group is an API-breaking change. If a function starts raising exception groups its callers need to be prepared for that. Realistically we think exception groups will be raised by new APIs. We tried and were unable to define exception group semantics for except that would be reasonable and backwards compatible. That's why we added except*.
tradeoffs are forcing us into awkward compromises. I only elaborated
on it b/c in your last email you said you didn't understand why this
was a problem :-). And except* is definitely useful. But I think there
are options for 'except' that haven't been considered fully.
Saying that you have to make a new API every time you start using
concurrency inside a function is extremely restrictive. The whole
point of structured concurrency (and structured programming in
general) is that function callers don't need to know about the control
flow decisions inside a function. So right now, the EG proposal is
like saying that if you every have a function that doesn't contain a
'for' loop, and then you want to add a 'for' loop, then you have to
define a whole new API instead of modifying the existing one.
I absolutely get why the proposal looks like that. I'm just making the
point that we should make sure we've exhausted all our options before
settling for that as a compromise.
>> > It is easy enough to write a denormalize() function in traceback.py that constructs this from the current EG structure, if you need it (use the leaf_generator from the PEP). I'm not sure I see why we should trouble the interpreter with this.
>>
>> In the current design, once an exception is wrapped in an EG, then it
>> can never be unwrapped, because its traceback information is spread
>> across the individual exception + the EG tree around it. This is
>> confusing to users ("how do I check errno?"), and makes the design
>> more complicated (the need for topology-preserving .split(), the
>> inability to define a sensible EG.__iter__, ...). The advantage of
>> making the denormalized form the native form is that now the leaf
>> exceptions would be self-contained objects like they are now, so you
>> don't need EG nesting at all, and users can write intuitive code like:
>>
>> except OSError as *excs:
>> remainder = [exc for exc in excs if exc.errno != ...]
>> if remainder:
>> raise ExceptionGroup(remainder)
>
>
> We have this precise example in the PEP:
> match, rest = excs.split(lambda e: e.errno != ...)
>
> You use split() instead of iteration for that. split() preserves all __context__, __cause__ and __traceback__ information, on all leaf and non-leaf exceptions.
to compare these two options: in the flat EG example, most Python
users could write and read that code without knowing anything except
"there can be multiple exceptions now". It's all old, well-known
constructs used in the obvious way.
For the .split() version, you have to write a lambda (which is allowed
to access parts of the exception object, but not all of it!), and use
this idiosyncratic method that only makes sense if you know about EG
tree structures. That's a lot more stuff that users have to understand
and hold in their head.
Or here's another example. Earlier you suggested:
> If you are using concurrency internally and don't want to raise EGs externally, then surely you will catch EGs, select one of the exceptions to raise and throw away all the others
out one of the exceptions to raise. The leaf exceptions aren't
standalone objects; you need some obscure traceback manipulation to do
this. I guarantee that users will get this wrong, even if we provide
the tools, because the explanation about when and why you need the
tools is complicated and people won't internalize it.
With flat EGs, this is trivial: it's just `raise
ExceptionGroup[the_selected_exception_index]`.
>> > For display purposes, it is probably nicer to look at a normalized traceback where common parts are not repeated.
>>
>> Yeah, I agree; display code would want to re-normalize before
>> printing. But now it's only the display code that needs to care about
>> figuring out shared parts of the traceback, rather than something that
>>
>> has to be maintained as an invariant everywhere.
>
> If you *do* want iteration, we show in the PEP how to write a leaf_generator() that gives you the leaf exceptions with their tracebacks (as a list of chunks). It is easy to copy the chunks into a single flat traceback. We didn't propose to add it to traceback.py yet because the use case is unclear but if people need it we're talking about 10-15 lines in traceback.py.
>
> So for your suggestion:
>
> Pros:
> 1. Those who want a denormalized traceback (if they exist) won't need to call traceback.denormalize().
>
> Cons:
> 1. A significant change in the interpreter that will make it less efficient (both time and space).
would be pretty simple. Whenever the exception handling code goes to
write back a traceback to an exception's __traceback__ attribute,
check if the exception is an ExceptionGroup; if so, then loop over
each exception in the group and push a clone of the traceback onto
that exception's __traceback__. It takes code of course, everything
does, but I'd be surprised if it was more than, say, 20 lines of C. I
think that's a pretty good tradeoff to get a simpler user-level
language semantics. Python is supposed to fit in your head :-).
Time/space efficiency is interesting, because yeah, with the nested EG
approach then common parts of the traceback are stored once, while the
flat EG approach needs to store a separate copy for each exception.
But:
- The extra cost is 1 traceback object per common stack frame, per
exception. So the worst case is if you have, like, a 1000 exceptions
in a single group, and then that group travels together for 1000 stack
frames. But... is that realistic? Very few programs recurse down 1000
levels and *then* spawn a bunch of tasks. And very few programs have
1000 tasks that all fail simultaneously with different exceptions. And
even if that did happen, 1000*1000 = 1 million traceback objects take
~56 megabytes of RAM and ~90 ms to instantiate on my laptop, which is
not a big deal. [2]
- This is assuming the most simple/naive implementation, where we
simply duplicate every traceback entry onto every exception. If it
turns out to be unacceptable, there are lots of tricks we could use to
speed it up, e.g. by updating __traceback__ lazily or letting
traceback entry objects be shared between tracebacks.
[2] Estimated based on 'sys.getsizeof(tb_object)' and '%timeit
types.TracebackType(None, frame, 0, 0)'.
> 2. Display code will need to normalize the traceback, which is much more complicated than denormalizing because you need to discover the shared parts.
>
> Am I missing something?
prefix problem. Given a tree-in-progress and a traceback, walk down
each of them in parallel until you run out of matching entries, then
insert a branch node and append the rest of the traceback.
>> >
>> > It sounds like you want some way to enrich exceptions. This should be optional (we wouldn't want EG to require additional metadata for exceptions) so yeah, I agree it should sit on the leaf exception and not on the group. In that sense it's orthogonal to this PEP.
>>
>> Well, the extra metadata would specifically be at "join" points in the
>> traceback, which are a thing that this PEP is creating :-). And it's
>> useful for every user of EGs, since by definition, an EG is
>> multiplexing exceptions from multiple sources, so it's nice to tell
>> the user which sources those are.
>>
>> That said, you're right, if we want to handle this by defining a new
>> kind of traceback entry that code like Trio/asyncio/hypothesis can
>> manually attach to exceptions, then that could be written as a
>> separate-but-complementary PEP. In my original design, instead of
>> defining a new kind of traceback entry, I was storing this on the EG
>> itself, so that's why I was thinking about it needing to be part of
>> this PEP.
>
>
> You can also create an ExceptionGroup subclass with whatever extra data you want to include.
should be included in traceback displays, similar to __cause__ and
__context__...
-n
--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Irit Katriel via Python-Dev
Apr 21, 2021, 6:55:10 PM4/21/21
to Nathaniel Smith, python-dev, Yury Selivanov
On Wed, Apr 21, 2021 at 11:22 PM Nathaniel Smith <n...@pobox.com> wrote:
Saying that you have to make a new API every time you start using
concurrency inside a function is extremely restrictive.
You don't need a new API when you start using concurrency inside a function. You need a new API when you start raising exception groups from a function because that changes the function's external behaviour.
-1 from me on the denormalized traceback idea. I've said what I had to say about it in my previous email.
Cheers
Irit
Guido van Rossum
Apr 21, 2021, 7:57:16 PM4/21/21
to Nathaniel Smith, Irit Katriel, python-dev, Yury Selivanov
On Wed, Apr 21, 2021 at 3:26 PM Nathaniel Smith <n...@pobox.com> wrote:
On Tue, Apr 20, 2021 at 3:15 AM Irit Katriel <iritk...@googlemail.com> wrote:
> On Tue, Apr 20, 2021 at 2:48 AM Nathaniel Smith <n...@pobox.com> wrote:
>>
>>
>> The problem is that most of the time, even if you're using concurrency
>> internally so multiple things *could* go wrong at once, only one thing
>> actually *does* go wrong. So it's unfortunate if some existing code is
>> prepared for a specific exception that it knows can be raised, that
>> exact exception is raised... and the existing code fails to catch it
>> because now it's wrapped in an EG.
>
> Yes, this was discussed at length on this list. Raising an exception group is an API-breaking change. If a function starts raising exception groups its callers need to be prepared for that. Realistically we think exception groups will be raised by new APIs. We tried and were unable to define exception group semantics for except that would be reasonable and backwards compatible. That's why we added except*.
Sure. This was in my list of reasons why the backwards compatibility
tradeoffs are forcing us into awkward compromises. I only elaborated
on it b/c in your last email you said you didn't understand why this
was a problem :-). And except* is definitely useful. But I think there
are options for 'except' that haven't been considered fully.
Do you have any suggestions, or are you just telling us to think harder? Because we've already thought as hard as we could and within all the constraints (backwards compatibility and otherwise) we just couldn't think of a better one.
Saying that you have to make a new API every time you start using
concurrency inside a function is extremely restrictive. The whole
point of structured concurrency (and structured programming in
general) is that function callers don't need to know about the control
flow decisions inside a function. So right now, the EG proposal is
like saying that if you every have a function that doesn't contain a
'for' loop, and then you want to add a 'for' loop, then you have to
define a whole new API instead of modifying the existing one.
That analogy doesn't fly. If "what exceptions can this raise" is not part of the function spec (so users who catch exceptions are already flying blind or guessing based on what they've seen happen in the past), then you can certainly switch to using EGs, and this *is* like adding a for-loop. But if the exceptions that a function raises *are* part of its spec, and users are catching them, then switching to concurrency (without catching and converting EGs in the function) *is* an API change, akin to taking a function that returns an int and changing it to (sometimes?) returning a list of ints.
I absolutely get why the proposal looks like that. I'm just making the
point that we should make sure we've exhausted all our options before
settling for that as a compromise.
But if you're not able to make constructive proposals I have to conclude that we *have* exhausted our options, and we should move on.
But it *is* a tree structure. The interior nodes of the tree have a meaning: they represent join points where exceptions were grouped together, and the selection process needs to traverse the tree.
Or here's another example. Earlier you suggested:
> If you are using concurrency internally and don't want to raise EGs externally, then surely you will catch EGs, select one of the exceptions to raise and throw away all the others
But with nested EGs, this is difficult, because you *can't* just pull
out one of the exceptions to raise. The leaf exceptions aren't
standalone objects; you need some obscure traceback manipulation to do
this. I guarantee that users will get this wrong, even if we provide
the tools, because the explanation about when and why you need the
tools is complicated and people won't internalize it.
With flat EGs, this is trivial: it's just `raise
ExceptionGroup[the_selected_exception_index]`.
But it's generally wrong to raise just one of the exceptions anyway, so you want to make it easier to get a wrong result?
I'd rather make this an option for asyncio.gather() or its TaskGroup equivalent, then the user doesn't have to do anything. gather() already has as its default behavior to raise the first exception it encounters, which seems good enough.
The real cost here is that we would need a new "TracebackGroup" concept, since the internal data structures and APIs keep the traceback chain and the exception object separated until the exception is caught. In our early design stages we actually explored this and the complexity of the data structures was painful. We eventually realized that we didn't need this concept at all, and the result is much clearer, despite what you seem to think.
> 2. Display code will need to normalize the traceback, which is much more complicated than denormalizing because you need to discover the shared parts.
>
> Am I missing something?
I think re-normalizing is very cheap and simple? It's just a common
prefix problem. Given a tree-in-progress and a traceback, walk down
each of them in parallel until you run out of matching entries, then
insert a branch node and append the rest of the traceback.
>> >
>> > It sounds like you want some way to enrich exceptions. This should be optional (we wouldn't want EG to require additional metadata for exceptions) so yeah, I agree it should sit on the leaf exception and not on the group. In that sense it's orthogonal to this PEP.
>>
>> Well, the extra metadata would specifically be at "join" points in the
>> traceback, which are a thing that this PEP is creating :-). And it's
>> useful for every user of EGs, since by definition, an EG is
>> multiplexing exceptions from multiple sources, so it's nice to tell
>> the user which sources those are.
>>
>> That said, you're right, if we want to handle this by defining a new
>> kind of traceback entry that code like Trio/asyncio/hypothesis can
>> manually attach to exceptions, then that could be written as a
>> separate-but-complementary PEP. In my original design, instead of
>> defining a new kind of traceback entry, I was storing this on the EG
>> itself, so that's why I was thinking about it needing to be part of
>> this PEP.
>
>
> You can also create an ExceptionGroup subclass with whatever extra data you want to include.
Unfortunately, that doesn't work either, because this is data that
should be included in traceback displays, similar to __cause__ and
__context__...
Fortunately in the flat design there wouldn't be a need for such metadata anyway...
Nathaniel Smith
Apr 23, 2021, 4:25:48 AM4/23/21
to Guido van Rossum, Irit Katriel, python-dev, Yury Selivanov
On Wed, Apr 21, 2021 at 4:50 PM Guido van Rossum <gu...@python.org> wrote:
> On Wed, Apr 21, 2021 at 3:26 PM Nathaniel Smith <n...@pobox.com> wrote:
>> Sure. This was in my list of reasons why the backwards compatibility
>> tradeoffs are forcing us into awkward compromises. I only elaborated
>> on it b/c in your last email you said you didn't understand why this
>> was a problem :-). And except* is definitely useful. But I think there
>> are options for 'except' that haven't been considered fully.
>
> Do you have any suggestions, or are you just telling us to think harder? Because we've already thought as hard as we could and within all the constraints (backwards compatibility and otherwise) we just couldn't think of a better one.
The main possibility that I don't think we've examined fully is to
> On Wed, Apr 21, 2021 at 3:26 PM Nathaniel Smith <n...@pobox.com> wrote:
>> Sure. This was in my list of reasons why the backwards compatibility
>> tradeoffs are forcing us into awkward compromises. I only elaborated
>> on it b/c in your last email you said you didn't understand why this
>> was a problem :-). And except* is definitely useful. But I think there
>> are options for 'except' that haven't been considered fully.
>
> Do you have any suggestions, or are you just telling us to think harder? Because we've already thought as hard as we could and within all the constraints (backwards compatibility and otherwise) we just couldn't think of a better one.
make 'except' blocks fire multiple times when there are multiple
exceptions. We ruled it out early b/c it's incompatible with nested
EGs, but if flat EGs are better anyway, then the balance shifts around
and it might land somewhere different. it's a tricky discussion
though, b/c both the current proposal and the alternative have very
complex implications and downsides. So we probably shouldn't get too
distracted by that until after the flat vs nested discussion has
settled down more.
I'm not trying to filibuster here -- I really want some form of EGs to
land. I think python has the potential to be the most elegant and
accessible language around for writing concurrent programs, and EGs
are a key part of that. I don't want to fight about anything; I just
want to work together to make sure we have a full picture of our
options, so we can be confident we're making the best choice.
> The real cost here is that we would need a new "TracebackGroup" concept, since the internal data structures and APIs keep the traceback chain and the exception object separated until the exception is caught. In our early design stages we actually explored this and the complexity of the data structures was painful. We eventually realized that we didn't need this concept at all, and the result is much clearer, despite what you seem to think.
think it can be done with exactly our current data structures, nothing
new.
- When an EG is raised, build the traceback for just that EG while
it's unwinding. This means if any C code peeks at exc_info while it's
in flight, it'll only see the current branch of the traceback tree,
but that seems fine.
- When the exception is caught and we go to write back the traceback
to its __traceback__ attribute, instead "peek through" the EG and
append the built-up traceback entries onto each of the constituent
exceptions.
You could get cleverer for efficiency, but that basic concept seems
pretty simple and viable to me. What am I missing?
-n
--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Chris Angelico
Apr 23, 2021, 5:44:58 AM4/23/21
to python-dev
On Fri, Apr 23, 2021 at 6:25 PM Nathaniel Smith <n...@pobox.com> wrote:
>
> On Wed, Apr 21, 2021 at 4:50 PM Guido van Rossum <gu...@python.org> wrote:
> > On Wed, Apr 21, 2021 at 3:26 PM Nathaniel Smith <n...@pobox.com> wrote:
> >> Sure. This was in my list of reasons why the backwards compatibility
> >> tradeoffs are forcing us into awkward compromises. I only elaborated
> >> on it b/c in your last email you said you didn't understand why this
> >> was a problem :-). And except* is definitely useful. But I think there
> >> are options for 'except' that haven't been considered fully.
> >
> > Do you have any suggestions, or are you just telling us to think harder? Because we've already thought as hard as we could and within all the constraints (backwards compatibility and otherwise) we just couldn't think of a better one.
>
> The main possibility that I don't think we've examined fully is to
> make 'except' blocks fire multiple times when there are multiple
> exceptions.
Vanilla except blocks? Not sure if I'm misunderstanding, but that
>
> On Wed, Apr 21, 2021 at 4:50 PM Guido van Rossum <gu...@python.org> wrote:
> > On Wed, Apr 21, 2021 at 3:26 PM Nathaniel Smith <n...@pobox.com> wrote:
> >> Sure. This was in my list of reasons why the backwards compatibility
> >> tradeoffs are forcing us into awkward compromises. I only elaborated
> >> on it b/c in your last email you said you didn't understand why this
> >> was a problem :-). And except* is definitely useful. But I think there
> >> are options for 'except' that haven't been considered fully.
> >
> > Do you have any suggestions, or are you just telling us to think harder? Because we've already thought as hard as we could and within all the constraints (backwards compatibility and otherwise) we just couldn't think of a better one.
>
> The main possibility that I don't think we've examined fully is to
> make 'except' blocks fire multiple times when there are multiple
> exceptions.
could cause some problems. Consider:
trn.begin()
try:
...
except BaseException:
trn.rollback()
raise
else:
trn.commit()
What happens if multiple exceptions get raised? Will the transaction
be rolled back more than once? What gets reraised?
If I'm completely misunderstanding you here, my apologies.
ChrisA
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/SGFE3XG4COLBC747U7IBVWRKWAULY43C/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Irit Katriel via Python-Dev
Apr 23, 2021, 7:12:23 AM4/23/21
to Nathaniel Smith, python-dev, Yury Selivanov
On Fri, Apr 23, 2021 at 9:22 AM Nathaniel Smith <n...@pobox.com> wrote:
On Wed, Apr 21, 2021 at 4:50 PM Guido van Rossum <gu...@python.org> wrote:
> On Wed, Apr 21, 2021 at 3:26 PM Nathaniel Smith <n...@pobox.com> wrote:
>> Sure. This was in my list of reasons why the backwards compatibility
>> tradeoffs are forcing us into awkward compromises. I only elaborated
>> on it b/c in your last email you said you didn't understand why this
>> was a problem :-). And except* is definitely useful. But I think there
>> are options for 'except' that haven't been considered fully.
>
> Do you have any suggestions, or are you just telling us to think harder? Because we've already thought as hard as we could and within all the constraints (backwards compatibility and otherwise) we just couldn't think of a better one.
The main possibility that I don't think we've examined fully is to
make 'except' blocks fire multiple times when there are multiple
exceptions. We ruled it out early b/c it's incompatible with nested
EGs, but if flat EGs are better anyway, then the balance shifts around
and it might land somewhere different. it's a tricky discussion
though, b/c both the current proposal and the alternative have very
complex implications and downsides. So we probably shouldn't get too
distracted by that until after the flat vs nested discussion has
settled down more.
I'm not trying to filibuster here -- I really want some form of EGs to
land.
I'm very glad to hear that. It's been hard to know where you stand, because you didn't even decline our invitation in October to work together on this, and several
later
invitations to look at the implementation and try it with Trio -- you just didn't reply. The only responses I saw were public, on this list, taking us right back to the drawing board (for example - the suggestion you mention in the previous paragraph as not-sufficiently-explored, actually appears in the rejected ideas section of the PEP, and the reason for rejection is not that it's incompatible with nesting). So thank you for clarifying that you are, ultimately, supportive of our efforts.
We do realize that we didn't adequately credit you for the contributions of your 2018 work to this PEP, and have now added an acknowledgements section for that purpose. Apologies for that.
I'm confused about the flattening suggestion - above you talk about "flat EG", but below about tracebacks. It's not clear to me whether you want EG to be flat (ie no nesting of EGs) or just the traceback to be flat (but you can still have a nested EG). I also don't know what problem you are trying to solve with this.
Fortunately we're not at the whiteboard stage anymore, we have a fully working implementation and you can use it to demonstrate any problems you see with what we did.
Irit
Nathaniel Smith
Apr 23, 2021, 11:41:50 AM4/23/21
to Python Dev
I just got the reply below sent directly to my personal account, and I'm confused about what's going on. If it's just a one off I'll chalk it up to random internet weirdness, but if other folks are getting these too it might be something the list admins should look into? Or... something?
---------- Forwarded message ---------
From: Hoi lam Poon <gillc...@gmail.com>
Date: Fri, Apr 23, 2021, 02:01
Subject: Re: [Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
To: Nathaniel Smith <n...@pobox.com>
From: Hoi lam Poon <gillc...@gmail.com>
Date: Fri, Apr 23, 2021, 02:01
Subject: Re: [Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
To: Nathaniel Smith <n...@pobox.com>
Stop pretending, I can definitely get the key control file, your working group, all past actions and instructions cannot be cleared in front of me at all. You have been playing around for a few days, and I won’t stop you. Your face? I won’t, you know, you can’t drive me away, and that file is all, after I get it, you will be convicted even if you disband, I swear
Jelle Zijlstra
Apr 23, 2021, 11:52:52 AM4/23/21
to Nathaniel Smith, Python Dev
I got a few of these too. I'm not sure anything can be done about it, since the mailing list is publicly archived, so anyone could send emails to anyone who posts here.
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/4BAOL763Y2O2AXLEILYGHSNG2VMZJIN6/
Pablo Galindo Salgado
Apr 23, 2021, 11:53:03 AM4/23/21
to Nathaniel Smith, Python Dev
I had I and still don't know what's going on. Mine was in a response to a release announcement so it was extra weird. Here is what I received:
I have now formally filed a final lawsuit against the manager of the python program company, because all of him is also a criminal act, and GNU has EU legal certification, only my key can log in, and the key must be recycled after the death of the holder, and gitlab allows to change It’s the most basic and important crime to log in by people who support the snatching of the key. I have to explain to you that the key is to be registered and authenticated. My girlfriend wants to authenticate me with this key, and my information is also there. The key is authenticated, so I will not log in now, and I have submitted a lawsuit against him and the authority of the key holder to the U.S. Supreme Court and the European Union. I will not log in until there is a judgment or the U.S. Supreme Court allows me. People will be litigated, and the information that has been changed online will be found out, and I have library files, I have all the original materials, please cooperate with me, my key is called the Boss key, all websites of the program, companies, Institutions, banks, third-party platforms, and only my keys can have them, including patents and copyrights.
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/4BAOL763Y2O2AXLEILYGHSNG2VMZJIN6/
Antoine Pitrou
Apr 23, 2021, 12:00:39 PM4/23/21
to pytho...@python.org
Perhaps an unconventional artistic performance ;-)
But more probably some kind of loony with a lot of free time...
David Mertz
Apr 23, 2021, 12:11:05 PM4/23/21
to Pablo Galindo Salgado, Python Dev
This reads like a much worse variation on GPT-3.
Sebastian Rittau
Apr 23, 2021, 12:16:57 PM4/23/21
to pytho...@python.org
Am 23.04.21 um 17:38 schrieb Nathaniel Smith:
style, which I have always connected to mental issues. To protect both
the sender and other list members, I would suggest to the list owners to
manually unsubscribe the sender from the mailing list.
- Sebastian
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/6SRZW4BI2ONGD4NG2AID5F5GEYNMLAJE/
> I just got the reply below sent directly to my personal account, and
> I'm confused about what's going on. If it's just a one off I'll chalk
> it up to random internet weirdness, but if other folks are getting
> these too it might be something the list admins should look into?
> Or... something?
I also got one of these from the same sender, Same rambling, incoherent
> I'm confused about what's going on. If it's just a one off I'll chalk
> it up to random internet weirdness, but if other folks are getting
> these too it might be something the list admins should look into?
> Or... something?
style, which I have always connected to mental issues. To protect both
the sender and other list members, I would suggest to the list owners to
manually unsubscribe the sender from the mailing list.
- Sebastian
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Nathaniel Smith
Apr 26, 2021, 12:31:38 AM4/26/21
to Chris Angelico, python-dev
On Fri, Apr 23, 2021 at 2:45 AM Chris Angelico <ros...@gmail.com> wrote:
>
> On Fri, Apr 23, 2021 at 6:25 PM Nathaniel Smith <n...@pobox.com> wrote:
> > The main possibility that I don't think we've examined fully is to
> > make 'except' blocks fire multiple times when there are multiple
> > exceptions.
>
> Vanilla except blocks? Not sure if I'm misunderstanding, but that
> could cause some problems. Consider:
>
> trn.begin()
> try:
> ...
> except BaseException:
> trn.rollback()
> raise
> else:
> trn.commit()
>
> What happens if multiple exceptions get raised? Will the transaction
> be rolled back more than once? What gets reraised?
>
> If I'm completely misunderstanding you here, my apologies.
Yeah, you've understood correctly, and you see why I wrote "both the
>
> On Fri, Apr 23, 2021 at 6:25 PM Nathaniel Smith <n...@pobox.com> wrote:
> > The main possibility that I don't think we've examined fully is to
> > make 'except' blocks fire multiple times when there are multiple
> > exceptions.
>
> Vanilla except blocks? Not sure if I'm misunderstanding, but that
> could cause some problems. Consider:
>
> trn.begin()
> try:
> ...
> except BaseException:
> trn.rollback()
> raise
> else:
> trn.commit()
>
> What happens if multiple exceptions get raised? Will the transaction
> be rolled back more than once? What gets reraised?
>
> If I'm completely misunderstanding you here, my apologies.
current proposal and the alternative have very complex implications
and downsides" :-)
A lot depends on the details, too. One possible design is "in general,
vanilla except blocks will trigger multiple times, but as a special
case, except: and except BaseException: will only fire once, receiving
the whole ExceptionGroup as a single exception". (Rationale for making
a special case: if you're saying "catch anything, I don't care what",
then you obviously didn't intend to do anything in particular with
those exceptions, plus this is the only case where the types work.) In
that design, your example becomes correct again.
Some other interesting cases:
# Filters out all OSError's that match the condition, while letting
all other OSError's continue
try:
...
except OSError as exc:
if exc.errno != errno.EWHATEVER:
raise
# If this gets an ExceptionGroup([ValueError, KeyboardInterrupt]),
then it logs the ValueError and
# then exits with the selected error code
try:
...
except Exception as exc:
logger.log_exception(exc)
except KeyboardInterrupt:
sys.exit(17)
OTOH you can still come up with cases that aren't handled correctly.
If you have old 'except' code mixed with new code raising
ExceptionGroup, then there's no way to make that do the right thing in
*all* cases. By definition, when the 'except' code was written, the
author wasn't thinking about ExceptionGroups! So the question is which
combination of semantics ends up doing the least harm on average
across all sorts of different real-world cases...
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/3V6YEKHOD5MYLSCPVDB7IYGX4EOO4KWB/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Chris Angelico
Apr 26, 2021, 3:37:37 AM4/26/21
to python-dev
On Mon, Apr 26, 2021 at 2:27 PM Nathaniel Smith <n...@pobox.com> wrote:
> Yeah, you've understood correctly, and you see why I wrote "both the
> current proposal and the alternative have very complex implications
> and downsides" :-)
>
> [chomp lots of very helpful summarizing]
> Yeah, you've understood correctly, and you see why I wrote "both the
> current proposal and the alternative have very complex implications
> and downsides" :-)
>
Gotcha, thanks!
ChrisA
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/LCQLKVQMBLHFX74WQD3Z5BIMYT5ARQFP/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Nathaniel Smith
Apr 28, 2021, 11:50:59 PM4/28/21
to Irit Katriel, python-dev, Yury Selivanov
On Fri, Apr 23, 2021 at 4:08 AM Irit Katriel <iritk...@googlemail.com> wrote:
>
> On Fri, Apr 23, 2021 at 9:22 AM Nathaniel Smith <n...@pobox.com> wrote:
>> I'm not trying to filibuster here -- I really want some form of EGs to
>> land.
>
> I'm very glad to hear that. It's been hard to know where you stand, because you didn't even decline our invitation in October to work together on this, and several later invitations to look at the implementation and try it with Trio -- you just didn't reply. The only responses I saw were public, on this list, taking us right back to the drawing board (for example - the suggestion you mention in the previous paragraph as not-sufficiently-explored, actually appears in the rejected ideas section of the PEP, and the reason for rejection is not that it's incompatible with nesting). So thank you for clarifying that you are, ultimately, supportive of our efforts.
Yes, I apologize again for the radio silence there -- I had real world
>
> On Fri, Apr 23, 2021 at 9:22 AM Nathaniel Smith <n...@pobox.com> wrote:
>> I'm not trying to filibuster here -- I really want some form of EGs to
>> land.
>
> I'm very glad to hear that. It's been hard to know where you stand, because you didn't even decline our invitation in October to work together on this, and several later invitations to look at the implementation and try it with Trio -- you just didn't reply. The only responses I saw were public, on this list, taking us right back to the drawing board (for example - the suggestion you mention in the previous paragraph as not-sufficiently-explored, actually appears in the rejected ideas section of the PEP, and the reason for rejection is not that it's incompatible with nesting). So thank you for clarifying that you are, ultimately, supportive of our efforts.
stuff that left me with no cope for open-source. I, uh, was feeling
guilty about not getting back to you the whole time, if that helps?
Probably that doesn't help.
My memory is that in our initial discussions, I suggested having each
'except Blah as exc' clause be executed once, receiving an
ExceptionGroup containing all the Blah exceptions. Guido pointed out
that this broke typing -- 'exc' would not longer have type 'Blah' --
and I was like... okay yeah that's a fatal flaw, never mind. And I
never seriously raised the 'execute a single clause multiple times'
option, because of the issue where in the nested design, taking
individual exceptions out of an ExceptionGroup breaks tracebacks.
Looking at the relevant section of the PEP again [1], it notes the
same fatal flaw with my first suggestion, and then says that the
multiple-except-executions option should be rejected because users
have written code like 'except SomeError: ...' with the expectation
that the 'except' clause would run exactly once. That's definitely
true, and it's a downside of the multiple-except-executions approach,
but I don't think it's convincing enough to rule this out on its own.
The problem is, *all* our options for how 'except' should interact
with ExceptionGroups will somehow break previous expectations.
Concretely: imagine you have a pre-existing 'except SomeError', and
some new code inside the 'try' block raises some number of
'SomeError's wrapped in an ExceptionGroup. There are three options:
- Execute the 'except' block multiple times. This breaks the
expectation that it should be executed at most once.
- Execute the 'except' block exactly once. But if there are multiple
SomeError's, this require they be grouped and delivered as a single
exception, which breaks typing.
- Execute the 'except' block zero times. This is what the current PEP
chooses, and breaks the expectation that 'except SomeError' should
catch 'SomeError'.
So we have to pick our poison.
[1] https://www.python.org/dev/peps/pep-0654/#extend-except-to-handle-exception-groups
> We do realize that we didn't adequately credit you for the contributions of your 2018 work to this PEP, and have now added an acknowledgements section for that purpose. Apologies for that.
was denying you credit for things that you had actually independently
re-invented.
> I'm confused about the flattening suggestion - above you talk about "flat EG", but below about tracebacks. It's not clear to me whether you want EG to be flat (ie no nesting of EGs) or just the traceback to be flat (but you can still have a nested EG).
In all my designs, the only reason I ever had nesting was because I
couldn't figure out any other way to make the tracebacks work. Do you
have some other motivation for wanting nesting? If so that would be
interesting, because it might point to why we're talking past each
other and help us understand the problem better...
> I also don't know what problem you are trying to solve with this.
(In my first message I explicitly said that it would be an improvement
over the status quo :-).) But I think that nesting will be really
counterintuitive/confusing for users in some ways. And concurrency
APIs will be offputting if they force you to use a different special
form of 'except' all the time. Basically the 'flat' version might be a
lot more ergonomic, and that's important for a language like Python.
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/D6G7Z4QVC5H6UMHOFR33FMVCRV4HL6SM/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Yury Selivanov
Apr 29, 2021, 12:21:14 PM4/29/21
to Nathaniel Smith, Irit Katriel, python-dev, Yury Selivanov
On Wed, Apr 28, 2021 at 8:53 PM Nathaniel Smith <n...@pobox.com> wrote:
> Looking at the relevant section of the PEP again [1], it notes the
> same fatal flaw with my first suggestion, and then says that the
> multiple-except-executions option should be rejected because users
> have written code like 'except SomeError: ...' with the expectation
> that the 'except' clause would run exactly once. That's definitely
> true, and it's a downside of the multiple-except-executions approach,
> but I don't think it's convincing enough to rule this out on its own.
> The problem is, *all* our options for how 'except' should interact
> with ExceptionGroups will somehow break previous expectations.
Well, this is where we respectfully disagree. The case of changing how
> Looking at the relevant section of the PEP again [1], it notes the
> same fatal flaw with my first suggestion, and then says that the
> multiple-except-executions option should be rejected because users
> have written code like 'except SomeError: ...' with the expectation
> that the 'except' clause would run exactly once. That's definitely
> true, and it's a downside of the multiple-except-executions approach,
> but I don't think it's convincing enough to rule this out on its own.
> The problem is, *all* our options for how 'except' should interact
> with ExceptionGroups will somehow break previous expectations.
the regular try..except works in a backwards incompatible way is a
very convincing blocker to us.
>
> Concretely: imagine you have a pre-existing 'except SomeError', and
> some new code inside the 'try' block raises some number of
> 'SomeError's wrapped in an ExceptionGroup. There are three options:
>
> - Execute the 'except' block multiple times. This breaks the
> expectation that it should be executed at most once.
> - Execute the 'except' block exactly once. But if there are multiple
> SomeError's, this require they be grouped and delivered as a single
> exception, which breaks typing.
> - Execute the 'except' block zero times. This is what the current PEP
> chooses, and breaks the expectation that 'except SomeError' should
> catch 'SomeError'.
>
> So we have to pick our poison.
> > I'm confused about the flattening suggestion - above you talk about "flat EG", but below about tracebacks. It's not clear to me whether you want EG to be flat (ie no nesting of EGs) or just the traceback to be flat (but you can still have a nested EG).
>
> Hmm, I was thinking about making both of them flat, so no nested EGs.
> In all my designs, the only reason I ever had nesting was because I
> couldn't figure out any other way to make the tracebacks work. Do you
> have some other motivation for wanting nesting? If so that would be
> interesting, because it might point to why we're talking past each
> other and help us understand the problem better...
>
> > I also don't know what problem you are trying to solve with this.
>
> I'm not saying that there's some fatal problem with the current PEP.
> (In my first message I explicitly said that it would be an improvement
> over the status quo :-).) But I think that nesting will be really
> counterintuitive/confusing for users in some ways. And concurrency
> APIs will be offputting if they force you to use a different special
> form of 'except' all the time. Basically the 'flat' version might be a
> lot more ergonomic, and that's important for a language like Python.
"important for a language like Python". The problem is that after all
these emails I still have absolutely no idea about:
- what exactly are you trying to propose?
- what specific problem do you want to address? (that the current PEP,
in your opinion, does not address)
- why do you think that what you propose would be more ergonomic or simple?
- what "flat EGs" or "flat tracebacks" even mean; I can't figure out
not only the high level API, I don't even understand what you're
talking about at the data structures level.
Nathaniel, at this point it's clear that this thread somehow does not
help us understand what you want. Could you please just write your own
PEP clearly outlining your proposal, its upsides and downsides?
Without a PEP from you this thread is just a distraction.
Yury
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/AR737RYK2KCTATCBW3QSDK5SSF7CH75M/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Jim J. Jewett
Apr 30, 2021, 4:30:39 PM4/30/21
to pytho...@python.org
Nathaniel Smith (in a conversation with Irit) wrote:
> ... suggested having each
> never seriously raised the 'execute a single clause multiple times'
> option, because of the issue where in the nested design, taking
> individual exceptions out of an ExceptionGroup breaks tracebacks.
This also seems like something that isn't hard to work around. The obvious way would involve shared (parts of) tracebacks, but even that isn't required.
> multiple-except-executions option should be rejected because users
> have written code like 'except SomeError: ...' with the expectation
> that the 'except' clause would run exactly once.
That is indeed a problem, because some error handlers would do bad things if rerun.
> The problem is, *all* our options for how 'except' should interact
> with ExceptionGroups will somehow break previous expectations.
agreed.
> Concretely: imagine you have a pre-existing 'except SomeError', and
> some new code inside the 'try' block raises some number of
> 'SomeError's wrapped in an ExceptionGroup. There are three options:
> - Execute the 'except' block multiple times. This breaks the
> expectation that it should be executed at most once.
Do we know how bad this really is? A second rollback or cancel or endConnection won't do what is expected, but they normally won't do much harm either. A second logError does the right thing.
> - Execute the 'except' block exactly once. But if there are multiple
> SomeError's, this require they be grouped and delivered as a single
> exception, which breaks typing.
I'm not worried about typing, but I am worried about silently dropping all but the "first" SomeError.
On the other hand, in practice were they all (including the "first") dropped before, in favor of some sort of superseding asyncCancelledError?
> - Execute the 'except' block zero times. This is what the current PEP
> chooses, and breaks the expectation that 'except SomeError' should
> catch 'SomeError'.
To me, this seems the worst of the three options, but ... maybe it is effectively the status quo often enough that it becomes the least bad?
> > I'm confused about the flattening suggestion - above you talk about "flat EG", but below about tracebacks. It's not clear to me whether you want EG to be flat (ie no nesting of EGs) or just the traceback to be flat (but you can still have a nested EG).
> > Hmm, I was thinking about making both of them flat, so no nested EGs.
> In all my designs, the only reason I ever had nesting was because I
> couldn't figure out any other way to make the tracebacks work. Do you
> have some other motivation for wanting nesting? If so that would be
> interesting, because it might point to why we're talking past each
> other and help us understand the problem better...
When I view this as strictly a typing problem, it matters which exceptions got joined at which junction; the shape of the tree is part of the type.
When I view this as a production support programmer, I really don't care about that ... I only care what triggered each SomeException so that I can account for all the bad data instead of just one piece. Having to dig through multiple layers of grouping to get to each original SomeException separately is just an annoyance that *will* cause juniors to sometimes miss part of the bad data. ("Open the door and check for problems" is a lot easier to learn and remember than a recursive "Open the door and check for problems or other doors, and then repeat on each door you find.")
-jJ
> ... suggested having each
> 'except Blah as exc' clause be executed once, receiving an
> ExceptionGroup containing all the Blah exceptions. Guido pointed out
> that this broke typing -- 'exc' would not longer have type 'Blah' --
> and I was like... okay yeah that's a fatal flaw, never mind.
The fact that "Blah as exc" could return either a single Blah or a collection of Blahs would be annoying, but not fatal -- that seems well within the level of ambiguity typing can deal with.
> ExceptionGroup containing all the Blah exceptions. Guido pointed out
> that this broke typing -- 'exc' would not longer have type 'Blah' --
> and I was like... okay yeah that's a fatal flaw, never mind.
> never seriously raised the 'execute a single clause multiple times'
> option, because of the issue where in the nested design, taking
> individual exceptions out of an ExceptionGroup breaks tracebacks.
> multiple-except-executions option should be rejected because users
> have written code like 'except SomeError: ...' with the expectation
> that the 'except' clause would run exactly once.
> The problem is, *all* our options for how 'except' should interact
> with ExceptionGroups will somehow break previous expectations.
> Concretely: imagine you have a pre-existing 'except SomeError', and
> some new code inside the 'try' block raises some number of
> 'SomeError's wrapped in an ExceptionGroup. There are three options:
> - Execute the 'except' block multiple times. This breaks the
> expectation that it should be executed at most once.
> - Execute the 'except' block exactly once. But if there are multiple
> SomeError's, this require they be grouped and delivered as a single
> exception, which breaks typing.
On the other hand, in practice were they all (including the "first") dropped before, in favor of some sort of superseding asyncCancelledError?
> - Execute the 'except' block zero times. This is what the current PEP
> chooses, and breaks the expectation that 'except SomeError' should
> catch 'SomeError'.
> > I'm confused about the flattening suggestion - above you talk about "flat EG", but below about tracebacks. It's not clear to me whether you want EG to be flat (ie no nesting of EGs) or just the traceback to be flat (but you can still have a nested EG).
> > Hmm, I was thinking about making both of them flat, so no nested EGs.
> In all my designs, the only reason I ever had nesting was because I
> couldn't figure out any other way to make the tracebacks work. Do you
> have some other motivation for wanting nesting? If so that would be
> interesting, because it might point to why we're talking past each
> other and help us understand the problem better...
When I view this as a production support programmer, I really don't care about that ... I only care what triggered each SomeException so that I can account for all the bad data instead of just one piece. Having to dig through multiple layers of grouping to get to each original SomeException separately is just an annoyance that *will* cause juniors to sometimes miss part of the bad data. ("Open the door and check for problems" is a lot easier to learn and remember than a recursive "Open the door and check for problems or other doors, and then repeat on each door you find.")
-jJ
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/VO5I7DMRYAB7EPUYSYEHHQFLGZAYWUB4/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Jeff Allen
May 2, 2021, 5:13:02 AM5/2/21
to pytho...@python.org
Yes, I got one from the same address today. Thanks for pointing out these are individual peformances: it was annoying when I thought it was spam to the list.
Although Hoi Lam Poon is a real (female) name, it may signify a
generated lampoon.
Jeff Allen
Jonathan Goble
May 4, 2021, 12:37:13 PM5/4/21
to Jeff Allen, Python Dev
I just got one now from the same person on the dependabot thread. Entirely in Chinese except for "GNU license" in the middle, along with an attached jpeg of a random SciPy-related tweet (according to Gmail's preview thumbnail; I didn't actually open the attachment for obvious reasons).
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/3EA53I3Y72ACEPDVG467NMNTXHRL3NXL/
Tim Peters
May 4, 2021, 2:12:34 PM5/4/21
to Jonathan Goble, Python Dev
FYI, I just force-unsubscribed this member (Hoi Lam Poon) from
python-dev. Normally I don't do things like that, since, e.g, we have
no way to know whether the sender address was spoofed in emails we
get. But in this case Hoi's name has come up several times as the
sender of individual spam, and if Hoi is actually a legit member they
should have noticed and spoken up.
Of course this can't stop them from spamming. Just intended to
increase the effort for them, if they're determined to continue.
[nothing new below[
> Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/JNYRFZ4FTIFUJFS7FPNMKI2HXW7SLL52/
python-dev. Normally I don't do things like that, since, e.g, we have
no way to know whether the sender address was spoofed in emails we
get. But in this case Hoi's name has come up several times as the
sender of individual spam, and if Hoi is actually a legit member they
should have noticed and spoken up.
Of course this can't stop them from spamming. Just intended to
increase the effort for them, if they're determined to continue.
[nothing new below[
> Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/LFULZM6GLDUAS5SFQ4F454QMRGEH3EGH/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Carol Willing
May 4, 2021, 2:37:49 PM5/4/21
to Tim Peters, Jonathan Goble, Python Dev
Thanks Tim.
Nathaniel Smith
May 6, 2021, 5:22:05 AM5/6/21
to Yury Selivanov, Irit Katriel, python-dev, Yury Selivanov
On Thu, Apr 29, 2021 at 9:14 AM Yury Selivanov <yseliv...@gmail.com> wrote:
> Nathaniel, at this point it's clear that this thread somehow does not
> help us understand what you want. Could you please just write your own
> PEP clearly outlining your proposal, its upsides and downsides?
> Without a PEP from you this thread is just a distraction.
If that's the best way to move forward, then ok. My main thing is just
> Nathaniel, at this point it's clear that this thread somehow does not
> help us understand what you want. Could you please just write your own
> PEP clearly outlining your proposal, its upsides and downsides?
> Without a PEP from you this thread is just a distraction.
that I don't want to make this some antagonistic me-vs-you thing.
After all, we all want the best design to be chosen, and none of us
know what that is yet, so there's no need for conflict :-).
Irit, Yury, would you be interested in co-authoring a PEP for the
"flat EG" approach? Basically trying to set down the best possible
version of each approach, so that we can put them next to each other?
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/LZBM5F6SAULEN7KEYNHBCLUPTB4JHBGO/
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Nathaniel Smith
May 6, 2021, 5:22:45 AM5/6/21
to Yury Selivanov, Irit Katriel, python-dev, Yury Selivanov
On Thu, May 6, 2021 at 2:17 AM Nathaniel Smith <n...@pobox.com> wrote:
>
> On Thu, Apr 29, 2021 at 9:14 AM Yury Selivanov <yseliv...@gmail.com> wrote:
> > Nathaniel, at this point it's clear that this thread somehow does not
> > help us understand what you want. Could you please just write your own
> > PEP clearly outlining your proposal, its upsides and downsides?
> > Without a PEP from you this thread is just a distraction.
>
> If that's the best way to move forward, then ok. My main thing is just
> that I don't want to make this some antagonistic me-vs-you thing.
> After all, we all want the best design to be chosen, and none of us
> know what that is yet, so there's no need for conflict :-).
>
> Irit, Yury, would you be interested in co-authoring a PEP for the
> "flat EG" approach? Basically trying to set down the best possible
> version of each approach, so that we can put them next to each other?
Uh, probably this is obvious, but I mean "co-authoring with me". I'm
>
> On Thu, Apr 29, 2021 at 9:14 AM Yury Selivanov <yseliv...@gmail.com> wrote:
> > Nathaniel, at this point it's clear that this thread somehow does not
> > help us understand what you want. Could you please just write your own
> > PEP clearly outlining your proposal, its upsides and downsides?
> > Without a PEP from you this thread is just a distraction.
>
> If that's the best way to move forward, then ok. My main thing is just
> that I don't want to make this some antagonistic me-vs-you thing.
> After all, we all want the best design to be chosen, and none of us
> know what that is yet, so there's no need for conflict :-).
>
> Irit, Yury, would you be interested in co-authoring a PEP for the
> "flat EG" approach? Basically trying to set down the best possible
> version of each approach, so that we can put them next to each other?
not suggesting you two go off to do it without me :-)
--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Irit Katriel via Python-Dev
May 6, 2021, 6:32:26 AM5/6/21
to Nathaniel Smith, python-dev, Yury Selivanov
Hi Nathaniel,
Philosophy of Science 101: nobody knows what TheBestSolution is and nobody ever will.
That said, when we decided to submit PEP 654, that was a point in time when we came to believe that we found the right solution. We made significant changes later following feedback we agreed with, but in the case of "flat EG" I see several disadvantages and don't quite understand what the win is. I appreciate the invitation, but I don't want to work on that idea (nor am I the right person to push it forward).
However, I think it's fantastic if you and others do pursue alternative options (independently of my biases). You will either find something better or validate PEP 654, and in either case the world wins. Feel free to adapt our implementation if that helps. You have a year before we have our next chance to commit to PEP 654.
Irit
Reply all
Reply to author
Forward
0 new messages