sentinel_exception argument to `iter`

[Ram Rachum]

Hi,

This time I'm posting on the right list :) Sorry for the mistake in the last thread.

`iter` has a very cool `sentinel` argument. I suggest an additional argument `sentinel_exception`; when it's supplied, instead of waiting for a sentinel value, we wait for a sentinel exception to be raised, and then the iteration is finished.

This'll be useful to construct things like this: 

    my_iterator = iter(my_deque.popleft, IndexError)

What do you think? 

Ram.

[Yury Selivanov]

Before starting discussion about the new parameter:
how passing 'my_deque.popleft' is supposed to work?

Yury
_______________________________________________
Python-ideas mailing list
Python...@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Ram Rachum]

Looks like someone needs to read the `iter` docs :)

--

--- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/UCaNfAHkBlQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Yury Selivanov]

My bad, never used it that way.
Since your propose a new parameter, it needs to be a keyword-only one,
like this:

iter(callback, sentinel_exception=IndexError)

Yury

>> python-ideas...@googlegroups.com.

>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>

[Yury Selivanov]

And, FWIW, take a look at itertools.takewhile.
Your particular example would look like

itertools.takewhile(lambda el:my_deque, my_deque)

On 2/6/2014, 7:17 PM, Ram Rachum wrote:

>> python-ideas...@googlegroups.com.

>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>

[Terry Reedy]

On 2/6/2014 7:10 PM, Ram Rachum wrote:

> `iter` has a very cool `sentinel` argument. I suggest an additional
> argument `sentinel_exception`; when it's supplied, instead of waiting
> for a sentinel value, we wait for a sentinel exception to be raised, and
> then the iteration is finished.
>
> This'll be useful to construct things like this:
>
> my_iterator = iter(my_deque.popleft, IndexError)
>
> What do you think?

I think this would be a great idea if simplified to reuse the current
parameter. It can work in Python because exceptions are objects like
anything else and can be passed as arguments. No new parameter is
needed. We only need to add 'or raises' to the two-parameter iter definition
"In the second form, the callable is called until it returns the sentinel."
to get
"In the second form, the callable is called until it returns or raises
the sentinel."

With this version of the proposal, your pop example would work as written.

The two-parameter form return a 'callable_iterator' object. Its __next__
method in Python might look like (untested)

def __next__(self):
x = self.func()
if x == self.sentinel:
raise StopIteration
else:
return x

The only change needed for the added behavior is to wrap the function
call and change an exception matching the sentinel to StopIteration.

def __next__(self):
try:
x = self.func()
except Exception as exc:
if isinstance(exc, self.sentinel):
raise StopIteration from None
if x == self.sentinel:
raise StopIteration
else:
return x

I do something similar to this in my custom function tester and it is
really handy to be able to pass in an expected exception as the expected
'output'. For example, my 'test table' for an int divide function might
contain these input and expected output pairs:
(2,0), ZeroDevisionError
(2,1), 2
(2,2), 1.

I consider the threat to backward compatibility, because of the added
test for exceptions, theoretical rather than actual. It is very rare to
write a function that returns an exception, rarer still to write one
that can also raise an instance of the same exception class. Also, using
an exception-returning function in two-parameter iter is tricky because
exceptions inherit the default equality comparison of only equal by
identity.

>>> ValueError('high') == ValueError('high')
False
>>> e=ValueError('high')
>>> e == e
True

The following tested example of code that would be broken by the
proposal could well be the first ever written. Warning: it is so twisted
and awful that it might to read it.

---
from random import random

e=ValueError('high value')

def crazy():
x = random()
if x > .999:
return e
elif x < .001:
raise ValueError('low value')
else:
return x

try:
for x in iter(crazy, e):
pass
else:
print(e.args[0])
except ValueError as exc:
print(exc.args[0])
#
prints 'high value' or 'low value' in less than a second

--
Terry Jan Reedy

[Chris Angelico]

On Fri, Feb 7, 2014 at 1:36 PM, Terry Reedy <tjr...@udel.edu> wrote:
> def __next__(self):
> try:
> x = self.func()
> except Exception as exc:
> if isinstance(exc, self.sentinel):
> raise StopIteration from None
> if x == self.sentinel:
> raise StopIteration
> else:
> return x

Forgive me if this is a stupid question, but wouldn't this suppress
any other thrown exception? I'm looking for a bare 'raise' inside the
except block, such that any exception is guaranteed to raise
something, but if it's a subclass of self.sentinel, it raises
StopIteration instead.

ChrisA

[Yury Selivanov]

On 2/6/2014, 9:36 PM, Terry Reedy wrote:
> I think this would be a great idea if simplified to reuse the current
> parameter. It can work in Python because exceptions are objects like
> anything else and can be passed as arguments. No new parameter is needed.

What will the following code print:

d = deque((42, IndexError, 'spam'))
print(list(iter(d.popleft, IndexError)))

?

Yury

[Chris Angelico]

On Fri, Feb 7, 2014 at 2:01 PM, Yury Selivanov <yseliv...@gmail.com> wrote:
>
> On 2/6/2014, 9:36 PM, Terry Reedy wrote:
>>
>> I think this would be a great idea if simplified to reuse the current
>> parameter. It can work in Python because exceptions are objects like
>> anything else and can be passed as arguments. No new parameter is needed.
>
> What will the following code print:
>
> d = deque((42, IndexError, 'spam'))
> print(list(iter(d.popleft, IndexError)))
>
> ?

Presumably it would stop once it reaches the IndexError, exactly per
current behaviour, and print [42]. So you can't depend on it actually
having caught IndexError, any more than you can depend on it actually
having found the element:

>>> def foo():
global idx
idx+=1
if idx==3: raise StopIteration()
return idx*10
>>> idx=0
>>> print(list(iter(foo, 20)))
[10]
>>> idx=0
>>> print(list(iter(foo, 50)))
[10, 20]

It'll stop at any of three conditions: a raised StopIteration from the
function, a returned value equal to the second argument, or a raised
exception that's a subclass of the second argument. All three will be
folded into the same result: StopIteration.

ChrisA

[Terry Reedy]

On 2/6/2014 9:45 PM, Chris Angelico wrote:
> On Fri, Feb 7, 2014 at 1:36 PM, Terry Reedy <tjr...@udel.edu> wrote:

>> def __next__(self):
>> try:
>> x = self.func()
>> except Exception as exc:
>> if isinstance(exc, self.sentinel):
>> raise StopIteration from None

else:
raise

>> if x == self.sentinel:
>> raise StopIteration
>> else:
>> return x
>
> Forgive me if this is a stupid question, but wouldn't this suppress
> any other thrown exception? I'm looking for a bare 'raise' inside the
> except block,

which should be there. As I said, but you snipped, '(untested)'

> such that any exception is guaranteed to raise
> something, but if it's a subclass of self.sentinel, it raises
> StopIteration instead.

--
Terry Jan Reedy

[Chris Angelico]

On Fri, Feb 7, 2014 at 3:15 PM, Terry Reedy <tjr...@udel.edu> wrote:
>> Forgive me if this is a stupid question, but wouldn't this suppress
>> any other thrown exception? I'm looking for a bare 'raise' inside the
>> except block,
>
>
> which should be there. As I said, but you snipped, '(untested)'

Ah okay. Makes sense :) Wasn't sure if it was intentional, or if there
was some other magic that would do what was wanted.

ChrisA

[Terry Reedy]

On 2/6/2014 11:15 PM, Terry Reedy wrote:
>> On Fri, Feb 7, 2014 at 1:36 PM, Terry Reedy

>>> def __next__(self):
>>> try:
>>> x = self.func()
>>> except Exception as exc:
>>> if isinstance(exc, self.sentinel):
>>> raise StopIteration from None
> else:
> raise

I just realized that the above is unnecessarily complicated because the
expression that follows 'except' is not limited to a builtin exception
class name or tuple thereof. (I have never before had reason to
dynamically determine the exception to be caught.) So, using a third
parameter, replace the 5 lines with 2.

except self.stop_exception:
raise StopIteration from None

>>> if x == self.sentinel:
>>> raise StopIteration
>>> else:
>>> return x

[Terry Reedy]

On 2/6/2014 10:01 PM, Yury Selivanov wrote:
>
> On 2/6/2014, 9:36 PM, Terry Reedy wrote:
>> I think this would be a great idea if simplified to reuse the current
>> parameter. It can work in Python because exceptions are objects like
>> anything else and can be passed as arguments. No new parameter is needed.
> What will the following code print:
>
> d = deque((42, IndexError, 'spam'))
> print(list(iter(d.popleft, IndexError)))

As Chris said, 42. To change current behavior before the function raises
an exception, the comparison of each function return to the sentinel
would have to be changed (or eliminated). My proposal does not do that.
It only changes behavior when there is an exception and iter has been
told that an exception is to be treated as having the same 'I am
finished' meaning as StopIteration.

As you showed, it is easy to construct callables that might return an
exception before raising it from finite collections with a prev or next
method (whether destructive or not).

>>> list(iter(['spam', IndexError, 42].pop, IndexError))
[42]
>>> list(iter({'spam', KeyError, 42}.pop, KeyError))
[42, 'spam'] # or [42] or ['spam'], depending on hashing

For these example, I guess Ram is right in suggesting a 3rd parameter. I
would, however, just call it something like 'exception' or 'stop_iter'.
That would make the description

"In the second form, the callable is called until it returns the

sentinel or raises an instance of stop_iter"

If sentinel is omitted, then the callable is iterated until 'completion'

The signature is a bit awkward because 'sentinel' is positional-only and
optional without a default. The C code must use the equivalent of *args
and switch on the number of args passed. So the new param would probably
have to be keyword-only. I remain in favor of the proposal.

--
Terry Jan Reedy

[Chris Angelico]

On Fri, Feb 7, 2014 at 5:05 PM, Terry Reedy <tjr...@udel.edu> wrote:
> As you showed, it is easy to construct callables that might return an
> exception before raising it from finite collections with a prev or next
> method (whether destructive or not).
>
>>>> list(iter(['spam', IndexError, 42].pop, IndexError))
> [42]
>>>> list(iter({'spam', KeyError, 42}.pop, KeyError))
> [42, 'spam'] # or [42] or ['spam'], depending on hashing
>
> For these example, I guess Ram is right in suggesting a 3rd parameter. I
> would, however, just call it something like 'exception' or 'stop_iter'.
> That would make the description

I honestly wouldn't worry. How often, in production code, will you
iterate over something that might return an exception, AND be testing
for the raising of that same exception? Or the converse - how often
would you be in a position to raise the thing you might want to
return, and be annoyed that the raised exception gets squashed into
StopIteration? Don't complicate a nice simple API for the sake of
that. If you're concerned about security implications of someone
manipulating the return values and chopping something off, then just
write your own generator:

def safe_iter(func, sentinel):
try:
while True:
yield func()
except sentinel:
pass

This will stop when the exception's raised, but not when it's
returned. The unusual case can be covered with so little code that the
API can afford to ignore it, imo.

ChrisA

[Andrew Barnert]

On Feb 6, 2014, at 22:03, Terry Reedy <tjr...@udel.edu> wrote:

> On 2/6/2014 11:15 PM, Terry Reedy wrote:
>>> On Fri, Feb 7, 2014 at 1:36 PM, Terry Reedy
>
>>>> def __next__(self):
>>>> try:
>>>> x = self.func()
>>>> except Exception as exc:
>>>> if isinstance(exc, self.sentinel):
>>>> raise StopIteration from None
>> else:
>> raise
>
> I just realized that the above is unnecessarily complicated because the expression that follows 'except' is not limited to a builtin exception class name or tuple thereof. (I have never before had reason to dynamically determine the exception to be caught.) So, using a third parameter, replace the 5 lines with 2.
>
> except self.stop_exception:
> raise StopIteration from None

Except that you don't have a stop_exception, you have a sentinel, which can be either an object or an exception type.

I'm actually not sure whether it's legal to use, say, 0 or "" as the except expression. In recent 3.4 builds, it seems to be accepted, and to never catch anything. So, if that's guaranteed by the language, it's just a simple typo to fix and your simplified implementation works perfectly.

[Andrew Barnert]

On Feb 6, 2014, at 22:52, Andrew Barnert <abar...@yahoo.com> wrote:

> On Feb 6, 2014, at 22:03, Terry Reedy <tjr...@udel.edu> wrote:
>
>> On 2/6/2014 11:15 PM, Terry Reedy wrote:
>>>> On Fri, Feb 7, 2014 at 1:36 PM, Terry Reedy
>>
>>>>> def __next__(self):
>>>>> try:
>>>>> x = self.func()
>>>>> except Exception as exc:
>>>>> if isinstance(exc, self.sentinel):
>>>>> raise StopIteration from None
>>> else:
>>> raise
>>
>> I just realized that the above is unnecessarily complicated because the expression that follows 'except' is not limited to a builtin exception class name or tuple thereof. (I have never before had reason to dynamically determine the exception to be caught.) So, using a third parameter, replace the 5 lines with 2.
>>
>> except self.stop_exception:
>> raise StopIteration from None
>
> Except that you don't have a stop_exception, you have a sentinel, which can be either an object or an exception type.
>
> I'm actually not sure whether it's legal to use, say, 0 or "" as the except expression. In recent 3.4 builds, it seems to be accepted, and to never catch anything. So, if that's guaranteed by the language, it's just a simple typo to fix and your simplified implementation works perfectly.

Reading the docs, it seems like it ought to be ok. In 8.4, it just says:

'For an except clause with an expression, the expression is evaluated, and the clause matches the exception if the resulting object is "compatible" with the exception. An object is compatible with an exception if it is the class or a base class of the exception object or a tuple containing an item compatible with the exception.'

So, it seems like 0 is a perfectly valid except expression, which can be checked for compatibility with any exception and will never match. Which is perfect.

[Chris Angelico]

On Fri, Feb 7, 2014 at 5:52 PM, Andrew Barnert <abar...@yahoo.com> wrote:
> I'm actually not sure whether it's legal to use, say, 0 or "" as the except expression. In recent 3.4 builds, it seems to be accepted, and to never catch anything. So, if that's guaranteed by the language, it's just a simple typo to fix and your simplified implementation works perfectly.
>

In 3.4b2:

>>> def f():
raise StopIteration

>>> try:
f()
except "":
print("Blank exception caught")

Traceback (most recent call last):
File "<pyshell#693>", line 2, in <module>
f()
File "<pyshell#691>", line 2, in f
raise StopIteration
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<pyshell#693>", line 3, in <module>
except "":
TypeError: catching classes that do not inherit from BaseException is
not allowed


It doesn't bomb until something gets raised. Is that changed in a
newer build? (This is the newest I have on here.)

ChrisA

[Steven D'Aprano]

On Thu, Feb 06, 2014 at 09:36:19PM -0500, Terry Reedy wrote:
> On 2/6/2014 7:10 PM, Ram Rachum wrote:
>
> >`iter` has a very cool `sentinel` argument. I suggest an additional
> >argument `sentinel_exception`; when it's supplied, instead of waiting
> >for a sentinel value, we wait for a sentinel exception to be raised, and
> >then the iteration is finished.
> >
> >This'll be useful to construct things like this:
> >
> > my_iterator = iter(my_deque.popleft, IndexError)
> >
> >What do you think?
>
> I think this would be a great idea if simplified to reuse the current
> parameter.

That would be a backwards-incompatible change, for exactly the reason
you give below:

> It can work in Python because exceptions are objects like
> anything else and can be passed as arguments.

Right. And there is a big difference between *returning* an exception
and *raising* an exception, which is why a new parameter (or a new
function) is required. A function might legitimately return exception
objects for some reason:

exceptions_to_be_tested = iter(
[IndexError(msg), ValueError, StopIteration, TypeError]
)

def func():
# pre- or post-processing might happen
return next(it)


for exception in iter(func, StopIteration):
# assume the exceptions are caught elsewhere
raise exception


With the current behaviour, that will raise IndexError and ValueError,
then stop. With the suggested change in behaviour, it will raise all
four exceptions.

We cannot assume that an exception is never a legitimate return result
from the callable. "Iterate until this exception is raised" and "iterate
until this value is returned" are very different things and it is folly
to treat them as the same.



[...]

> I consider the threat to backward compatibility, because of the added
> test for exceptions, theoretical rather than actual. It is very rare to
> write a function that returns an exception,

Rare or not, I've done it, it's allowed by the language, and it is
inappropriate to conflate returning a class or instance with raising an
exception.

It doesn't matter whether it is rare. It is rare to write:

iter(func, ({}, {}))

nevertheless it would be poor design to have iter treat tuples of
exactly two dicts as a special case.

Exceptions are first-class values like strings, ints, and tuples
containing exactly two dicts. They should be treated exactly the same as
any other first-class value.



--
Steven

[Terry Reedy]

On 2/7/2014 1:52 AM, Andrew Barnert wrote:
> On Feb 6, 2014, at 22:03, Terry Reedy <tjr...@udel.edu> wrote:
>
>> On 2/6/2014 11:15 PM, Terry Reedy wrote:
>>>> On Fri, Feb 7, 2014 at 1:36 PM, Terry Reedy
>>
>>>>> def __next__(self):
>>>>> try:
>>>>> x = self.func()
>>>>> except Exception as exc:
>>>>> if isinstance(exc, self.sentinel):
>>>>> raise StopIteration from None
>>> else:
>>> raise
>>
>> I just realized that the above is unnecessarily complicated because the expression that follows 'except' is not limited to a builtin exception class name or tuple thereof. (I have never before had reason to dynamically determine the exception to be caught.) So, using a third parameter, replace the 5 lines with 2.
>>
>> except self.stop_exception:
>> raise StopIteration from None
>
> Except that you don't have a stop_exception, you have a sentinel, which can be either an object or an exception type.

I wrote the above with the idea that there would be a third parameter to
provide an exception. It would have to have a private default like

class _NoException(Exception): pass

> I'm actually not sure whether it's legal to use, say, 0 or "" as the except expression. In recent 3.4 builds, it seems to be accepted, and to never catch anything. So, if that's guaranteed by the language, it's just a simple typo to fix and your simplified implementation works perfectly.

Easy to test. The result is that a non-exception expression is
syntactically legal, but not when an exception occurs.

>>> try: 3
except 1: 4

3
>>> try: 1/0
except 1: 4

Traceback (most recent call last):

File "<pyshell#56>", line 1, in <module>
try: 1/0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "<pyshell#56>", line 2, in <module>
except 1: 4

TypeError: catching classes that do not inherit from BaseException is
not allowed

[Steven D'Aprano]

On Fri, Feb 07, 2014 at 05:41:32PM +1100, Chris Angelico wrote:

> I honestly wouldn't worry. How often, in production code, will you
> iterate over something that might return an exception, AND be testing
> for the raising of that same exception? Or the converse - how often
> would you be in a position to raise the thing you might want to
> return, and be annoyed that the raised exception gets squashed into
> StopIteration?

Doesn't matter how rare it is. Doesn't matter if nobody has ever written
production that does it. Or for that matter, if nobody has ever written
*non-production* code that does it. "Special cases aren't special enough
to break the rules."

Exceptions are values just like strings and lists and floats. Probably
nobody has ever written this particular piece of code:

iter(func, "NOBODY expects the Spanish Inquisition!!!")

but would you think it is okay to treat that particular sentinel value
differently from every other sentinel value?

> Don't complicate a nice simple API for the sake of
> that.

Your idea of simple is not the same as mine. The current behaviour

"the callable is called until it returns the sentinel"

is simpler than the proposed behaviour:

"the callable is called until it returns the sentinel, unless the
sentinel is an exception instance or class, in which case the
callable is called until it raises that exception, or one which is
compatible with it"


This is an ugly API that violates the Zen of Python. I think that is why
Raymond has listed this as a recipe rather than modify the function to
behave as suggested:

http://code.activestate.com/recipes/577155


--
Steven

[Terry Reedy]

On 2/7/2014 1:59 AM, Chris Angelico wrote:
> On Fri, Feb 7, 2014 at 5:52 PM, Andrew Barnert <abar...@yahoo.com> wrote:
>> I'm actually not sure whether it's legal to use, say, 0 or "" as the except expression. In recent 3.4 builds, it seems to be accepted, and to never catch anything. So, if that's guaranteed by the language, it's just a simple typo to fix and your simplified implementation works perfectly.
>>
>
> In 3.4b2:
>
>>>> def f():
> raise StopIteration
>
>>>> try:
> f()
> except "":
> print("Blank exception caught")
>
> Traceback (most recent call last):
> File "<pyshell#693>", line 2, in <module>
> f()
> File "<pyshell#691>", line 2, in f
> raise StopIteration
> StopIteration
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "<pyshell#693>", line 3, in <module>
> except "":
> TypeError: catching classes that do not inherit from BaseException is
> not allowed
>
>
> It doesn't bomb until something gets raised. Is that changed in a
> newer build? (This is the newest I have on here.)

As of a week ago, no.



--
Terry Jan Reedy

[Terry Reedy]

On 2/7/2014 3:28 AM, Steven D'Aprano wrote:
> On Thu, Feb 06, 2014 at 09:36:19PM -0500, Terry Reedy wrote:
>> On 2/6/2014 7:10 PM, Ram Rachum wrote:
>>
>>> `iter` has a very cool `sentinel` argument. I suggest an additional
>>> argument `sentinel_exception`; when it's supplied, instead of waiting
>>> for a sentinel value, we wait for a sentinel exception to be raised, and
>>> then the iteration is finished.
>>>
>>> This'll be useful to construct things like this:
>>>
>>> my_iterator = iter(my_deque.popleft, IndexError)
>>>
>>> What do you think?
>>
>> I think this would be a great idea if simplified to reuse the current
>> parameter.
>
> That would be a backwards-incompatible change, for exactly the reason
> you give below:

In a later message, I reversed myself, in spite of the actual C
signature making it a bit messy. Although help says 'iter(callable,
sentinel)', the actual signature is iter(*args), with any attempt to
pass an arg by keyword raising
TypeError: iter() takes no keyword arguments

Let n = len(args). Then the code switches as follows:
n=0: TypeError: iter expected at least 1 arguments, got 0
n>2: TypeError: iter expected at most 2 arguments, got 3
n=1: Return __iter__ or __getitem__ wrapper.
n=2: if callable(args[0]): return callable_iterator(*args),
else: TypeError: iter(v, w): v must be callable

If the current signature were merely extended, then I believe the new
signature would have to be (if possible)
iter(*args, *, stop_iter=<private exception>)
But having parameter args[1] (sentinel) be position-only, with no
default, while added parameter stop_iter is keyword only, with a
(private) default, would be a bit weird.

So instead I would suggest making the new signature be
iter(iter_or_call, sentinel=<private object>, stop_iter=<private
exception>). If sentinel and stop_iter are both default, use current n=1
code, else pass all 3 args to modified callable_iterator that compares
sentinel to return values and catches stop_iter exceptions.

Either way, the user could choose to only stop on a return value, only
stop on an exception, or stop on either with the two values not having
to be the same. The only thing that would break is code that depends on
a TypeError, but we allow ourselves to do that to extend functions.

>> It can work in Python because exceptions are objects like
>> anything else and can be passed as arguments.
>
> Right. And there is a big difference between *returning* an exception
> and *raising* an exception, which is why a new parameter (or a new
> function) is required. A function might legitimately return exception
> objects for some reason:
>
> exceptions_to_be_tested = iter(
> [IndexError(msg), ValueError, StopIteration, TypeError]
> )
>
> def func():
> # pre- or post-processing might happen
> return next(it)

Did you mean next(exceptions_to_be_tested)

> for exception in iter(func, StopIteration):
> # assume the exceptions are caught elsewhere
> raise exception
>
>
> With the current behaviour, that will raise IndexError and ValueError,
> then stop. With the suggested change in behaviour, it will raise all
> four exceptions.

No, it would still only raise the first two as it would still stop with
the return of StopIteration. But the certainty that people would be
confused by the double use of one parameter is reason enough not to do it.

> We cannot assume that an exception is never a legitimate return result
> from the callable. "Iterate until this exception is raised" and "iterate
> until this value is returned" are very different things and it is folly
> to treat them as the same.

>> [...]
>> I consider the threat to backward compatibility, because of the added
>> test for exceptions, theoretical rather than actual. It is very rare to
>> write a function that returns an exception,

While *writing* such a function might be very rare, Yuri showed how easy
it is to create such a callable by binding a collection instance to a
method.

> Rare or not, I've done it, it's allowed by the language, and it is
> inappropriate to conflate returning a class or instance with raising an
> exception.
>
> It doesn't matter whether it is rare. It is rare to write:
>
> iter(func, ({}, {}))
>
> nevertheless it would be poor design to have iter treat tuples of
> exactly two dicts as a special case.
>
> Exceptions are first-class values like strings, ints, and tuples
> containing exactly two dicts. They should be treated exactly the same as
> any other first-class value.

Agreed.

--
Terry Jan Reedy

[Stephen J. Turnbull]

Steven D'Aprano writes:

> iter(func, "NOBODY expects the Spanish Inquisition!!!")

+1 Code Snippet of the Week!

--
スティーヴェン・ジョン・ターンブル准教授
筑波大学 システム情報系 社会工学域
81+(0)29-853-5091 turn...@sk.tsukuba.ac.jp
http://turnbull.sk.tsukuba.ac.jp/

[Ram Rachum]

On Friday, February 7, 2014 11:49:12 AM UTC+2, Stephen J. Turnbull wrote:

Steven D'Aprano writes:

 > iter(func, "NOBODY expects the Spanish Inquisition!!!")

+1 Code Snippet of the Week!

A++, would run again :)

Now back to serious discussion: I do prefer a separate keyword argument rather than reusing `sentinel`. If `sentinel_exception` is too verbose, then an alternative is `exception`, but just not `stop_iter` which is very undescriptive.

Also, I'd support having three separate implementations for the `iter` function, one for just sentinel, one for exception, and one for both, so the iteration would be as quick as possible after the iterator was created.

Thanks,

Ram.

[Serhiy Storchaka]

07.02.14 10:36, Terry Reedy написав(ла):

> I wrote the above with the idea that there would be a third parameter to
> provide an exception. It would have to have a private default like
>
> class _NoException(Exception): pass

Natural default is StopIteration or ().

[Ram Rachum]

Oh yeah, we definitely want to allow a sequence of possible exceptions, but if feeding in a single exception, not require a sequence for it. So () would be good default.

--

--- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/UCaNfAHkBlQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe@googlegroups.com.

[Terry Reedy]

On 2/7/2014 3:39 AM, Steven D'Aprano wrote:

> is simpler than the proposed behaviour:
>
> "the callable is called until it returns the sentinel, unless the
> sentinel is an exception instance or class, in which case the
> callable is called until it raises that exception, or one which is
> compatible with it"

To repeat, for the 3rd or 4th time, this is not the behavior that either
I or Ram proposed. I even provided code to be clear as to what I mean at
the time.

--
Terry Jan Reedy

[Steven D'Aprano]

On Fri, Feb 07, 2014 at 04:54:04PM -0500, Terry Reedy wrote:
> On 2/7/2014 3:39 AM, Steven D'Aprano wrote:
>
> >is simpler than the proposed behaviour:
> >
> > "the callable is called until it returns the sentinel, unless the
> > sentinel is an exception instance or class, in which case the
> > callable is called until it raises that exception, or one which is
> > compatible with it"
>
> To repeat, for the 3rd or 4th time, this is not the behavior that either
> I or Ram proposed. I even provided code to be clear as to what I mean at
> the time.

You did write:

No new parameter is needed. We only need to add 'or raises' to the
two-parameter iter definition "In the second form, the callable is
called until it returns the sentinel." to get "In the second form,

the callable is called until it returns or raises the sentinel."


I agree that what I described is not what Ram suggested, but I believe
that it was you who first suggested overloading the meaning of the
sentinel argument depending on whether it was an exception or not. If
you've changed your mind, I'll be glad to hear it, and you can stop
reading here. If you haven't, then I admit I'm still confused, and would
be grateful for a short, simple, obvious example of what you are
proposing and how it differs from the current behaviour.

I must admit that I haven't studied your code snippets in detail,
particularly since you descibed at least one of them as "twisted and
awful". I don't believe that an iterable that returns exceptions (either
exception classes, or exception instances) is either twisted or awful.

However, I do believe that overloading the sentinel argument depending
on the specific type of value provided *is* twisted and awful. That sort
of behaviour should only be allowed when there is otherwise no possible
reason for the function to return that kind of value. E.g. contrast
iter() with (say) str.replace. The optional count argument, if given,
must be an int. There is no valid reason for supplying (say) count='a
string'. So we might, hypothetically, overload the count operator to say
"if count is a string, then do this special behaviour instead". Even
that is dubious, but at least it is backwards-compatible.

With iter(), on the other hand, the callable can return values of any
type, it is completely general. Likewise the sentinel can be of any
type. We wouldn't even dream of overloading the behaviour of iter when
sentinel happens to be a string. We shouldn't do so if it happens to be
<insert any type here>, no matter how unusual the type.


--
Steven

[Terry Reedy]

On 2/7/2014 7:30 PM, Steven D'Aprano wrote:
> On Fri, Feb 07, 2014 at 04:54:04PM -0500, Terry Reedy wrote:
>> On 2/7/2014 3:39 AM, Steven D'Aprano wrote:

This

>>> "the callable is called until it returns the sentinel, unless the
>>> sentinel is an exception instance or class, in which case the
>>> callable is called until it raises that exception, or one which is
>>> compatible with it"
>>
>> To repeat, for the 3rd or 4th time, this is not the behavior that either
>> I or Ram proposed. I even provided code to be clear as to what I mean at
>> the time.

is critically different from what I actually wrote for how to change the
one-line docstring summary

> You did write:
>
> No new parameter is needed. We only need to add 'or raises' to the
> two-parameter iter definition "In the second form, the callable is
> called until it returns the sentinel." to get "In the second form,
> the callable is called until it returns or raises the sentinel."

and the code that followed because it adds 'unless the sentinel is an
exception instance or class'. Your addtion would change existing
behavior before an exception is raised (if ever). I explicitly said in
both code and text that I was not proposing to do that, but only to
change what happened if and when an exception were raised (at which
point there would be no more return values values to worry about.

> If you've changed your mind, I'll be glad to hear it,

As I already responded to you at 4:28 EST Friday, I withdrew my
reuse-the-parameter proposal, in response to Yuri, 2 hours before the
timestamp I have on your first response. In that response to you, I
discussed in several paragraphs the actual existing parameter and how to
add a third parameter. Since you did not read it, here it is again.

'''

'''

iter(iter_or_call, sentinel=<private object>, stop_iter=<private exception>)

is still my current proposal. Serhiy suggested 'stop_iter=StopIteration'
and I will explain separately why I think that this would not work.

--
Terry Jan Reedy

[Stefan Behnel]

Terry Reedy, 08.02.2014 08:32:

> iter(iter_or_call, sentinel=<private object>, stop_iter=<private exception>)
> is still my current proposal.

+1, the name "stop_iter" resembles StopIteration, which IMHO is enough of a
hint at what it does.

Also, +1 for the general proposal. While I'd rarely use either of the two,
my guess is that there aren't even as many use cases for iterating up to a
sentinel value (and I used that already) as for iterating up to an
exception (and I'm sure I would have used that as well, if it had been there).

> Serhiy suggested 'stop_iter=StopIteration'
> and I will explain separately why I think that this would not work.

It suggests that it swallows a raised StopIteration *instance* and raises
its own, which it shouldn't.

Stefan

[Terry Reedy]

On 2/7/2014 9:28 AM, Serhiy Storchaka wrote:
> 07.02.14 10:36, Terry Reedy написав(ла):
>> I wrote the above with the idea that there would be a third parameter to
>> provide an exception. It would have to have a private default like
>>
>> class _NoException(Exception): pass
>
> Natural default is StopIteration or ().

That does not work because iter has to know if the user called iter with
only one argument, an iterable, or with two or (with the proposal)
three, with the first argument being a callable.

def iter(iter_or_call, sentinel=<private object>, stop_iter=<private
exception>):
if sentinel == <private object> and stop_iter == <private exception>:
# iter_or_call must be an iterable
<do what iter(iterable) does now,
which is to return iterable.__iter__()
or getitem_iterable(iterable)>
else:
# iter_or_call must be a callable
return callable_iterator(iter_or_call, sentinel, stop_iter)

where callable_iterator has been modified to take the third parameter
and raise StopIteration if iter_or_call raises stop_iter.

class Callable_iterator:
def __next__(self):
try:
val = self.func()
except self.stop_iter:
raise StopIteration from None
if self.sentinel == value:
raise StopIteration
else:
return val

If a user passes sentinel but not stop_iter, the except clause should
have no effect and the method should work as it does currently. A
default of StopIteration would accomplish that but it does not work for
the iter switch. A private exception class also works since no exception
raised by self.func should be an instance of such a class (unless the
user violates convention).

If a user passes stop_iter but not sentinel, the if clause should have
no effect and the iteration should continue until there is an exception.
So the default sentinel should never compare equal to any value returned
by self.func. A private instance of object on the left of == will invoke
object.__eq__ which compares by identity. Unless a user violates
convention by extracting and using the private instance, the function
will never return it. Since value could compare equal to everything, the
order for == matters.

--
Terry Jan Reedy

[Ram Rachum]

HI everybody,

I see that discussion has stalled on this suggestion. What can we do to move this forward?

[Terry Reedy]

On 2/10/2014 4:39 AM, Ram Rachum wrote:

> I see that discussion has stalled on this suggestion.

I think of it as more or less finished here and time to open an issue on
the tracker, which I plan to do soon, with the specific signature and
Python equivalent code I have suggested, as well as your example. If you
open one, add me as nosy so I can add the above.

> What can we do to move this forward?

Remind me if neither of us have not opened an issue within a week.

[Serhiy Storchaka]

08.02.14 10:32, Terry Reedy написав(ла):

> On 2/7/2014 9:28 AM, Serhiy Storchaka wrote:
>> 07.02.14 10:36, Terry Reedy написав(ла):
>>> I wrote the above with the idea that there would be a third parameter to
>>> provide an exception. It would have to have a private default like
>>>
>>> class _NoException(Exception): pass
>>
>> Natural default is StopIteration or ().
>
> That does not work because iter has to know if the user called iter with
> only one argument, an iterable, or with two or (with the proposal)
> three, with the first argument being a callable.

Agree.