StopIteration (was Re: [pythonpune] Inserting colon in a string)

[Dhananjay Nene]

On Tue, Apr 30, 2013 at 10:50 PM, steve <st...@lonetwin.net> wrote:

>>> s = "0x21000024ff3afd2b"
>>> i = iter(s)
>>> for x in range(10000):
...     next(i)
...
'0'
'x'
'2'
'1'
'0'
'0'
'0'
'0'
'2'
'4'
'f'
'f'
'3'
'a'
'f'
'd'
'2'
'b'
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
StopIteration
>>> ''.join(next(i) for _ in range(10000))
'0x21000024ff3afd2b'

Is that (not raising a StopIteration) valid (ie: defined) behaviour ?

cheers,
- steve

Not sure if I understood your question correctly

>>> for i in range(5) :

...     raise StopIteration()

... 

Traceback (most recent call last):

  File "<stdin>", line 2, in <module>

StopIteration

When a stop iteration is raised as a part of the range function it is dealt with by the for loop, but when raised in the body of the for loop the exception is simply passed upwards.

So if the context in which the iteration is being done can deal with the stopiteration, then the iteration gracefully terminates, in other contexts, the exception is just propagated.

In the example you state, the stopiteration is coming from the body of the for loop. not from the iterator the for loop is running on. Whereas I suspect in the   ''.join(next(i) for _ in range(10000)) statement, the comprehension is geared enough to deal with the stop iteration that the next(i) raises.

[Dhananjay Nene]

I did investigate it a bit more. I could find that somehow if a StopIteration is raised from the body part of a "[ body for var in iterable]", it raises a StopIteration which is not caught. However if one instead uses a generator expression ie. (body for var in iterable) it does not seem to quite work that way (probably because the StopIteration is executed only later). 

So :

for var in iterable :

    body

will raise an exception, as will

[body for var in iterable] and

for var in [body for var in iterable]

but the following will not

[var for var in (body for var in iterable)] nor will

delim.join(body for var in iterable) and many other situations nor will

list(body for var in iterable)

So in that sense the [body for var in iterable] behaves most like a for loop. And a StopIteration exception from body will propagate outwards. 

I looked for specific information in the python documentation, but at least could not find it in one casual search. My hypothesis is that unlike most situations where "body for var in iterable" is a generator, "[body for var in iterable]" forces an evaluation thus materialising the results of the generator. This materialisation triggers the StopIteration which propagates / bubbles up. When the output of the expression is a generator, typically the recipient of such an expression has the necessary logic to terminate the iteration on receipt of a StopIteration. The best guess I could make is that when the parameter of a function or a broader expression is a list comprehension, such a comprehension is materialised independently (just like expressions in a function invocation are evaluated before passing the results to a function) and if that raises a StopIteration in the body of the comprehension then there is no outer try/catch yet in place to capture it and terminate the loop (since the list must be first materialised before getting passed to the function or rest of the expression that it is a part of.

That might sound confusing .. but perhaps in a less tired state I just might be able to explain better.

Any links to python language specification sections which might better explain this behaviour would be appreciated. 

--
----------------------------------------------------------------------------------------------------------------------------------
http://blog.dhananjaynene.com twitter: @dnene google plus: http://gplus.to/dhananjaynene

[Dhananjay Nene]

You might also want to look at the question and the comments to the accepted answer here http://stackoverflow.com/questions/1106903/python-stopiteration-exception-and-list-comprehensions

[Anand Chitipothu]

The issue is with the behavior of raising StopIteration in the body of the generator expression.

Peter Norvig reported a bug about this.

http://bugs.python.org/issue14845

Anand

On Wed, May 1, 2013 at 2:15 AM, Dhananjay Nene <dhananj...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "Python Pune" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pythonpune+...@googlegroups.com.
To post to this group, send email to pytho...@googlegroups.com.
Visit this group at http://groups.google.com/group/pythonpune?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Anand
http://anandology.com/

[steve]

On Wednesday 01 May 2013 08:19 AM, Anand Chitipothu wrote:
> The issue is with the behavior of raising StopIteration in the body of the
> generator expression.
>
> Peter Norvig reported a bug about this.
>
> http://bugs.python.org/issue14845
>

Thanks Anand, Dhananjay for explaining/investigating this. That was good to know.

cheers,
- steve

[...snip...]

[Dhananjay Nene]

On Wed, May 1, 2013 at 8:19 AM, Anand Chitipothu <anand...@gmail.com> wrote:

The issue is with the behavior of raising StopIteration in the body of the generator expression.

Peter Norvig reported a bug about this.

http://bugs.python.org/issue14845

If I read this correctly then it would be summarised as follows :

• StopIteration exceptions in the body of a comprehension are "intended" to propagate beyond the comprehension
• "Except" in case of a generator expression where an StopIteration raised within a genexp is indistinguishable from a StopIteration raised by the genexp, thus forcing the context to always capture and not propagate the exception
  
• The side effect of this (in my understanding) being for blocks and list comprehensions will propagate that exception - most other typical usages will not.

Pls. Comment if you have a different understanding.

[Anand Chitipothu]

On Wed, May 1, 2013 at 10:03 AM, Dhananjay Nene <dhananj...@gmail.com> wrote:

On Wed, May 1, 2013 at 8:19 AM, Anand Chitipothu <anand...@gmail.com> wrote:

The issue is with the behavior of raising StopIteration in the body of the generator expression.

Peter Norvig reported a bug about this.

http://bugs.python.org/issue14845

If I read this correctly then it would be summarised as follows :

• StopIteration exceptions in the body of a comprehension are "intended" to propagate beyond the comprehension

All exceptions are treated in the same way in list comprehensions. StopIteration is not an exception to this. 

• "Except" in case of a generator expression where an StopIteration raised within a genexp is indistinguishable from a StopIteration raised by the genexp, thus forcing the context to always capture and not propagate the exception
  

Yes. In case of generator expressions, StopIteration is treated separately. 

• The side effect of this (in my understanding) being for blocks and list comprehensions will propagate that exception - most other typical usages will not.
  

Or rather put it other way. StopIteration raised in the body of generator-expression is consumed. will be propagated in every other case.

Anand