extend with list vs extend with iterator

[Martin R]

While working on https://trac.sagemath.org/ticket/31897 Tejasvi (gsoc21) and I stumbled across the following python behaviour:

sage: def f(n):
....:     print(len(l))
....:     
sage: l = []; l.extend(f(n) for n in range(3))
0
1
2
sage: l = []; l.extend([f(n) for n in range(3)])
0
0
0

Is this behaviour we can rely on?  I could not find in the python doc, in any case.

Martin

[Dima Pasechnik]

see https://docs.python.org/3/tutorial/datastructures.html

Martin

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/53c97d28-fc5e-43ee-9371-bf8d2b646554n%40googlegroups.com.

[Martin R]

Thank you for the pointer, but of course I looked it up before posting.  It says

list.extend(iterable)

Extend the list by appending all the items from the iterable. Equivalent to a[len(a):] = iterable.

However, this description is simply not true:

sage: l = []; l[len(l):] = (f(n) for n in range(3))
0
0
0

[Jean-Florent Raymond]

Related:
https://stackoverflow.com/questions/57194152/is-python-list-extenditerator-guaranteed-to-be-lazy

It seems they are indeed not equivalent, despite what is claimed in the
documentation.

Le 10/06/2021 à 10:20, 'Martin R' via sage-devel a écrit :
> Thank you for the pointer, but of course I looked it up before posting. It
> says
>

> list.extend(*iterable*)

>>> <https://groups.google.com/d/msgid/sage-devel/53c97d28-fc5e-43ee-9371-bf8d2b646554n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
>

[Martin R]

I found a relevant discussion.  Bad news, it seems.

https://stackoverflow.com/questions/57194152/is-python-list-extenditerator-guaranteed-to-be-lazy

[Martin R]

Oh, sorry, emails crossed.  The problem is, that the python crew seems to refuse to clarify whether this is unspecified or intended.  Unless I find a completely different approach (which I hope), this is a shame, because extend is very likely faster than the loop equivalent.

[Sébastien Labbé]

In the second example, the list [f(n) for n in range(3)] is created before the method extend is called which explains why 0 is printed 3 times.

What is the question?

Sébastien

[Martin R]

The question is, whether we can rely on behaviour of the reference implementation of python which contradicts its documentation:

list.extend(iterable)

Extend the list by appending all the items from the iterable. Equivalent to a[len(a):] = iterable.

sage: def f(n):

....:     print(len(l))
....:     
sage: l = []; l.extend(f(n) for n in range(3))
0
1
2

sage: l = []; l[len(l):] = (f(n) for n in range(3))

0
0
0

Martin

[Michael Orlitzky]

On Thu, 2021-06-10 at 09:47 -0700, 'Martin R' via sage-devel wrote:
> The question is, whether we can rely on behaviour of the reference
> implementation of python which contradicts its documentation:
>

The cynic's take: you can't rely on anything in a language that has no
specification.

Can you write a doctest for the current behavior at the same time as
you write the code that relies upon it?

[Sébastien Labbé]

On Thursday, June 10, 2021 at 6:47:37 PM UTC+2 axio...@yahoo.de wrote:

The question is, whether we can rely on behaviour of the reference implementation of python which contradicts its documentation:

list.extend(iterable)

Extend the list by appending all the items from the iterable. Equivalent to a[len(a):] = iterable.

I believe it is equivalent:

sage: def f(n):
....:     return None
....:                                                                    
sage: l = []; l.extend(f(n) for n in range(3)); l
[None, None, None]
sage: l = []; l[len(l):] = (f(n) for n in range(3)); l                                                                         
[None, None, None]

Sébastien

[Martin R]

So, my example is fake?

[Vincent Delecroix]

It is not fake. If your iterator uses a reference to
the underlying list, then the behavior is different

sage: l = []
sage: iterator = (len(l) for i in range(3))
sage: l.extend(iterator)
sage: print(l)
[0, 1, 2]

sage: l = []
sage: iterator = (len(l) for i in range(3))
sage: l[:] = iterator
sage: print(l)
[0, 0, 0]

Le 13/06/2021 à 12:00, 'Martin R' via sage-devel a écrit :
> So, my example is fake?
>
> Sébastien Labbé schrieb am Sonntag, 13. Juni 2021 um 08:42:47 UTC+2:
>
>> On Thursday, June 10, 2021 at 6:47:37 PM UTC+2 axio...@yahoo.de wrote:
>>
>>> The question is, whether we can rely on behaviour of the reference
>>> implementation of python which contradicts its documentation:
>>>

>>> list.extend(*iterable*)

[Sébastien Labbé]

On Sunday, June 13, 2021 at 12:00:44 PM UTC+2 axio...@yahoo.de wrote:

So, my example is fake?

I think it has nothing to do with extend method:

sage: def f(n):
....:     print(len(l))
....:                                                                           
sage: l = []                                                                    

sage: for fn in [f(n) for n in range(3)]:
....:     l.append(fn)
....:                                                                           
0
0
0

sage: l                                                                         
[None, None, None]

sage: l = []                                                                    
sage: for fn in (f(n) for n in range(3)):
....:     l.append(fn)
....:                                                                           

0
1
2
sage: l                                                                         

[None, None, None]

It is more about the difference between [f(n) for n in range(3)] which is a list and (f(n) for n in range(3)) which is an iterator. The list [f(n) for n in range(3)] is computed completely before being used, whereas the iterator is lazy:

sage: l = []

sage: L = [f(n) for n in range(3)]                                              
0
0
0
sage: it = (f(n) for n in range(3))