Multi-dimensional list initialization

[Demian Brecht]

So, here I was thinking "oh, this is a nice, easy way to initialize a 4D matrix" (running 2.7.3, non-core libs not allowed):

m = [[None] * 4] * 4

The way to get what I was after was:

m = [[None] * 4, [None] * 4, [None] * 4, [None * 4]]

(Obviously, I could have just hardcoded the initialization, but I'm too lazy to type all that out ;))

The behaviour I encountered seems a little contradictory to me. [None] * 4 creates four distinct elements in a single array while [[None] * 4] * 4 creates one distinct array of four distinct elements, with three references to it:

>>> a = [None] * 4
>>> a[0] = 'a'
>>> a
['a', None, None, None]

>>> m = [[None] * 4] * 4
>>> m[0][0] = 'm'
>>> m
[['m', None, None, None], ['m', None, None, None], ['m', None, None, None], ['m', None, None, None]]

Is this expected behaviour and if so, why? In my mind either result makes sense, but the inconsistency is what throws me off.

Demian Brecht
@demianbrecht
http://demianbrecht.github.com

[Andrew Robinson]

On 11/04/2012 10:27 PM, Demian Brecht wrote:
> So, here I was thinking "oh, this is a nice, easy way to initialize a 4D matrix" (running 2.7.3, non-core libs not allowed):
>
> m = [[None] * 4] * 4
>
> The way to get what I was after was:
>
> m = [[None] * 4, [None] * 4, [None] * 4, [None * 4]]

FYI: The behavior is the same in python 3.2
m=[[None]*4]*4
produces a nested list with all references being to the first instance
of the inner list construction.

I agree, the result is very counter-intuitive; hmmm... but I think you
meant:

m = [[None] * 4, [None] * 4, [None] * 4, [None] *4 ]
rather than:

m = [[None] * 4, [None] * 4, [None] * 4, [None * 4]]

? :) ?

I asked a why question on another thread, and watched several dodges to
the main question; I'll be watching to see if you get anything other
than "That's the way it's defined in the API". IMHO -- that's not a
real answer.

My guess is that the original implementation never considered anything
beyond a 1d list.
:)

A more precise related question might be: is there a way to force the
replication operator to use copying rather than referencing?
:/

[Chris Rebert]

On Sun, Nov 4, 2012 at 10:27 PM, Demian Brecht <demian...@gmail.com> wrote:
> So, here I was thinking "oh, this is a nice, easy way to initialize a 4D matrix" (running 2.7.3, non-core libs not allowed):
>
> m = [[None] * 4] * 4
>
> The way to get what I was after was:
>
> m = [[None] * 4, [None] * 4, [None] * 4, [None * 4]]
>

> (Obviously, I could have just hardcoded the initialization, but I'm too lazy to type all that out ;))
>
> The behaviour I encountered seems a little contradictory to me.
> [None] * 4 creates four distinct elements in a single array
> while [[None] * 4] * 4 creates one distinct array of four distinct elements, with three references to it:

Incorrect. In /both/ cases, the result is a list of length 4, whose
elements are 4 (references to) the exact same object as the original
list's element.
Put simply, the list multiplication operator never copies objects; it
just makes additional references to them.

However, unlike a list object (as in your latter example), the object
`None` is completely immutable (and what's more, a singleton value),
so you just-so-happen *not to be able to* run into the same problem of
mutating an object (assignment to an index of a list constitutes
mutation of that list) that is referenced in multiple places, for you
cannot mutate None in the first place!:
>>> x = None
>>> x.a = 42
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'a'
>>> # it doesn't overload any mutating operators:
>>> type(None).__dict__.keys()
['__hash__', '__repr__', '__doc__']
>>> # and it obviously has no instance variables,
>>> # so, we can't modify it in any way whatsoever!
(Lists, on the other hand, define item assignment, .pop(), .remove(),
and a few other mutator methods.)

>>>> a = [None] * 4
>>>> a[0] = 'a'
>>>> a
> ['a', None, None, None]
>

>>>> m = [[None] * 4] * 4

>>>> m[0][0] = 'm'
>>>> m
> [['m', None, None, None], ['m', None, None, None], ['m', None, None, None], ['m', None, None, None]]
>

> Is this expected behavior

Yes. It's also a FAQ:
http://docs.python.org/2/faq/programming.html#how-do-i-create-a-multidimensional-list

> and if so, why?

It's a general (albeit AFAIK unstated) principle that Python never
copies objects unless you explicitly ask it to. You have encountered
one example of this rule in action.

> In my mind either result makes sense, but the inconsistency is what throws me off.

It is perfectly consistent, once you understand what list
multiplication actually does.

Cheers,
Chris
--
http://rebertia.com

[Chris Angelico]

On Mon, Nov 5, 2012 at 6:07 PM, Chris Rebert <cl...@rebertia.com> wrote:
>>>> x = None
>>>> x.a = 42
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> AttributeError: 'NoneType' object has no attribute 'a'

Python needs a YouGottaBeKiddingMeError for times when you do
something utterly insane like this. Attributes of None??!? :)

ChrisA

[Andrew Robinson]

Hmmm? Everything in Python is an object.
Therefore! SURE. None *does* have attributes! ( even if not useful ones... )

eg: " None.__getattribute__( "__doc__" ) " doesn't produce an error.

In C, in Linux, at the end of the file "errno.h", where all error codes
are listed eg:( EIO, EAGAIN, EBUSY, E....) They had a final error like
the one you dreamed up, it was called "EIEIO"; and the comment read
something like, "All the way around Elmer's barn".

:)

The poster just hit that strange wall -- *all* built in types are
injection proof; and that property is both good and bad...

[Chris Angelico]

On Mon, Nov 5, 2012 at 6:54 PM, Andrew Robinson
<and...@r3dsolutions.com> wrote:
> On 11/04/2012 11:27 PM, Chris Angelico wrote:
>>
>> On Mon, Nov 5, 2012 at 6:07 PM, Chris Rebert<cl...@rebertia.com> wrote:
>>>>>>
>>>>>> x = None
>>>>>> x.a = 42
>>>
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in<module>
>>> AttributeError: 'NoneType' object has no attribute 'a'
>>
>> Python needs a YouGottaBeKiddingMeError for times when you do
>> something utterly insane like this. Attributes of None??!? :)
>>
>> ChrisA
>
> Hmmm? Everything in Python is an object.
> Therefore! SURE. None *does* have attributes! ( even if not useful ones... )
>
> eg: " None.__getattribute__( "__doc__" ) " doesn't produce an error.

Eh, I meant mutating None's attributes, which is just as insane as I said.

> In C, in Linux, at the end of the file "errno.h", where all error codes are
> listed eg:( EIO, EAGAIN, EBUSY, E....) They had a final error like the one
> you dreamed up, it was called "EIEIO"; and the comment read something like,
> "All the way around Elmer's barn".

There's been a collection of those around the place. A few memorable ones:

EMILYPOST: Bad fork()
ETOBACCO: Read on empty pipe
EHORSE: Mount failed

I may be misremembering, but I'm sure the originals can be found at
the other end of a web search.

ChrisA

[Hans Mulder]

On 5/11/12 07:27:52, Demian Brecht wrote:
> So, here I was thinking "oh, this is a nice, easy way to initialize a 4D matrix"
> (running 2.7.3, non-core libs not allowed):
>
> m = [[None] * 4] * 4
>
> The way to get what I was after was:
>
> m = [[None] * 4, [None] * 4, [None] * 4, [None * 4]]

Or alternateively:

m = [[None] * 4 for _ in range(4)]

> (Obviously, I could have just hardcoded the initialization, but I'm too
> lazy to type all that out ;))
>
> The behaviour I encountered seems a little contradictory to me.
> [None] * 4 creates four distinct elements in a single array

Actually, it creates a list with four references to the same object.
But then, this object is immutable, so you won't notice that it's the
same object.

> while [[None] * 4] * 4 creates one distinct array of four distinct
> elements, with three references to it:

We usually phrase that as "a list with four references to the
same list". The first reference is not special in any way.

>>>> a = [None] * 4
>>>> a[0] = 'a'
>>>> a
> ['a', None, None, None]
>
>>>> m = [[None] * 4] * 4
>>>> m[0][0] = 'm'
>>>> m
> [['m', None, None, None], ['m', None, None, None], ['m', None, None, None], ['m', None, None, None]]
>
> Is this expected behaviour

Yes.

> and if so, why? In my mind either result makes sense, but the
> inconsistency is what throws me off.

There's no inconsistency: in both cases you get a list with four
references to the same object. The only difference is that in the
fist case, the references are to an immutable object, so the fact
that it's the same object won't hurt you.


Hope this helps,

-- HansM

[wxjm...@gmail.com]

----------

You probably mean a two-dimensional matrix not a 4D matrix.

>>> def DefMatrix(nrow, ncol, val):
... return [[val] * ncol for i in range(nrow)]
...
>>> aa = DefMatrix(2, 3, 1.0)
>>> aa
>>> aa = DefMatrix(2, 3, 1.0)
>>> aa
[[1.0, 1.0, 1.0], [1.0, 1.0, 1.0]]
>>> aa[0][0] = 3.14
>>> aa[1][2] = 2.718
>>> aa
[[3.14, 1.0, 1.0], [1.0, 1.0, 2.718]]
>>>
>>> bb = DefMatrix(2, 3, None)
>>> bb
[[None, None, None], [None, None, None]]
>>> bb[0][0] = 3.14
>>> bb[1][2] = 2.718
>>> bb
[[3.14, None, None], [None, None, 2.718]]


jmf

[Demian Brecht]

On 2012-11-04, at 10:44 PM, Andrew Robinson <and...@r3dsolutions.com> wrote:
> but I think you meant:
>

> m = [[None] * 4, [None] * 4, [None] * 4, [None] *4 ]
> rather than:

> m = [[None] * 4, [None] * 4, [None] * 4, [None * 4]]

Yes, I meant the former, thanks for catching the typo.

[Demian Brecht]

On 2012-11-04, at 11:07 PM, Chris Rebert <cl...@rebertia.com> wrote:

> However, unlike a list object (as in your latter example), the object
> `None` is completely immutable (and what's more, a singleton value),
> so you just-so-happen *not to be able to* run into the same problem of
> mutating an object (assignment to an index of a list constitutes
> mutation of that list) that is referenced in multiple places, for you
> cannot mutate None in the first place!

Thanks for clearing that up Chris (and the link to the FAQ). I had thought about that after going to bed ("D'oh.. None is immutable.. *That's* gotta be why").

[Oscar Benjamin]

On 5 November 2012 09:13, Hans Mulder <han...@xs4all.nl> wrote:
> On 5/11/12 07:27:52, Demian Brecht wrote:
>> So, here I was thinking "oh, this is a nice, easy way to initialize a 4D matrix"
>> (running 2.7.3, non-core libs not allowed):
>>
>> m = [[None] * 4] * 4
>>
>> The way to get what I was after was:
>>
>> m = [[None] * 4, [None] * 4, [None] * 4, [None * 4]]
>
> Or alternateively:
>
> m = [[None] * 4 for _ in range(4)]

That's the way to do it.

I've seen this question many times between here and the python-tutor
list. It does seem to be a common gotcha.

I was just thinking to myself that it would be a hard thing to change
because the list would need to know how to instantiate copies of all
the different types of the elements in the list. Then I realised it
doesn't. It is simply a case of how the list multiplication operator
is implemented and whether it chooses to use a reference to the same
list or make a copy of that list. Since all of this is implemented
within the same list type it is a relatively easy change to make
(ignoring backward compatibility concerns).

I don't see this non-copying list multiplication behaviour as
contradictory but has anyone ever actually found a use for it?


Oscar

[Chris Angelico]

On Tue, Nov 6, 2012 at 12:32 PM, Oscar Benjamin
<oscar.j....@gmail.com> wrote:
> I was just thinking to myself that it would be a hard thing to change
> because the list would need to know how to instantiate copies of all
> the different types of the elements in the list. Then I realised it
> doesn't. It is simply a case of how the list multiplication operator
> is implemented and whether it chooses to use a reference to the same
> list or make a copy of that list. Since all of this is implemented
> within the same list type it is a relatively easy change to make
> (ignoring backward compatibility concerns).
>
> I don't see this non-copying list multiplication behaviour as
> contradictory but has anyone ever actually found a use for it?

Stupid example of why it can't copy:

bad = [open("test_file")] * 4

How do you clone something that isn't Plain Old Data? Ultimately,
that's where the problem comes from. It's easy enough to clone
something that's all scalars (strings, integers, None, etc) and
non-recursive lists/dicts of scalars, but anything more complicated
than that is rather harder.

If you want a deep copy and are prepared to handle any issues that
might result, you can do this:

>>> import copy
>>> a=[[2,3,4]]
>>> a.extend(copy.deepcopy(a))
>>> a[0][1]=10
>>> a
[[2, 10, 4], [2, 3, 4]]

And some things just won't work:
>>> bad.extend(copy.deepcopy(bad))

Traceback (most recent call last):

File "<pyshell#17>", line 1, in <module>
bad.extend(copy.deepcopy(bad))
File "C:\Python32\lib\copy.py", line 147, in deepcopy
y = copier(x, memo)
File "C:\Python32\lib\copy.py", line 209, in _deepcopy_list
y.append(deepcopy(a, memo))
File "C:\Python32\lib\copy.py", line 166, in deepcopy
rv = reductor(2)
TypeError: cannot serialize '_io.TextIOWrapper' object

The default behaviour is safe and reliable. When you want something
other than the default, there are ways of doing it.

ChrisA

[Oscar Benjamin]

On 6 November 2012 02:01, Chris Angelico <ros...@gmail.com> wrote:
> On Tue, Nov 6, 2012 at 12:32 PM, Oscar Benjamin
> <oscar.j....@gmail.com> wrote:
>> I was just thinking to myself that it would be a hard thing to change
>> because the list would need to know how to instantiate copies of all
>> the different types of the elements in the list. Then I realised it
>> doesn't. It is simply a case of how the list multiplication operator
>> is implemented and whether it chooses to use a reference to the same
>> list or make a copy of that list. Since all of this is implemented
>> within the same list type it is a relatively easy change to make
>> (ignoring backward compatibility concerns).
>>
>> I don't see this non-copying list multiplication behaviour as
>> contradictory but has anyone ever actually found a use for it?
>
> Stupid example of why it can't copy:
>
> bad = [open("test_file")] * 4
>
> How do you clone something that isn't Plain Old Data? Ultimately,
> that's where the problem comes from. It's easy enough to clone
> something that's all scalars (strings, integers, None, etc) and
> non-recursive lists/dicts of scalars, but anything more complicated
> than that is rather harder.

That's not what I meant. But now you've made me realise that I was
wrong about what I did mean. In the case of

stuff = [[obj] * n] * m

I thought that the multiplication of the inner list ([obj] * n) by m
could create a new list of lists using copies. On closer inspection I
see that the list being multiplied is in fact [[obj] * n] and that
this list can only know that it is a list of lists by inspecting its
element(s) which makes things more complicated.

I retract my claim that this change would be easy to implement.


Oscar

[Andrew Robinson]

Hi Oscar,

In general, people don't use element multiplication (that I have *ever*
seen) to make lists where all elements of the outer most list point to
the same sub-*list* by reference. The most common use of the
multiplication is to fill an array with a constant, or short list of
constants; Hence, almost everyone has to work around the issue as the
initial poster did by using a much longer construction.

The most compact notation in programming really ought to reflect the
most *commonly* desired operation. Otherwise, we're really just making
people do extra typing for no reason.

Further, list comprehensions take quite a bit longer to run than low
level copies; by a factor of roughly 10. SO, it really would be worth
implementing the underlying logic -- even if it wasn't super easy.

I really don't think doing a shallow copy of lists would break anyone's
program.
The non-list elements, whatever they are, can be left as reference
copies -- but any element which is a list ought to be shallow copied.
The behavior observed in the opening post where modifying one element of
a sub-list, modifies all elements of all sub-lists is never desired as
far as I have ever witnessed.

The underlying implementation of Python can check an object type
trivially, and the only routine needed is a shallow list copy. So, no
it really isn't a complicated operation to do shallow copies of lists.

:)

[Chris Angelico]

On Tue, Nov 6, 2012 at 4:51 PM, Andrew Robinson
<and...@r3dsolutions.com> wrote:
> I really don't think doing a shallow copy of lists would break anyone's
> program.

Well, it's a change, a semantic change. It's almost certainly going to
break _something_. But for the sake of argument, we can suppose that
the change could be made. Would it be the right thing to do?

Shallow copying by default would result in extremely weird behaviour.
All the same confusion would result, only instead of comparing
[None]*4 with [[None]]*4, there'd be confusion over the difference
between [[None]]*4 and [[[None]]]*4.

I don't think it would help anything, and it'd result in a lot more
work for no benefit.

ChrisA

[Andrew Robinson]

I don't follow.
a=[ None ]*4 would give a=[ None, None, None, None ] as usual.
All four None's would be the same object, but there are automatically 4
different pointers to it.
Hence,
a[0]=1 would give a=[ 1, None, None, None ] as usual.

a=[ [None] ]*4 would give a=[ [None], [None], [None], [None] ] as usual
BUT:
a[0][0] = 1 would no longer give a=[ [1],[1],[1],[1] ] *Rather* it would
give
a=[ [1].[None].[None],[None] ]

The None objects are all still the same one, BUT the lists themselves
are different.

Again, a=[ ["alpha","beta"] * 4 ] would give:
a=[ ["alpha","beta"], ["alpha","beta"], ["alpha","beta"], ["alpha","beta"] ]

All four strings, "alpha", are the same object -- but there are 5
different lists; The pointers inside the initial list are copied four
times -- not the string objects;
But the *lists* themselves are created new for each replication.

If you nest it another time;
[[[None]]]*4, the same would happen; all lists would be independent --
but the objects which aren't lists would be refrenced-- not copied.

a=[[["alpha","beta"]]]*4 would yield:
a=[[['alpha', 'beta']], [['alpha', 'beta']], [['alpha', 'beta']],
[['alpha', 'beta']]]
and a[0][0]=1 would give [[1],[['alpha', 'beta']], [['alpha', 'beta']],
[['alpha', 'beta']]]]
rather than a=[[1], [1], [1], [1]]

Or at another level down: a[0][0][0]=1 would give: a=[[[1, 'beta']],
[['alpha', 'beta']], [['alpha', 'beta']], [['alpha', 'beta']] ]
rather than a=[[[1, 'beta']], [[1, 'beta']], [[1, 'beta']], [[1, 'beta']]]

The point is, there would be no difference at all noticed in what data
is found where in the array;
the *only* thing that would change is that replacing an item by
assignment would only affect the *location* assigned to -- all other
locations would not be affected.

That really is what people *generally* want.
If the entire list is meant to be read only -- the change would affect
*nothing* at all.

See if you can find *any* python program where people desired the
multiplication to have the die effect that changing an object in one of
the sub lists -- changes all the objects in the other sub lists.

I'm sure you're not going to find it -- and even if you do, it's going
to be 1 program in 1000's.

[Steven D'Aprano]

On Mon, 05 Nov 2012 21:51:24 -0800, Andrew Robinson wrote:

> The most compact notation in programming really ought to reflect the
> most *commonly* desired operation. Otherwise, we're really just making
> people do extra typing for no reason.

There are many reasons not to put minimizing of typing ahead of all other
values:

* Typically, code is written once and read many times. Minimizing
typing might save you a second or two once, and then cost you many
seconds every time you read the code. That's why we tell people to
choose meaningful variable names, instead of naming everything "a"
and "b".

* Consistency of semantics is better than a plethora of special
cases. Python has a very simple and useful rule: objects should
not be copied unless explicitly requested to be copied. This is
much better than having to remember whether this operation or
that operation makes a copy. The answer is consistent:

(pardon me for belabouring the point here)

Q: Does [0]*10 make ten copies of the integer object?
A: No, list multiplication doesn't make copies of elements.

Q: How about [0.0]*10?
A: No, the elements are never copied.

Q: What if I use a singleton? Does [None]*10 try to copy?
A: No, the elements are never copied.

Q: How about things like file objects that can't be copied?
A: No, the elements are never copied.

Q: What about [[]]*10?
A: No, the elements are never copied.

Q: How about if the elements are subclasses of list?
A: No, the elements are never copied.

Q: What about other mutable objects like sets or dicts?
A: No, the elements are never copied.

Q: What about instances of custom classes?
A: No, the elements are never copied.

Q: What if they are old-style Classic classes?
A: No, the elements are never copied.

Q: What if I do some funny tricks with the metaclass?
A: No, the elements are never copied.

Q: How about on Tuesdays? I bet they're copied on Tuesdays.
A: No, the elements are never copied.



Your proposal throws away consistency for a trivial benefit on a rare use-
case, and replaces it with a bunch of special cases:

Q: What about [[]]*10?
A: Oh yeah, I forgot about lists, they're copied.

Q: How about if the elements are subclasses of list?
A: Hmmm, that's a good one, I'm not actually sure.

Q: How about if I use delegation to proxy a list?
A: Oh no, they definitely won't be copied.

Q: What about other mutable objects like sets or dicts?
A: No, definitely not. Unless people complain enough.

Q: What about instances of custom classes?
A: That's a definite maybe.

Q: How about on Tuesdays? I bet they're copied on Tuesdays.
A: Only if you're in Belgium.


Losing consistency in favour of saving a few characters for something as
uncommon as list multiplication is a poor tradeoff. That's why this
proposal has been rejected again and again and again every time it has
been suggested.

List multiplication [x]*n is conceptually equivalent to:

newlist = []
for i in range(n):
newlist.append(x)

or if you prefer a list comp:

[x for i in range(n)]

This is nice and simple and efficient. Some objects cannot be copied at
all. Copying other objects is slow and inefficient. Keeping list
multiplication consistent, and fast, is MUCH more important than making
it work as expected for the rare case of 2D arrays:

[[0]*n]*m

where there are other alternatives.

> Further, list comprehensions take quite a bit longer to run than low
> level copies; by a factor of roughly 10. SO, it really would be worth
> implementing the underlying logic -- even if it wasn't super easy.

Copying those elements does not come for free.

It is true that list multiplication can be much faster than a list comp.
But that's because the list multiply doesn't have to inspect the
elements, copy them, or engage the iteration machinery. Avoiding copying
gives you a big saving:


[steve@ando ~]$ python3.3 -m timeit -s "x = range(1000)"
"[x for _ in range(100)]" # not copied
100000 loops, best of 3: 11.9 usec per loop

[steve@ando utilities]$ python3.3 -m timeit -s "x = range(1000)"
"[x[:] for _ in range(100)]" # copied
10000 loops, best of 3: 103 usec per loop

So there's a factor of ten difference right there. If list multiplication
had to make copies, it would lose much of its speed advantage. For large
enough lists, or complicated enough objects, it would become slower than
a list comprehension.

It would be even slower if list multiplication had to inspect each
element first and decide whether or not to copy.

> I really don't think doing a shallow copy of lists would break anyone's
> program.

Anyone who is currently using list multiplication with mutable objects is
expecting that they will be the same object, and relying on that fact.
Otherwise they wouldn't be using list multiplication.

You're suggesting a semantic change. Therefore they will be expecting
something different from what actually happens. Result: broken code.

It's not just mutable objects. It's also objects that can't be copied.
Result: mylist*3 used to work, now it raises an exception. And
performance issues: what used to be fast is now slow.

Even if this change was allowed, it would have to go through a multi-year
process. Python 3.3 is too late -- the absolute earliest would be Python
3.4, which is scheduled for about 18 months from now. So in Python 3.4
you could write:

from __future__ import list_multiplication_copying

to get the behaviour you want, and then in Python 3.5 it would become the
default. That's three years until it becomes the standard. Meanwhile,
there will still be millions of people using Python 2.7 or 3.2, and their
code will behave differently from your code.

Conservatively, if you write code to support three previous releases,
that means you can't use this feature until Python 3.7. So that's about
six years before it can be used widely.

If the problem being solved was big enough, this would be worth doing.
But it's not.

> The non-list elements, whatever they are, can be left as reference
> copies -- but any element which is a list ought to be shallow copied.

That's even worse than "list multiplication always copies". At least that
is simple and consistent, even if it isn't consistent with the rest of
the language, at least it is self-consistent. You are proposing something
much worse: special cases to remember. "Objects aren't copied, except for
lists, which are copied."

And then people will wonder why sets aren't copied, and dicts. People
will make a 2D array like so:

[[0]*5]*10

and it will work. Then they'll write this:

[{}]*5

and wonder why it doesn't work the way they expect. Consistency is *much*
more valuable than ad hoc DWIM semantics. Languages that try to Do What I
Mean somehow end up Doing What Somebody Else Meant, But Not Me.



--
Steven

[Ian Kelly]

On Tue, Nov 6, 2012 at 1:21 AM, Andrew Robinson
<and...@r3dsolutions.com> wrote:
> If you nest it another time;
> [[[None]]]*4, the same would happen; all lists would be independent -- but
> the objects which aren't lists would be refrenced-- not copied.
>
> a=[[["alpha","beta"]]]*4 would yield:
> a=[[['alpha', 'beta']], [['alpha', 'beta']], [['alpha', 'beta']], [['alpha',
> 'beta']]]
> and a[0][0]=1 would give [[1],[['alpha', 'beta']], [['alpha', 'beta']],
> [['alpha', 'beta']]]]
> rather than a=[[1], [1], [1], [1]]
>
> Or at another level down: a[0][0][0]=1 would give: a=[[[1, 'beta']],
> [['alpha', 'beta']], [['alpha', 'beta']], [['alpha', 'beta']] ]
> rather than a=[[[1, 'beta']], [[1, 'beta']], [[1, 'beta']], [[1, 'beta']]]

You wrote "shallow copy". When the outer-level list is multiplied,
the mid-level lists would be copied. Because the copies are shallow,
although the mid-level lists are copied, their contents are not. Thus
the inner-level lists would still be all referencing the same list.
To demonstrate:

>>> from copy import copy
>>> class ShallowCopyList(list):
... def __mul__(self, number):
... new_list = ShallowCopyList()
... for _ in range(number):
... new_list.extend(map(copy, self))
... return new_list
...
>>> a = ShallowCopyList([[["alpha", "beta"]]])
>>> a
[[['alpha', 'beta']]]
>>> b = a * 4
>>> b
[[['alpha', 'beta']], [['alpha', 'beta']], [['alpha', 'beta']],
[['alpha', 'beta']]]
>>> b[0][0][0] = 1
>>> b

[[[1, 'beta']], [[1, 'beta']], [[1, 'beta']], [[1, 'beta']]]

>>> b[0][0] = 1
>>> b
[[1], [[1, 'beta']], [[1, 'beta']], [[1, 'beta']]]

This shows that assignments at the middle level are independent with a
shallow copy on multiplication, but assignments at the inner level are
not. In order to achieve the behavior you describe, a deep copy would
be needed.

> That really is what people *generally* want.
> If the entire list is meant to be read only -- the change would affect
> *nothing* at all.

The time and memory cost of the multiplication operation would become
quadratic instead of linear.

> See if you can find *any* python program where people desired the
> multiplication to have the die effect that changing an object in one of the
> sub lists -- changes all the objects in the other sub lists.
>
> I'm sure you're not going to find it -- and even if you do, it's going to be
> 1 program in 1000's.

Per the last thread where we discussed extremely rare scenarios,
shouldn't you be rounding "1 in 1000s" up to 20%? ;-)

[Shambhu Rajak]

Well said Steve, I agree with you...
-Shambhu

[Prasad, Ramit]

Ian Kelly wrote:
>
> On Tue, Nov 6, 2012 at 1:21 AM, Andrew Robinson
>

[snip]

> > See if you can find *any* python program where people desired the
> > multiplication to have the die effect that changing an object in one of the
> > sub lists -- changes all the objects in the other sub lists.
> >
> > I'm sure you're not going to find it -- and even if you do, it's going to be
> > 1 program in 1000's.
>
> Per the last thread where we discussed extremely rare scenarios,
> shouldn't you be rounding "1 in 1000s" up to 20%? ;-)

Actually, I would be surprised if it was even 1 in 1000.
Of course, consistency makes it easier to learn and *remember*.
I value that far more than a minor quirk that is unlikely to
bother me now that I know of it. Well, at least not as long as
I do not forget my morning coffee/tea :)


~Ramit


This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.

[Andrew Robinson]

On 11/06/2012 06:35 AM, Oscar Benjamin wrote:
>
> > In general, people don't use element multiplication (that I have
> *ever* seen) to make lists where all elements of the outer most list
> point to the same sub-*list* by reference. The most common use of the
> multiplication is to fill an array with a constant, or short list of
> constants; Hence, almost everyone has to work around the issue as
> the initial poster did by using a much longer construction.
>

> That's what I have seen as well. I've never seen an example where
> someone wanted this behaviour.

>
> >
> > The most compact notation in programming really ought to reflect the
> most *commonly* desired operation. Otherwise, we're really just
> making people do extra typing for no reason.
>

> It's not so much the typing as the fact that this a common gotcha.
> Apparently many people expect different behaviour here. I seem to
> remember finding this surprising at first.
>
:) That's true as well.

>
> >
> > Further, list comprehensions take quite a bit longer to run than low
> level copies; by a factor of roughly 10. SO, it really would be worth
> implementing the underlying logic -- even if it wasn't super easy.
> >

> > I really don't think doing a shallow copy of lists would break
> anyone's program.

> > The non-list elements, whatever they are, can be left as reference
> copies -- but any element which is a list ought to be shallow copied.

> The behavior observed in the opening post where modifying one element
> of a sub-list, modifies all elements of all sub-lists is never desired
> as far as I have ever witnessed.
>

> It is a semantic change that would, I imagine, break many things in
> subtle ways.
>
?? Do you have any guesses, how ?

>
> >
> > The underlying implementation of Python can check an object type
> trivially, and the only routine needed is a shallow list copy. So, no
> it really isn't a complicated operation to do shallow copies of lists.
>

> Yes but if you're inspecting the object to find out whether to copy it
> what do you test for? If you check for a list type what about
> subclasses? What if someone else has a custom list type that is not a
> subclass? Should there be a dunder method for this?
>
No dunder methods. :)
Custom non-subclass list types aren't a common usage for list
multiplication in any event.
At present one has to do list comprehensions for that, and that would
simply remain so.

Subclasses, however, are something I hadn't considered...

> I don't think it's such a simple problem.
>
> Oscar
>
You made a good point, Oscar; I'll have to think about the subclassing a
bit.
:)

[Andrew Robinson]

On 11/06/2012 09:32 AM, Prasad, Ramit wrote:
> Ian Kelly wrote:
>> On Tue, Nov 6, 2012 at 1:21 AM, Andrew Robinson
>>
> [snip]
>>> See if you can find *any* python program where people desired the
>>> multiplication to have the die effect that changing an object in one of the
>>> sub lists -- changes all the objects in the other sub lists.
>>>
>>> I'm sure you're not going to find it -- and even if you do, it's going to be
>>> 1 program in 1000's.
>> Per the last thread where we discussed extremely rare scenarios,
>> shouldn't you be rounding "1 in 1000s" up to 20%? ;-)

:D -- Ian -- also consider that I *am* willing to use extra memory.
Not everything can be shrunk to nothing and still remain functional. :)
So, it isn't *all* about *micro* optimization -- it's also about
psychology and flexibility.

> Actually, I would be surprised if it was even 1 in 1000.
> Of course, consistency makes it easier to learn and *remember*.
> I value that far more than a minor quirk that is unlikely to
> bother me now that I know of it. Well, at least not as long as
> I do not forget my morning coffee/tea :)

But, having it copy lists -- when the only purpose of multiplication is
for lists;
is only a minor quirk as well.

>
>
> ~Ramit

[Ian Kelly]

On Tue, Nov 6, 2012 at 2:36 PM, Andrew Robinson
<and...@r3dsolutions.com> wrote:
> I meant all lists are shallow copied from the innermost level out.
> Equivalently, it's a deep copy of list objects -- but a shallow copy of any list contents except other lists.

Why only list objects, though? When a user writes [[]] * 10, they
probably want a list containing ten distinct nested lists. Likewise,
when a user writes [{}] * 10, they probably want a list containing ten
distinct dicts, which is not at all an uncommon thing to want. It
seems very inconsistent that the former should work while the latter
should not. This is especially true when you start mixing the two
paradigms; the user might expect [[{}] * 10] * 10 to create a a 10x10
matrix where each element is a distinct dict, but this still would not
work, even though the nested lists would all have different
identities.

What about ([],) * 10? This is perhaps best interpreted as a request
to create a matrix of ten rows where the rows themselves are mutable
but the collection of rows is not. If list multiplication were to
copy nested lists, then should tuple multiplication do the same?

[Andrew Robinson]

On 11/06/2012 01:04 AM, Steven D'Aprano wrote:
> On Mon, 05 Nov 2012 21:51:24 -0800, Andrew Robinson wrote:
>
>> The most compact notation in programming really ought to reflect the
>> most *commonly* desired operation. Otherwise, we're really just making
>> people do extra typing for no reason.
> There are many reasons not to put minimizing of typing ahead of all other
> values:

I didn't. I put it ahead of *some* values for the sake of practicality
and human psychology.
" Practicality beats purity. "

>
> * Typically, code is written once and read many times. Minimizing
> typing might save you a second or two once, and then cost you many
> seconds every time you read the code. That's why we tell people to
> choose meaningful variable names, instead of naming everything "a"
> and "b".

Yes. But this isn't going to cost any more time than figuring out
whether or not the list multiplication is going to cause quirks,
itself. Human psychology *tends* (it's a FAQ!) to automatically assume
the purpose of the list multiplication is to pre-allocate memory for the
equivalent (using lists) of a multi-dimensional array. Note the OP even
said "4d array".

The OP's original construction was simple, elegant, easy to read and
very commonly done by newbies learning the language because it's
*intuitive*. His second try was still intuitive, but less easy to read,
and not as elegant.

>
> * Consistency of semantics is better than a plethora of special
> cases. Python has a very simple and useful rule: objects should
> not be copied unless explicitly requested to be copied. This is
> much better than having to remember whether this operation or
> that operation makes a copy. The answer is consistent:

Bull. Even in the last thread I noted the range() object produces
special cases.
>>> range(0,5)[1]
1
>>> range(0,5)[1:3]
range(1, 3)
>>>

The principle involved is that it gives you what you *usually* want; I
read some of the documentation on why Python 3 chose to implement it
this way.

>
> (pardon me for belabouring the point here)
>
> Q: Does [0]*10 make ten copies of the integer object?
> A: No, list multiplication doesn't make copies of elements.

Neither would my idea for the vast majority of things on your first list.

Q: What about [[]]*10?
A: No, the elements are never copied.

YES! For the obvious reason that such a construction is making mutable
lists that the user wants to populate later. If they *didn't* want to
populate them later, they ought to have used tuples -- which take less
overhead. Who even does this thing you are suggesting?!

>>> a=[[]]*10
>>> a
[[], [], [], [], [], [], [], [], [], []]
>>> a[0].append(1)
>>> a
[[1], [1], [1], [1], [1], [1], [1], [1], [1], [1]]

Oops! Damn, not what anyone normal wants....

> Q: How about if the elements are subclasses of list?
> A: No, the elements are never copied.

Another poster brought that point up -- it's something I would have to
study before answering.
It's a valid objection.

>
> Q: What about other mutable objects like sets or dicts?
> A: No, the elements are never copied.

They aren't list multiplication compatible in any event! It's a total
nonsense objection.

If these are inconsistent in my idea -- OBVIOUSLY -- they are
inconsistent in Python's present implementation. You can't even
reference duplicate them NOW.

>>> { 1:'a', 2:'b', 3:'c' } * 2

Traceback (most recent call last):

File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for *: 'dict' and 'int'

> Q: How about on Tuesdays? I bet they're copied on Tuesdays.
> A: No, the elements are never copied.

That's really a stupid objection, and everyone knows it.
" Although that way may not be obvious at first unless you're Dutch. "

>
> Your proposal throws away consistency for a trivial benefit on a rare use-
> case, and replaces it with a bunch of special cases:

RARE!!!! You are NUTS!!!!

> Q: What about [[]]*10?
> A: Oh yeah, I forgot about lists, they're copied.

Yup.

> Q: How about if the elements are subclasses of list?
> A: Hmmm, that's a good one, I'm not actually sure.
>
> Q: How about if I use delegation to proxy a list?
> A: Oh no, they definitely won't be copied.

Give an example usage of why someone would want to do this. Then we can
discuss it.

> Q: What about other mutable objects like sets or dicts?
> A: No, definitely not. Unless people complain enough.

now you're just repeating yourself to make your contrived list longer --
but there's no new objections...

> Losing consistency in favour of saving a few characters for something as
> uncommon as list multiplication is a poor tradeoff. That's why this
> proposal has been rejected again and again and again every time it has
> been suggested.

Please link to the objection being proposed to the developers, and their
reasoning for rejecting it.
I think you are exaggerating.

> List multiplication [x]*n is conceptually equivalent to:

> <snip>

> This is nice and simple and efficient.

No it isn't efficient. It's *slow* when done as in your example.

> Copying other objects is slow and inefficient. Keeping list
> multiplication consistent, and fast, is MUCH more important than making
> it work as expected for the rare case of 2D arrays:

I don't think so -- again, look at range(); it was made to work
inconsistent for a "common" case.

Besides, 2D arrays are *not* rare and people *have* to copy internals of
them very often.
The copy speed will be the same or *faster*, and the typing less -- and
the psychological mistakes *less*, the elegance more.

It's hardly going to confuse anyone to say that lists are copied with
list multiplication, but the elements are not.

Every time someone passes a list to a function, they *know* that the
list is passed by value -- and the elements are passed by reference.
People in Python are USED to lists being "the" way to weird behavior
that other languages don't do.

>
> Copying those elements does not come for free.
>
> It is true that list multiplication can be much faster than a list comp.
> But that's because the list multiply doesn't have to inspect the
> elements, copy them, or engage the iteration machinery. Avoiding copying
> gives you a big saving:
>
>
> [steve@ando ~]$ python3.3 -m timeit -s "x = range(1000)"
> "[x for _ in range(100)]" # not copied
> 100000 loops, best of 3: 11.9 usec per loop
>
> [steve@ando utilities]$ python3.3 -m timeit -s "x = range(1000)"
> "[x[:] for _ in range(100)]" # copied
> 10000 loops, best of 3: 103 usec per loop
>
> So there's a factor of ten difference right there. If list multiplication
> had to make copies, it would lose much of its speed advantage.

And when multiplication doesn't make copies of *lists*, it's going
"nowhere fast", because people don't want the results that gives.

So what difference does it make? People won't make the construction
unless they wanted to make the copies in the first place. If they want
the copies, well -- copies are *slow*. Big deal.

> For large
> enough lists, or complicated enough objects, it would become slower than
> a list comprehension.

Huh? You're nuts.

> It would be even slower if list multiplication had to inspect each
> element first and decide whether or not to copy.

A single pointer comparison in a 'C' for loop takes less than 5 nano
seconds on a 1Ghz machine.
(I'll bet yours is faster than that...!)
Consider: list objects have a pointer which points back to the generic
list object -- that's all it takes to determine what the "type" is.

Your measured loop times, doing list comprehensions takes over 10
microseconds *per loop*.
Compared to what you're proposing -- The pointer compare is a mere 0.05%
change; You can't even measure that with "timeit!". BUT: The increase
in speed for not running tokenized "for" loops is *much* bigger than the
loss for a single pointer compare; so it will *usually* be a *serious*
net gain.

>> I really don't think doing a shallow copy of lists would break anyone's
>> program.
> Anyone who is currently using list multiplication with mutable objects is
> expecting that they will be the same object, and relying on that fact.
> Otherwise they wouldn't be using list multiplication.

yes, and I'm not changing that -- except for lists; and *no* one is
using that.
Find two examples of it from existing non contrived web examples of
Python code.
*ask* around.

>
> You're suggesting a semantic change. Therefore they will be expecting
> something different from what actually happens. Result: broken code.

Even if it was; So are many semantic changes happening between python 2
and python 3.
Look at what python 2 did:

>>> range(0,5)[0]
0
>>> range(0,5)[1:3]
[1, 2]

That's a *semantic* change.
Also; if you complain that xrange has been renamed range; then look:

>>> xrange(0,5)[0]
0
>>> xrange(0,5)[1:3]

Traceback (most recent call last):

File "<stdin>", line 1, in <module>
TypeError: sequence index must be integer, not 'slice'

WOW. WOW. WOW. An even BIGGER semantic change.

> It's not just mutable objects. It's also objects that can't be copied.
> Result: mylist*3 used to work, now it raises an exception. And
> performance issues: what used to be fast is now slow.

Where do you get off??; a list can be copied -- the contents might not.

> Even if this change was allowed, it would have to go through a multi-year
> process.

Fine. if that's normal -- then let them process it the normal way.
That's not my concern in the slightest.

> to get the behaviour you want, and then in Python 3.5 it would become the
> default. That's three years until it becomes the standard. Meanwhile,
> there will still be millions of people using Python 2.7 or 3.2, and their
> code will behave differently from your code.

Uh, they aren't *using* the construction I am proposing now -- they are
avoiding it like the plague.
Hence, it will merely become a new ability in a few years -- not
'differently' behaving code.

The rest of your repetitive nonsense has been deleted.
:(

[Prasad, Ramit]

Andrew Robinson wrote:
>
> On 11/06/2012 01:04 AM, Steven D'Aprano wrote:

> > On Mon, 05 Nov 2012 21:51:24 -0800, Andrew Robinson wrote:
> >

[snip]

> > Q: What about other mutable objects like sets or dicts?
> > A: No, the elements are never copied.

> They aren't list multiplication compatible in any event! It's a total
> nonsense objection.
>
> If these are inconsistent in my idea -- OBVIOUSLY -- they are
> inconsistent in Python's present implementation. You can't even
> reference duplicate them NOW.
>
> >>> { 1:'a', 2:'b', 3:'c' } * 2
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> TypeError: unsupported operand type(s) for *: 'dict' and 'int'

>>> z = [ {'a':1} ]*10
>>> z[0]['b'] = 4
>>> z
[{'a': 1, 'b': 4}, {'a': 1, 'b': 4}, {'a': 1, 'b': 4},{'a': 1, 'b': 4},
{'a': 1, 'b': 4}, {'a': 1, 'b': 4}, {'a': 1, 'b': 4}, {'a': 1, 'b': 4},
{'a': 1, 'b': 4}, {'a': 1, 'b': 4}]

Should that copy the dictionary? According to logical reasoning
it should copy the dictionary as well. How do you draw the line of
what should be copied and what should not?

>
> > Q: How about on Tuesdays? I bet they're copied on Tuesdays.
> > A: No, the elements are never copied.

> That's really a stupid objection, and everyone knows it.

Agreed. [snip]

> > Q: How about if I use delegation to proxy a list?
> > A: Oh no, they definitely won't be copied.

> Give an example usage of why someone would want to do this. Then we can
> discuss it.

IIRC, someone wanted to do something very similar for dictionaries to
prevent editing of global variables.

> > Q: What about other mutable objects like sets or dicts?
> > A: No, definitely not. Unless people complain enough.

> now you're just repeating yourself to make your contrived list longer --
> but there's no new objections...

This is my main objection and one of the flaws of your argument.
You want to handle one type of mutable objects completely separately
than other mutable objects. Why is list any different than dictionary
in this respect? The only reason I can imagine is because lists
end up being used for 2d (or higher) "matrices".

>
> > Losing consistency in favour of saving a few characters for something as
> > uncommon as list multiplication is a poor tradeoff. That's why this
> > proposal has been rejected again and again and again every time it has
> > been suggested.

> Please link to the objection being proposed to the developers, and their
> reasoning for rejecting it.
> I think you are exaggerating.

I reject (as a developer) it because it forces me to remember a very
specific quirk versus a simple (logical) rule that applies to all objects. Not to mention that the quirk is not even that useful except for beginners.

>
> > List multiplication [x]*n is conceptually equivalent to:

> > <snip>

> > This is nice and simple and efficient.

> No it isn't efficient. It's *slow* when done as in your example.
>

> > Copying other objects is slow and inefficient. Keeping list
> > multiplication consistent, and fast, is MUCH more important than making
> > it work as expected for the rare case of 2D arrays:

> I don't think so -- again, look at range(); it was made to work
> inconsistent for a "common" case.
>
> Besides, 2D arrays are *not* rare and people *have* to copy internals of
> them very often.
> The copy speed will be the same or *faster*, and the typing less -- and
> the psychological mistakes *less*, the elegance more.
>
> It's hardly going to confuse anyone to say that lists are copied with
> list multiplication, but the elements are not.
>
> Every time someone passes a list to a function, they *know* that the
> list is passed by value -- and the elements are passed by reference.
> People in Python are USED to lists being "the" way to weird behavior
> that other languages don't do.

I think you just lost 90% of your credibility (with me). When did lists
get passed by value? Python uses call by sharing[0].

Terminology aside, lists are handled exactly the same way as all
other objects; the rules regarding their mutability in the callee
are the same as dictionaries, sets, or any mutable type (including
non-builtins).

>
> >
> > Copying those elements does not come for free.
> >
> > It is true that list multiplication can be much faster than a list comp.
> > But that's because the list multiply doesn't have to inspect the
> > elements, copy them, or engage the iteration machinery. Avoiding copying
> > gives you a big saving:
> >
> >
> > [steve@ando ~]$ python3.3 -m timeit -s "x = range(1000)"
> > "[x for _ in range(100)]" # not copied
> > 100000 loops, best of 3: 11.9 usec per loop
> >
> > [steve@ando utilities]$ python3.3 -m timeit -s "x = range(1000)"
> > "[x[:] for _ in range(100)]" # copied
> > 10000 loops, best of 3: 103 usec per loop
> >
> > So there's a factor of ten difference right there. If list multiplication
> > had to make copies, it would lose much of its speed advantage.

> And when multiplication doesn't make copies of *lists*, it's going
> "nowhere fast", because people don't want the results that gives.
>
> So what difference does it make? People won't make the construction
> unless they wanted to make the copies in the first place. If they want
> the copies, well -- copies are *slow*. Big deal.
>

> > For large
> > enough lists, or complicated enough objects, it would become slower than
> > a list comprehension.

> Huh? You're nuts.

>
> > It would be even slower if list multiplication had to inspect each
> > element first and decide whether or not to copy.

> A single pointer comparison in a 'C' for loop takes less than 5 nano
> seconds on a 1Ghz machine.
> (I'll bet yours is faster than that...!)
> Consider: list objects have a pointer which points back to the generic
> list object -- that's all it takes to determine what the "type" is.
>
> Your measured loop times, doing list comprehensions takes over 10
> microseconds *per loop*.
> Compared to what you're proposing -- The pointer compare is a mere 0.05%
> change; You can't even measure that with "timeit!". BUT: The increase
> in speed for not running tokenized "for" loops is *much* bigger than the
> loss for a single pointer compare; so it will *usually* be a *serious*
> net gain.
>

> >> I really don't think doing a shallow copy of lists would break anyone's
> >> program.
> > Anyone who is currently using list multiplication with mutable objects is
> > expecting that they will be the same object, and relying on that fact.
> > Otherwise they wouldn't be using list multiplication.

> yes, and I'm not changing that -- except for lists; and *no* one is
> using that.
> Find two examples of it from existing non contrived web examples of
> Python code.
> *ask* around.

I am positive that majority of code is not examples--web or otherwise.

>
> >
> > You're suggesting a semantic change. Therefore they will be expecting
> > something different from what actually happens. Result: broken code.

> Even if it was; So are many semantic changes happening between python 2
> and python 3.
> Look at what python 2 did:
>
> >>> range(0,5)[0]
> 0
> >>> range(0,5)[1:3]
> [1, 2]
>
> That's a *semantic* change.
> Also; if you complain that xrange has been renamed range; then look:
>
> >>> xrange(0,5)[0]
> 0
> >>> xrange(0,5)[1:3]
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> TypeError: sequence index must be integer, not 'slice'
>
> WOW. WOW. WOW. An even BIGGER semantic change.

So because one thing has a semantic change that gives license
for semantic changes everywhere? Bah, ridiculous!

[snip]

>
> > to get the behaviour you want, and then in Python 3.5 it would become the
> > default. That's three years until it becomes the standard. Meanwhile,
> > there will still be millions of people using Python 2.7 or 3.2, and their
> > code will behave differently from your code.

> Uh, they aren't *using* the construction I am proposing now -- they are
> avoiding it like the plague.
> Hence, it will merely become a new ability in a few years -- not
> 'differently' behaving code.

How in the name of <insert deity (or religion)> do you have any clue
about that? Granted, as an educated you *may* be right, but you
may not be. I have no idea how you could know this definitively
or with any great degree of certainty. [snip]


~Ramit

[0] http://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_sharing

[Ian Kelly]

On Tue, Nov 6, 2012 at 3:41 PM, Andrew Robinson
<and...@r3dsolutions.com> wrote:
>> Q: What about other mutable objects like sets or dicts?
>> A: No, the elements are never copied.
>

> They aren't list multiplication compatible in any event! It's a total
> nonsense objection.
>
> If these are inconsistent in my idea -- OBVIOUSLY -- they are inconsistent
> in Python's present implementation. You can't even reference duplicate them
> NOW.
>
>>>> { 1:'a', 2:'b', 3:'c' } * 2
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> TypeError: unsupported operand type(s) for *: 'dict' and 'int'

The objection is not nonsense; you've merely misconstrued it. If
[[1,2,3]] * 4 is expected to create a mutable matrix of 1s, 2s, and
3s, then one would expect [[{}]] * 4 to create a mutable matrix of
dicts. If the dicts are not copied, then this fails for the same
reason

>> Q: How about if I use delegation to proxy a list?
>> A: Oh no, they definitely won't be copied.
>

> Give an example usage of why someone would want to do this. Then we can
> discuss it.

Seriously? Read a book on design patterns. You might start at SO:

http://stackoverflow.com/questions/832536/when-to-use-delegation-instead-of-inheritance

>> Losing consistency in favour of saving a few characters for something as
>> uncommon as list multiplication is a poor tradeoff. That's why this
>> proposal has been rejected again and again and again every time it has
>> been suggested.
>

> Please link to the objection being proposed to the developers, and their
> reasoning for rejecting it.
> I think you are exaggerating.

>From Google:

http://bugs.python.org/issue1408
http://bugs.python.org/issue12597
http://bugs.python.org/issue9108
http://bugs.python.org/issue7823

Note that in two out of these four cases, the reporter was trying to
multiply lists of dicts, not just lists of lists.

> Besides, 2D arrays are *not* rare and people *have* to copy internals of
> them very often.
> The copy speed will be the same or *faster*, and the typing less -- and the
> psychological mistakes *less*, the elegance more.

List multiplication is not potentially useful for copying 2D lists,
only for initializing them. For copying an existing nested list,
you're still stuck with either copy.deepcopy() or a list
comprehension.

> It's hardly going to confuse anyone to say that lists are copied with list
> multiplication, but the elements are not.
>
> Every time someone passes a list to a function, they *know* that the list is
> passed by value -- and the elements are passed by reference. People in
> Python are USED to lists being "the" way to weird behavior that other
> languages don't do.

Incorrect. Python uses what is commonly known as call-by-object, not
call-by-value or call-by-reference. Passing the list by value would
imply that the list is copied, and that appends or removes to the list
inside the function would not affect the original list. This is not
what Python does; the list inside the function and the list passed in
are the same list. At the same time, the function does not have
access to the original reference to the list and cannot reassign it by
reassigning its own reference, so it is not call-by-reference
semantics either.

[MRAB]

On 2012-11-06 23:52, Ian Kelly wrote:
> On Tue, Nov 6, 2012 at 3:41 PM, Andrew Robinson
> <and...@r3dsolutions.com> wrote:

>>> Q: What about other mutable objects like sets or dicts?
>>> A: No, the elements are never copied.
>>

>> They aren't list multiplication compatible in any event! It's a total
>> nonsense objection.
>>
>> If these are inconsistent in my idea -- OBVIOUSLY -- they are inconsistent
>> in Python's present implementation. You can't even reference duplicate them
>> NOW.
>>
>>>>> { 1:'a', 2:'b', 3:'c' } * 2
>>
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> TypeError: unsupported operand type(s) for *: 'dict' and 'int'
>
> The objection is not nonsense; you've merely misconstrued it. If
> [[1,2,3]] * 4 is expected to create a mutable matrix of 1s, 2s, and
> 3s, then one would expect [[{}]] * 4 to create a mutable matrix of
> dicts. If the dicts are not copied, then this fails for the same
> reason
>

>>> Q: How about if I use delegation to proxy a list?
>>> A: Oh no, they definitely won't be copied.
>>

>> Give an example usage of why someone would want to do this. Then we can
>> discuss it.
>
> Seriously? Read a book on design patterns. You might start at SO:
>
> http://stackoverflow.com/questions/832536/when-to-use-delegation-instead-of-inheritance
>

>>> Losing consistency in favour of saving a few characters for something as
>>> uncommon as list multiplication is a poor tradeoff. That's why this
>>> proposal has been rejected again and again and again every time it has
>>> been suggested.
>>

I prefer the term "reference semantics".

[Steven D'Aprano]

On Tue, 06 Nov 2012 14:41:24 -0800, Andrew Robinson wrote:

> Yes. But this isn't going to cost any more time than figuring out
> whether or not the list multiplication is going to cause quirks, itself.
> Human psychology *tends* (it's a FAQ!) to automatically assume the
> purpose of the list multiplication is to pre-allocate memory for the
> equivalent (using lists) of a multi-dimensional array. Note the OP even
> said "4d array".

I'm not entirely sure what your point is here. The OP screwed up -- he
didn't generate a 4-dimensional array. He generated a 2-dimensional
array. If his intuition about the number of dimensions is so poor, why
should his intuition about list multiplication be treated as sacrosanct?

As they say, the only truly intuitive interface is the nipple. There are
many places where people's intuition about programming fail. And many
places where Fred's intuition is the opposite of Barney's intuition.

Even more exciting, there are places where people's intuition is
*inconsistent*, where they expect a line of code to behave differently
depending on their intention, rather than on the code. And intuition is
often sub-optimal: e.g. isn't it intuitively obvious that "42" + 1 should
give 43? (Unless it is intuitively obvious that it should give 421.)

So while I prefer intuitively obvious behaviour where possible, it is not
the holy grail, and I am quite happy to give it up.

> The OP's original construction was simple, elegant, easy to read and
> very commonly done by newbies learning the language because it's
> *intuitive*. His second try was still intuitive, but less easy to read,
> and not as elegant.

Yes. And list multiplication is one of those areas where intuition is
suboptimal -- it produces a worse outcome overall, even if one minor use-
case gets a better outcome.

I'm not disputing that [[0]*n]*m is intuitively obvious and easy. I'm
disputing that this matters. Python would be worse off if list
multiplication behaved intuitively.

An analogy: the intuitively obvious thing to do with a screw is to bang
it in with a hammer. It's long, thin, has a point at the end, and a flat
head that just screams "hit me". But if you do the intuitive thing, your
carpentry will be *much worse* than the alternatives -- a hammered in
screw holds much less strongly than either a nail or a screwed in screw.
The surface area available for gripping is about 2% compared to a nail
and about 0.01% compared to a screw used correctly.

Having list multiplication copy has consequences beyond 2D arrays. Those
consequences make the intuitive behaviour you are requesting a negative
rather than a positive. If that means that newbie programmers have to
learn not to hammer screws in, so be it. It might be harder, slower, and
less elegant to drill a pilot hole and then screw the screw in, but the
overall result is better.

>> * Consistency of semantics is better than a plethora of special
>> cases. Python has a very simple and useful rule: objects should not
>> be copied unless explicitly requested to be copied. This is much
>> better than having to remember whether this operation or that
>> operation makes a copy. The answer is consistent:
>
> Bull. Even in the last thread I noted the range() object produces
> special cases.
> >>> range(0,5)[1]
> 1
> >>> range(0,5)[1:3]
> range(1, 3)

What's the special case here? What do you think is copied?

You take a slice of a tuple, you get a new tuple.

You take a slice of a list, you get a new list.

You take a slice of a range object, you get a new range object.

I'm honestly not getting what you think is inconsistent about this.

> The principle involved is that it gives you what you *usually* want;

Who is the "you" that decides what "you" usually want? And how do they
know what is "usual"?

Two-dimensional arrays in Python using lists are quite rare. Anyone who
is doing serious numeric work where they need 2D arrays is using numpy,
not lists. There are millions of people using Python, so it's hardly
surprising that once or twice a year some newbie trips over this. But
it's not something that people tend to trip over again and again and
again, like C's "assignment is an expression" misfeature.

> I read some of the documentation on why Python 3 chose to implement it
> this way.

What documentation is this? Because this is a design decision that goes
all the way back to at least Python 1.5:

[steve@ando ~]$ python1.5
Python 1.5.2 (#1, Aug 27 2012, 09:09:18) [GCC 4.1.2 20080704 (Red Hat
4.1.2-52)] on linux2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> x = [[0]*5]*3
>>> x[0][1] = 99
>>> x
[[0, 99, 0, 0, 0], [0, 99, 0, 0, 0], [0, 99, 0, 0, 0]]


So I expect the design decision for Python 3 was "we made the right
decision before, there's no need to change it".

>> (pardon me for belabouring the point here)
>>
>> Q: Does [0]*10 make ten copies of the integer object? A: No, list
>> multiplication doesn't make copies of elements.
>
> Neither would my idea for the vast majority of things on your first
> list.

Um, yes? The point is that "vast majority" is not "everything". Hence,
your suggested behaviour is inconsistent.

> Q: What about [[]]*10?
> A: No, the elements are never copied.
>
> YES! For the obvious reason that such a construction is making mutable
> lists that the user wants to populate later. If they *didn't* want to
> populate them later, they ought to have used tuples -- which take less
> overhead. Who even does this thing you are suggesting?!

Who knows? Who cares? Nobody does:

n -= n

instead of just n=0, but that doesn't mean that we should give it some
sort of special meaning different from n -= m. If it turns out that the
definition of list multiplication is such that NOBODY, EVER, uses [[]]*n,
that is *still* not a good reason for special-casing it. All it means is
that this will be a less-obscure example of the billions of things which
can be done in Python but nobody wants to.

You have quoted from the Zen of Python a few times in this post. Perhaps
you missed one of the most critical ones?

Special cases aren't special enough to break the rules.

There are perfectly good ways to generate a 2D array out of lists, and
even better reasons not to use lists for that in the first place. (Numpy
arrays are much better suited for serious work.)

>> Q: What about other mutable objects like sets or dicts? A: No, the
>> elements are never copied.
>
> They aren't list multiplication compatible in any event! It's a total
> nonsense objection.

I'm afraid you've just lost an awful lot of credibility there.

py> x = [{}]*5
py> x
[{}, {}, {}, {}, {}]
py> x[0]['key'] = 1
py> x
[{'key': 1}, {'key': 1}, {'key': 1}, {'key': 1}, {'key': 1}]

And similarly for any other mutable object.

If you don't understand that lists can contain other mutable objects
apart from lists, then you really shouldn't be discussing this issue.

>> Your proposal throws away consistency for a trivial benefit on a rare
>> use- case, and replaces it with a bunch of special cases:
>
> RARE!!!! You are NUTS!!!!

Yes, rare. I base that on about 15 years of Python coding and many
thousands (tens of thousands?) of hours on Python forums like this one.
What's your opinion based on?

List multiplication is rare enough, but when it is used, it is usually
used to generate a 1D array like this:

values = [None]*n # or 0 is another popular starting value

Using it twice to generate a 2D array is even rarer.

>> Q: How about if I use delegation to proxy a list? A: Oh no, they
>> definitely won't be copied.
>
> Give an example usage of why someone would want to do this. Then we can
> discuss it.

Proxying objects is hardly a rare scenario. Delegation is less common
since you can subclass built-ins, but it is still used. It is a standard
design pattern.

>> Losing consistency in favour of saving a few characters for something
>> as uncommon as list multiplication is a poor tradeoff. That's why this
>> proposal has been rejected again and again and again every time it has
>> been suggested.
>
> Please link to the objection being proposed to the developers, and their
> reasoning for rejecting it.
> I think you are exaggerating.

Python is a twenty year old language. Do you really think this is the
first time somebody has noticed it?

It's hard to search for discussions on the dev list, because the obvious
search terms bring up many false positives. But here are a couple of bug
reports closed as "won't fix":

http://bugs.python.org/issue1408
http://bugs.python.org/issue12597

I suspect it is long past time for a PEP so this can be rejected once and
for all.

>> List multiplication [x]*n is conceptually equivalent to: <snip>
>> This is nice and simple and efficient.
> No it isn't efficient. It's *slow* when done as in your example.

Well of course it is slow*er* when you move it from low-level C to high
level Python, but it is still fast.

>> Copying other objects is slow and inefficient. Keeping list
>> multiplication consistent, and fast, is MUCH more important than making
>> it work as expected for the rare case of 2D arrays:
>
> I don't think so -- again, look at range(); it was made to work
> inconsistent for a "common" case.

You mentioned range before, but it isn't clear to me what you think is
inconsistent about it.

> Besides, 2D arrays are *not* rare and people *have* to copy internals of
> them very often.

So you say.

> The copy speed will be the same or *faster*, and the typing less -- and
> the psychological mistakes *less*, the elegance more.

You think that it is *faster* to copy a list than to make a new pointer
to it? Your credibility is not looking too good here.

> It's hardly going to confuse anyone to say that lists are copied with
> list multiplication, but the elements are not.

Well, that confuses me. What about a list where the elements are lists?
Are they copied?

What about other mutable objects? Are they copied?

What about mutable objects which are uncopyable, like file objects?

> Every time someone passes a list to a function, they *know* that the
> list is passed by value -- and the elements are passed by reference.

And there goes the last of your credibility. *You* might "know" this, but
that doesn't make it so.

Python does not use either call-by-value or call-by-reference, and it
certainly doesn't use different calling conventions for different
arguments or parts of arguments. Everything is passed using the same
calling convention. Start here:

http://mail.python.org/pipermail/tutor/2010-December/080505.html

> People in Python are USED to lists being "the" way to weird behavior
> that other languages don't do.

Python's calling behaviour is identical to that used by languages
including Java (excluding unboxed primitives) and Ruby, to mention only
two.

You're starting to shout and yell, so perhaps it's best if I finish this
here.


--
Steven

[rusi]

On Nov 7, 5:26 am, MRAB <pyt...@mrabarnett.plus.com> wrote:
> I prefer the term "reference semantics".

Ha! That hits the nail on the head.

To go back to the OP:

On Nov 5, 11:28 am, Demian Brecht <demianbre...@gmail.com> wrote:
> So, here I was thinking "oh, this is a nice, easy way to initialize a 4D matrix" (running 2.7.3, non-core libs not allowed):
>
> m = [[None] * 4] * 4
>
> The way to get what I was after was:
>
> m = [[None] * 4, [None] * 4, [None] * 4, [None * 4]]
>

> (Obviously, I could have just hardcoded the initialization, but I'm too lazy to type all that out ;))
>

> The behaviour I encountered seems a little contradictory to me. [None] * 4 creates four distinct elements in a single array while [[None] * 4] * 4 creates one distinct array of four distinct elements, with three references to it:

>
> >>> a = [None] * 4
> >>> a[0] = 'a'
> >>> a
>
> ['a', None, None, None]
>
> >>> m = [[None] * 4] * 4
> >>> m[0][0] = 'm'
> >>> m
>
> [['m', None, None, None], ['m', None, None, None], ['m', None, None, None], ['m', None, None, None]]
>

> Is this expected behaviour and if so, why? In my mind either result makes sense, but the inconsistency is what throws me off.
>

m=[[None] * 2] * 3

is the same as

m=[[None]*2, [None]*2, [None]*2]

until one starts doing things like

m[0][0] = 'm'

So dont do it!

And to get python to help you by saying the same that I am saying do
m=((None) * 2) * 3
(well almost... its a bit more messy in practice)
m=(((None,) * 2),)*3

After that try assigning to m[0][0] and python will kindly say NO!

tl;dr version:
reference semantics is ok
assignment is ok (well up to a point)
assignment + reference semantics is not

[Steven D'Aprano]

On Wed, 07 Nov 2012 00:23:44 +0000, MRAB wrote:

>> Incorrect. Python uses what is commonly known as call-by-object, not
>> call-by-value or call-by-reference. Passing the list by value would
>> imply that the list is copied, and that appends or removes to the list
>> inside the function would not affect the original list. This is not
>> what Python does; the list inside the function and the list passed in
>> are the same list. At the same time, the function does not have access
>> to the original reference to the list and cannot reassign it by
>> reassigning its own reference, so it is not call-by-reference semantics
>> either.
>>
> I prefer the term "reference semantics".

Oh good, because what the world needs is yet another name for the same
behaviour.

- call by sharing
- call by object sharing
- call by object reference
- call by object
- call by value, where "values" are references
(according to the Java community)
- call by reference, where "references" refer to objects, not variables
(according to the Ruby community)
- reference semantics


Anything else?

http://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_sharing




--
Steven

[Roy Smith]

In article <5099ec1d$0$21759$c3e8da3$7649...@news.astraweb.com>,

Call by social network? The called function likes the object.
Depending on how it feels, it can also comment on some of the object's
attributes.

[Gregory Ewing]

Roy Smith wrote:
> Call by social network? The called function likes the object.
> Depending on how it feels, it can also comment on some of the object's
> attributes.

And then finds that it has inadvertently shared all its
private data with other functions accessing the object.

--
Greg

[Gregory Ewing]

If anything is to be done in this area, it would be better
as an extension of list comprehensions, e.g.

[[None times 5] times 10]

which would be equivalent to

[[None for _i in xrange(5)] for _j in xrange(10)]

--
Greg

[Demian Brecht]

On 2012-11-06, at 5:55 PM, Steven D'Aprano <steve+comp....@pearwood.info> wrote:
> I'm not entirely sure what your point is here. The OP screwed up -- he
> didn't generate a 4-dimensional array. He generated a 2-dimensional
> array. If his intuition about the number of dimensions is so poor, why
> should his intuition about list multiplication be treated as sacrosanct?

Yep, I may have mis-worded the explanation a bit (although I *did* express that it was a 4D matrix in the OP). I was using a 2D list to represent a 4D matrix in order to easily iterate over 90 degree rotations with zip(*matrix[::-1]). It wasn't for production code (otherwise I *would* be using numpy), it was for an online programming challenge in which external libs are not supported.

> As they say, the only truly intuitive interface is the nipple. There are
> many places where people's intuition about programming fail. And many
> places where Fred's intuition is the opposite of Barney's intuition.

I couldn't agree more with this. My question was *not* based on what I perceive to be intuitive (although most of this thread has now seemed to devolve into that and become more of a philosophical debate), but was based on what I thought may have been inconsistent behaviour (which was quickly cleared up with None being immutable and causing it to *seem* that the behaviour was inconsistent to the forgetful mind). As you touch on here, "intuition" is entirely subjective. If you're coming from a C/C++ background, I'd think that your intuition would be that everything's passed by value unless explicitly stated. Someone coming from another background (Lua perhaps?) would likely have entirely different intuition.

> So while I prefer intuitively obvious behaviour where possible, it is not
> the holy grail, and I am quite happy to give it up.

I fail to see where there has been any giving up on intuitiveness in the context of this particular topic. In my mind, intuitiveness is generally born of repetitiveness and consistency. As everything in Python is a reference, it would seem to me to be inconsistent to treat expressions such as [[obj]*4]*4 un-semantically (Pythonically speaking) and making it *less* intuitive. I agree that Python would definitely be worse off.

[Jussi Piitulainen]

Something else:

There's a call-by-* versus pass-by-* distinction, where the call-by-*
would be rather different from any of the above:

- call-by-value is what most languages now use: argument expressions
are reduced to values before they are passed to the function /
procedure / method / whatever.

- call-by-name was something Algol 60 had by default: something like
evaluating the argument expression every time its value is needed

- call-by-need: argument expression is reduced to a value the first
time its value is needed (if ever)

- call-by-lazy (increasingly silly terminology, and I don't quite have
an idea what it means in contrast to call-by-need)

The modern confusions would then be mostly over the pass-by-* family,
invariably using call-by-value in the above sense. The terminology for
these tends to produce more heat than light, but I think the relevant
distinctions are mostly just these:

- can one modify the argument effectively [Python: yes]

- can one modify the parameter with abandon [Python: don't]

- can one swap [Python: no]

- possibly: is it expensive to pass large objects? [Python: no]

The actual rule in Scheme, Java, and Python is the same simple and
sane rule: what are passed are values (argument expressions are fully
evaluated before the actual call takes place), parameter passing does
not involve any (observable) copying, and the arguments are bound to
fresh variables (no aliasing of variables).

Different communities use different words. Sometimes they use the same
words about different things. Resulting in more heat than light :(

(I'd have a few more things in the something-else department, but this
is already much longer than I thought. Ends.)

[wxjm...@gmail.com]

Le mercredi 7 novembre 2012 02:55:10 UTC+1, Steven D'Aprano a écrit :

>
>
>
>
>
>
> Two-dimensional arrays in Python using lists are quite rare. Anyone who
>
> is doing serious numeric work where they need 2D arrays is using numpy,
>
> not lists. There are millions of people using Python, so it's hardly
>
> surprising that once or twice a year some newbie trips over this. But
>
> it's not something that people tend to trip over again and again and
>
> again, like C's "assignment is an expression" misfeature.
>
>

--------------------


>>> from vecmat6 import *
>>> from vmio5 import *

Traceback (most recent call last):

File "<eta last command>", line 1, in <module>
ImportError: No module named vmio5
>>> from vmio6 import *
>>> from svdecomp6 import *
>>> mm = NewMat(3, 3)
>>> mm
[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]
>>> mm[0][0] = 1.0; mm[0][1] = 2.0; mm[0][2] = 3.0
>>> mm[1][0] = 11.0; mm[1][1] = 12.0; mm[1][2] = 13.0
>>> mm[2][0] = 21.0; mm[2][1] = 22.0; mm[2][2] = 23.0
>>> pr(mm, 'mm=')
mm=
( 1.00000e+000 2.00000e+000 3.00000e+000 )
( 1.10000e+001 1.20000e+001 1.30000e+001 )
( 2.10000e+001 2.20000e+001 2.30000e+001 )
>>> aa, b, cc = SVDecomp(mm)
>>> pr(aa, 'aa=')
aa=
( -8.08925e-002 -9.09280e-001 4.08248e-001 )
( -4.77811e-001 -3.24083e-001 -8.16497e-001 )
( -8.74730e-001 2.61114e-001 4.08248e-001 )
>>> pr(b, 'b=')
b=
( 4.35902e+001 1.37646e+000 1.93953e-016 )
>>> pr(cc, 'cc=')
cc=
( -5.43841e-001 7.33192e-001 4.08248e-001 )
( -5.76726e-001 2.68499e-002 -8.16497e-001 )
( -6.09610e-001 -6.79492e-001 4.08248e-001 )
>>> bb = VecToDiagMat(b)
>>> cct = TransposeMat(cc)
>>> oo = MatMulMatMulMat(aa, bb, cct)
>>> pr(oo, 'aa * bb * cct=')
aa * bb * cct=
( 1.00000e+000 2.00000e+000 3.00000e+000 )
( 1.10000e+001 1.20000e+001 1.30000e+001 )
( 2.10000e+001 2.20000e+001 2.30000e+001 )
>>>
>>> # or
>>> oo
[[0.9999999999999991, 1.9999999999999993, 2.9999999999999982],
[10.999999999999995, 11.99999999999999, 12.999999999999996],
[20.999999999999986, 21.999999999999975, 22.999999999999986]]



jmf

[Oscar Benjamin]

On 7 November 2012 13:39, Joshua Landau <joshua.l...@gmail.com> wrote:
>
> On 7 November 2012 11:11, Oscar Benjamin <oscar.j....@gmail.com> wrote:
>>
>> A more modest addition for the limited case described in this thread could
>> be to use exponentiation:
>>
>> >>> [0] ** (2, 3)
>> [[0, 0, 0], [0, 0, 0]]
>
> Hold on: why not just use multiplication?
>
>>>> [0] * (2, 3)
>
> is an error now, and it makes total sense. Additionally, it's not breaking
> the "no copy -- _ever_" rule because none of the lists existed before. The
> values inside the list would be by reference, as before, so lst * (x,) would
> be the same as lst * x if x is an integer.

The problem is that this operation is asymmetric. Currently int/list
multiplication is commutative so that:

['a', 'b'] * 2 == 2 * ['a', 'b']

If you use this kind of multiplication what happens to the other
cases? e.g. what do you give for:

>>> [0] * [2, 3]

>>> [2, 3] * [0]

>>> (2, 3) * [0]

>>> (2, 3) * (4, 5)

and so on. Although Python does not guarantee commutativity of
multiplication in general I think that since for lists it has always
been commutative it would be bad to change that.

Exponentiation is expected to be asymmetric and is currently unused so
there is no ambiguity. The problem is if someone has already
subclassed list and added an exponentiation method.


Oscar

[Ethan Furman]

After this post the only credibility you have left (with me, anyway) is that you seem to be willing
to learn. So learn the way Python works before you try to reimplement it.

~Ethan~

[Ethan Furman]

Oscar Benjamin wrote:
> On Nov 7, 2012 5:41 AM, "Gregory Ewing" <greg....@canterbury.ac.nz

> I think you're right that the meaning of list-int multiplication
> can't/shouldn't be changed if this way.
>
> A multidimensional list comprehension would be useful even for people
> who are using numpy as it's common to use a list comprehension to
> initialise a numpy array.

>
> A more modest addition for the limited case described in this thread
> could be to use exponentiation:
>
> >>> [0] ** (2, 3)
> [[0, 0, 0], [0, 0, 0]]

What would happen with

--> [{}] ** (2, 3)

or

--> [my_custom_container()] ** (2, 3)

?

~Ethan~

[Prasad, Ramit]

And this is where Dihedral 8888 (or whatever the bot is called)
tells you that Python has no "private" variables. :)


~Ramit

[MRAB]

The disadvantage of calling it "call by ..." is that it suggests that
you're just talking about calling functions.

What about binding in general, eg "x = y"? Does it make sense to still
call it "call by ..."?

[Ian Kelly]

On Wed, Nov 7, 2012 at 12:51 PM, Andrew Robinson
<and...@r3dsolutions.com> wrote:
> Interesting, you avoided the main point "lists are copied with list
> multiplication".

It seems that each post is longer than the last. If we each responded
to every point made, this thread would fill a book.

Anyway, your point was to suggest that people would not be confused by
having list multiplication copy lists but not other objects, because
passing lists into functions as parameters works in basically the same
way. Except that it does not work the same way, because when lists
are passed into functions, they are not copied at all. Nor are are
any of their contents copied, lists or not. So actually I did address
this point with the "call-by-object" tangent; I just did not
explicitly link it back to your thesis.

> But, in any event:
> Pass by value (not call by value) is a term stretching back 30 years; eg:
> when I learned the meaning of the words. Rewording it as "Call by value" is
> something that happened later, and the nuance is lost on those without a
> very wide programming knowledge *and* age.

Potayto, potahto. The distinction that you're describing is between
"strict" versus "non-strict" evaluation strategies. Hinging the
distinction on the non-descriptive words "call" and "pass" is lazy
terminology that should never have been introduced in the first place.

> In any event:
> All objects in Python are based on pointers; all parameters passed to
> functions, etc, are *copies* of those pointers; (by pointer value).

No, all parameters passed to functions are *objects*. Python itself
has no concept of pointers. What you describe is true as an
implementation detail for CPython but not necessarily true for other
implementations, and not true at all for an abstract
(implementation-independent) view of the language.

> I made the distinction between contents of the list and the list object
> itself for that reason; I gave an explicit correction to the pass by "value"
> generalization by saying: ("the elements are passed by reference").

The elements are not passed anywhere. Only the list object is passed
to the function, which is completely agnostic of the fact that the
list object happens to contain other objects.

[Mark Lawrence]

On 07/11/2012 22:02, Andrew Robinson wrote:
>

You're doing extremely well, you've overtaken Xah Lee as the biggest
waste of space on this list.

--
Cheers.

Mark Lawrence.

[Steven D'Aprano]

On Wed, 07 Nov 2012 17:17:02 +0000, MRAB wrote:

> The disadvantage of calling it "call by ..." is that it suggests that
> you're just talking about calling functions.

*shrug*

There are already two synonyms for this, "call by ..." and "pass by ...".
They are old, venerable terms dating back to Algol and possibly even
older. All the way back to Fortran perhaps?

> What about binding in general, eg "x = y"? Does it make sense to still
> call it "call by ..."?

Sure, why not? The person who prepares beef tartare or sushimi is still
called the cook.


--
Steven

[Oscar Benjamin]

On 7 November 2012 22:16, Joshua Landau <joshua.l...@gmail.com> wrote:

> On 7 November 2012 14:00, Oscar Benjamin <oscar.j....@gmail.com> wrote:
>> On 7 November 2012 13:39, Joshua Landau <joshua.l...@gmail.com>
>> wrote:
>> > On 7 November 2012 11:11, Oscar Benjamin <oscar.j....@gmail.com>
>> > wrote:

>> >> A more modest addition for the limited case described in this thread
>> >> could
>> >> be to use exponentiation:
>> >>
>> >> >>> [0] ** (2, 3)
>> >> [[0, 0, 0], [0, 0, 0]]
>>

>> Exponentiation is expected to be asymmetric and is currently unused so
>> there is no ambiguity. The problem is if someone has already
>> subclassed list and added an exponentiation method.
>

> How is that a problem? They just wont get the functionality.

This is absolutely contrived but:
Library A defines a subclass of list that adds an exponentiation
operator thinking that it's okay to still use these objects as lists.
Library B has an API that expects a list and tries to use the list
copy-exponentiation on its input. A user passes a list type object
from library A into library B and hopefully gets an error but possibly
gets a subtle bug that is hard to track down.

It doesn't sound plausible to me but at least in principle there is a
backward compatibility problem.

> That said, losing:
> [0] * (2, 3) == [0] * [2, 3]
> would mean losing duck-typing in general. *Thus*, I fully agree with your
> choice of exponentiation.

Also there's no reason why tuples couldn't have the same
exponentiation operator (although for them it would be no different
from repeated multiplication).


Oscar

[Ian Kelly]

On Wed, Nov 7, 2012 at 3:02 PM, Andrew Robinson
<and...@r3dsolutions.com> wrote:
> Draw up some use cases for the multiplication operator (I'm calling on your
> experience, let's not trust mine, right?); What are all the Typical ways
> people *Do* to use it now?
>
> If those use cases do not *primarily* center around *wanting* an effect
> explicitly caused by reference duplication -- then it may be better to
> abolish list multiplication all together; and rather, improve the list
> comprehensions to overcome the memory, clarity, and speed pitfalls in the
> most common case of initializing a list.

Why? Just to get rid of an FAQ?

Here's one of the more interesting uses from my own code:

values = zip(samples, times * num_groups)
if len(values) < len(times) * num_groups:
# raise an error

Converting that multiplication to a generator expression would look like this:

values = zip(samples, (t for _ in range(num_groups) for t in times))

That's not particularly hairy, but I do assert that it is
substantially less readable, and more so because it loses the symmetry
with the following if condition.

The recipes in the itertools docs also include this example, which
notably depends on the list containing multiple references to the same
iterator:

def grouper(n, iterable, fillvalue=None):
"Collect data into fixed-length chunks or blocks"
# grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx
args = [iter(iterable)] * n
return izip_longest(fillvalue=fillvalue, *args)

Replacing the list multiplication in that function with a list
comprehension would be awkward, as the obvious replacement of
[iter(iterable) for _ in range(n)] would produce different results.


> For example, in initialization use cases; often the variable of a for loop
> isn't needed and all the initializers have parameters which only need to be
> evaluated *once* (no side effects).
>
> Hence, there is an opportunity for speed and memory gains,while maintaining
> clarity and *consistency*.
>
> Some ideas of use cases:
> [ (0) in xrange(10) ] # The function to create a tuple cache's the
> parameter '0', makes 10 (0)'s
> [ dict.__new__(dict) in xrange(10) ] # dict.__new__, The dict parameter is
> cached -- makes 10 dicts.
> [ lambda x:(0) in xrange(10) ] # lambda caches (0), returns a *reference* to
> it multiple times.

How exactly do you propose to indicate to the compiler which parts of
the expressions are meant to be cached, and which are not?

>>> Bull. Even in the last thread I noted the range() object produces
>>> special cases.
>>> >>> range(0,5)[1]
>>> 1
>>> >>> range(0,5)[1:3]
>>> range(1, 3)
>>
>> What's the special case here? What do you think is copied?
>>
>>

>> You take a slice of a range object, you get a new range object.
>

> You were'nt paying attention, OCCASIONALLY, get an integer, or a list.
>>>> range(3)[2]
> 2
>
> LOOOOK! That's not a range object, that's an integer. Use Python 3.2 and
> try it.

Of course you got an integer. You took an index of the range object,
not a slice. The rule is that taking an index of a sequence returns
an element; taking a slice of a sequence returns a sub-sequence. You
still have not shown any inconsistency here.

> Game programmers routinely use 2D lists to represent the screen layout;
> For example, they might use 'b' to represent a brick tile, and 'w' to
> represent a water tile.

In many cases it may be simpler to use a plain list of strings:

screen = [
"sssss",
"ssbss",
"sbbbs",
"bbbbb",
]

> py> x = [{}]*5
> py> x
> [{}, {}, {}, {}, {}]
>

> No, I showed what happed when you do {}*3;
> That *DOESN'T* work; You aren't multiplying the dictionary, you are
> multiplying the LIST of dictionaries. Very different things.
> You were complaining that my method doesn't multiply them -- well, gee --
> either mine DOES or python DOESN'T. Double standards are *crap*.

No, he wasn't. He was talking about multiplying lists of dicts, and
whether the dicts are then copied or not, just like every other Q&A
item in that dialogue was concerning whether item X in a list should
expect to be copied when the containing list is multiplied.

You are the only one talking about applying the multiplication
operator to dicts.

> Huh?
> I'm not yelling any more than you are. Are ???YOU??? yelling?

Perhaps you're not aware that on the Internet, TYPING IN ALL CAPS is
commonly construed as SHOUTING.

[Steven D'Aprano]

Andrew, it appears that your posts are being eaten or rejected by my
ISP's news server, because they aren't showing up for me. Possibly a side-
effect of your dates being in the distant past? So if you have replied to
any of my posts, I haven't seen them.

In any case, I wanted to ask a question:

On Wed, 07 Nov 2012 14:01:19 -0700, Ian Kelly wrote:
> On Wed, Nov 7, 2012 at 12:51 PM, Andrew Robinson
> <and...@r3dsolutions.com> wrote:

[...]

>> But, in any event:
>> Pass by value (not call by value) is a term stretching back 30 years;
>> eg: when I learned the meaning of the words. Rewording it as "Call by
>> value" is something that happened later, and the nuance is lost on
>> those without a very wide programming knowledge *and* age.

Every now and again I come across somebody who tries to distinguish
between "call by foo" and "pass by foo", but nobody has been able to
explain the difference (if any) to me. When you CALL a function, you PASS
values to it. Hence the two terms are effectively synonyms, and both
refer to the evaluation strategy when binding arguments to parameters.

If you believe that is incorrect, can you point me to something
explaining the difference?



--
Steven

[Andrew Robinson]

On 11/07/2012 01:01 PM, Ian Kelly wrote:
> On Wed, Nov 7, 2012 at 12:51 PM, Andrew Robinson
> <and...@r3dsolutions.com> wrote:

>> Interesting, you avoided the main point "lists are copied with list
>> multiplication".
> It seems that each post is longer than the last. If we each responded
> to every point made, this thread would fill a book.

It already is :)

>
> Anyway, your point was to suggest that people would not be confused by
> having list multiplication copy lists but not other objects, because
> passing lists into functions as parameters works in basically the same
> way.

Not quite; Although I wasn't clear; The variable passed in is by
*value* in contradistinction to the list which is by reference. Python
does NOT always default copy by reference *when it could*; that's the point.

Hence the programmer has to remember in foo( x,y ), the names x and y
when assigned to -- *DONT* affect the variables from which they came.
But any object internals do affect the objects everywhere.

A single exception exists; My thesis is for a single exception as well
-- I think Python allows that kind of thinking.

> So actually I did address
> this point with the "call-by-object" tangent; I just did not
> explicitly link it back to your thesis.

My apology for not proof reading my statements for clarity. It was
definitely time for a nap back then.

> Potayto, potahto. The distinction that you're describing is between
> "strict" versus "non-strict" evaluation strategies. Hinging the
> distinction on the non-descriptive words "call" and "pass" is lazy
> terminology that should never have been introduced in the first place.

I would do it again. Other's have already begun to discuss terminology
with you -- I won't double team you.

[Oscar Benjamin]

On 8 November 2012 00:00, Steven D'Aprano

Did you also miss MRAB's post above? It made sense to me.

MRAB wrote:
> The disadvantage of calling it "call by ..." is that it suggests that
> you're just talking about calling functions.
>

[Greg Ewing]

On 08/11/12 12:06, Oscar Benjamin wrote:
> On 7 November 2012 22:16, Joshua Landau<joshua.l...@gmail.com> wrote:
>> That said, losing:
>> [0] * (2, 3) == [0] * [2, 3]
>> would mean losing duck-typing in general.

There are precedents for this kind of thing; the
string % operator treats tuples specially, for
example.

I don't think it's all that bad if you regard
the tuple as effectively part of the syntax.

--
Greg

[Steven D'Aprano]

On Thu, 08 Nov 2012 00:30:53 +0000, Oscar Benjamin wrote:

>> Every now and again I come across somebody who tries to distinguish
>> between "call by foo" and "pass by foo", but nobody has been able to
>> explain the difference (if any) to me. When you CALL a function, you
>> PASS values to it. Hence the two terms are effectively synonyms, and
>> both refer to the evaluation strategy when binding arguments to
>> parameters.
>>
>> If you believe that is incorrect, can you point me to something
>> explaining the difference?
>
> Did you also miss MRAB's post above? It made sense to me.

You mean MRABs post which I replied to?

Yes, I must have missed it :-P

But seriously, no I didn't miss it. He doesn't give any evidence that
there is a difference between "call by ..." and "pass by ..." when
talking about binding arguments to formal parameters. His objection to
"call by ..." is that it doesn't make it clear that the evaluation rules
apply to simple binding/assignment as well as calling functions.



--
Steven

[Steven D'Aprano]

On Wed, 07 Nov 2012 16:24:22 -0800, Andrew Robinson wrote:

> On 11/07/2012 01:01 PM, Ian Kelly wrote:

[...]

>> Anyway, your point was to suggest that people would not be confused by
>> having list multiplication copy lists but not other objects, because
>> passing lists into functions as parameters works in basically the same
>> way.
>
> Not quite; Although I wasn't clear; The variable passed in is by
> *value* in contradistinction to the list which is by reference. Python
> does NOT always default copy by reference *when it could*; that's the
> point.

It isn't clear to me whether you are describing what you think Python
*actually* does, versus what you wish it *would* do, or what it *could*
do in some abstract hypothetical sense.

It certainly is not true that Python passes "the variable" by value, and
lists "by reference". Arguments are not passed to functions either by
value or by reference.

There is a trivial test for pass-by-value semantics: does the value get
copied? We can see that Python does not copy arguments:

py> def test(x):
... print id(x)
...
py> spam = []
py> print id(spam); test(spam)
3071264556
3071264556

The argument is not copied, therefore Python is not pass-by-value.

There is also an easy test for pass-by-reference semantics: can you write
a procedure which, given two variables, swaps the contents of the
variables? In Pascal, that is trivial.

procedure swap(var a: int, var b: int):
var
tmp: int;
begin
tmp := a;
a := b;
b := a;
end;

swap(x, y);

(if I've remembered my Pascal syntax correctly).


In Python, you can swap two values like this:

a, b = b, a

but that's not sufficient. The test is to do the swap inside a function:

def swap(a, b):
return b, a

b, a = swap(a, b)

But that fails too, since the assignment is still taking place outside
the function.

It turns out that there is no way in Python to write such a swap
function. Tricks such as passing the variable names as strings, then
using exec, are hacks and don't count. Python is not pass by reference
either.

> Hence the programmer has to remember in foo( x,y ), the names x and y
> when assigned to -- *DONT* affect the variables from which they came.
> But any object internals do affect the objects everywhere.

Ummm.... yes?

The programmer has to remember Python's execution model in order to
correctly predict what Python will do. What's your point?


> A single exception exists;

There is no such exception in Python. Python always uses the same
argument passing (parameter binding) semantics.



--
Steven

[Andrew Robinson]

On 11/07/2012 04:00 PM, Steven D'Aprano wrote:
> Andrew, it appears that your posts are being eaten or rejected by my
> ISP's news server, because they aren't showing up for me. Possibly a side-
> effect of your dates being in the distant past?

Date has been corrected since two days ago. It will remain until a
reboot....
Ignorance, though, might be bliss...

> Every now and again I come across somebody who tries to distinguish
> between "call by foo" and "pass by foo", but nobody has been able to
> explain the difference (if any) to me.

I think the "Call by foo" came into vogue around the time of C++; Eg:
It's in books like C++ for C programmers; I never saw it used before
then so I *really* don't know for sure...

I know "Pass by value" existed all the way back to the 1960's. I see
"pass by" in my professional books from those times and even most newer
ones; but I only find "Call by value" in popular programming books of
more recent times. (Just my experience) So -- I "guess" the reason is
that when invoking a subroutine, early hardware often had an assembler
mnemonic by the name "call".

See for example: Intelx86 hardware books from the 1970's;

Most early processors (like the MC6809E, and 8080) allow both direct and
indirect *references* to a function (C would call them function
pointers); So, occasionally early assembly programs comment things like:
"; dynamic VESA libraries are called by value in register D."; And they
meant that register D is storing a function call address from two or
more vesa cards. It had little to do with the function's parameters,
(which might be globals anyway) (It procedural dynamic binding!)

Today, I don't know for sure -- so I just don't use it.
"pass" indicates a parameter of the present call; but not the present
call itself.

[Ian Kelly]

On Wed, Nov 7, 2012 at 8:13 PM, Andrew Robinson
<and...@r3dsolutions.com> wrote:
> OK, and is this a main use case? (I'm not saying it isn't I'm asking.)

I have no idea what is a "main" use case.

> There is a special keyword which signals the new type of comprehension; A
> normal comprehension would say eg: '[ foo for i in xrange ]'; but when the
> 'for i in' is reduced to a specific keyword such as 'ini' (instead of
> problematic 'in') the caching form of list comprehension would start.

FYI, the Python devs are not very fond of adding new keywords. Any
time a new keyword is added, existing code that uses that word as a
name is broken. 'ini' is particularly bad, because 1) it's not a
word, and 2) it's the name of a common type of configuration file and
is probably frequently used as a variable name in relation to such
files.

> So, then, just like a comprehension -- the interpreter will begin to
> evaluate the code from the opening bracket '['; But anything other than a
> function/method will raise a type error (people might want to change that,
> but it's safe).
>
> The interpreter then caches all functions/initialiser methods it comes into
> contact with.
> Since every function/method has a parameter list (even if empty); The
> interpreter would evaluate the parameter list on the first pass through the
> comprehension, and cache each parameter list with it's respective function.
>
> When the 'ini' keyword is parsed a second time, Python would then evaluate
> each cached function on its cached parameter list; and the result would be
> stored in the created list.
> This cached execution would be repeated as many times as is needed.
>
> Now, for your example:

>
> values = zip(samples, times * num_groups)
> if len(values) < len(times) * num_groups:
> # raise an error
>

> Might be done with:
>
> values = zip( samples, [ lambda:times, ini xrange(num_groups) ] )

>
> if len(values) < len(times) * num_groups
>

> The comma after the lambda is questionable, and this construction would be
> slower since lambda automatically invokes the interpreter; but it's correct.

How is this any better than the ordinary list comprehension I already
suggested as a replacement? For that matter, how is this any better
than list multiplication? Your basic complaint about list
multiplication as I understand it is that the non-copying semantics
are unintuitive. Well, the above is even less intuitive. It is
excessively complicated and almost completely opaque. If I were to
come across it outside the context of this thread, I would have no
idea what it is meant to be doing.

> As an aside, how would you do the lambda inside a list comprehension?

As a general rule, I wouldn't. I would use map instead.

> [lambda:6 for i in xrange(10) ] # Nope.

Thak constructs a list of 10 functions and never calls them. If you
want to actually call the lambda, then:

[(lambda: 6)() for i in range(10)]

or:

map(lambda i: 6, range(10))

But note that the former creates equivalent 10 functions and calls
each of them once, whereas the latter creates one function and calls
it ten times.

>> Of course you got an integer. You took an index of the range object, not a
>> slice. The rule is that taking an index of a sequence returns an element;
>> taking a slice of a sequence returns a sub-sequence. You still have not
>> shown any inconsistency here.
>
>

> Because it's an arbitrary rule which operates differently than the
> traditional idea shown in python docs?
>
> slice.indices() is *for* (QUOTE)"representing the set of indices specified
> by range(start, stop, step)"
> http://docs.python.org/2/library/functions.html#slice

slice.indices() has nothing to do with it. Indexing a sequence and
calling the .indices() method on a slice are entirely different
operations. The slice.indices method is a utility method meant to be
called by __getitem__ implementations when doing slicing, not an
implementation of indexing. When a sequence is indexed, there is no
slice. That method is not related in any way to the semantics of
indexing a sequence.

[w...@mac.com]

On Nov 7, 2012, at 11:51 PM, Andrew Robinson <and...@r3dsolutions.com> wrote:

> On 11/07/2012 04:00 PM, Steven D'Aprano wrote:

>> Andrew, it appears that your posts are being eaten or rejected by my
>> ISP's news server, because they aren't showing up for me. Possibly a side-
>> effect of your dates being in the distant past?

> Date has been corrected since two days ago. It will remain until a reboot....
> Ignorance, though, might be bliss...
>

>> Every now and again I come across somebody who tries to distinguish between "call by foo" and "pass by foo", but nobody has been able to explain the difference (if any) to me.

> I think the "Call by foo" came into vogue around the time of C++; Eg: It's in books like C++ for C programmers; I never saw it used before then so I *really* don't know for sure…
>

Just as an aside - there is a famous quote from Niklaus Wirt, who, when asked how he pronounced his name, is said to have replied: "Well you can call me by name, Veert, or you can call me by value, Worth."

That would have been sometime in the early 60s, when he was at Stanford.

-Bill

[Ian Kelly]

On Thu, Nov 8, 2012 at 1:26 AM, Andrew Robinson
<and...@r3dsolutions.com> wrote:
> OK: Then copy by reference using map....:
>
> values = zip( map( lambda:times, xrange(num_groups) ) )
> if len(values) < len(times) * num_groups ...
>
> Done. It's clearer than a list comprehension and you still really don't
> need a list multiply.

That is not equivalent to the original. Even had you not omitted some parts:

values = zip(samples, map(lambda i: times, range(num_groups)))

This still has the problem that map returns a list of num_groups
elements, each of which is times. The desired value to be passed into
zip is a *single* sequence containing len(times) * num_groups
elements. This is easily handled by list multiplication, but not so
easily by map or by a single list comprehension. Looking back at the
'ini' solution you proposed before, I see that this also would be a
problem there. Fixing the above, it would have to be something like:

values = zip(samples, reduce(operator.add, map(lambda i: times,
range(num_groups)), []))

Or from how I understand the 'ini' syntax to work:

values = zip(samples, reduce(operator.add, [lambda: times, ini
xrange(num_groups)], []))

Which brings to mind another point that I want to get to in a moment.
But when I said that I would use map instead, I meant that *if* the
body of the list comprehension is just a function application, then I
would prefer to use map over the list comprehension. But in the above
I see no benefit in using a lambda in the first place.

Getting back to that other point, notice what we ended up doing in
both of those constructions above: repeated list concatenation as a
substitute for multiplication. In fact, when we multiply (aList * 5),
this should be the equivalent of (aList + aList + aList + aList
+aList), should it not? Clearly, however, there should be no implicit
copying involved in mere list concatenation. For one thing, if the
user wants to concatenate copies, that is quite easily done
explicitly: (aList[:] + aList[:]) instead of (aList + aList). For
another, list concatenation is less likely to be used for an
initialization process. If list multiplication were to copy nested
lists, then, this would break the intuitive notion that list
multiplication is equivalent to repeated list concatenation.

> Yes, but you're very blind to history and code examples implementing the
> slice operation.
> slice usually depends on index; index does not depend on slice.
> Slice is suggested to be implemented by multiple calls to single indexes in
> traditional usage and documentation.

...and then by composing the elements located at those indexes into a
subsequence.

> The xrange(,,)[:] implementation breaks the tradition, because it doesn't
> call index multiple times; nor does it return a result equivalent identical
> to doing that.

Whether any given __getitem__ slicing implementation recursively calls
__getitem__ with a series of indexes or not is an implementation
detail. If it were possible to index a range object multiple times
and then stuff the results into another range object, then the slicing
result would be equivalent. The only reason it is not is that you
cannot construct a range object in that fashion.

I think that what you're expecting is that range(5)[:] should return a
list in Python 3 because it returns a list in Python 2. This does not
represent a change in slicing behavior -- in fact, all you got by
slicing an xrange object in Python 2 was a TypeError. This represents
an intentional break in backward compatibility between Python 2 and
Python 3, which was the purpose of Python 3 -- to fix a lot of
existing warts in Python by breaking them all at once, rather than
progressively over a long string of versions. Users porting their
scripts from Python 2 to Python 3 are advised to replace "range(...)"
with "list(range(...))" if what they actually want is a list, and I
believe the 2to3 tool does this automatically. Once the range object
is converted to a list, there is no further break with Python 2 --
slicing a list gives you a list, just as it always has.

In a nutshell, yes: range(...)[:] produces a different result in
Python 3 than in Python 2, just as it does without the slicing
operation tacked on. It was never intended that scripts written for
Python 2 should be able to run in Python 3 unchanged without careful
attention to detail.

[88888 Dihedral]

On Monday, November 5, 2012 3:07:12 PM UTC+8, Chris Rebert wrote:
> On Sun, Nov 4, 2012 at 10:27 PM, Demian Brecht <demian...@gmail.com> wrote:
>
> > So, here I was thinking "oh, this is a nice, easy way to initialize a 4D matrix" (running 2.7.3, non-core libs not allowed):
>
> >
>
> > m = [[None] * 4] * 4
This is not clear in a name binding objective
programming language.

b=[1,2,3,4]*4
mb=[ b]*4

# check the behaviors and usages of reference copies
# and shadow value copies and deep-value copies





>
> >
>
> > The way to get what I was after was:
>
> >
>
> > m = [[None] * 4, [None] * 4, [None] * 4, [None * 4]]
>
> >
>
> > (Obviously, I could have just hardcoded the initialization, but I'm too lazy to type all that out ;))
>
> >
>
> > The behaviour I encountered seems a little contradictory to me.
>
> > [None] * 4 creates four distinct elements in a single array
>
> > while [[None] * 4] * 4 creates one distinct array of four distinct elements, with three references to it:
>
>
>
> Incorrect. In /both/ cases, the result is a list of length 4, whose
>
> elements are 4 (references to) the exact same object as the original
>
> list's element.
>
> Put simply, the list multiplication operator never copies objects; it
>
> just makes additional references to them.
>
>
>
> However, unlike a list object (as in your latter example), the object
>
> `None` is completely immutable (and what's more, a singleton value),
>
> so you just-so-happen *not to be able to* run into the same problem of
>
> mutating an object (assignment to an index of a list constitutes
>
> mutation of that list) that is referenced in multiple places, for you
>
> cannot mutate None in the first place!:
>
> >>> x = None
>
> >>> x.a = 42

>
> Traceback (most recent call last):
>

> File "<stdin>", line 1, in <module>
>
> AttributeError: 'NoneType' object has no attribute 'a'
>
> >>> # it doesn't overload any mutating operators:
>
> >>> type(None).__dict__.keys()
>
> ['__hash__', '__repr__', '__doc__']
>
> >>> # and it obviously has no instance variables,
>
> >>> # so, we can't modify it in any way whatsoever!
>
> (Lists, on the other hand, define item assignment, .pop(), .remove(),
>
> and a few other mutator methods.)
>
>
>
> >>>> a = [None] * 4
>
> >>>> a[0] = 'a'
>
> >>>> a
>
> > ['a', None, None, None]
>
> >
>
> >>>> m = [[None] * 4] * 4
>
> >>>> m[0][0] = 'm'
>
> >>>> m
>
> > [['m', None, None, None], ['m', None, None, None], ['m', None, None, None], ['m', None, None, None]]
>
> >
>
> > Is this expected behavior
>
>
>
> Yes. It's also a FAQ:
>
> http://docs.python.org/2/faq/programming.html#how-do-i-create-a-multidimensional-list
>
>
>
> > and if so, why?
>
>
>
> It's a general (albeit AFAIK unstated) principle that Python never
>
> copies objects unless you explicitly ask it to. You have encountered
>
> one example of this rule in action.
>
>
>
> > In my mind either result makes sense, but the inconsistency is what throws me off.
>
>
>
> It is perfectly consistent, once you understand what list
>
> multiplication actually does.
>
>
>
> Cheers,
>
> Chris
>
> --
>
> http://rebertia.com

[88888 Dihedral]

[Mark Lawrence]

On 07/11/2012 01:55, Steven D'Aprano wrote:
>
> Who knows? Who cares? Nobody does:
>
> n -= n
>

But I've seen this scattered through code:

x := x - x - x

--
Cheers.

Mark Lawrence.

[Chris Angelico]

On Fri, Nov 9, 2012 at 12:39 PM, Mark Lawrence <bream...@yahoo.co.uk> wrote:
> On 07/11/2012 01:55, Steven D'Aprano wrote:
>>
>>
>> Who knows? Who cares? Nobody does:
>>
>> n -= n
>>
>
> But I've seen this scattered through code:
>
> x := x - x - x

Can you enlighten us as to how this is better than either:
x := -x
or
x := 0 - x
? I'm not seeing it. And I'm not seeing any nonnumeric that would
benefit from being subtracted from itself twice (strings, arrays,
sets, you can subtract them from one another but not usefully more
than once).

ChrisA

[Steven D'Aprano]

On Fri, 09 Nov 2012 17:07:09 +1100, Chris Angelico wrote:

> On Fri, Nov 9, 2012 at 12:39 PM, Mark Lawrence <bream...@yahoo.co.uk>
> wrote:
>> On 07/11/2012 01:55, Steven D'Aprano wrote:
>>>
>>>
>>> Who knows? Who cares? Nobody does:
>>>
>>> n -= n
>>>
>>>
>> But I've seen this scattered through code:
>>
>> x := x - x - x
>
> Can you enlighten us as to how this is better than either:
> x := -x
> or
> x := 0 - x
> ? I'm not seeing it.

I'm hoping that Mark intended it as an example of crappy code he has
spotted in some other language rather than a counter-example of something
you would do.

To be pedantic... there may very well be some (rare) cases where you
actually do want x -= x rather than just x = 0. Consider the case where x
could be an INF or NAN. Then x -= x should give x = NAN rather than zero.
That may be desirable in some cases.

At the very least, the compiler should NOT optimize away x = x - x to
x = 0 if x could be a float, complex or Decimal.

> And I'm not seeing any nonnumeric that would
> benefit from being subtracted from itself twice (strings, arrays, sets,
> you can subtract them from one another but not usefully more than once).

How do you subtract strings?



--
Steven

[Chris Angelico]

On Fri, Nov 9, 2012 at 5:37 PM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
> On Fri, 09 Nov 2012 17:07:09 +1100, Chris Angelico wrote:
>> Can you enlighten us as to how this is better than either:
>> x := -x
>> or
>> x := 0 - x
>> ? I'm not seeing it.
>
> I'm hoping that Mark intended it as an example of crappy code he has
> spotted in some other language rather than a counter-example of something
> you would do.

Ohh. Yeah, that figures. Huh.

> To be pedantic... there may very well be some (rare) cases where you
> actually do want x -= x rather than just x = 0. Consider the case where x
> could be an INF or NAN. Then x -= x should give x = NAN rather than zero.
> That may be desirable in some cases.
>
> At the very least, the compiler should NOT optimize away x = x - x to
> x = 0 if x could be a float, complex or Decimal.

Yep. In the specific case of integers, though, and in the specific
instance of CPU registers in assembly language, it's reasonable to
optimize it the *other* way - MOV reg,0 is a one-byte opcode and 1, 2,
or 4 bytes of immediate data, while SUB reg,reg (or XOR reg,reg) is a
two-byte operation regardless of data size. But that's
microoptimization that makes, uhh, itself-subtracted-from-itself sense
in Python.

>> And I'm not seeing any nonnumeric that would
>> benefit from being subtracted from itself twice (strings, arrays, sets,
>> you can subtract them from one another but not usefully more than once).
>
> How do you subtract strings?

The same way you subtract sets. Same with arrays. Python doesn't do
either, but Python also doesn't do the ":=" operator that the example
code demonstrated, so I didn't assume Python.

Pike v7.8 release 700 running Hilfe v3.5 (Incremental Pike Frontend)
> "Hello, world!"-"l";
(1) Result: "Heo, word!"
> ({1,2,3,3,2,3,1,2,1})-({2});
(2) Result: ({ /* 6 elements */
1,
3,
3,
3,
1,
1
})

Python spells it differently:
>>> "Hello, world!".replace("l","")
'Heo, word!'

Not sure how to do array subtraction other than with filter:
>>> list(filter(lambda x: x!=2,[1,2,3,3,2,3,1,2,1]))
[1, 3, 3, 3, 1, 1]
But there's probably a way (list.remove only takes out the first
occurrence, so it's not equivalent).

In any case, subtracting something from _itself_ is only going to give
you an empty string, array, set, or whatever, and doing so a second
time is going to achieve nothing. Hence my comment.

But poor code we will always have with us, to paraphrase the Gospel of Matthew.

ChrisA

[Mark Lawrence]

On 09/11/2012 06:37, Steven D'Aprano wrote:
> On Fri, 09 Nov 2012 17:07:09 +1100, Chris Angelico wrote:
>
>> On Fri, Nov 9, 2012 at 12:39 PM, Mark Lawrence <bream...@yahoo.co.uk>
>> wrote:
>>> On 07/11/2012 01:55, Steven D'Aprano wrote:
>>>>
>>>>
>>>> Who knows? Who cares? Nobody does:
>>>>
>>>> n -= n
>>>>
>>>>
>>> But I've seen this scattered through code:
>>>
>>> x := x - x - x
>>
>> Can you enlighten us as to how this is better than either:
>> x := -x
>> or
>> x := 0 - x
>> ? I'm not seeing it.
>
> I'm hoping that Mark intended it as an example of crappy code he has
> spotted in some other language rather than a counter-example of something
> you would do.

Correct, CORAL 66 and pointed out to me by a colleague when another team
member had resigned.

>
> To be pedantic... there may very well be some (rare) cases where you
> actually do want x -= x rather than just x = 0. Consider the case where x
> could be an INF or NAN. Then x -= x should give x = NAN rather than zero.
> That may be desirable in some cases.

Interesting what comes up when we get chatting here. I hope we don't
get punished for going off topic :)

>
> At the very least, the compiler should NOT optimize away x = x - x to
> x = 0 if x could be a float, complex or Decimal.
>

X was an int so almost certainly optimised away by the SDL compiler on
VMS of 1986 or 1987.

--
Cheers.

Mark Lawrence.

[rusi]

On Nov 9, 11:37 am, Steven D'Aprano <steve

+comp.lang.pyt...@pearwood.info> wrote:
> On Fri, 09 Nov 2012 17:07:09 +1100, Chris Angelico wrote:

> > On Fri, Nov 9, 2012 at 12:39 PM, Mark Lawrence <breamore...@yahoo.co.uk>

> > wrote:
> >> On 07/11/2012 01:55, Steven D'Aprano wrote:
>
> >>> Who knows? Who cares? Nobody does:
>
> >>> n -= n
>
> >> But I've seen this scattered through code:
>
> >> x := x - x - x
>
> > Can you enlighten us as to how this is better than either:
> >  x := -x
> > or
> >  x := 0 - x
> > ? I'm not seeing it.
>
> I'm hoping that Mark intended it as an example of crappy code he has
> spotted in some other language rather than a counter-example of something
> you would do.
>
> To be pedantic... there may very well be some (rare) cases where you
> actually do want x -= x rather than just x = 0. Consider the case where x
> could be an INF or NAN. Then x -= x should give x = NAN rather than zero.
> That may be desirable in some cases.

In x86 assembler
mov ax, 0
is 4 bytes
sub ax, ax
is 2
and therefore better (at least for those brought up on Peter Norton);
the most common being
xor ax, ax

[Chris Angelico]

On Sat, Nov 10, 2012 at 2:05 AM, rusi <rusto...@gmail.com> wrote:
> In x86 assembler
> mov ax, 0
> is 4 bytes

Three bytes actually, B8 00 00 if my memory hasn't failed me. BA for
DX, B9 ought to be BX and BB CX, I think. But yes, the xor or sub is
two bytes and one clock.

ChrisA

[Prasad, Ramit]

Dennis Lee Bieber wrote:
>
> On Fri, 9 Nov 2012 17:07:09 +1100, Chris Angelico <ros...@gmail.com>
> declaimed the following in gmane.comp.python.general:

>
> > On Fri, Nov 9, 2012 at 12:39 PM, Mark Lawrence <bream...@yahoo.co.uk> wrote:
> > > On 07/11/2012 01:55, Steven D'Aprano wrote:
> > >>
> > >>
> > >> Who knows? Who cares? Nobody does:
> > >>
> > >> n -= n
> > >>
> > >
> > > But I've seen this scattered through code:
> > >
> > > x := x - x - x
> >
> > Can you enlighten us as to how this is better than either:
> > x := -x
> > or
> > x := 0 - x
>

> Of course, if one has a language that, for some reason, evaluates
> right-to-left (APL, anyone), then

>
> x := x - x - x
>

> becomes
>
> x := x - 0
>

Is that not the same as x:=-x?


~Ramit


This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.

[Ethan Furman]

Prasad, Ramit wrote:

> Dennis Lee Bieber wrote:
>> Of course, if one has a language that, for some reason, evaluates
>> right-to-left (APL, anyone), then
>>
>> x := x - x - x
>>
>> becomes
>>
>> x := x - 0
>
> Is that not the same as x:=-x?

No, its the same as 'x = x'.

~Ethan~