Re: copy on write

[Eduardo Suarez-Santana]

El 13/01/12 11:33, Eduardo Suarez-Santana escribió:
> I wonder whether this is normal behaviour.
>
Even simpler:

$ python
Python 2.7.2 (default, Oct 31 2011, 11:54:55)
[GCC 4.5.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> r={'a':1};
>>> d={};
>>> d['x']=r;
>>> d['y']=r;
>>> d['x']['a']=3
>>> d['y']
{'a': 3}
>>>

[Eduardo Suarez-Santana]

I wonder whether this is normal behaviour.

I would expect equal sign to copy values from right to left. However, it
seems there is a copy-on-write mechanism that is not working.

Anyone can explain and provide a working example?

Thanks,
-Eduardo

$ python
Python 2.7.2 (default, Oct 31 2011, 11:54:55)
[GCC 4.5.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> class n:
... def __init__(self, id, cont):
... self.id = id;
... self.cont = cont;
...

>>> r={'a':1};
>>> d={};
>>> d['x']=r;
>>> d['y']=r;

>>> x1 = n('x',d['x']);
>>> y1 = n('y',d['y']);
>>> x1.cont['a']=2;
>>> y1.cont
{'a': 2}
>>>

[Jean-Michel Pichavant]

Eduardo Suarez-Santana wrote:
> El 13/01/12 11:33, Eduardo Suarez-Santana escribió:

>> I wonder whether this is normal behaviour.
>>

> Even simpler:

>
> $ python
> Python 2.7.2 (default, Oct 31 2011, 11:54:55)
> [GCC 4.5.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.

> >>> r={'a':1};
> >>> d={};
> >>> d['x']=r;
> >>> d['y']=r;

> >>> d['x']['a']=3
> >>> d['y']
> {'a': 3}
> >>>
>
yes it is.

>>> d['x']=r;
>>> d['y']=r;

means that both d['x'] and d['y'] name the same object r. If you change
r, you'll see these changes wheter using d['x'] or d['y'].

The operator '=' does not copy objects, it binds an object to a name,
and an object can have multiple names.
Use the dictionary copy method to copy a dictionary:

d['x'] = r.copy()

JM

[Steven D'Aprano]

On Fri, 13 Jan 2012 11:33:24 +0000, Eduardo Suarez-Santana wrote:

> I wonder whether this is normal behaviour.
>
> I would expect equal sign to copy values from right to left.

Assignment in Python never copies values.

> However, it
> seems there is a copy-on-write mechanism that is not working.

There is no copy-on-write.

Assignment in Python is name binding: the name on the left hand side is
bound to the object on the right. An object can have zero, one or many
names. If the object is mutable, changes to the object will be visible
via any name:

>>> x = [] # lists are mutable objects
>>> y = x # not a copy of x, but x and y point to the same object
>>> x.append(42) # mutates the object in place
>>> print y
[42]

The same rules apply not just to names, but also to list items and dict
items, as well as attributes, and any other reference:

>>> z = [x, y] # z is a list containing the same sublist twice
>>> z[0].append(23)
>>> print z
[[42, 23], [42, 23]]

When you work with floats, ints or strings, you don't notice this because
those types are immutable: you can't modify those objects in place. So
for example:

>>> a = 42 # binds the name 'a' to the object 42
>>> b = a # a and b point to the same object
>>> a += 1 # creates a new object, and binds it to a
>>> print b # leaving b still pointing to the old object
42


--
Steven

[Chris Angelico]

On Fri, Jan 13, 2012 at 11:10 PM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
>>>> z = [x, y]  # z is a list containing the same sublist twice
>>>> z[0].append(23)
>>>> print z
> [[42, 23], [42, 23]]
>
> When you work with floats, ints or strings, you don't notice this because
> those types are immutable: you can't modify those objects in place. So
> for example:
>
>>>> a = 42  # binds the name 'a' to the object 42
>>>> b = a  # a and b point to the same object
>>>> a += 1  # creates a new object, and binds it to a
>>>> print b  # leaving b still pointing to the old object
> 42

I was about to say that it's a difference between ".append()" which is
a method on the object, and "+=" which is normally a rebinding, but
unfortunately:

>>> a=[]
>>> b=a
>>> a+=[1]
>>> a
[1]
>>> b
[1]
>>> b+=[2]
>>> a
[1, 2]
>>> a
[1, 2]
>>> a=a+[3]
>>> a
[1, 2, 3]
>>> b
[1, 2]

(tested in Python 3.2 on Windows)

It seems there's a distinct difference between a+=b (in-place
addition/concatenation) and a=a+b (always rebinding), which is sorely
confusing to C programmers. But then, there's a lot about Python
that's sorely confusing to C programmers.

ChrisA

[Steven D'Aprano]

On Fri, 13 Jan 2012 23:30:56 +1100, Chris Angelico wrote:

> It seems there's a distinct difference between a+=b (in-place
> addition/concatenation) and a=a+b (always rebinding),

Actually, both are always rebinding. It just happens that sometimes a+=b
rebinds to the same object that it was originally bound to.

In the case of ints, a+=b creates a new object (a+b) and rebinds a to it.
In the case of lists, a+=b nominally creates a list a+b, but in fact it
implements that as an in-place operation a.extend(b), and then rebinds
the name a to the list already bound to a.

It does that because the Python VM doesn't know at compile time whether
a+=b will be in-place or not, and so it has to do the rebinding in order
to support the fall-back case of a+=b => a=a+b. Or something -- go read
the PEP if you really care :)

Normally this is harmless, but there is one interesting little glitch you
can get:

>>> t = ('a', [23])
>>> t[1] += [42]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> t
('a', [23, 42])

> which is sorely
> confusing to C programmers. But then, there's a lot about Python
> that's sorely confusing to C programmers.

I prefer to think of it as "there's a lot about C that is sorely
confusing to anyone who isn't a C programmer" <wink>



--
Steven

[Devin Jeanpierre]

On Fri, Jan 13, 2012 at 7:30 AM, Chris Angelico <ros...@gmail.com> wrote:
> It seems there's a distinct difference between a+=b (in-place

> addition/concatenation) and a=a+b (always rebinding), which is sorely

> confusing to C programmers. But then, there's a lot about Python
> that's sorely confusing to C programmers.

I think this is confusing to just about everyone, when they first encounter it.

-- Devin

[Grant Edwards]

That depends on what languages they've used in the past and whether
they skip reading any documentation and just assume that all languages
work the same way.

I would agree that for the majority of new users, they previously used
only languages where an assignment operator does a "copy value", and
that 90+ percent of the time those new users they assume all languages
work that way.

I'm not sure what we can do about that -- Python's semantics are well
documented.

--
Grant Edwards grant.b.edwards Yow! If our behavior is
at strict, we do not need fun!
gmail.com

[Devin Jeanpierre]

On Fri, Jan 13, 2012 at 10:13 AM, Grant Edwards <inv...@invalid.invalid> wrote:
> On 2012-01-13, Devin Jeanpierre <jeanpi...@gmail.com> wrote:
>> On Fri, Jan 13, 2012 at 7:30 AM, Chris Angelico <ros...@gmail.com> wrote:
>>> It seems there's a distinct difference between a+=b (in-place
>>> addition/concatenation) and a=a+b (always rebinding), which is sorely
>>> confusing to C programmers. But then, there's a lot about Python
>>> that's sorely confusing to C programmers.
>>
>> I think this is confusing to just about everyone, when they first
>> encounter it.
>
> That depends on what languages they've used in the past and whether
> they skip reading any documentation and just assume that all languages
> work the same way.
>
> I would agree that for the majority of new users, they previously used
> only languages where an assignment operator does a "copy value", and
> that 90+ percent of the time those new users they assume all languages
> work that way.

That isn't what I was referring to. Specifically, it confuses almost
everyone the first time they encounter it that "a += b" is not the
same as "a = a + b".

And sure, it's documented. That's a bit of a cop-out though... it
isn't in the tutorial, and even if it were, it's not as if people
remember everything they read. It's not about whether you _can_ know
it as much as whether it is """obvious"". There's a bit of a feeling
that code should "do what it looks like" and be sort of understandable
without exactly understanding everything. Maybe this idea is wrong if
taken to an extreme (since it's really impossible to do completely),
but the feeling of it is probably decent. It's why we use "+" for
addition and "-" for subtraction, and not the other way around. You
don't need to know the details of operator overloading and
NotImplemented and so on to get what X + Y means for numbers, or even
for lists.

I feel like "a += b" is sort of implicitly understood by most
programmers to be the same as "a = a + b". If you asked someone what
it meant, their first answer would be "Oh, it means a = a + b"[*].
That is why it's confusing -- even to people that weren't already
exposed to that idea that these are equivalent, they get infected
fast. And then expectations get broken, because they're only *usually*
equivalent.

[*] Before posting this, I actually tried this on a Python IRC channel
-- and it happened exactly as so.

-- Devin

[Neil Cerutti]

On 2012-01-13, Devin Jeanpierre <jeanpi...@gmail.com> wrote:
> On Fri, Jan 13, 2012 at 10:13 AM, Grant Edwards <inv...@invalid.invalid> wrote:
>> On 2012-01-13, Devin Jeanpierre <jeanpi...@gmail.com> wrote:
>>> On Fri, Jan 13, 2012 at 7:30 AM, Chris Angelico <ros...@gmail.com> wrote:
>>>> It seems there's a distinct difference between a+=b (in-place
>>>> addition/concatenation) and a=a+b (always rebinding), which is sorely
>>>> confusing to C programmers. But then, there's a lot about Python
>>>> that's sorely confusing to C programmers.
>>>
>>> I think this is confusing to just about everyone, when they first
>>> encounter it.
>>
>> That depends on what languages they've used in the past and whether
>> they skip reading any documentation and just assume that all languages
>> work the same way.
>>
>> I would agree that for the majority of new users, they previously used
>> only languages where an assignment operator does a "copy value", and
>> that 90+ percent of the time those new users they assume all languages
>> work that way.
>
> That isn't what I was referring to. Specifically, it confuses
> almost everyone the first time they encounter it that "a += b"
> is not the same as "a = a + b".

If you've ever implemented operator=, operator+, and operator+=
in C++ you'll know how and why they are different. A C++
programmer would be wondering how either can work on immutable
objects, and that's where Python's magical rebinding semantics
come into play.

--
Neil Cerutti

[Grant Edwards]

On 2012-01-13, Neil Cerutti <ne...@norwich.edu> wrote:

> If you've ever implemented operator=, operator+, and operator+=
> in C++ you'll know how and why they are different.

That assumes that C++ programmers understand C++.

;)

> A C++ programmer would be wondering how either can work on immutable
> objects, and that's where Python's magical rebinding semantics come
> into play.

--

Grant Edwards grant.b.edwards Yow! Thousands of days of
at civilians ... have produced
gmail.com a ... feeling for the
aesthetic modules --

[Chris Angelico]

On Sat, Jan 14, 2012 at 5:15 AM, Grant Edwards <inv...@invalid.invalid> wrote:
> That assumes that C++ programmers understand C++.

I understand C++ very well. That's why I use Python or Pike.

(With apologies to Larry Wall)

ChrisA

[Grant Edwards]

Were one inclined to troll a bit, one might be tempted to claim that
using C++ is prima facie evidence of not understanding C++.

Not that I would ever claim something inflamitory like that...

[Ethan Furman]

Steven D'Aprano wrote:
> Normally this is harmless, but there is one interesting little glitch you
> can get:
>
>>>> t = ('a', [23])
>>>> t[1] += [42]
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> TypeError: 'tuple' object does not support item assignment
>>>> t
> ('a', [23, 42])

There is one other glitch, and possibly my only complaint:

--> a = [1, 2, 3]
--> b = 'hello, world'
--> a = a + b

Traceback (most recent call last):
File "<stdin>", line 1, in <module>

TypeError: can only concatenate list (not "str") to list
--> a += b
--> a
[1, 2, 3, 'h', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd']

IMO, either both + and += should succeed, or both should fail.

~Ethan~

[Neil Cerutti]

On 2012-01-13, Grant Edwards <inv...@invalid.invalid> wrote:
> On 2012-01-13, Chris Angelico <ros...@gmail.com> wrote:
>> On Sat, Jan 14, 2012 at 5:15 AM, Grant Edwards
>> <inv...@invalid.invalid> wrote:
>>> That assumes that C++ programmers understand C++.
>>
>> I understand C++ very well. That's why I use Python or Pike.
>>
>> (With apologies to Larry Wall)
>
> Were one inclined to troll a bit, one might be tempted to claim
> that using C++ is prima facie evidence of not understanding
> C++.
>
> Not that I would ever claim something inflamitory like that...

On the Python newsgroup, it's funny. ;)

--
Neil Cerutti

[Evan Driscoll]

On 01/13/2012 10:54 AM, Neil Cerutti wrote:
> If you've ever implemented operator=, operator+, and operator+=
> in C++ you'll know how and why they are different.

At the same time, you'd also know that that implementing them in such a
way that 'a += b' does *not* perform the same action as 'a = a + b' is
considered very bad-mannered.

In fact, it's often suggested (e.g. in "More Effective C++"'s Item 22,
though this is not the main thrust of that section) to implement
operator+ in terms of += to ensure that this is the case:
MyType operator+ (MyType left, MyType right) {
MyType copy = left; copy += right; return copy;

}

> A C++
> programmer would be wondering how either can work on immutable
> objects, and that's where Python's magical rebinding semantics
> come into play.

IMO a C++ programmer wouldn't be likely to wonder that much at all
because he or she wouldn't view the objects as immutable to begin with.
:-) 'x = 5; x += 1;' makes perfect sense in C++, just for a somewhat
different reason.

Evan

[Neil Cerutti]

On 2012-01-13, Evan Driscoll <edri...@wisc.edu> wrote:
> On 01/13/2012 10:54 AM, Neil Cerutti wrote:
>> If you've ever implemented operator=, operator+, and operator+=
>> in C++ you'll know how and why they are different.
>
> At the same time, you'd also know that that implementing them
> in such a way that 'a += b' does *not* perform the same action
> as 'a = a + b' is considered very bad-mannered.
>
> In fact, it's often suggested (e.g. in "More Effective C++"'s Item 22,
> though this is not the main thrust of that section) to implement
> operator+ in terms of += to ensure that this is the case:
> MyType operator+ (MyType left, MyType right) {
> MyType copy = left; copy += right; return copy;
> }

They perform the same action, but their semantics are different.
operator+ will always return a new object, thanks to its
signature, and operator+= shall never do so. That's the main
difference I was getting at.

>> A C++ programmer would be wondering how either can work on
>> immutable objects, and that's where Python's magical rebinding
>> semantics come into play.
>
> IMO a C++ programmer wouldn't be likely to wonder that much at
> all because he or she wouldn't view the objects as immutable to
> begin with. :-) 'x = 5; x += 1;' makes perfect sense in C++,
> just for a somewhat different reason.

I was thinking of const objects, but you are correct that
immutable isn't really a C++ concept.

--
Neil Cerutti

[88888 Dihedral]

Ethan Furman於 2012年1月14日星期六UTC+8上午2時40分47秒寫道：

The += operator is not only for value types in the above example.

An operator of two operands and an operator of three operands of
general object types are two different operators.

[88888 Dihedral]

Ethan Furman於 2012年1月14日星期六UTC+8上午2時40分47秒寫道：

[Evan Driscoll]

On 01/13/2012 03:20 PM, Neil Cerutti wrote:
> They perform the same action, but their semantics are different.
> operator+ will always return a new object, thanks to its
> signature, and operator+= shall never do so. That's the main
> difference I was getting at.

I was talking about the combination of + and =, since the discussion is
about 'a = a + b' vs 'a += b', not 'a + b' vs 'a += b' (where the
differences are obvious).

And I stand by my statement. In 'a = a + b', operator+ obviously returns
a new object, but operator= should then go and assign the result to and
return a reference to 'a', just like how 'a += b' will return a
reference to 'a'.

If you're working in C++ and overload your operators so that 'a += b'
and 'a = a + b' have different observable behaviors (besides perhaps
time), then either your implementation is buggy or your design is very
bad-mannered.

Evan

[John O'Hagan]

On Fri, 13 Jan 2012 10:40:47 -0800
Ethan Furman <et...@stoneleaf.us> wrote:

> Steven D'Aprano wrote:
> > Normally this is harmless, but there is one interesting little
> > glitch you can get:
> >
> >>>> t = ('a', [23])
> >>>> t[1] += [42]
> > Traceback (most recent call last):
> > File "<stdin>", line 1, in <module>
> > TypeError: 'tuple' object does not support item assignment
> >>>> t
> > ('a', [23, 42])

IMHO, this is worthy of bug-hood: shouldn't we be able to conclude from the TypeError that the assignment failed?

> There is one other glitch, and possibly my only complaint:
>
> --> a = [1, 2, 3]
> --> b = 'hello, world'

> --> a = a + b

> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>

> TypeError: can only concatenate list (not "str") to list
> --> a += b
> --> a
> [1, 2, 3, 'h', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd']
>
> IMO, either both + and += should succeed, or both should fail.
>
> ~Ethan~

This also happens for tuples, sets, generators and range objects (probably any iterable), AFAIK only when the left operand is a list. Do lists get special treatment in terms of implicitly converting the right-hand operand?

The behaviour of the "in-place" operator could be more consistent across types:

>>> a=[1,2]
>>> a+=(3,4)
>>> a
[1, 2, 3, 4]
>>> a=(1,2)
>>> a+=(3,4)
>>> a
(1, 2, 3, 4)
>>> a=(1,2)
>>> a+=[3,4]

Traceback (most recent call last):
File "<stdin>", line 1, in <module>

TypeError: can only concatenate tuple (not "list") to tuple


John

[Rick Johnson]

On Jan 13, 10:48 am, Devin Jeanpierre <jeanpierr...@gmail.com> wrote:
> On Fri, Jan 13, 2012 at 10:13 AM, Grant Edwards <inva...@invalid.invalid> wrote:

> > On 2012-01-13, Devin Jeanpierre <jeanpierr...@gmail.com> wrote:
> >> On Fri, Jan 13, 2012 at 7:30 AM, Chris Angelico <ros...@gmail.com> wrote:
> There's a bit of a feeling
> that code should "do what it looks like" and be sort of understandable
> without exactly understanding everything.

Yeah there's a word for that; INTUITIVE, And I've been preaching its
virtues (sadly in vain it seems!) to these folks for some time now.

[Steven D'Aprano]

On Wed, 01 Feb 2012 19:51:13 -0800, Rick Johnson wrote:

> Yeah there's a word for that; INTUITIVE, And I've been preaching its
> virtues (sadly in vain it seems!) to these folks for some time now.

Intuitive to whom?

Expert Python programmers?

VB coders?

Perl hackers?

School children who have never programmed before?

Mathematicians?

Babies?

Rocket scientists?

Hunter-gatherers from the Kalahari desert?


My intuition tells me you have never even considered that intuition
depends on who is doing the intuiting.


--
Steven

[Devin Jeanpierre]

On Wed, Feb 1, 2012 at 10:18 PM, John O'Hagan <rese...@johnohagan.com> wrote:
> On Fri, 13 Jan 2012 10:40:47 -0800
> Ethan Furman <et...@stoneleaf.us> wrote:
>
>> Steven D'Aprano wrote:

>> > Normally this is harmless, but there is one interesting little
>> > glitch you can get:
>> >
>> >>>> t = ('a', [23])
>> >>>> t[1] += [42]
>> > Traceback (most recent call last):
>> >   File "<stdin>", line 1, in <module>
>> > TypeError: 'tuple' object does not support item assignment
>> >>>> t
>> > ('a', [23, 42])
>

> IMHO, this is worthy of bug-hood: shouldn't we be able to conclude from the TypeError that the assignment failed?

It did fail. The mutation did not.

I can't think of any way out of this misleadingness, although if you
can that would be pretty awesome.

-- Devin

[John O'Hagan]

On Thu, 2 Feb 2012 01:34:48 -0500
Devin Jeanpierre <jeanpi...@gmail.com> wrote:

> On Wed, Feb 1, 2012 at 10:18 PM, John O'Hagan
> <rese...@johnohagan.com> wrote:
> > On Fri, 13 Jan 2012 10:40:47 -0800
> > Ethan Furman <et...@stoneleaf.us> wrote:
> >
> >> Steven D'Aprano wrote:

> >> > Normally this is harmless, but there is one interesting little
> >> > glitch you can get:
> >> >
> >> >>>> t = ('a', [23])
> >> >>>> t[1] += [42]
> >> > Traceback (most recent call last):
> >> >   File "<stdin>", line 1, in <module>
> >> > TypeError: 'tuple' object does not support item assignment
> >> >>>> t
> >> > ('a', [23, 42])
> >

> > IMHO, this is worthy of bug-hood: shouldn't we be able to conclude
> > from the TypeError that the assignment failed?
>
> It did fail. The mutation did not.

You're right, in fact, for me the surprise is that "t[1] +=" is interpreted as an assignment at all, given that for lists (and other mutable objects which use "+=") it is a mutation. Although as Steven says elsewhere, it actually is an assignment, but one which ends up reassigning to the same object.

But it shouldn't be both. I can't think of another example of (what appears to be but is not) a single operation failing with an exception, but still doing exactly what you intended.

>
> I can't think of any way out of this misleadingness, although if you
> can that would be pretty awesome.

In the case above, the failure of the assignment is of no consequence. I think it would make more sense if applying "+=" to a tuple element were treated (by the interpreter I suppose) only on the merits of the element, and not as an assignment to the tuple.

John

[Steven D'Aprano]

On Thu, 02 Feb 2012 19:11:53 +1100, John O'Hagan wrote:

> You're right, in fact, for me the surprise is that "t[1] +=" is
> interpreted as an assignment at all, given that for lists (and other
> mutable objects which use "+=") it is a mutation. Although as Steven
> says elsewhere, it actually is an assignment, but one which ends up
> reassigning to the same object.
>
> But it shouldn't be both.

Do you expect that x += 1 should succeed? After all, "increment and
decrement numbers" is practically THE use-case for the augmented
assignment operators.

How can you expect x += 1 to succeed without an assignment? Numbers in
Python are immutable, and they have to stay immutable. It would cause
chaos and much gnashing of teeth if you did this:

x = 2
y = 7 - 5
x += 1
print y * 100
=> prints 300

So if you want x += 1 to succeed, += must do an assignment.

Perhaps you are thinking that Python could determine ahead of time
whether x[1] += y involved a list or a tuple, and not perform the finally
assignment if x was a tuple. Well, maybe, but such an approach (if
possible!) is fraught with danger and mysterious errors even harder to
debug than the current situation. And besides, what should Python do
about non-built-in types? There is no way in general to predict whether
x[1] = something will succeed except to actually try it.

> I can't think of another example of (what
> appears to be but is not) a single operation failing with an exception,
> but still doing exactly what you intended.

Neither can I, but that doesn't mean that the current situation is not
the least-worst alternative.

>> I can't think of any way out of this misleadingness, although if you
>> can that would be pretty awesome.
>
> In the case above, the failure of the assignment is of no consequence. I
> think it would make more sense if applying "+=" to a tuple element were
> treated (by the interpreter I suppose) only on the merits of the
> element, and not as an assignment to the tuple.

How should the interpreter deal with other objects which happen to raise
TypeError? By always ignoring it?

x = [1, None, 3]
x[1] += 2 # apparently succeeds

Or perhaps by hard-coding tuples and only ignoring errors for tuples? So
now you disguise one error but not others?



--
Steven

[Thomas Rachel]

Am 13.01.2012 13:30 schrieb Chris Angelico:

> It seems there's a distinct difference between a+=b (in-place
> addition/concatenation) and a=a+b (always rebinding),

There is indeed.

a = a + b is a = a.__add__(b), while

a += b is a = a.__iadd__(b).

__add__() is supposed to leave the original object intact and return a
new one, while __iadd__() is free to modify (preference, to be done if
possible) or return a new one.

A immutable object can only return a new one, and its __iadd__()
behaviour is the same as __add__().

A mutable object, however, is free to and supposed to modify itself and
then return self.


Thomas

[Hrvoje Niksic]

Steven D'Aprano <steve+comp....@pearwood.info> writes:

> Perhaps you are thinking that Python could determine ahead of time
> whether x[1] += y involved a list or a tuple, and not perform the
> finally assignment if x was a tuple. Well, maybe, but such an approach
> (if possible!) is fraught with danger and mysterious errors even
> harder to debug than the current situation. And besides, what should
> Python do about non-built-in types? There is no way in general to
> predict whether x[1] = something will succeed except to actually try
> it.

An alternative approach is to simply not perform the final assignment if
the in-place method is available on the contained object. No prediction
is needed to do it, because the contained object has to be examined
anyway. No prediction is needed, just don't. Currently,
lhs[ind] += rhs is implemented like this:

item = lhs[ind]
if hasattr(item, '__iadd__'):
lhs.__setitem__(ind, item.__iadd__(rhs))
else:
lhs.__setitem__(ind, item + rhs)
# (Note item assignment in both "if" branches.)

It could, however, be implemented like this:

item = lhs[ind]
if hasattr(item, '__iadd__'):
item += rhs # no assignment, item supports in-place change
else:
lhs.__setitem__(ind, lhs[ind] + rhs)

This would raise the exact same exception in the tuple case, but without
executing the in-place assignment. On the other hand, some_list[ind] += 1
would continue working exactly the same as it does now.

In the same vein, in-place methods should not have a return value
(i.e. they should return None), as per Python convention that functions
called for side effect don't return values.

The alternative behavior is unfortunately not backward-compatible (it
ignores the return value of augmented methods), so I'm not seriously
proposing it, but I believe it would have been a better implementation
of augmented assignments than the current one. The present interface
doesn't just bite those who try to use augmented assignment on tuples
holding mutable objects, but also those who do the same with read-only
properties, which is even more reasonable. For example, obj.list_attr
being a list, one would expect that obj.list_attr += [1, 2, 3] does the
same thing as obj.list_attr.extend([1, 2, 3]). And it almost does,
except it also follows up with an assignment after the list has already
been changed, and the assignment to a read-only property raises an
exception. Refusing to modify the list would have been fine, modifying
it without raising an exception (as described above) would have been
better, but modifying it and *then* raising an exception is a surprise
that takes some getting used to.

[88888 Dihedral]

在 2012年1月14日星期六UTC+8上午6时48分29秒，Evan Driscoll写道：

> On 01/13/2012 03:20 PM, Neil Cerutti wrote:
> > They perform the same action, but their semantics are different.
> > operator+ will always return a new object, thanks to its
> > signature, and operator+= shall never do so. That's the main
> > difference I was getting at.
>
> I was talking about the combination of + and =, since the discussion is
> about 'a = a + b' vs 'a += b', not 'a + b' vs 'a += b' (where the
> differences are obvious).
>
> And I stand by my statement. In 'a = a + b', operator+ obviously returns
> a new object, but operator= should then go and assign the result to and
> return a reference to 'a', just like how 'a += b' will return a
> reference to 'a'.
>

The operation a+b means add(a,b) and returns a result instance, furthermore a and b can't be modified.

The expression a = a+b are two operations not one. But in C or C++ the problem is mixing operations and expressions in a free style allowed.

The operation a+=b means a modified by b and b can't be changed.
Note that no new instance is necessary in a+=b.

> If you're working in C++ and overload your operators so that 'a += b'
> and 'a = a + b' have different observable behaviors (besides perhaps
> time), then either your implementation is buggy or your design is very
> bad-mannered.
>
> Evan

Do you mean the result instances after 'a+=b' and 'a=a+b' or
the actions of behaviors of instances involved in performing 'a+=b' and 'a=a+b'?

[88888 Dihedral]

在 2012年1月14日星期六UTC+8上午6时48分29秒，Evan Driscoll写道：

> On 01/13/2012 03:20 PM, Neil Cerutti wrote:
> > They perform the same action, but their semantics are different.
> > operator+ will always return a new object, thanks to its
> > signature, and operator+= shall never do so. That's the main
> > difference I was getting at.
>
> I was talking about the combination of + and =, since the discussion is
> about 'a = a + b' vs 'a += b', not 'a + b' vs 'a += b' (where the
> differences are obvious).
>
> And I stand by my statement. In 'a = a + b', operator+ obviously returns
> a new object, but operator= should then go and assign the result to and
> return a reference to 'a', just like how 'a += b' will return a
> reference to 'a'.
>

The operation a+b means add(a,b) and returns a result instance, furthermore a and b can't be modified.

The expression a = a+b are two operations not one. But in C or C++ the problem is mixing operations and expressions in a free style allowed.

The operation a+=b means a modified by b and b can't be changed.
Note that no new instance is necessary in a+=b.

> If you're working in C++ and overload your operators so that 'a += b'
> and 'a = a + b' have different observable behaviors (besides perhaps
> time), then either your implementation is buggy or your design is very
> bad-mannered.
>
> Evan

[John O'Hagan]

On 02 Feb 2012 09:16:40 GMT

Steven D'Aprano <steve+comp....@pearwood.info> wrote:

> On Thu, 02 Feb 2012 19:11:53 +1100, John O'Hagan wrote:
>
> > You're right, in fact, for me the surprise is that "t[1] +=" is
> > interpreted as an assignment at all, given that for lists (and other
> > mutable objects which use "+=") it is a mutation. Although as Steven
> > says elsewhere, it actually is an assignment, but one which ends up
> > reassigning to the same object.
> >
> > But it shouldn't be both.
>
> Do you expect that x += 1 should succeed? After all, "increment and
> decrement numbers" is practically THE use-case for the augmented
> assignment operators.
>
> How can you expect x += 1 to succeed without an assignment?

I don't; obviously, for immutable objects assignment is the only possibility.

[...]

>
> Perhaps you are thinking that Python could determine ahead of time
> whether x[1] += y involved a list or a tuple, and not perform the
> finally assignment if x was a tuple. Well, maybe, but such an
> approach (if possible!) is fraught with danger and mysterious errors
> even harder to debug than the current situation. And besides, what
> should Python do about non-built-in types? There is no way in general
> to predict whether x[1] = something will succeed except to actually
> try it.

It's not so much about the type of x but that of x[1]. Wouldn't it be possible to omit the assignment simply if the object referred to by x[1] uses "+=" without creating a new object? That way, some_tuple[i] += y will succeed if some_tuple[i] is a list but not with, say, an int. That seems reasonable to me.

[...]

> >
> > In the case above, the failure of the assignment is of no
> > consequence. I think it would make more sense if applying "+=" to a
> > tuple element were treated (by the interpreter I suppose) only on
> > the merits of the element, and not as an assignment to the tuple.
>
> How should the interpreter deal with other objects which happen to
> raise TypeError? By always ignoring it?
>
> x = [1, None, 3]
> x[1] += 2 # apparently succeeds
>
> Or perhaps by hard-coding tuples and only ignoring errors for tuples?
> So now you disguise one error but not others?

I'm not suggesting either of those. None can't be modified in place. But for objects which can, wouldn't omitting the final assignment prevent the TypeError in the first place?

John

[MRAB]

[snip]
Could it not perform the assignment if the reference returned by
__iadd__ is the same as the current reference?

For example:

t[0] += x

would do:

r = t[0].__iadd__(x)
if t[0] is not r:
t[0] = r

Should failed assignment be raising TypeError? Is it really a type
error?

[Devin Jeanpierre]

On Thu, Feb 2, 2012 at 11:28 AM, MRAB <pyt...@mrabarnett.plus.com> wrote:
> Should failed assignment be raising TypeError? Is it really a type
> error?

A failed setitem should be a TypeError as much as a failed getitem
should. Should 1[0] be a TypeError?

-- Devin

[Terry Reedy]

On 2/2/2012 9:17 AM, John O'Hagan wrote:

> It's not so much about the type of x but that of x[1]. Wouldn't it be
> possible to omit the assignment simply if the object referred to by
> x[1] uses "+=" without creating a new object? That way, some_tuple[i]
> += y will succeed if some_tuple[i] is a list but not with, say, an
> int. That seems reasonable to me.

There was considerable discussion of the exact semantics of augmented
operations when they were introduced. I do not remember if that
particular idea was suggested (and rejected) or not. You could try to
look at the PEP, if there is one, or the dicussion ( probably on pydev
list).

--
Terry Jan Reedy

[Evan Driscoll]

On 01/-10/-28163 01:59 PM, 88888 Dihedral wrote:
>> If you're working in C++ and overload your operators so that 'a +='
>> and 'a = + b' have different observable behaviors (besides perhaps

>> time), then either your implementation is buggy or your design is very
>> bad-mannered.
>>
>> Evan
>

> Do you mean the result instances after 'a+= and 'a=a+b' or
> the actions of behaviors of instances involved in performing 'a+= and 'a=a+b'?
>

I mean "if which operation you called is distinguishable in any way
besides the time it takes to run or by tracing it through in a debugger"

That means:

1. The value of 'a' should be the same after executing 'a+=b' and
'a=a+b'
2. The actual result of the expression should be the same in both cases
(in both cases it should be a reference to a)
3. Any additional side effects performed (ew!) should be the same in
both cases

Evan

[John O'Hagan]

I think we're 12 years late on this one. It's PEP 203 from 2000 and the key phrase was:

"The in-place function should always return a new reference, either
to the old `x' object if the operation was indeed performed
in-place, or to a new object."

If this had read:

"The in-place function should return a reference to a new object
if the operation was not performed in-place."

or something like that, we wouldn't be discussing this.

The discussion on py-dev at the time was quite limited but there was some lively debate on this list the following year (in the context of widespread controversy over new-fangled features which also included list comprehensions and generators), to which the BDFL's response was:

"You shouldn't think "+= is confusing because sometimes it modifies an
object and sometimes it does". Gee, there are lots of places where
something that's *spelled* the same has a different effect depending
on the object types involved."


That's true, but I don't think there should be a different effect depending on what _name_ we use for an operand:

>>> t=([],)
>>> l=t[0]
>>> l is t[0]
True
>>> l+=[1]
>>> t
([1],)
>>> t[0]+=[1]

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> t

([1, 1],)
>>> l is t[0]
True

Same object, same operator, different name, different outcome. Maybe that was obvious from the foregoing discussion, but it shocked me when put that way.

John

[Steven D'Aprano]

On Fri, 03 Feb 2012 14:08:06 +1100, John O'Hagan wrote:

> I think we're 12 years late on this one. It's PEP 203 from 2000 and the
> key phrase was:
>
> "The in-place function should always return a new reference, either to
> the old `x' object if the operation was indeed performed in-place, or to
> a new object."
>
> If this had read:
>
> "The in-place function should return a reference to a new object if the
> operation was not performed in-place."
>
> or something like that, we wouldn't be discussing this.

And what should it return if the operation *is* performed in-place?
"Don't return anything" is not an option, Python doesn't have procedures.
That implies that __iadd__ etc. should return None. But two problems come
to mind:

1) Using None as an out-of-band signal to the interpreter to say "don't
perform the assignment" makes it impossible for the augmented assignment
method to return None as the result. If we only think about numeric
operations like x += 1 then we might not care, but once you consider the
situation more widely the problem is clear:

x = Fact(foo)
y = Fact(bar)
x & y # Returns a composite Fact, or None if they are contradictory

With your suggestion, x &= y fails to work, but only sometimes. And when
it fails, it doesn't fail with an explicit exception, but silently fails
and then does the wrong thing. This makes debugging a horror.

2) And speaking of debugging, sometimes people forget to include the
return statement in methods. Normally, the left hand side of the
assignment then gets set to None, and the error is pretty obvious as soon
as you try to do something with it. But with your suggestion, instead of
getting an exception, it silently fails, and your code does the wrong
thing.

I suppose that they could have invented a new sentinel, or a special
exception to be raised as a signal, but that's piling complication on top
of complication, and it isn't clear to me that it's worth it for an
obscure corner case.

Yes, the current behaviour is a Gotcha, but it's a Gotcha that makes good
sense compared to the alternatives.

Ultimately, augmented assignment is *assignment*, just like it says on
the tin. t[1] += x is syntactic sugar for t[1] = t[1].__iadd__(x). It
can't and shouldn't fail to raise an exception if t is a tuple, because
tuple item assignment *must* fail.

The problem is that lists treat __iadd__ as an in-place optimization, and
this clashes with tuple immutability. But if lists *didn't* treat
__iadd__ as in-place, people would complain when they used it directly
without a tuple wrapper.

Perhaps lists shouldn't define += at all, but then people will complain
that mylist += another_list is slow. Telling them to use mylist.extend
instead just makes them cranky. After all, mylist + another_list works,
so why shouldn't += work?

Ultimately, there is no right answer, because the multitude of
requirements are contradictory. No matter what Python did, somebody would
complain.



--
Steven

[Chris Angelico]

On Fri, Feb 3, 2012 at 4:04 PM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
> No matter what Python did, somebody would complain.

+1

This is, I think, the ultimate truth of the matter.

ChrisA

[Antoon Pardon]

On 02/03/2012 06:04 AM, Steven D'Aprano wrote:
> Ultimately, there is no right answer, because the multitude of
> requirements are contradictory. No matter what Python did, somebody would
> complain.

Which makes me wonder why it was introduced at all, or at least so fast
If you see the difference in speed in introducing augmented assignment
vs how long it took to get conditional expression I start thinking of a
bikeshed. In the first case we have something that raises semantic
questions that are difficult to resolve, in the second case the
semantics were clear, the big problem that delayed introduction was what
syntax to use.

But the second took a lot longer to become part of the language than the
first, which seems very odd to me.

--
Antoon Pardon

[John O'Hagan]

On 03 Feb 2012 05:04:39 GMT

Steven D'Aprano <steve+comp....@pearwood.info> wrote:

> On Fri, 03 Feb 2012 14:08:06 +1100, John O'Hagan wrote:
>
> > I think we're 12 years late on this one. It's PEP 203 from 2000 and
> > the key phrase was:
> >
> > "The in-place function should always return a new reference, either
> > to the old `x' object if the operation was indeed performed
> > in-place, or to a new object."
> >
> > If this had read:
> >
> > "The in-place function should return a reference to a new object if
> > the operation was not performed in-place."
> >
> > or something like that, we wouldn't be discussing this.
>
> And what should it return if the operation *is* performed in-place?

Not knowing anything about the inner workings of the interpreter, I'm agnostic on that as long as it's not "a new reference". Perhaps the old reference?

[...snip undoubted reasons why returning None wouldn't work...]

I don't know what would work. Maybe it is insoluble. But didn't Hrvoje Niksic's post in this thread suggest it could have been implemented to work the way I'm saying, even supplying code to demonstrate it?

All I'm saying is that however it's implemented, x[i] += y should simply mutate x[i] in-place if x[i] implements that, otherwise it should do x[i] = x[i] + y. If I can say it under 25 words, surely it's implementable? (Whether it's practical to do so is another question.)

The x[i] in x[i] += y can be seen as a reference to an object to be incremented rather than an assignment (despite the name). In that view, whether the name x[i] needs to be rebound to a new object, resulting in an assignment, depends on the capabilities of x[i], not x.

>
> Yes, the current behaviour is a Gotcha, but it's a Gotcha that makes
> good sense compared to the alternatives.

I think it's worse than a Gotcha. IMHO a Gothcha is, for example, the mutable default arguments thing, which makes sense once you get it. This one has the bizarre consequence that what happens when you operate on an object depends on which name you use for the object. Not to mention that it succeeds after raising an exception.

> Ultimately, augmented assignment is *assignment*, just like it says
> on the tin. t[1] += x is syntactic sugar for t[1] = t[1].__iadd__(x).
> It can't and shouldn't fail to raise an exception if t is a tuple,
> because tuple item assignment *must* fail.

That makes sense if we view it strictly as assignment (but in that case the mutation of t[1] should not occur either).
But isn't it equally true if we say that z = t[1], then t[1] += x is syntactic sugar for z = z.__iadd__(x)? Why should that fail, if z can handle it?

[...]

>
> Ultimately, there is no right answer, because the multitude of
> requirements are contradictory. No matter what Python did, somebody
> would complain.
>

Not complaining, just trying to contribute to the best of my ability. :)

John

[Rick Johnson]

On Feb 2, 11:28 pm, Chris Angelico <ros...@gmail.com> wrote:
> On Fri, Feb 3, 2012 at 4:04 PM, Steven D'Aprano
>

> <steve+comp.lang.pyt...@pearwood.info> wrote:
> > No matter what Python did, somebody would complain.
>
> +1
>
> This is, I think, the ultimate truth of the matter.

People would not complain if they did not care. The only useless
complaint is people complaining about other people complaining. And
the only thing worse than that is rabid fanboi brown-nosing!

[OKB (not okblacke)]

Steven D'Aprano wrote:

> Perhaps lists shouldn't define += at all, but then people will
> complain that mylist += another_list is slow. Telling them to use
> mylist.extend instead just makes them cranky. After all, mylist +
> another_list works, so why shouldn't += work?

It would work, it just wouldn't work in-place.

--
--OKB (not okblacke)
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is
no path, and leave a trail."
--author unknown

[88888 Dihedral]

在 2012年1月14日星期六UTC+8上午6时48分29秒，Evan Driscoll写道：

> On 01/13/2012 03:20 PM, Neil Cerutti wrote:
> > They perform the same action, but their semantics are different.
> > operator+ will always return a new object, thanks to its
> > signature, and operator+= shall never do so. That's the main
> > difference I was getting at.

Well, in any associative operation with an identity implemented in a computer language personally I believe the operator overloading part in C++ is another teasing trick.


Do we have to work out the algebra here?

[88888 Dihedral]

在 2012年1月14日星期六UTC+8上午6时48分29秒，Evan Driscoll写道：

> On 01/13/2012 03:20 PM, Neil Cerutti wrote:
> > They perform the same action, but their semantics are different.
> > operator+ will always return a new object, thanks to its
> > signature, and operator+= shall never do so. That's the main
> > difference I was getting at.

[Wolfram Hinderer]

On 3 Feb., 11:47, John O'Hagan <resea...@johnohagan.com> wrote:

> But isn't it equally true if we say that z = t[1], then t[1] += x is syntactic sugar for z = z.__iadd__(x)? Why should that fail, if z can handle it?

It's more like syntactic sugar for
y = t; z = y.__getitem__(1); z.__iadd__(x); y.__setitem__(1, z)

It's clear that only the last expression fails, after the mutation has
taken place.

Just in case you wonder about the y: you need it for more complicated
cases.
t[1][1] += [4] is syntactic sugar for
y = t.__getitem__(1); z = y.__getitem__(1); z.__iadd__([4]);
y.__setitem__(1, z)

That makes clear why there's no exception in this code:
>>> t = (0, [1, [2, 3]])
>>> t[1][1] += [4]
>>> t
(0, [1, [2, 3, 4]])