Appending to []

[Jan Sipke]

Can you explain why there is a difference between the following two
statements?

>>> a = []
>>> a.append(1)
>>> print a
[1]

>>> print [].append(1)
None

Best regards,
Jan Sipke

[Rotwang]

On 20/04/2012 21:03, Jan Sipke wrote:
> Can you explain why there is a difference between the following two
> statements?
>
>>>> a = []
>>>> a.append(1)
>>>> print a
> [1]
>
>>>> print [].append(1)
> None

append is a method of the list object []. Methods, in general, both do
something to the objects of which they are attributes, and return a
value (in fact they work pretty much like any other Python function; if
a is an instance of type A, then calling a.method(x, y, etc) is the same
thing as calling A.method(a, x, y, etc), which is no different from
calling any other function). Both of the two code examples you posted
are doing the same thing, namely appending the value one to a list and
returning None. But in the first case you can't see that the method is
returning None since the Python interpreter doesn't bother to write a
function's output when that output is None. But you would have seen it
if you had explicitly asked the interpreter to show it, like so:

>>> a = []
>>> print a.append(1)
None

Similarly, the second example is changing the list on which it was
called in the same way that the first example changed a; but since you
didn't assign the list in question to any variable, there's no way for
you to refer to it to see its new value (in fact it just gets deleted
right after being used since its reference count is zero).

In general there's no reason why

>>> a.method(arguments)
>>> print a

will print the same thing as

>>> print a.method(arguments)

since a method doesn't assign the value it returns to the instance on
which it is called; what it does to the instance and what it returns are
two completely different things.

--
Hate music? Then you'll hate this:

http://tinyurl.com/psymix

[Kiuhnm]

On 4/20/2012 22:03, Jan Sipke wrote:
> Can you explain why there is a difference between the following two
> statements?
>
>>>> a = []
>>>> a.append(1)
>>>> print a
> [1]
>
>>>> print [].append(1)
> None

Try this one:
a = []
print a.append(1)

Does that answer your question?

Kiuhnm

[Bernd Nawothnig]

On 2012-04-20, Rotwang wrote:
> since a method doesn't assign the value it returns to the instance on
> which it is called; what it does to the instance and what it returns are
> two completely different things.

Returning a None-value is pretty useless. Why not returning self, which would be
the resulting list in this case? Returning self would make the
language a little bit more functional, without any drawback.

Then nested calls like

a = [].append('x').append('y').append('z')

would be possible with a containing the resulting list

['x', 'y', 'z'].

That is the way I expect any append to behave.




Bernd

--
"Die Antisemiten vergeben es den Juden nicht, dass die Juden Geist
haben - und Geld." [Friedrich Nietzsche]

[Kiuhnm]

On 4/21/2012 14:48, Bernd Nawothnig wrote:
> On 2012-04-20, Rotwang wrote:
>> since a method doesn't assign the value it returns to the instance on
>> which it is called; what it does to the instance and what it returns are
>> two completely different things.
>
> Returning a None-value is pretty useless. Why not returning self, which would be
> the resulting list in this case? Returning self would make the
> language a little bit more functional, without any drawback.
>
> Then nested calls like
>
> a = [].append('x').append('y').append('z')

You just answered to your own question: append returns None so that
people can't use it the way you did.
You make the reader believe that you're adhering to the functional
paradigm whereas 'append' has actually side effects!
Moreover, you use an assignment just to reinforce this wrong belief.

Kiuhnm

[Bernd Nawothnig]

On 2012-04-21, Kiuhnm wrote:
>> Returning a None-value is pretty useless. Why not returning self, which would be
>> the resulting list in this case? Returning self would make the
>> language a little bit more functional, without any drawback.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^

>>
>> Then nested calls like
>>
>> a = [].append('x').append('y').append('z')
>
> You just answered to your own question: append returns None so that
> people can't use it the way you did.

That is one possible way to design the method, but not the only
possible way.

> You make the reader believe that you're adhering to the functional
> paradigm whereas 'append' has actually side effects!
> Moreover, you use an assignment just to reinforce this wrong belief.

I know about side effects and I know that letting append return self
would not make Python a purely functional language with only immutable
data.

I just asked a simple question about a detail I personally would
consider it to be useful.

Please no further religious war about that ;-)

[Kiuhnm]

On 4/21/2012 17:41, Bernd Nawothnig wrote:
> On 2012-04-21, Kiuhnm wrote:
>>> Returning a None-value is pretty useless. Why not returning self, which would be
>>> the resulting list in this case? Returning self would make the
>>> language a little bit more functional, without any drawback.
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>
>>> Then nested calls like
>>>
>>> a = [].append('x').append('y').append('z')
>>
>> You just answered to your own question: append returns None so that
>> people can't use it the way you did.
>
> That is one possible way to design the method, but not the only
> possible way.
>
>> You make the reader believe that you're adhering to the functional
>> paradigm whereas 'append' has actually side effects!
>> Moreover, you use an assignment just to reinforce this wrong belief.
>
> I know about side effects and I know that letting append return self
> would not make Python a purely functional language with only immutable
> data.
>
> I just asked a simple question about a detail I personally would
> consider it to be useful.
>
> Please no further religious war about that ;-)

Sorry if I wasn't clear. I meant that one should either relies on
side-effects and write something like
a.append('x').append('t').append('z')
or use a more functional style and write
a = a + [x] + [z]
Mixing the two doesn't seem very elegant to me.
I also think that if you need to repeatedly modify an object in the same
expression, then you should prefer the more functional approach.
I wasn't suggesting that you didn't know what functional programming and
side effects are.
Mine was a justification (maybe opinable) of why append was designed
that way.

Kiuhnm

[Kiuhnm]

Oops... I forgot the single quotes and the 't'.

Kiuhnm

[Bernd Nawothnig]

On 2012-04-21, Kiuhnm wrote:
> Sorry if I wasn't clear. I meant that one should either relies on
> side-effects and write something like
> a.append('x').append('t').append('z')
> or use a more functional style and write
> a = a + [x] + [z]
> Mixing the two doesn't seem very elegant to me.
> I also think that if you need to repeatedly modify an object in the same
> expression, then you should prefer the more functional approach.
> I wasn't suggesting that you didn't know what functional programming and
> side effects are.
> Mine was a justification (maybe opinable) of why append was designed
> that way.

Ok, understood and accepted

[Steven D'Aprano]

On Sat, 21 Apr 2012 14:48:44 +0200, Bernd Nawothnig wrote:

> On 2012-04-20, Rotwang wrote:
>> since a method doesn't assign the value it returns to the instance on
>> which it is called; what it does to the instance and what it returns
>> are two completely different things.
>
> Returning a None-value is pretty useless. Why not returning self, which
> would be the resulting list in this case? Returning self would make the
> language a little bit more functional, without any drawback.

It is a deliberate design choice, and there would be a drawback.

A method like append could have three obvious designs:

1) Functional, no side-effects: return a new list with the item appended.

2) Functional, with side-effect: return the same list, after appending
the item.

3) Procedural, with side-effect: append the item, don't return anything
(like a procedure in Pascal, or void in C).

Python chooses 3) as the design, as it is the cleanest, most pure choice
for a method designed to operate by side-effect. Unfortunately, since
Python doesn't have procedures, that clean design is slightly spoilt due
to the need for append to return None (instead of not returning anything
at all).

How about 1), the pure functional design? The downside of that is the
usual downside of functional programming -- it is inefficient to
duplicate a list of 100 million items just to add one more item to that
list. Besides, if you want a pure functional append operation, you can
simply use mylist + [item] instead.

But what about 2), the mixed (impure) functional design? Unfortunately,
it too has a failure mode: by returning a list, it encourages the error
of assuming the list is a copy rather than the original:

mylist = [1, 2, 3, 4]
another_list = mylist.append(5)
# many pages of code later...
do_something_with(mylist)


This is especially a pernicious error because instead of giving an
exception, your program will silently do the wrong thing.

"I find it amusing when novice programmers believe their main
job is preventing programs from crashing. More experienced
programmers realize that correct code is great, code that
crashes could use improvement, but incorrect code that doesn’t
crash is a horrible nightmare."
-- Chris Smith


Debugging these sorts of bugs can become very difficult, and design 2) is
an attractive nuisance: it looks good because you can chain appends:

mylist.append(17).append(23).append(42)
# but why not use mylist.extend([17, 23, 42]) instead?

but the disadvantage in practice far outweighs the advantage in theory.

This is the same reason why list.sort, reverse and others also return
None.

> Then nested calls like
>
> a = [].append('x').append('y').append('z')
>
> would be possible with a containing the resulting list
>
> ['x', 'y', 'z'].
>
> That is the way I expect any append to behave.

That would be possible, but pointless. Why not use:

a = ['x', 'y', 'z']

directly instead of constructing an empty list and then make three
separate method calls? Methods which operate by side-effect but return
self are an attractive nuisance: they seem like a good idea but actually
aren't, because they encourage the user to write inefficient, or worse,
incorrect, code.



--
Steven

[Bernd Nawothnig]

On 2012-04-22, Steven D'Aprano wrote:
> On Sat, 21 Apr 2012 14:48:44 +0200, Bernd Nawothnig wrote:
>
>> On 2012-04-20, Rotwang wrote:
>>> since a method doesn't assign the value it returns to the instance on
>>> which it is called; what it does to the instance and what it returns
>>> are two completely different things.
>>
>> Returning a None-value is pretty useless. Why not returning self, which
>> would be the resulting list in this case? Returning self would make the
>> language a little bit more functional, without any drawback.
>
> It is a deliberate design choice, and there would be a drawback.
>
> A method like append could have three obvious designs:
>
> 1) Functional, no side-effects: return a new list with the item appended.
>
> 2) Functional, with side-effect: return the same list, after appending
> the item.
>
> 3) Procedural, with side-effect: append the item, don't return anything
> (like a procedure in Pascal, or void in C).

Correct.

> Python chooses 3) as the design, as it is the cleanest, most pure choice
> for a method designed to operate by side-effect. Unfortunately, since
> Python doesn't have procedures, that clean design is slightly spoilt due
> to the need for append to return None (instead of not returning anything
> at all).
>
> How about 1), the pure functional design? The downside of that is the
> usual downside of functional programming -- it is inefficient to
> duplicate a list of 100 million items just to add one more item to that
> list.

In general I always prefer the pure functional approach. But you are
right, if it is too costly, one has to weigh the pros and contras.

> Besides, if you want a pure functional append operation, you can
> simply use mylist + [item] instead.

That ist true. I will keep that in mind :-)

> But what about 2), the mixed (impure) functional design? Unfortunately,
> it too has a failure mode: by returning a list, it encourages the error
> of assuming the list is a copy rather than the original:
>
> mylist = [1, 2, 3, 4]
> another_list = mylist.append(5)
> # many pages of code later...
> do_something_with(mylist)

Yes, but mutable data is in general a candidate for unexpected
behaviour, regardless wether you use an impure functional notation or
not:

mylist = [1, 2, 3, 4]

mylist.append(5)
another_list = mylist

# many pages of code later...
do_something_with(mylist)

avoids that impure function call but can perfectly lead to the same
unexpected behaviour. Your "many pages of code later" and that it is
simply difficult or impossible to keep in mind all these possible
state changes of variables is the real problem here.

> This is especially a pernicious error because instead of giving an
> exception, your program will silently do the wrong thing.
>
> "I find it amusing when novice programmers believe their main
> job is preventing programs from crashing. More experienced
> programmers realize that correct code is great, code that
> crashes could use improvement, but incorrect code that doesn’t
> crash is a horrible nightmare."
> -- Chris Smith

Absolutely corrrect!

> Debugging these sorts of bugs can become very difficult, and design 2) is
> an attractive nuisance: it looks good because you can chain appends:
>
> mylist.append(17).append(23).append(42)
> # but why not use mylist.extend([17, 23, 42]) instead?
>
> but the disadvantage in practice far outweighs the advantage in theory.
>
> This is the same reason why list.sort, reverse and others also return
> None.

Yeah, understood.

>> Then nested calls like
>>
>> a = [].append('x').append('y').append('z')
>>
>> would be possible with a containing the resulting list
>>
>> ['x', 'y', 'z'].
>>
>> That is the way I expect any append to behave.
>
> That would be possible, but pointless. Why not use:
>
> a = ['x', 'y', 'z']
>
> directly instead of constructing an empty list and then make three
> separate method calls? Methods which operate by side-effect but return
> self are an attractive nuisance: they seem like a good idea but actually
> aren't, because they encourage the user to write inefficient, or worse,
> incorrect, code.

In the past I often wrote methods that returned self instead of void,
None, or Nil depending on the used language.

But your arguments against that are not bad.

Thanks!

Instead of thinking about impure designs I should dig deeper into
Haskell :-)

[Kiuhnm]

On 4/22/2012 10:29, Bernd Nawothnig wrote:
[...]

> In general I always prefer the pure functional approach. But you are
> right, if it is too costly, one has to weigh the pros and contras.

Here's some stupid trick I came up with after reading this thread. It's
of very limited use, of course and I don't even know whether it's
reliable or not.

-->
import sys

class MyList: # just a test
def __init__(self, iter):
self.__start_cnt = sys.getrefcount(self)
self.__elems = list(iter)

def __add__(self, other):
global optimizations
cnt = sys.getrefcount(self)
if cnt == self.__start_cnt and len(self.__elems) > 100:
print('optimized')
optimizations += 1
self.__elems.extend(other.__elems)
return self
print('normal behavior')
return MyList(self.__elems + other.__elems)

def __eq__(self, other):
return self.__elems == other.__elems

def __iter__(self):
return self.__elems.__iter__()

optimizations = 0

print('#1')
y = MyList([1000, 1001])
x = MyList(range(0, 1000)) + y
assert x == MyList(range(0, 1002))
assert optimizations == 1

optimizations = 0

print('#2')
x = MyList(range(0, 1000))
z = x + MyList([1000, 1001])
assert x == MyList(range(0, 1000))
assert z == MyList(range(0, 1002))
assert optimizations == 0

optimizations = 0

print('#3')

def comp_list(from_, to_):
# Some long computation which results in a list.
return MyList(range(from_, to_))

x = comp_list(0, 1000) + comp_list(1000, 2000) + MyList([2000, 2001])
assert x == MyList(range(0, 2002))
assert optimizations == 2
<--

Kiuhnm