Immutability and Python

[andrea crotti]

I have a philosofical doubt about immutability, that arised while doing
the SCALA functional programming course.

Now suppose I have a simple NumWrapper class, that very stupidly does:

class NumWrapper(object):
def __init__(self, number):
self.number = number

and we want to change its state incrementing the number, normally I
would do this

def increment(self):
self.number += 1


But the immutability purists would instead suggest to do this:

def increment(self):
return NumWrapper(self.number + 1)


Now on one hand I would love to use only immutable data in my code, but
on the other hand I wonder if it makes so much sense in Python.

My impression is that things get more clumsy in the immutable form, for
example in the mutable form I would do simply this:

number = NumWrapper(1)
number.increment()

while with immutability I have to do this instead:
new_number = number.increment()

But more importantly normally classes are way more complicated than my
stupid example, so recreating a new object with the modified state might
be quite complex.

Any comments about this? What do you prefer and why?

[Jean-Michel Pichavant]

> --
> http://mail.python.org/mailman/listinfo/python-list
>

"return NumWrapper(self.number + 1) "

still returns a(nother) mutable object.

So what's the point of all this ?

JM

-- IMPORTANT NOTICE:

The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[andrea crotti]

2012/10/29 Jean-Michel Pichavant <jeanm...@sequans.com>:

>
> "return NumWrapper(self.number + 1) "
>
> still returns a(nother) mutable object.
>
> So what's the point of all this ?
>
> JM
>

Well sure but it doesn't modify the first object, just creates a new
one. There are in general good reasons to do that, for example I can
then compose things nicely:

num.increment().increment()

or I can parallelize operations safely not caring about the order of
operations.

But while I do this all the time with more functional languages, I
don't tend to do exactly the same in Python, because I have the
impression that is not worth, but maybe I'm wrong..

[andrea crotti]

2012/10/29 andrea crotti <andrea....@gmail.com>:

>>
>
> Well sure but it doesn't modify the first object, just creates a new
> one. There are in general good reasons to do that, for example I can
> then compose things nicely:
>
> num.increment().increment()
>
> or I can parallelize operations safely not caring about the order of
> operations.
>
> But while I do this all the time with more functional languages, I
> don't tend to do exactly the same in Python, because I have the
> impression that is not worth, but maybe I'm wrong..

By the way on this topic there is a great talk by the creator of
Clojure: http://www.infoq.com/presentations/Value-Values

[Mark Lawrence]

I prefer practicality beats purity.

--
Cheers.

Mark Lawrence.

[Paul Rubin]

andrea crotti <andrea....@gmail.com> writes:
> and we want to change its state incrementing the number ...

> the immutability purists would instead suggest to do this:
> def increment(self):
> return NumWrapper(self.number + 1)

Immutability purists would say that numbers don't have "state" and if
you're trying to change a number's state by incrementing it, that's not
immutability. You end up with a rather different programming style than
imperative programming, for example using tail recursion (maybe wrapped
in an itertools-like higher-order function) instead of indexed loops to
iterate over a structure.

[Jean-Michel Pichavant]

----- Original Message -----
> 2012/10/29 Jean-Michel Pichavant <jeanm...@sequans.com>:
> >
> > "return NumWrapper(self.number + 1) "
> >
> > still returns a(nother) mutable object.
> >
> > So what's the point of all this ?
> >
> > JM
> >
>

> Well sure but it doesn't modify the first object, just creates a new
> one. There are in general good reasons to do that, for example I can
> then compose things nicely:
>
> num.increment().increment()
>
> or I can parallelize operations safely not caring about the order of
> operations.
>
> But while I do this all the time with more functional languages, I
> don't tend to do exactly the same in Python, because I have the
> impression that is not worth, but maybe I'm wrong..
>

In an OOP language num.increment() is expected to modify the object in place.
So I think you're right when you say that functional languages technics do not necessarily apply to Python, because they don't.

I would add that what you're trying to suggest in the first post was not really about immutability, immutable objects in python are ... well immutable, they can be used as a dict key for instance, your NumWrapper object cannot.

[Chris Angelico]

In that case, rename increment to next_integer and TYAOOYDAO. [1]
You're not changing the state of this number, you're locating the
number which has a particular relationship to this one (in the same
way that GUI systems generally let you locate the next and previous
siblings of any given object).

ChrisA
[1] "there you are, out of your difficulty at once" - cf WS Gilbert's "Iolanthe"

[andrea crotti]

2012/10/29 Jean-Michel Pichavant <jeanm...@sequans.com>:

>
>
> In an OOP language num.increment() is expected to modify the object in place.
> So I think you're right when you say that functional languages technics do not necessarily apply to Python, because they don't.
>
> I would add that what you're trying to suggest in the first post was not really about immutability, immutable objects in python are ... well immutable, they can be used as a dict key for instance, your NumWrapper object cannot.
>
>
> JM

Yes right immutable was not the right word, I meant that as a contract
with myself I'm never going to modify its state.

Also because how doi I make an immutable object in pure Python?

But the example with the dictionary is not correct though, because this:

In [145]: class C(object):
.....: def __hash__(self):
.....: return 42
.....:

In [146]: d = {C(): 1}

works perfectly, but an object of class C can mutate as much as it
wants, as my NumWrapper instance..

[andrea crotti]

2012/10/29 Chris Angelico <ros...@gmail.com>:

> On Tue, Oct 30, 2012 at 2:55 AM, Paul Rubin <no.e...@nospam.invalid> wrote:

> In that case, rename increment to next_integer and TYAOOYDAO. [1]
> You're not changing the state of this number, you're locating the
> number which has a particular relationship to this one (in the same
> way that GUI systems generally let you locate the next and previous
> siblings of any given object).
>
> ChrisA
> [1] "there you are, out of your difficulty at once" - cf WS Gilbert's "Iolanthe"

> --
> http://mail.python.org/mailman/listinfo/python-list


Yes the name should be changed, but the point is that they are both
ways to implement the same thing.

For example suppose I want to have 10 objects (for some silly reason)
that represent the next number, in the first case I would do:

numbers = [NumWrapper(orig.number)] * 10
for num in numbers:
num.increment()

while in the second is as simple as:
numbers = [orig.next_number()] * 10

composing things become much easier, but as a downside it's not always
so easy and convienient to write code in this way, it probably depends
on the use case..

[Paul Rubin]

andrea crotti <andrea....@gmail.com> writes:
> Also because how doi I make an immutable object in pure Python?

Numbers in Python are already immutable. What you're really looking for
is a programming style where you don't bind any variable more than once.

This gives rise to a programming style that Python can support to a
certain extent, but for which some other languages are designed from the
beginning.

You might look at http://learnyouahaskell.com (online book) if you want
to try the above approach.

[andrea crotti]

2012/10/29 Paul Rubin <no.e...@nospam.invalid>:

> --
> http://mail.python.org/mailman/listinfo/python-list


I meant how do I create new immutables classes myself, I guess that's
possible writing C extensions but I don't see in pure Python..

Anyway I know haskell already, and that's one of the reasons why I was
trying to evaluate if the same techniques could be useful in Python as
well..

[Terry Reedy]

On 10/29/2012 11:20 AM, andrea crotti wrote:
> I have a philosofical doubt about immutability, that arised while doing
> the SCALA functional programming course.

In real life, the physical world, things have mutable state, at least
down to the atomic level. Do you only want to model mathematical worlds,
or also physical worlds.

Even mathematically, I do not think there necessarily a problem with
mutable collections. The fact that sets are defined by content does not
mean that everything has to be.

--
Terry Jan Reedy

[Evan Driscoll]

On 10/29/2012 12:05 PM, andrea crotti wrote:
> I meant how do I create new immutables classes myself, I guess that's
> possible writing C extensions but I don't see in pure Python..

The short answer is: you don't, not really, except by using NamedTuple
if that gives you what you want.

The longer answer:

You can kinda get it somewhat if you define your own
__getattribute__/__setattribute__ functions. __setattribute__ of course
should never do anything except raise an error (one way or another
you'll need to make an exception for your __init__ function of course).
__getattribute__ should make sure no mutable references are returned:
e.g. you'll probably want to make it so someone can't side-step your
setter by saying someobject.__dict__["foo"] = "bar". (I return a copy of
the dict.) It will still be possible to bypass these protections though.

To really get true immutability in pure Python, you'll have to inherit
from tuple or NamedTuple (which inherits from tuple, I think). You can
see some discussion on Stack Overflow and some other places about this;
having played around with this a bit, I think it's not worth the hassle
and have done the __getattribute__/__setattribute__ thing the couple of
times I wanted immutability.

Evan

[Terry Reedy]

On 10/29/2012 1:05 PM, andrea crotti wrote:

> I meant how do I create new immutables classes myself, I guess that's
> possible writing C extensions but I don't see in pure Python..

If you mean class with immutable instances, mutate new instances in
__new__ instead of __init__ and write a custom .__setattr__ that
prevents changes thereafter.

If you want the class itself to be immutable (after creation), write a
custom metaclass.

You may also need to think about .__getattribute__, but I never studied
the detail completely and have forgotten what I learned.

--
Terry Jan Reedy

[Devin Jeanpierre]

On Mon, Oct 29, 2012 at 12:46 PM, Paul Rubin <no.e...@nospam.invalid> wrote:
> andrea crotti <andrea....@gmail.com> writes:
>> Also because how doi I make an immutable object in pure Python?
>
> Numbers in Python are already immutable. What you're really looking for
> is a programming style where you don't bind any variable more than once.

No, they were looking for a way to create classes whose instances are immutable.

Also, immutability has nothing to do with the presence or lack of for loops.

-- Devin

[Ian Kelly]

On Mon, Oct 29, 2012 at 10:12 AM, andrea crotti
<andrea....@gmail.com> wrote:
> Also because how doi I make an immutable object in pure Python?

I sometimes use namedtuples for this.

from collections import namedtuple

MyImmutableClass = namedtuple('MyImmutableClass', 'field1 field2 field3 field4')

If you want default arguments then use a factory function. Or if you
want the class to have methods, then subclass it:

_MyImmutableClass = namedtuple('MyImmutableClass', 'field1 field2
field3 field4')

class MyImmutableClass(_MyImmutableClass):

def __new__(cls, field1, field2, field3=None, field4=42):
return super().__new__(cls, field1, field2, field3, field4)

def get_sum(self):
return self.field1 + self.field2

Cheers,
Ian

[Chris Angelico]

On Tue, Oct 30, 2012 at 6:23 AM, Ian Kelly <ian.g...@gmail.com> wrote:
> _MyImmutableClass = namedtuple('MyImmutableClass', 'field1 field2
> field3 field4')
>
> class MyImmutableClass(_MyImmutableClass):

Question: Is it clearer to take advantage of the fact that the base
class can be an arbitrary expression?

class MyImmutableClass(namedtuple('MyImmutableClass', 'field1 field2
field3 field4')):

You lose the unnecessary temporary and triplication of name, but gain
instead a rather long line.

ChrisA

[Ian Kelly]

On Mon, Oct 29, 2012 at 1:36 PM, Chris Angelico <ros...@gmail.com> wrote:
> Question: Is it clearer to take advantage of the fact that the base
> class can be an arbitrary expression?
>
> class MyImmutableClass(namedtuple('MyImmutableClass', 'field1 field2
> field3 field4')):
>
> You lose the unnecessary temporary and triplication of name, but gain
> instead a rather long line.

I think it's more readable if separated, but YMMV.

[Steven D'Aprano]

On Mon, 29 Oct 2012 17:05:07 +0000, andrea crotti wrote:

> I meant how do I create new immutables classes myself, I guess that's
> possible writing C extensions but I don't see in pure Python..

Well, you can't *quite* make a truly immutable class in pure-Python,
because if *your* Python code can manipulate the class during
construction then so can the caller's Python code after construction.

The trivial way to make an immutable class in Python is to inherit from
an already immutable class and add behaviour but no state:

class MyInt(int):
def inc(self):
return self.__class__(self + 1)


Otherwise, you can add private state and rely on the caller not shooting
themselves in the foot by accessing single-underscore names, use
properties to protect private state, etc.

See the source code for collections.namedtuple and decimal.Decimal for
some examples.

Warning: namedtuple is special, because it uses some runtime exec magic;
most immutable classes do not need that. And Decimal is seriously large
and complicated. But you can get some ideas from them both.

Also, see this:

http://northernplanets.blogspot.com.au/2007/01/immutable-instances-in-python.html



--
Steven

[Steven D'Aprano]

On Tue, 30 Oct 2012 06:36:52 +1100, Chris Angelico wrote:

> On Tue, Oct 30, 2012 at 6:23 AM, Ian Kelly <ian.g...@gmail.com>
> wrote:
>> _MyImmutableClass = namedtuple('MyImmutableClass', 'field1 field2
>> field3 field4')
>>
>> class MyImmutableClass(_MyImmutableClass):
>
> Question: Is it clearer to take advantage of the fact that the base
> class can be an arbitrary expression?
>
> class MyImmutableClass(namedtuple('MyImmutableClass', 'field1 field2
> field3 field4')):

I'm too lazy to google for it, but if you read the examples provided by
namedtuple's creator, Raymond Hettinger, that is precisely one of the
styles he uses. No need to explicitly declare the base class before using
it.



--
Steven

[Chris Kaynor]

On Mon, Oct 29, 2012 at 3:30 PM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
> On Mon, 29 Oct 2012 17:05:07 +0000, andrea crotti wrote:
>
>> I meant how do I create new immutables classes myself, I guess that's
>> possible writing C extensions but I don't see in pure Python..
>
> Well, you can't *quite* make a truly immutable class in pure-Python,
> because if *your* Python code can manipulate the class during
> construction then so can the caller's Python code after construction.
>
> The trivial way to make an immutable class in Python is to inherit from
> an already immutable class and add behaviour but no state:
>
> class MyInt(int):
> def inc(self):
> return self.__class__(self + 1)
>
>
> Otherwise, you can add private state and rely on the caller not shooting
> themselves in the foot by accessing single-underscore names, use
> properties to protect private state, etc.
>

You'd also need to add __slots__ = () to the class definition to make
it immutable. Otherwise they still can shoot themselves in the foot by
adding new attributes.

>>> class MyInt(int):
... def inc(self):
... return self.__class__(self+1)
...
>>> a = MyInt()
>>> a.b = 1 # Oops. Mutated "a".
>>> a.b
1



>>> class MyInt(int):
... __slots__ = ()
... def inc(self):
... return self.__class__(self + 1)
...
>>> a = MyInt()
>>> a.b = 1
AttributeError: 'MyInt' object has no attribute 'b'
Traceback (most recent call last):
File "<stdin-inspect>", line 1, in <module>
AttributeError: 'MyInt' object has no attribute 'b'

[Steven D'Aprano]

On Mon, 29 Oct 2012 15:20:02 +0000, andrea crotti wrote:

> I have a philosofical doubt about immutability, that arised while doing
> the SCALA functional programming course.

"Philosophical". Like most words derived from the ancient Greeks, the "F"
sound uses "ph" rather than "f".

> Now suppose I have a simple NumWrapper class, that very stupidly does:
>
> class NumWrapper(object):
> def __init__(self, number):
> self.number = number
>
> and we want to change its state incrementing the number, normally I
> would do this
>
> def increment(self):
> self.number += 1

That's a perfectly fine (although incomplete) design for a mutable
numeric class. But as the basis of an immutable class, it's lousy.

> But the immutability purists would instead suggest to do this:
>
> def increment(self):
> return NumWrapper(self.number + 1)

Only if they don't know Python very well :-)

In this example, the right way to get an immutable class is:

class NumWrapper(int): # not exactly a *wrapper*
def increment(self):
return self.__class__(self + 1)


and you're done. Immutability for free, because you don't store state
anywhere that pure-Python code can get to it. (Technically, using ctypes
you could mutate it, so don't do that.)

Here's a sketch of another technique:

class MyNum(object):
__slots__ = '_num'
def __new__(cls, arg):
instance = object.__new__(cls)
instance._num = int(arg)
return instance
@property
def value(self):
return self._num
def increment(self):
return self.__class__(self.value + 1)

> Now on one hand I would love to use only immutable data in my code, but
> on the other hand I wonder if it makes so much sense in Python.

You can go a long, long way using only immutable primitives and
functional style in Python, and I recommend it.

On the other hand, a *purely* functional approach doesn't make a lot of
sense for some tasks. Python is not a pure functional language, and
doesn't force you to hammer round pegs into the square hole of the
functional style.

Some problems are best modelled by an object that holds state and can
change over time, e.g. a database or a dict. Other problems are best
modelled by constants which do not change, but can be replaced by other
constants, e.g. numbers. Some problems fall into a grey area, e.g. lists,
arrays, sets, sequences, strings.

My advice is to always be alert for square pegs in your code, and write
them in functional style using immutable instances, but don't be a
purist. If you have a round peg, write that part of your code using a
mutable instance with in-place mutator methods, and be happy.

The beauty of Python is that you can use whichever style suits the
problem best.

> My impression is that things get more clumsy in the immutable form, for
> example in the mutable form I would do simply this:
>
> number = NumWrapper(1)
> number.increment()
>
> while with immutability I have to do this instead:
> new_number = number.increment()

Why is this clumsy? Do you have problems with this?

x = 1
y = x+1



--
Steven

[Steven D'Aprano]

On Mon, 29 Oct 2012 15:45:59 -0700, Chris Kaynor wrote:

> On Mon, Oct 29, 2012 at 3:30 PM, Steven D'Aprano
> <steve+comp....@pearwood.info> wrote:
>> On Mon, 29 Oct 2012 17:05:07 +0000, andrea crotti wrote:
>>
>>> I meant how do I create new immutables classes myself, I guess that's
>>> possible writing C extensions but I don't see in pure Python..
>>
>> Well, you can't *quite* make a truly immutable class in pure-Python,
>> because if *your* Python code can manipulate the class during
>> construction then so can the caller's Python code after construction.
>>
>> The trivial way to make an immutable class in Python is to inherit from
>> an already immutable class and add behaviour but no state:
>>
>> class MyInt(int):
>> def inc(self):
>> return self.__class__(self + 1)
>>
>>
>> Otherwise, you can add private state and rely on the caller not
>> shooting themselves in the foot by accessing single-underscore names,
>> use properties to protect private state, etc.
>>
>>
> You'd also need to add __slots__ = () to the class definition to make it
> immutable. Otherwise they still can shoot themselves in the foot by
> adding new attributes.

"Doctor, it hurts when I do this."

"Then don't do that."


I'm not a big fan of preventatively using __slots__ merely to prevent the
caller from tagging an object with extra data. Why do you care if the
caller sticks a postit note on the object? It doesn't hurt the object,
and if the caller loses track of which object has a postit note, that's
their responsibility, not yours.

I often wish I could sick an attribute on built-ins, e.g. after
calculating some numeric result as a float, stick an error estimate on
it. Callers who care about the error estimate can inspect it; those who
don't, will never even notice it.

If you have a good reason for using __slots__, then go right ahead.
Otherwise, don't be paternalistic. This is Python, we have the right to
shoot ourselves in the foot if we like.


--
Steven

[rusi]

On Oct 29, 8:20 pm, andrea crotti <andrea.crott...@gmail.com> wrote:
<snipped>

> Any comments about this? What do you prefer and why?

Im not sure how what the 'prefer' is about -- your specific num
wrapper or is it about the general question of choosing mutable or
immutable types?

If the latter I would suggest you read
http://en.wikipedia.org/wiki/Alexander_Stepanov#Criticism_of_OOP

[And remember that Stepanov is the author of C++ STL, he is arguably
as important in the C++ world as Stroustrup]

[Neal Becker]

The usual calls for immutability are not related to OO. They have to do with
optimization, and specifically with parallel processing.

[rusi]

From the time of Backus' Turing award
http://www.thocp.net/biographies/papers/backus_turingaward_lecture.pdf
it is standard fare that
assignment = imperative programming (which he collectively and
polemically called the von Neumann bottleneck)
That what he decried as 'conventional programming languages' today
applies to OO languages; see
http://www.cs.tufts.edu/~nr/backus-lecture.html

A more modern viewpoint:

--------------
Object-oriented programming is eliminated entirely from the
introductory curriculum, because it is both anti-modular and anti-
parallel by its very nature, and hence unsuitable for a modern CS
curriculum. A proposed new course on object-oriented design
methodology will be offered at the sophomore level for those students
who wish to study this topic.
----------------

from http://existentialtype.wordpress.com/2011/03/15/teaching-fp-to-freshmen/

Call it polemical if you like; noting that that's Carnegie Mellon.

[Thomas Rachel]

Am 29.10.2012 16:20 schrieb andrea crotti:

> Now on one hand I would love to use only immutable data in my code, but
> on the other hand I wonder if it makes so much sense in Python.

You can have both. Many mutable types distinguish between them with
their operators.

To pick up your example,

class NumWrapper(object):
def __init__(self, number):
self.number = number

def __iadd__(self, x):
self.number += x
return self
def __add__(self, x):
return NumWrapper(self.number + x)

So with

number += 1

you keep the same object and modify it, while with

number = number + 1

or

new_number = number + 1

you create a new object.

> But more importantly normally classes are way more complicated than my
> stupid example, so recreating a new object with the modified state might
> be quite complex.
>

> Any comments about this? What do you prefer and why?

That's why I generally prefer mutable objects, but it can depend.


Thomas