Bizarre behavior with mutable default arguments

3 views
Skip to first unread message

bukzor

unread,
Dec 29, 2007, 12:50:53 PM12/29/07
to
I've found some bizzare behavior when using mutable values (lists,
dicts, etc) as the default argument of a function. I want to get the
community's feedback on this. It's easiest to explain with code.

This example is trivial and has design issues, but it demonstrates a
problem I've seen in production systems:

def main(argv = ['TESTING']):
'print out arguments with BEGIN and END'
argv.insert(1, "BEGIN")
argv.append("END")
for arg in argv: print arg

if __name__ == '__main__':
from sys import argv, exit
exit(main(argv))


Now lets try this out at the terminal:

>>> import example
>>> example.main()
BEGIN
TESTING
END
>>> example.main(["TESTING"])
BEGIN
TESTING
END
>>> example.main()
BEGIN
BEGIN
TESTING
END
END
>>> example.main(["TESTING"])
BEGIN
TESTING
END

The function does different things if you call it with ["TESTING"] as
the argument, even though that is identical to the default value!! It
seems that default values are only allocated once. If the default
value is mutable and is changed during the function's execution, this
has the side effect of making the default value change on each
subsequent execution.

Is this functionality intended? It seems very unintuitive. This has
caused a bug in my programs twice so far, and both times I was
completely mystified until I realized that the default value was
changing.

I'd like to draw up a PEP to remove this from py3k, if I can get some
community backing.


--Buck

"Martin v. Löwis"

unread,
Dec 29, 2007, 1:11:44 PM12/29/07
to bukzor
> Is this functionality intended?

Google for "Python mutable default arguments" (you can actually
leave out Python).

It's part of the language semantics, yes.

Regards,
Martin

Istvan Albert

unread,
Dec 29, 2007, 1:17:29 PM12/29/07
to
On Dec 29, 12:50 pm, bukzor <workithar...@gmail.com> wrote:

> Is this functionality intended? It seems very unintuitive. This has
> caused a bug in my programs twice so far, and both times I was
> completely mystified until I realized that the default value was
> changing.

it is only unintuitive when you do not know about it

once you realize how it works and what it does it can actually be very
useful

i.

Istvan Albert

unread,
Dec 29, 2007, 1:49:58 PM12/29/07
to
On Dec 29, 1:11 pm, "Martin v. Löwis" <mar...@v.loewis.de> wrote:

> Google for "Python mutable default arguments"

and a mere 30 minutes later this thread is already one of the results
that come up

bukzor

unread,
Dec 29, 2007, 2:14:30 PM12/29/07
to
Here's the answer to the question:
http://www.python.org/doc/faq/general/#why-are-default-values-shared-between-objects

It looks like Guido disagrees with me, so the discussion is closed.


For the record, I still think the following would be an improvement to
py3k:

In python25:
def f(a=None):
if a is None: a = []
...

In py3k becomes:
def f(a=[])
...


In python25 (this function from the FAQ linked above):
def f(a, _cache={}):
# Callers will never provide a third parameter for this function.
(then why is it an argument?)
...

In py3k becomes:
_cache = {}
def f(a):
global _cache
...

This follows the "explicit is better" and "one best way" principles of
Python, and greatly improves the intuitiveness. Also since the first
example is much more common, it reduces the overall verbosity of the
language.

Just my parting two cents,
--Buck

Steven D'Aprano

unread,
Dec 29, 2007, 7:27:33 PM12/29/07
to
On Sat, 29 Dec 2007 11:14:30 -0800, bukzor wrote:


> In python25 (this function from the FAQ linked above):
> def f(a, _cache={}):
> # Callers will never provide a third parameter for this function.
> (then why is it an argument?)

The caller might want to provide it's own pre-prepared cache. Say, for
testing.

I think that this behaviour is a little unintuitive, and by a little I
mean a lot. Nevertheless, I am used to it, and I don't see any reason to
change it. There's very little about programming that is intuitive --
there's no intuitive reason to think that dictionary lookups are O(1)
while list lookups are O(n).

In the absence of a better solution, I'm very comfortable with keeping
the behaviour as is. Unfortunately, there's no good solution in Python to
providing functions with local storage that persists across calls to the
function:


(1) Use a global variable.

cache = {}
def foo():
global cache
print cache


(2) Use a function attribute.

def foo():
print foo.cache
foo.cache = {}


def foo():
try:
foo.cache
except AttributeError:
foo.cache = {}
print foo.cache

(3) Use an argument that isn't actually an argument.

def foo(cache={}):
print cache


#1, the global variable, is probably the worst solution of the lot.
Global variables are rightly Considered Harmful.


#2 has the disadvantages that you initialize the value *after* you write
the code that relies on it. Either that, or you waste time on every call
checking to see it if has been initialized. Also, like recursive
functions, it is difficult to rename the function.


#3 is, I think, the least-worse solution, but I would hardly call it
ideal.

> _cache = {}
> def f(a):
> global _cache
> ...
>
> This follows the "explicit is better" and "one best way" principles of
> Python,

Declaring an argument is equally explicit.

And you are confused -- the Zen doesn't say "one best way". People so
often get it wrong.

The Zen says:

"There should be one-- and PREFERABLY only one --OBVIOUS way to do it."
(Emphasis added.)

At least you're not saying "there should be only one way to do it". I
give you credit for that!


> and greatly improves the intuitiveness. Also since the first
> example is much more common, it reduces the overall verbosity of the
> language.

I question that it is "much more common". How do you know? Where's your
data?

--
Steven

Steven D'Aprano

unread,
Dec 29, 2007, 7:43:55 PM12/29/07
to
On Sat, 29 Dec 2007 09:50:53 -0800, bukzor wrote:

> I've found some bizzare behavior when using mutable values (lists,
> dicts, etc) as the default argument of a function.

This FAQ is so Frequently Asked that I sometimes wonder if Python should,
by default, print a warning when it compiles a function with a list or
dict as as default value.

There's precedence for such a thing: the sum() built-in (un)helpfully
raises an exception if you try to use it on strings.

I say unhelpfully because the one time I wanted to use sum() on strings
was specifically to demonstrate the difference between O(n**2) behaviour
and O(n). I was quite put out that Python, which normally allows you to
shoot yourself in the foot if you insist, was so unnecessarily protective
in this case. Give me a warning, if you wish, but don't stop me.

--
Steven

bukzor

unread,
Dec 29, 2007, 10:50:18 PM12/29/07
to
> I think that this behaviour is a little unintuitive, and by a little I
> mean a lot.

Thanks for acknowledging it.

> I question that it is "much more common". How do you know? Where's your
> data?

I did a dumb grep of my Python25/Lib folder and found 33 occurances of
the first pattern above. (Use None as the default value, then check
for None and assign empty list/dict)

Although I spent at least double the amount of time looking for the
second pattern, I found no occurances. (Use dict/list as default value
and modify it in place.) Every single function that used a list or
dict as a default value treated these variables as read-only. However,
I did find two new ways to accomplish the above (further violating the
Zen).

/c/Python25/Lib/site-packages/wx-2.8-msw-ansi/wx/lib/
customtreectrl.py:
def FillArray(self, item, array=[]):
if not array:
array = []

/c/Python25/Lib/site-packages/wx-2.8-msw-ansi/wx/lib/floatcanvas/
FloatCanvas.py:
def __init__(self, ObjectList=[], InForeground = False, IsVisible =
True):
self.ObjectList = list(ObjectList)

--Buck

bukzor

unread,
Dec 29, 2007, 11:21:08 PM12/29/07
to
Just for completeness, the mutable default value problem also affects
classes:

class c:
def __init__(self, list = []):
self.list = list
self.list.append("LIST END")
def __repr__(self):
return "<Class a: %s>" % self.list

>>> import example2
>>> print example2.c()
<Class a: ['LIST END']>
>>> print example2.c([])
<Class a: ['LIST END']>
>>> print example2.c()
<Class a: ['LIST END', 'LIST END']>
>>> print example2.c([])
<Class a: ['LIST END']>

Again, we get different results if we supply an argument that is
identical to the default value. There are many instances in the
standard library where class values are assigned directly from the
initializer, which has list or dict default values, so there is chance
for errors cropping up here.

The error scenario is this:
1. Use a mutable value as default value in a class constructor.
2. Assign class property from constructor arguments.
3. Instantiate class using default value.
4. Modify class property in place.
5. Instantiate (again) class using default value.

The second instance will behave strangely because data from the first
instance has leaked over. The standard library is not affected because
it avoids one of these five steps. Most classes simply don't have
mutable default values (1). Those that do generally treat them as read-
only (4). Some classes are not useful using the default values (3).
Some classes are not useful to be instantiated twice (5). The classes
that don't avoid the problem at one of these four steps have to avoid
it at (2) by using one of the three above patterns.

--Buck

John Machin

unread,
Dec 30, 2007, 12:17:27 AM12/30/07
to
On Dec 30, 3:21 pm, bukzor <workithar...@gmail.com> wrote:
> Just for completeness, the mutable default value problem also affects
> classes:

Simply, because methods are functions, and can have default arguments.
You don't need to nail *another* zillion theses to the cathedral
door :-)

thebjorn

unread,
Dec 30, 2007, 5:23:40 AM12/30/07
to

I agree it is a potentially useful feature, yet it can still bite you
even after a decade of Python... Scenario: long running server
process, Bug report: "people aren't getting older", Code:

def age(dob, today=datetime.date.today()):
...

None of my unit tests caught that one :-)

-- bjorn

Istvan Albert

unread,
Dec 30, 2007, 8:45:46 AM12/30/07
to
On Dec 30, 5:23 am, thebjorn <BjornSteinarFjeldPetter...@gmail.com>
wrote:

> def age(dob, today=datetime.date.today()):
> ...
>
> None of my unit tests caught that one :-)

interesting example I can see how it caused some trouble. A quick fix
would be to write it:

def age(dob, today=datetime.date.today ):

and inside the definition invoke it as today() rather than just today.
That way it still keeps the original spirit of the definition.

i.

Istvan Albert

unread,
Dec 30, 2007, 9:07:09 AM12/30/07
to
On Dec 29, 11:21 pm, bukzor <workithar...@gmail.com> wrote:

> The standard library is not affected because

the people who wrote code into it know how python works.

Programming abounds with cases that some people think should work
differently:

a = b = []
a.append(1)

is b empty or not at this point? Get informed, remember the rules, be
happy and move on to write some cool code.

There is little new in what you say. Every so often someone is having
a confusing time with a feature and therefore proposes that the
language be changed to match his/her expectations.

i.

Stephe...@gmail.com

unread,
Dec 30, 2007, 11:25:59 AM12/30/07
to

George Sakkis

unread,
Dec 30, 2007, 11:26:37 AM12/30/07
to
On Dec 29, 9:14 pm, bukzor <workithar...@gmail.com> wrote:
> Here's the answer to the question:http://www.python.org/doc/faq/general/#why-are-default-values-shared-...

>
> It looks like Guido disagrees with me, so the discussion is closed.

Note that the FAQ mainly explains *what* happens, not *why* was this
decision taken. Although it shows an example where "this feature can
be useful", it's neither the only way to do it nor is memoization as
common as wanting fresh default arguments on every call.

> For the record, I still think the following would be an improvement to
> py3k:
>
> In python25:
> def f(a=None):
> if a is None: a = []
> ...
>
> In py3k becomes:
> def f(a=[])
> ...
>
> In python25 (this function from the FAQ linked above):
> def f(a, _cache={}):
> # Callers will never provide a third parameter for this function.
> (then why is it an argument?)
> ...
>
> In py3k becomes:
> _cache = {}
> def f(a):
> global _cache
> ...
>
> This follows the "explicit is better" and "one best way" principles of
> Python, and greatly improves the intuitiveness. Also since the first
> example is much more common, it reduces the overall verbosity of the
> language.

I'm with you on this one; IMHO it's one of the relatively few language
design missteps of Python, favoring the rare case as the default
instead of the common one. Unfortunately, many Pythoneers become so
immersed with the language and whatever the current status quo is that
they rarely question the rationale of the few counter-intuitive design
choices.

George

thebjorn

unread,
Dec 30, 2007, 12:57:49 PM12/30/07
to

The purpose of the argument was to be able to calculate the age at a
given point in time -- i.e. was the person 18 y/o at the time of the
incident?

Our coding standard now dictates:

def foo(arg=None):
if arg is None:
arg = <default mutable value>

(unless there's a very good reason to do it otherwise :-)

a close runner-up, that we specifically opted not to allow was

def foo(arg={}):
arg = arg or {}

even though it looks sexy and it's perhaps a bit more self-documenting
in some IDEs, it was rejected because it prevents "false" overrides of
the default argument.

For purely practical reasons we couldn't consider

def foo(arg=None):
arg = <default mutable value> if arg is None else arg

-- bjorn

bukzor

unread,
Dec 30, 2007, 3:11:50 PM12/30/07
to
On Dec 30, 2:23 am, thebjorn <BjornSteinarFjeldPetter...@gmail.com>
wrote:

> Scenario: long running server process,
> Bug report: "people aren't getting older", Code:
>
> def age(dob, today=datetime.date.today()):
> ...

A very interesting example, thanks.

On Dec 30, 8:25 am, StephenRF...@gmail.com wrote:
> This is a well-known python gotcha. See:http://www.ferg.org/projects/python_gotchas.html#contents_item_6

Just because it's well known doesn't mean we shouldn't think about it.
For example, in the same list you linked, "3. Integer division" is
being fixed in py3k.


On Dec 30, 8:26 am, George Sakkis <george.sak...@gmail.com> wrote:
> I'm with you on this one; IMHO it's one of the relatively few language
> design missteps of Python, favoring the rare case as the default
> instead of the common one.

Well put. Although I've seen 'potentially useful' often used about
this, I havn't found an instance of its use in production code.

On Dec 30, 9:57 am, thebjorn <BjornSteinarFjeldPetter...@gmail.com>
wrote:


> Our coding standard now dictates:
>
> def foo(arg=None):
> if arg is None:
> arg = <default mutable value>
>
> (unless there's a very good reason to do it otherwise :-)

I believe this is very similar to most people's coding practices. It's
the one I adopted, it's in the above FAQ, the above 'gotchas' list,
and it's all over the standard library. This is a case of a language
feature requiring coding practices to prevent bugs, with very little
value added elsewhere.

--Buck

Istvan Albert

unread,
Dec 30, 2007, 3:32:04 PM12/30/07
to
On Dec 30, 11:26 am, George Sakkis <george.sak...@gmail.com> wrote:

> I'm with you on this one; IMHO it's one of the relatively few language
> design missteps of Python, favoring the rare case as the default
> instead of the common one.

George, you pointed this out this link in a different thread

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/521877

how would you rewrite the code below if you could not use mutable
default arguments (global variables not accepted)? Maybe there is a
way, but I can't think of it as of now.

---------------------------------------

def blocks(s, start, end):
def classify(c, ingroup=[0]):
klass = c==start and 2 or c==end and 3 or ingroup[0]
ingroup[0] = klass==1 or klass==2
return klass
return [tuple(g) for k, g in groupby(s, classify) if k == 1]

print blocks('the {quick} brown {fox} jumped', start='{', end='}')

bukzor

unread,
Dec 30, 2007, 3:41:57 PM12/30/07
to

Extremely simple

def blocks(s, start, end):
ingroup=[0]
def classify(c):


klass = c==start and 2 or c==end and 3 or ingroup[0]
ingroup[0] = klass==1 or klass==2
return klass
return [tuple(g) for k, g in groupby(s, classify) if k == 1]

print blocks('the {quick} brown {fox} jumped', start='{', end='}')


No globals, as you specified. BTW, it's silly not to 'allow' globals
when they're called for, otherwise we wouldn't need the 'global'
keyword.

--Buck

Istvan Albert

unread,
Dec 30, 2007, 3:54:25 PM12/30/07
to
On Dec 30, 3:41 pm, bukzor <workithar...@gmail.com> wrote:

> No globals, as you specified. BTW, it's silly not to 'allow' globals
> when they're called for, otherwise we wouldn't need the 'global'
> keyword.

okay, now note that you do not actually use the ingroup list for
anything else but getting and setting its first element. So why would
one really need it be a list? Let's replace it with a variable called
ingroup that is not a list anymore. See it below (run it to see what
happens):

----------------------

def blocks(s, start, end):
ingroup = 0


def classify(c):
klass = c==start and 2 or c==end and 3 or ingroup

ingroup = klass==1 or klass==2

Message has been deleted

Steven D'Aprano

unread,
Dec 30, 2007, 6:34:07 PM12/30/07
to
On Sun, 30 Dec 2007 12:41:57 -0800, bukzor wrote:

> BTW, it's silly not to 'allow' globals when they're called for,
> otherwise we wouldn't need the 'global' keyword.

Nobody argues against allowing globals variables *when they're called
for*, just pointing out that ninety-nine times out of a hundred, people
use them when they're not called for and are positively harmful.

And according to functional programmers, they're NEVER called for.


--
Steven.

Steven D'Aprano

unread,
Dec 30, 2007, 7:01:23 PM12/30/07
to
On Sun, 30 Dec 2007 13:34:07 -0800, Dennis Lee Bieber wrote:

> On Sun, 30 Dec 2007 12:11:50 -0800 (PST), bukzor
> <workit...@gmail.com> declaimed the following in comp.lang.python:


>
>
>> Just because it's well known doesn't mean we shouldn't think about it.
>> For example, in the same list you linked, "3. Integer division" is
>> being fixed in py3k.
>>

> IMHO -- Py3K is /breaking/ integer division... as the division of
> two integers will differ from what happens in all the other languages I
> have used... All the others, to get a floating result from dividing two
> integers requires one to explicitly convert at least one term to a float
> first

You need to use more languages :)

Prolog uses / for division and // for integer division, just like Python.

Apple's Hypertalk (and derivatives) don't distinguish between integer and
floating point division. The / operator returns an integer result if the
floating point result happens to be an integer.

e.g. 10.0/5.0 => 2 while 11.0/5.0 => 2.2)

I believe that Javascript behaves the same way.

According to this table here:
http://msdn2.microsoft.com/en-us/library/2hxce09y.aspx

VisualBasic uses / for floating point division and \ for integer
division, and both JScript and Visual FoxPro don't even offer integer
division at all.

No doubt there are others...

> -- as I would do with the current Python. The forthcoming change
> is going to require one to remember that if they want an integer result
> from two integers, they must use a different operator instead.

How is that different from needing to remember to use a different
algorithm if you want a floating point result?

--
Steven.

bukzor

unread,
Dec 30, 2007, 11:00:14 PM12/30/07
to
On Dec 30, 3:34 pm, Steven D'Aprano <st...@REMOVE-THIS-


I think you struck at the heart of the matter earlier when you noted
that this is the simplest way to declare a static variable in python.
Using the 'global' keyword is the other way, and is much more
explicit, and much more widely used. I also see this as the main use
of the 'notlocal' keyword to be introduced in py3k (it also fixes the
example given by Istvan above).

If the main value of this behavior is to declare a static variable, it
seems like an argument to create a more explicit syntax for static
variables. In the example above, the function only needed a static
integer, but created a one-length list instead because this quirk
doesn't work for immutable values.

Steven D'Aprano

unread,
Dec 31, 2007, 2:01:51 AM12/31/07
to
On Sun, 30 Dec 2007 20:00:14 -0800, bukzor wrote:

> I think you struck at the heart of the matter earlier when you noted
> that this is the simplest way to declare a static variable in python.
> Using the 'global' keyword is the other way, and is much more explicit,
> and much more widely used. I also see this as the main use of the
> 'notlocal' keyword to be introduced in py3k (it also fixes the example
> given by Istvan above).

There doesn't appear to be any reference to a "notlocal" keyword in
Python 3 that I can find. Have I missed something? It sounds like an
April Fool's gag to me. Do you have a reference to a PEP or other
official announcement?

--
Steven

Gabriel Genellina

unread,
Dec 31, 2007, 2:35:44 AM12/31/07
to pytho...@python.org
En Mon, 31 Dec 2007 05:01:51 -0200, Steven D'Aprano
<st...@REMOVE-THIS-cybersource.com.au> escribió:

No, it's a real keyword in python 3, but it's spelled "nonlocal".
See http://www.python.org/dev/peps/pep-3104/

--
Gabriel Genellina

Odalrick

unread,
Dec 31, 2007, 5:58:45 AM12/31/07
to
On 30 Dec, 17:26, George Sakkis <george.sak...@gmail.com> wrote:
> On Dec 29, 9:14 pm, bukzor <workithar...@gmail.com> wrote:
>
> > Here's the answer to the question:http://www.python.org/doc/faq/general/#why-are-default-values-shared-...
>
> > It looks like Guido disagrees with me, so the discussion is closed.
>
> Note that the FAQ mainly explains *what* happens, not *why* was this
> decision taken. Although it shows an example where "this feature can
> be useful", it's neither the only way to do it nor is memoization as
> common as wanting fresh default arguments on every call.
>

I'm surprised noone has said anything about the why of default
mutables. I think it is becasue it isn't easy to do it an other way.

def some_function( an_integer=1,pointless_list=[],
random_fuction_value=random_function()):
pass

To you and me it is obvious that this is an integer, a list and a
function call, but to python it is just 3 objects. Python'd have to
check each argument carefully to determine if it is mutable or not. Or
always copy each object, adding additional overhead to function calls,
and making passing arguments to functions expensive.

Even if these problems were solved, it would only make the problem
less common, not extinct.

# hypothetical
def another_function( still_alive=([],) ):
still_alive[0].append('spam')
print still_alive

>>> another_function()
(['spam'],)
>>> another_function()
(['spam', 'spam'],)

(Could of course be solved by always making deep copies of all
arguments.)

While I would welcome making mutable defaults work differently, I
don't see any way to make such a change without making unacceptable
tradeoffs.


--

Incidentally, I wrote a program a while back, with a bug caused by
mutable defaults. Never bothered to change it, it was the behaviour I
wanted, just not the one I thought I had implemented. -- Python, so
good even the bugs make the program better.

Arnaud Delobelle

unread,
Dec 31, 2007, 12:22:14 PM12/31/07
to
On Dec 31, 10:58 am, Odalrick <odalr...@hotmail.com> wrote:
> On 30 Dec, 17:26, George Sakkis <george.sak...@gmail.com> wrote:
>
> > On Dec 29, 9:14 pm, bukzor <workithar...@gmail.com> wrote:
>
> > > Here's the answer to the question:http://www.python.org/doc/faq/general/#why-are-default-values-shared-...
>
> > > It looks like Guido disagrees with me, so the discussion is closed.
>
> > Note that the FAQ mainly explains *what* happens, not *why* was this
> > decision taken. Although it shows an example where "this feature can
> > be useful", it's neither the only way to do it nor is memoization as
> > common as wanting fresh default arguments on every call.
>
> I'm surprised noone has said anything about the why of default
> mutables. I think it is becasue it isn't easy to do it an other way.

[...]

There is an easy enough way: evaluate default values when the function
is called rather than when it is defined. This behaviour comes with
its own caveats as well I imagine, and it's not 'as easy' to implement
as the current one.

What's good about the current behaviour is that it is easy to reason
with (once you know what happens), even though you almost have to get
bitten once. But using this to have static variable is extremely ugly
IMHO.

--
Arnaud

Odalrick

unread,
Dec 31, 2007, 3:08:22 PM12/31/07
to
On 31 Dec, 18:22, Arnaud Delobelle <arno...@googlemail.com> wrote:
> On Dec 31, 10:58 am, Odalrick <odalr...@hotmail.com> wrote:
>
> > On 30 Dec, 17:26, George Sakkis <george.sak...@gmail.com> wrote:
>
> > > On Dec 29, 9:14 pm, bukzor <workithar...@gmail.com> wrote:
>
> > > > Here's the answer to the question:http://www.python.org/doc/faq/general/#why-are-default-values-shared-...
>
> > > > It looks like Guido disagrees with me, so the discussion is closed.
>
> > > Note that the FAQ mainly explains *what* happens, not *why* was this
> > > decision taken. Although it shows an example where "this feature can
> > > be useful", it's neither the only way to do it nor is memoization as
> > > common as wanting fresh default arguments on every call.
>
> > I'm surprised noone has said anything about the why of default
> > mutables. I think it is becasue it isn't easy to do it an other way.
>
> [...]
>
> There is an easy enough way: evaluate default values when the function
> is called rather than when it is defined. This behaviour comes with
> its own caveats as well I imagine, and it's not 'as easy' to implement
> as the current one.
>

Adding overhead to *all* function calls, even the ones without mutable
defaults. That doesn't sound like an attractive tradeoff.

Chris Mellon

unread,
Dec 31, 2007, 4:30:06 PM12/31/07
to pytho...@python.org
On Dec 31, 2007 2:08 PM, Odalrick <odal...@hotmail.com> wrote:
> On 31 Dec, 18:22, Arnaud Delobelle <arno...@googlemail.com> wrote:
> > On Dec 31, 10:58 am, Odalrick <odalr...@hotmail.com> wrote:
> >
> > > On 30 Dec, 17:26, George Sakkis <george.sak...@gmail.com> wrote:
> >
> > > > On Dec 29, 9:14 pm, bukzor <workithar...@gmail.com> wrote:
> >
> > > > > Here's the answer to the question:http://www.python.org/doc/faq/general/#why-are-default-values-shared-...
> >
> > > > > It looks like Guido disagrees with me, so the discussion is closed.
> >
> > > > Note that the FAQ mainly explains *what* happens, not *why* was this
> > > > decision taken. Although it shows an example where "this feature can
> > > > be useful", it's neither the only way to do it nor is memoization as
> > > > common as wanting fresh default arguments on every call.
> >
> > > I'm surprised noone has said anything about the why of default
> > > mutables. I think it is becasue it isn't easy to do it an other way.
> >
> > [...]
> >
> > There is an easy enough way: evaluate default values when the function
> > is called rather than when it is defined. This behaviour comes with
> > its own caveats as well I imagine, and it's not 'as easy' to implement
> > as the current one.
> >
>
> Adding overhead to *all* function calls, even the ones without mutable
> defaults. That doesn't sound like an attractive tradeoff.
>

And also removing the only way you can currently do early binding in
Python. I agree that it's a gotcha, but unless someone comes up with
an answer to the following questions, I'll stick with the status quo
(Note that this is not blind Python group-think as a previous poster
implied, but a pragmatic decision that this is the most practical
solution):

a) If we don't evaluate default arguments at function compilation,
when do we do it?
b) If you do it at call time, how do you implement early binding?
c) If you want to introduce new syntax for the current behavior, what
is it and can you justify it?
d) What are the performance implications of your proposal versus the
current behavior?

Note that the desired behavior can be implemented under the current
behavior, at the expense of verbosity - using factories and sentinel
values as the default arguments, and then expanding them in the
function. It's not possible to implement the current behavior of
early-bound arguments if default arguments are evaluated with every
call. This alone is a good reason to keep the current behavior until
someone actually has a good alternative that covers the current use
cases and isn't just upset by the behavior.

NickC

unread,
Jan 1, 2008, 12:26:38 AM1/1/08
to
On Jan 1, 3:22 am, Arnaud Delobelle <arno...@googlemail.com> wrote:
> On Dec 31, 10:58 am, Odalrick <odalr...@hotmail.com> wrote:
>
> > I'm surprised noone has said anything about the why of default
> > mutables. I think it is becasue it isn't easy to do it an other way.
>
> [...]
>
> There is an easy enough way: evaluate default values when the function
> is called rather than when it is defined. This behaviour comes with
> its own caveats as well I imagine, and it's not 'as easy' to implement
> as the current one.

As Odalrick notes, there is no way to give different calls to a
function their own copies of mutable default arguments without re-
evaluating the defaults every time the function is called. The
horrendous performance implications mean that that simply isn't going
to happen. So the status quo, where the defaults are calculated once
when the function is defined and the result cached in the function
object is unlikely to change.

> What's good about the current behaviour is that it is easy to reason
> with (once you know what happens), even though you almost have to get
> bitten once. But using this to have static variable is extremely ugly
> IMHO.

The only thing it doesn't give you is a static variable that isn't
visible to the caller. Py3k's keyword-only arguments (PEP 3102) will
make those cases a little tidier, since it won't be possible to
accidentally replace the static variables by providing too many
positional arguments.

I believe the suggestion of permitting static variables after the **
entry in a function's parameter list was raised during the PEP 3102
discussions, but never gained much traction over a '_cache={}' keyword-
only argument approach (and the latter has the distinct advantage of
being *much* easier to test, since you can override the cache from the
test code to ensure it is being handled correctly).

Cheers,
Nick.

Arnaud Delobelle

unread,
Jan 1, 2008, 4:32:18 AM1/1/08
to
On Jan 1, 5:26 am, NickC <ncogh...@gmail.com> wrote:
> On Jan 1, 3:22 am, Arnaud Delobelle <arno...@googlemail.com> wrote:
>
> > On Dec 31, 10:58 am, Odalrick <odalr...@hotmail.com> wrote:
>
> > > I'm surprised noone has said anything about the why of default
> > > mutables. I think it is becasue it isn't easy to do it an other way.
>
> > [...]
>
> > There is an easy enough way: evaluate default values when the function
> > is called rather than when it is defined.  This behaviour comes with
> > its own caveats as well I imagine, and it's not 'as easy' to implement
> > as the current one.
>
> As Odalrick notes, there is no way to give different calls to a
> function their own copies of mutable default arguments without re-
> evaluating the defaults every time the function is called. The
> horrendous performance implications mean that that simply isn't going
> to happen. So the status quo, where the defaults are calculated once
> when the function is defined and the result cached in the function
> object is unlikely to change.

I'm in no way advocating a change, in fact I wouldn't like things to
change. I was just saying that it was not difficult (technically) to
alter the behaviour, but that this change wouldn't be desirable
because it would make code more difficult to reason on. OTOH a very
common idiom in python is

def foo(x, y, z=None):
if z is None: z = ['a', 'mutable', 'object']
# stuff that foo does

This the current way to say "I want the default value of z to be
reevaluated each time it is used". I use this much more often than

def bar(x, y, z=ExpensiveImmutableCreation())

So I'm not so convinced with the performance argument at face value
(though it's probably pertinent:)

> > What's good about the current behaviour is that it is easy to reason
> > with (once you know what happens), even though you almost have to get
> > bitten once.  But using this to have static variable is extremely ugly
> > IMHO.
>
> The only thing it doesn't give you is a static variable that isn't
> visible to the caller. Py3k's keyword-only arguments (PEP 3102) will
> make those cases a little tidier, since it won't be possible to
> accidentally replace the static variables by providing too many
> positional arguments.

I was always a bit puzzled by this PEP. If this is one of the
underlying reasons for it, then I am even more puzzled.

> I believe the suggestion of permitting static variables after the **
> entry in a function's parameter list was raised during the PEP 3102
> discussions, but never gained much traction over a '_cache={}' keyword-
> only argument approach (and the latter has the distinct advantage of
> being *much* easier to test, since you can override the cache from the
> test code to ensure it is being handled correctly).

Well I'm glad that didn't go through, argument lists in function
definitions are complicated enough already!

--
Arnaud

bukzor

unread,
Jan 1, 2008, 11:47:11 AM1/1/08
to
On Dec 30 2007, 11:01 pm, Steven D'Aprano <st...@REMOVE-THIS-

I got it slightly wrong. It's 'nonlocal':
http://www.python.org/dev/peps/pep-3104/

--Buck

bukzor

unread,
Jan 1, 2008, 12:00:09 PM1/1/08
to
On Dec 31 2007, 1:30 pm, "Chris Mellon" <arka...@gmail.com> wrote:

I'm confused by what you mean by 'early binding'. Can you give a quick-
n-dirty example?

Thanks,
--Buck

bukzor

unread,
Jan 1, 2008, 12:45:00 PM1/1/08
to

Is an 'early bound' variable synonymous with a 'static' variable (in
C)?

Gabriel Genellina

unread,
Jan 1, 2008, 1:48:33 PM1/1/08
to pytho...@python.org
En Tue, 01 Jan 2008 15:45:00 -0200, bukzor <workit...@gmail.com>
escribi�:

> On Jan 1, 9:00 am, bukzor <workithar...@gmail.com> wrote:
>> On Dec 31 2007, 1:30 pm, "Chris Mellon" <arka...@gmail.com> wrote:
>>
>> > And also removing the only way you can currently do early binding in
>> > Python. I agree that it's a gotcha, but unless someone comes up with
>> > an answer to the following questions, I'll stick with the status quo
>> > (Note that this is not blind Python group-think as a previous poster
>> > implied, but a pragmatic decision that this is the most practical
>> > solution):
>>
>> > a) If we don't evaluate default arguments at function compilation,
>> > when do we do it?
>> > b) If you do it at call time, how do you implement early binding?
>>
>> I'm confused by what you mean by 'early binding'. Can you give a quick-
>> n-dirty example?
> Is an 'early bound' variable synonymous with a 'static' variable (in
> C)?

No. It means, in which moment the name gets its value assigned. Usually
Python does "late binding", that is, names are resolved at the time the
code is executed, not when it's compiled or defined.
Consider this example:

z = 1
def foo(a)
print a+z
foo(3) # prints 4
z = 20
foo(3) # prints 23

The second time it prints 23, not 4, because the value for z is searched
when the code is executed, so the relevant value for z is 20.
Note that if you later assign a non-numeric value to z, foo(3) will fail.

If you want to achieve the effect of "early binding", that is, you want to
"freeze" z to be always what it was at the time the function was defined,
you can do that using a default argument:

z = 1
def foo(a, z=z)
print a+z
z = None
foo(3) # prints 4

This way, foo(3) will always print 4, independently of the current value
of z. Moreover, you can `del z` and foo will continue to work.

This is what I think Chris Mellon was refering to. This specific default
argument semantics allows one to achieve the effect of "early binding" in
a language which is mostly "late binding". If someone changes this, he has
to come with another way of faking early binding semantics at least as
simple as this, else we're solving an [inexistant for me] problem but
creating another one.

--
Gabriel Genellina

Arnaud Delobelle

unread,
Jan 1, 2008, 3:26:23 PM1/1/08
to
On Jan 1, 6:48 pm, "Gabriel Genellina" <gagsl-...@yahoo.com.ar> wrote:
> En Tue, 01 Jan 2008 15:45:00 -0200, bukzor <workithar...@gmail.com>  
> escribi�:
[...]

Let me say again that I believe the current behaviour to be the
correct one. But I don't think this 'early binding' is critical for
this sort of example. There are lots of ways to solve the problem of
having persistent state across function calls, for example:

* using classes
* using modules
* or simply nested functions:

def getfoo(z):
def foo(a):
print a + z
return foo

>>> z = 1
>>> foo = getfoo(z)
>>> z = None
>>> foo(3)
4

And with nonlocal, we could even modify z inside foo and this change
would persist across calls. This will be a much cleaner solution than
the current def bar(x, y, _hidden=[startvalue]).

Also, note that it's easy to implement default arguments in pure
python-without-default-arguments using a decorator:

def default(**defaults):
defaults = defaults.items()
def decorator(f):
def decorated(*args, **kwargs):
for name, val in defaults:
kwargs.setdefault(name, val)
return f(*args, **kwargs)
return decorated
return decorator

Here is your example:

>>> z=1
>>> @default(z=z)
... def foo(a, z):
... print a + z
...
>>> z=None
>>> foo(3)
4

Another example, using mutables:

>>> @default(history=[])
... def bar(x, history):
... history.append(x)
... return list(history)
...
>>> map(bar, 'spam')
[['s'], ['s', 'p'], ['s', 'p', 'a'], ['s', 'p', 'a', 'm']]

--
Arnaud

Ali

unread,
Jan 2, 2008, 5:56:54 AM1/2/08
to
On Dec 30 2007, 12:27 am, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.au> wrote:

> In the absence of a better solution, I'm very comfortable with keeping
> the behaviour as is. Unfortunately, there's no good solution in Python to
> providing functions with local storage that persists across calls to the
> function:

...

(4) Instances handle this pretty well, just s/functions/methods/.

Reply all
Reply to author
Forward
0 new messages