generator expression works in shell, NameError in script

[ssc]

Hello,

I am trying to generate this list of tuples:
[(0, ''), (1, 'Dr'), (2, 'Miss'), (3, 'Mr'), (4, 'Mrs'), (5, 'Ms')]

My code works fine in the Python shell:

>>> titles = ['Dr', 'Miss', 'Mr', 'Mrs', 'Ms',]
>>> title_choices = [(0, '')] + list((titles.index(t)+1, t) for t in titles)
>>> title_choices
[(0, ''), (1, 'Dr'), (2, 'Miss'), (3, 'Mr'), (4, 'Mrs'), (5, 'Ms')]

The same code run in a script fails with
NameError: global name 'titles' is not defined

Does anybody know why ? How can I fix the error ?

Thank you very much :-)

Steve

[Emile van Sebille]

On 6/17/2009 3:19 PM ssc said...

> Hello,
>
> I am trying to generate this list of tuples:
> [(0, ''), (1, 'Dr'), (2, 'Miss'), (3, 'Mr'), (4, 'Mrs'), (5, 'Ms')]
>
> My code works fine in the Python shell:
>
>>>> titles = ['Dr', 'Miss', 'Mr', 'Mrs', 'Ms',]
>>>> title_choices = [(0, '')] + list((titles.index(t)+1, t) for t in titles)
>>>> title_choices
> [(0, ''), (1, 'Dr'), (2, 'Miss'), (3, 'Mr'), (4, 'Mrs'), (5, 'Ms')]
>
> The same code run in a script fails with
> NameError: global name 'titles' is not defined

You get this because titles doesn't exist in the builtin, local or
global namespaces.

Post the code that fails. It's hard to debug working code. :)

Emile

[Chris Rebert]

See what Emile said, but here's a nicer way to code it, IMHO:

titles = ['Dr', 'Miss', 'Mr', 'Mrs', 'Ms']
title_choices = zip(range(len(titles)+1), ['']+titles)

zip() to the rescue!

Cheers,
Chris
--
http://blog.rebertia.com

[Jason Tackaberry]

On Wed, 2009-06-17 at 15:38 -0700, Chris Rebert wrote:
> See what Emile said, but here's a nicer way to code it, IMHO:
>
> titles = ['Dr', 'Miss', 'Mr', 'Mrs', 'Ms']
> title_choices = zip(range(len(titles)+1), ['']+titles)
>
> zip() to the rescue!

How about:

enumerate([''] + titles)

[Jon Clements]

Why are you doing this? I'm assuming a code to title look up is
required (don't forget military, royal and honorable titles
etc... :) )

Why not just:
>>> titles = ['', 'Dr', 'Miss', 'Mr', 'Mrs', 'Ms',]
>>> lookup = dict(enumerate(titles))
>>> print lookup
{0: '', 1: 'Dr', 2: 'Miss', 3: 'Mr', 4: 'Mrs', 5: 'Ms'}

or if you don't want a dict, then just:
>>> print list(enumerate(titles))

[Emile van Sebille]

On 6/17/2009 3:41 PM Jason Tackaberry said...

> How about:
>
> enumerate([''] + titles)
>

or perhaps, depending on usage...

list(enumerate(titles))

Emile

[ssc]

Wow! Didn't expect that kind of instant support. Thank you very much,
I'll give both zip and enumerate a try.

The code I've shown is actually copied pretty straight from a Django
form class, but I didn't want to mention that as not to dilute the
conversation. Don't think it matters, anyway. This is the relevant
excerpt:

from django.forms import Form

class SignupForm(Form):

titles = ['Dr', 'Miss', 'Mr', 'Mrs', 'Ms',]
title_choices = [(0, '')] + list((titles.index(t)+1, t) for t in
titles)

Now that I look at it again, it seems odd to me to not have the code
e.g. in __init__(...), but just 'class-global'.
Still, that does not seem a reason for titles not to be not defined,
as I do define it just in the line above.

Does the generator expression have its own little namespace or so ?

[Chris Rebert]

No, which leads to the common "WTF?" reaction upon seeing stuff like:

>>> funcs = ((lambda: x) for x in range(7))
>>> list_of = list(funcs)
>>> list_of[0]()
6

[Emile van Sebille]

On 6/17/2009 3:54 PM ssc said...

> Wow! Didn't expect that kind of instant support. Thank you very much,
> I'll give both zip and enumerate a try.
>
> The code I've shown is actually copied pretty straight from a Django
> form class, but I didn't want to mention that as not to dilute the
> conversation. Don't think it matters, anyway. This is the relevant
> excerpt:
>
> from django.forms import Form
>
> class SignupForm(Form):
>
> titles = ['Dr', 'Miss', 'Mr', 'Mrs', 'Ms',]
> title_choices = [(0, '')] + list((titles.index(t)+1, t) for t in
> titles)
>
> Now that I look at it again, it seems odd to me to not have the code
> e.g. in __init__(...), but just 'class-global'.

And as class is an executable statement, I imagine titles exists in a
namespace that hasn't yet been completely defined.

> Still, that does not seem a reason for titles not to be not defined,
> as I do define it just in the line above.

Well, again, titles will exists in the SignupForm namespace once it
exists, which it doesn't yet when you get to title_choices and want to
access it. Define titles above the class and you should be OK.

>
> Does the generator expression have its own little namespace or so ?

Sometimes -- in some python versions generator expressions leak...

Emile

[Steven Samuel Cole]

Both zip and enumerate do the trick. Thanks for the pointers, that's 2
more built-ins under my belt :-)
Still don't really understand why my initial code didn't work, though...

Thanks everyone! :-)

[guthrie]

On Jun 17, 6:38 pm, Steven Samuel Cole <steven.samuel.c...@gmail.com>
wrote:

> Still don't really understand why my initial code didn't work, though...

Your code certainly looks reasonable, and looks to me like it "should"
work. The comment of partial namespace is interesting, but
unconvincing (to me) - but I am not a Python expert! It would
certainly seem that within that code block it is in the local
namespace, not removed from that scope to be in another.

Seems that it should either be a bug, or else is a design error in the
language!

Just as in the above noted "WTF" - non-intuitive language constructs
that surprise users are poor design.

[nn]

This is certainly an odd one. This code works fine under 2.6 but fails
in Python 3.1.

>>> class x:
... lst=[2]
... gen=[lst.index(e) for e in lst]
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in x
File "<stdin>", line 3, in <listcomp>
NameError: global name 'lst' is not defined
>>>

[ryles]

> Does the generator expression have its own little namespace or so ?

Yes, generators create a new scope:

http://docs.python.org/reference/expressions.html#grammar-token-generator_expression

[ryles]

I believe it works in 2.x because unlike generator expressions, list
comprehensions do not create a new scope. However, in 3.0 list
comprehensions are actually treated as list(<generator>).

http://docs.python.org/reference/expressions.html
http://www.python.org/dev/peps/pep-0289

[greg]

ssc wrote:

> class SignupForm(Form):
>
> titles = ['Dr', 'Miss', 'Mr', 'Mrs', 'Ms',]
> title_choices = [(0, '')] + list((titles.index(t)+1, t) for t in
> titles)
>

> Does the generator expression have its own little namespace or so ?

Yes. The generator expression is a function, with its
own local namespace. Since the class scope is not visible
from inside functions declared within it, the behaviour
you're seeing results.

--
Greg

[greg]

nn wrote:

> This is certainly an odd one. This code works fine under 2.6 but fails
> in Python 3.1.
>
>>>>class x:
>
> ... lst=[2]
> ... gen=[lst.index(e) for e in lst]

In 3.x it was decided that the loop variables in a list
comprehension should be local, and not leak into the
surrounding scope. This was implemented by making the
list comprehension into a nested function.

Unfortunately this leads to the same unintuitive
behaviour as a genexp when used in a class scope.

--
Greg

[Bruno Desthuilliers]

Emile van Sebille a �crit :

> On 6/17/2009 3:54 PM ssc said...
>> Wow! Didn't expect that kind of instant support. Thank you very much,
>> I'll give both zip and enumerate a try.
>>
>> The code I've shown is actually copied pretty straight from a Django
>> form class, but I didn't want to mention that as not to dilute the
>> conversation. Don't think it matters, anyway. This is the relevant
>> excerpt:
>>
>> from django.forms import Form
>>
>> class SignupForm(Form):
>>
>> titles = ['Dr', 'Miss', 'Mr', 'Mrs', 'Ms',]
>> title_choices = [(0, '')] + list((titles.index(t)+1, t) for t in
>> titles)
>>
>> Now that I look at it again, it seems odd to me to not have the code
>> e.g. in __init__(...), but just 'class-global'.
>
> And as class is an executable statement, I imagine titles exists in a
> namespace that hasn't yet been completely defined.

>> Still, that does not seem a reason for titles not to be not defined,
>> as I do define it just in the line above.
>
> Well, again, titles will exists in the SignupForm namespace once it
> exists, which it doesn't yet when you get to title_choices and want to
> access it.

The namespace itself is as defined at any point of the class statement's
body as the local namespace of a function at any given point of the
function's body. Else decorators (or their alternate pre '@'
syntactic-sugar version) wouldn't work.

FWIW, the following code works JustFine(tm):

bruno@bruno:~$ python
Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
[GCC 4.3.2] on linux2
>>> class Foo(object):
... bar = ['a', 'b', 'c']
... baaz = list(enumerate(bar))

as well as this one:

>>> class Foo(object):
... bar = ['a', 'b', 'c']
... baaz = [(bar.index(t)+1, t) for t in bar]

and this one:

>>> class Foo(object):
... bar = ['a', 'b', 'c']
... baaz = list((b, b) for b in bar)

but it indeed looks like using bar.index *in a generator expression*
fails (at least in 2.5.2) :

>>> class Foo(object):
... bar = ['a', 'b', 'c']
... baaz = list((bar.index(b), b) for b in bar)

...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>

File "<stdin>", line 3, in Foo
File "<stdin>", line 3, in <genexpr>
NameError: global name 'bar' is not defined

Looks like a bug to me :-/

[Mark Dickinson]

On Jun 19, 1:45 pm, Bruno Desthuilliers <bruno.
42.desthuilli...@websiteburo.invalid> wrote:
> [...]

> but it indeed looks like using bar.index *in a generator expression*
> fails (at least in 2.5.2) :
>
>   >>> class Foo(object):
> ...     bar = ['a', 'b', 'c']
> ...     baaz = list((bar.index(b), b) for b in bar)
> ...
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "<stdin>", line 3, in Foo
>    File "<stdin>", line 3, in <genexpr>
> NameError: global name 'bar' is not defined
>
> Looks like a bug to me :-/

I think it's working as intended: name resolution for
nested scopes skips intermediate class scopes. The
'Discussion' section of PEP 227 (Statically Nested Scopes)
is illuminating; here's a snippet:

"""Names in class scope are not accessible. Names are resolved in
the innermost enclosing function scope. If a class definition
occurs in a chain of nested scopes, the resolution process skips
class definitions. This rule prevents odd interactions between
class attributes and local variable access."""

Mark

[greg]

ryles wrote:
> However, in 3.0 list
> comprehensions are actually treated as list(<generator>).

That's not quite true -- it would be rather inefficient
if it was. The code generated for the LC is much the same
as it was in 2.x. But the whole LC is put into a nested
function so that the loop variables are local to it.

--
Greg

[guthrie]

-- I don't get this - the only local loop variable is "e", not lst.
And lst is used twice in the generator expression, which is being
complained about?

It would generally be the case that a nested function would have
access to its enclosing scope.

In any case, even if/when the current behavior is explained or
rationalized, it certainly appears un-intuitive, and that is another
level of "error"! :-)

[Miles Kaufmann]

On Jun 19, 2009, at 8:45 AM, Bruno Desthuilliers wrote:
> >>> class Foo(object):
> ... bar = ['a', 'b', 'c']
> ... baaz = list((b, b) for b in bar)
>
> but it indeed looks like using bar.index *in a generator expression*
> fails (at least in 2.5.2) :
>
> >>> class Foo(object):
> ... bar = ['a', 'b', 'c']
> ... baaz = list((bar.index(b), b) for b in bar)
> ...
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "<stdin>", line 3, in Foo
> File "<stdin>", line 3, in <genexpr>
> NameError: global name 'bar' is not defined

The reason that the first one works but the second fails is clearer if
you translate each generator expression to the approximately
equivalent generator function:

class Foo(object):

bar = ['a', 'b', 'c']

def _gen(_0):
for b in _0:
yield (b, b)
baaz = list(_gen(iter(bar))

# PEP 227: "the name bindings that occur in the class block
# are not visible to enclosed functions"
class Foo(object):

bar = ['a', 'b', 'c']

def _gen(_0):
for b in _0:
yield (bar.index(b), b)
baaz = list(_gen(iter(bar))

-Miles

[greg]

guthrie wrote:

> -- I don't get this - the only local loop variable is "e", not lst.

Yes, but the whole list comprehension gets put into
a nested function, including the part that evaluates
lst. (It's not strictly *necessary* to do that, but
that's the way it happens to be implemented at the
moment.)

> It would generally be the case that a nested function would have
> access to its enclosing scope.

Usually, but class namespaces are a special case --
they're not considered to be enclosing scopes, even
though textually they're written that way.

> it certainly appears un-intuitive

It is, but it's hard to see what could be done to
improve the situation without introducing worse
problems.

--
Greg

[ssc]

On Jun 18, 10:49 am, Jon Clements <jon...@googlemail.com> wrote:
> Why are you doing this? I'm assuming a code to title look up is
> required (don't forget military, royal and honorable titles
> etc... :) )

I'm in New Zealand. Hardly any need for military titles, rarely any
for royal and damn sure none for honorable :-D (scnr)

[Aahz]

In article <13e776b0-6ca8-479e...@a5g2000pre.googlegroups.com>,
ssc <steven.sa...@gmail.com> wrote:

+1 QOTW
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/

"as long as we like the same operating system, things are cool." --piranha

[alito]

Arghh. I see thousands of future wtf!? posts to c.l.p. triggered by
this new feature. Might as well save some time and add it to the FAQ
already.