Asymmetry in globals __getitem__/__setitem__

41 views
Skip to first unread message

Robert Lehmann

unread,
Jun 12, 2014, 2:18:29 PM6/12/14
to pytho...@python.org
Hi all,

I have noticed there is a slight asymmetry in the way the interpreter (v3.3.5, reproduced also in v3.5.x) loads and stores globals.  While loading globals from a custom mapping triggers __getitem__ just fine, writing seems to silently ignore __setitem__.

class Namespace(dict):
    def __getitem__(self, key):
        print("getitem", key)
    def __setitem__(self, key, value):
        print("setitem", key, value)

def fun():
    global x, y
    x  # should call globals.__getitem__
    y = 1  # should call globals.__setitem__

exec(fun.__code__, Namespace())
# => getitem x

I would have expected "setitem y 1" to show up as well, but to no avail.  Am I doing something wrong?  Is this on purpose?

Cheers,
Robert

PS.  I found a 3.3.x commit (e3ab8aa) which fixed the LOAD_GLOBAL opcode to support other types than dict, but STORE_GLOBAL seems to use bare PyDict_SetItem instead of dispatching to PyObject_SetItem.

Ian Kelly

unread,
Jun 12, 2014, 2:48:48 PM6/12/14
to Python
On Thu, Jun 12, 2014 at 12:18 PM, Robert Lehmann <ma...@robertlehmann.de> wrote:
> Hi all,
>
> I have noticed there is a slight asymmetry in the way the interpreter
> (v3.3.5, reproduced also in v3.5.x) loads and stores globals. While loading
> globals from a custom mapping triggers __getitem__ just fine, writing seems
> to silently ignore __setitem__.
>
> class Namespace(dict):
> def __getitem__(self, key):
> print("getitem", key)
> def __setitem__(self, key, value):
> print("setitem", key, value)
>
> def fun():
> global x, y
> x # should call globals.__getitem__
> y = 1 # should call globals.__setitem__
>
> exec(fun.__code__, Namespace())
> # => getitem x
>
> I would have expected "setitem y 1" to show up as well, but to no avail. Am
> I doing something wrong? Is this on purpose?

Seems like a bug to me. I note that the STORE_NAME opcode does call
__setitem__:

>>> code = compile('x = 1', '', 'exec')
>>> dis.dis(code)
1 0 LOAD_CONST 0 (1)
3 STORE_NAME 0 (x)
6 LOAD_CONST 1 (None)
9 RETURN_VALUE
>>> exec(code, Namespace())
setitem x 1

But STORE_GLOBAL does not:

>>> code = compile('global x; x = 1', '', 'exec')
>>> dis.dis(code)
1 0 LOAD_CONST 0 (1)
3 STORE_GLOBAL 0 (x)
6 LOAD_CONST 1 (None)
9 RETURN_VALUE
>>> exec(code, Namespace())

Chris Angelico

unread,
Jun 12, 2014, 3:47:53 PM6/12/14
to pytho...@python.org
On Fri, Jun 13, 2014 at 4:18 AM, Robert Lehmann <ma...@robertlehmann.de> wrote:
> PS. I found a 3.3.x commit (e3ab8aa) which fixed the LOAD_GLOBAL opcode to
> support other types than dict, but STORE_GLOBAL seems to use bare
> PyDict_SetItem instead of dispatching to PyObject_SetItem.
>

This looks like something for a tracker issue. I agree with Ian, seems
like a bug.

ChrisA

Gregory Ewing

unread,
Jun 12, 2014, 8:38:24 PM6/12/14
to
Robert Lehmann wrote:
> I have noticed there is a slight asymmetry in the way the interpreter
> (v3.3.5, reproduced also in v3.5.x) loads and stores globals. While
> loading globals from a custom mapping triggers __getitem__ just fine,
> writing seems to silently ignore __setitem__.

I didn't think that using a custom mapping object for
globals was officially supported. Has that changed?

--
Greg

Marko Rauhamaa

unread,
Jun 13, 2014, 2:07:45 AM6/13/14
to
Gregory Ewing <greg....@canterbury.ac.nz>:

> I didn't think that using a custom mapping object for globals was
> officially supported. Has that changed?

The documentation is a bit vague about it:

If only globals is provided, it must be a dictionary, which will be
used for both the global and the local variables. If globals and
locals are given, they are used for the global and local variables,
respectively. If provided, locals can be any mapping object.

Now, if you define:

class Namespace(dict):
...

A Namespace *is* a dict[ionary] in the classic OOP sense.


Marko

rob...@robertlehmann.de

unread,
Jun 13, 2014, 6:13:03 AM6/13/14
to
On Friday, June 13, 2014 8:07:45 AM UTC+2, Marko Rauhamaa wrote:
>
> The documentation is a bit vague about it:
>
> If only globals is provided, it must be a dictionary, which will be
> used for both the global and the local variables. If globals and
> locals are given, they are used for the global and local variables,
> respectively. If provided, locals can be any mapping object.


Interesting. This paragraph explicitly states "locals can be any mapping object," but that seems to be false:


class Namespace(dict):
def __getitem__(self, key):
print("getitem", key)
def __setitem__(self, key, value):
print("setitem", key, value)

def fun():
x # should call locals.__getitem__
y = 1 # should call locals.__setitem__

exec(fun.__code__, {}, Namespace())


Neither __getitem__ nor __setitem__ seem to be called on the local variables.

Peter Otten

unread,
Jun 13, 2014, 6:53:54 AM6/13/14
to pytho...@python.org
rob...@robertlehmann.de wrote:

> On Friday, June 13, 2014 8:07:45 AM UTC+2, Marko Rauhamaa wrote:
>>
>> The documentation is a bit vague about it:
>>
>> If only globals is provided, it must be a dictionary, which will be
>> used for both the global and the local variables. If globals and
>> locals are given, they are used for the global and local variables,
>> respectively. If provided, locals can be any mapping object.
>
>
> Interesting. This paragraph explicitly states "locals can be any mapping
> object," but that seems to be false:
>
>
> class Namespace(dict):
> def __getitem__(self, key):
> print("getitem", key)
> def __setitem__(self, key, value):
> print("setitem", key, value)
>
> def fun():
> x # should call locals.__getitem__

No, x is a global here.

> y = 1 # should call locals.__setitem__
>
> exec(fun.__code__, {}, Namespace())
>
>
> Neither __getitem__ nor __setitem__ seem to be called on the local
> variables.

Accessing fun.__code__ is clever, but unfortunately the compiler produces
different bytecodes for loading/storing variables inside a function.
Compare:

>>> import dis
>>> def fun(x=2):
... x
... y = 1
...
>>> dis.dis(fun.__code__)
2 0 LOAD_FAST 0 (x)
3 POP_TOP

3 4 LOAD_CONST 1 (1)
7 STORE_FAST 1 (y)
10 LOAD_CONST 0 (None)
13 RETURN_VALUE
>>> dis.dis(compile("x\ny=2", "<nofile>", "exec"))
1 0 LOAD_NAME 0 (x)
3 POP_TOP

2 4 LOAD_CONST 0 (2)
7 STORE_NAME 1 (y)
10 LOAD_CONST 1 (None)
13 RETURN_VALUE

Only the latter works as advertised:

>>> exec("x\ny=1", {}, Namespace())
getitem x
setitem y 1



Paul Sokolovsky

unread,
Jun 13, 2014, 7:28:36 AM6/13/14
to Peter Otten, pytho...@python.org
Hello,

On Fri, 13 Jun 2014 12:53:54 +0200
Peter Otten <__pet...@web.de> wrote:

[]

> > exec(fun.__code__, {}, Namespace())
> >
> >
> > Neither __getitem__ nor __setitem__ seem to be called on the local
> > variables.
>
> Accessing fun.__code__ is clever, but unfortunately the compiler
> produces different bytecodes for loading/storing variables inside a
> function. Compare:

Compiler produces different bytecodes, and allocates local variables on
stack (just like C) very fortunately, steps like that allowed Python to
drop moniker of "ridiculously slow language". And people should decide
what they really want - fast language which can stand against the
competition, or language with dynamicity and reflection capabilities
beyond any practical need. I make first choice any time. And then it
makes sense to just accept that any function can be JIT (or AOT)
compiled, and there's nothing to fish inside of it (but for the raw
machine code of unknown architecture).


--
Best regards,
Paul mailto:pmi...@gmail.com

Marko Rauhamaa

unread,
Jun 13, 2014, 8:32:13 AM6/13/14
to
Paul Sokolovsky <pmi...@gmail.com>:

> And people should decide what they really want - fast language which
> can stand against the competition, or language with dynamicity and
> reflection capabilities beyond any practical need. I make first choice
> any time.

I'm in the latter camp, absolutely, except that I have a lot of
practical needs for much of that dynamism.

Admittedly, the topic of this thread is a bit funky. I'm wondering what
the application is: a profiler? a debugger? malware? self-awareness?

> And then it makes sense to just accept that any function can be JIT
> (or AOT) compiled, and there's nothing to fish inside of it (but for
> the raw machine code of unknown architecture).

I've been talking about the need for effective JIT so we can get rid of
Java et co. I wouldn't dream of taking away any of Python's dynamism,
though. In particular, type annotations etc are a big no in my book.


Marko
Reply all
Reply to author
Forward
0 new messages