Observations on the three pillars of Python execution
Eric Snow
for their instantiation, during which a code object is generated:
modules, classes, and functions. Each has its own role to play and
the differences between them mostly reflect that. Here are some
observations (based on default behavior):
Object creation relative to code object creation:
(M) before
(C) after
(F) after
Object creation relative to code object execution:
(M) before
(C) after
(F) before
Object available during code object execution:
(M) no
(C) no
(F) no
Code object destiny:
(M) executed once at definition time, then discarded
(C) executed once at definition time, then discarded
(F) not executed at definition time, stored on the function
object for execution with each __call__() invocation.
Execution locals is available during code object execution:
(M) as globals()
(C) as locals()
(F) as locals()
Object namespace is execution locals of code object:
(M) yes
(C) more or less
(F) completely distinct
Unique syntax:
(M) 'import' statement
(C) 'class' statement
(F) 'def' statement
Mechanism triggered by respective syntax:
(M) import machinery (import.c/importlib)
(C) internal, but partially exposed via metaclasses and the
__build_class__ builtin
(F) internal, nothing exposed
Override capability:
(M) complete capability through builtin __import__ and PEP
302 import hooks
(C) partial capability, before code object execution through
metaclass __prepare__() and after through __build_class__()
and metaclass __call__(), __new__(), and __init__()
(F) no capability
Post instantiation modification capability:
(M) yes
(C) yes
(F) mostly (some attributes are read-only)
Mechanism to instantiate outside of respective unique syntax:
(M) imp.new_module(), types.ModuleType(), type(<module>)()
(C) type()
(F) types.FunctionType(), type(f)()
Type extensibility:
(M) Not relative to 'import' syntax (by default)
(C) Complete
(F) No
Name available during definition execution:
(M) as __name__
(C) only through metaclass __prepare__()
(F) through inspect.currentframe().f_code.co_name
Name available on object as __name__:
(M) yes
(C) yes
(F) yes
Corrections, additions, and comment are welcome.
-eric
Thomas Jollans
> Object available during code object execution:
> (M) no
> (C) no
> (F) no
cf. recursion.
Steven D'Aprano
> In Python, three types of objects have special syntax and mechanics
> for their instantiation, during which a code object is generated:
> modules, classes, and functions.
I believe you are labouring under a misapprehension. Modules and classes
don't generate code objects.
The simplest possible module just has a name:
>>> import sys
>>> module = type(sys)
>>> module("name")
<module 'name' (built-in)>
Or if you prefer to do it the "normal" way by importing from source code,
the simplest source code of all is just an empty file.
A module is an object with a __dict__, a __name__, and a __docstring__ which
defaults to None. That's it. No code objects, unless you populate the
__dict__ with functions, but it is the *functions* that have the code
objects, not the module itself. The module is just a container.
Classes are similar. The simplest class:
class K:
pass
or if you prefer:
>>> type('K', (), {})
<class '__main__.K'>
Again, no code objects unless you populate the dict with functions (which
get turned into methods). Again, the class object is just a container. A
more complex container than modules, with a richer set of inherited
behaviour, but still just a container.
The signature for creating a function is very different:
help(type(lambda: None)) =>
class function(object)
| function(code, globals[, name[, argdefs[, closure]]])
|
| Create a function object from a code object and a dictionary.
| The optional name string overrides the name from the code object.
| The optional argdefs tuple specifies the default argument values.
| The optional closure tuple supplies the bindings for free variables.
The simplest function I can think of:
>>> co = compile('pass', '', 'single')
>>> f = type(lambda: None)(co, {})
Even simpler than
def f():
pass
Can you see why?
If you inspect a function object, you will see that it always has a code
object, even if the code object does nothing but return None:
>>> import dis
>>> dis.dis(f.func_code)
1 0 LOAD_CONST 0 (None)
3 RETURN_VALUE
There may be some other obscure built-in type that includes code objects,
but I can't imagine what it would be. I feel confident in saying that
functions, and functions alone, contain code. Even methods are just
wrappers around functions. Even built-in functions like len don't contain
code! (Or at least, their code isn't accessible from Python.) Which makes
sense, if you think about it: their code is part of the Python virtual
machine, not the object.
--
Steven
Eric Snow
<steve+comp....@pearwood.info> wrote:
> Eric Snow wrote:
>
>> In Python, three types of objects have special syntax and mechanics
>> for their instantiation, during which a code object is generated:
>> modules, classes, and functions.
>
> I believe you are labouring under a misapprehension. Modules and classes
> don't generate code objects.
Sorry for any confusion. I was talking specifically about the their
instantiation through their respective special syntax. During that
process a code object is generated for each. For classes and modules
the code object is executed and then discarded. I covered this
explicitly in one of the observations.
Agreed that [non-builtin] functions are the only objects that have a
code object attribute. However, they are not the only objects for
which a code object is generated and executed.
I've always found your frequent responses very insightful and
thoughtful, Steven, and I appreciate that. In this case I can only
imagine that my opening statement was unclear enough that you did not
continue to read the rest, where the context of my point was more
clear (I hope :). Regardless, thanks for taking a look.
-eric
>
>
> --
> Steven
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
Steven D'Aprano
> On Fri, Aug 5, 2011 at 8:36 AM, Steven D'Aprano
> <steve+comp....@pearwood.info> wrote:
>> Eric Snow wrote:
>>
>>> In Python, three types of objects have special syntax and mechanics
>>> for their instantiation, during which a code object is generated:
>>> modules, classes, and functions.
>>
>> I believe you are labouring under a misapprehension. Modules and classes
>> don't generate code objects.
>
> Sorry for any confusion. I was talking specifically about the their
> instantiation through their respective special syntax. During that
> process a code object is generated for each.
Do you believe that this process of generating a code object and throwing it
away is a part of the Python language specification, which any compiler
must do in order to call itself "Python", or a mere implementation detail?
Is this documented somewhere? If it is not documented, what makes you think
that it happens at all? You are writing as if it were self-evidently true,
but I don't believe it is self-evident at all. I think you need to
demonstrate the truth of two of those three pillars, not just take them for
granted.
> For classes and modules
> the code object is executed and then discarded. I covered this
> explicitly in one of the observations.
I think your definition of "explicitly" and mine differ here.
> Agreed that [non-builtin] functions are the only objects that have a
> code object attribute. However, they are not the only objects for
> which a code object is generated and executed.
Are you talking about the fact that importing a module, class statements and
function statements all involve executing a block of code?
How does that differ from executing any other fragment of code?
Modules are special, of course, because they can get compiled to byte-code,
but I trust you're not talking about .pyc files.
--
Steven
Mel
> There may be some other obscure built-in type that includes code objects,
> but I can't imagine what it would be. I feel confident in saying that
> functions, and functions alone, contain code. Even methods are just
> wrappers around functions. Even built-in functions like len don't contain
> code! (Or at least, their code isn't accessible from Python.) Which makes
> sense, if you think about it: their code is part of the Python virtual
> machine, not the object.
Interesting question. Iterators seem to have code objects:
Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> a = [1, 2, 3]
>>> b = (x for x in a)
>>> dir(b)
['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__',
'__hash__', '__init__', '__iter__', '__name__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__',
'__subclasshook__', 'close', 'gi_code', 'gi_frame', 'gi_running', 'next',
'send', 'throw']
>>> for name in dir(b):
... print name, type (getattr (b, name))
...
__class__ <type 'type'>
__delattr__ <type 'method-wrapper'>
__doc__ <type 'NoneType'>
__format__ <type 'builtin_function_or_method'>
__getattribute__ <type 'method-wrapper'>
__hash__ <type 'method-wrapper'>
__init__ <type 'method-wrapper'>
__iter__ <type 'method-wrapper'>
__name__ <type 'str'>
__new__ <type 'builtin_function_or_method'>
__reduce__ <type 'builtin_function_or_method'>
__reduce_ex__ <type 'builtin_function_or_method'>
__repr__ <type 'method-wrapper'>
__setattr__ <type 'method-wrapper'>
__sizeof__ <type 'builtin_function_or_method'>
__str__ <type 'method-wrapper'>
__subclasshook__ <type 'builtin_function_or_method'>
close <type 'builtin_function_or_method'>
gi_code <type 'code'>
gi_frame <type 'frame'>
gi_running <type 'int'>
next <type 'method-wrapper'>
send <type 'builtin_function_or_method'>
throw <type 'builtin_function_or_method'>
in the form of the gi_code attribute. No idea what it's for, although no
reason to believe it shouldn't be there. (Very interesting demo you gave of
primitive object creation. I' awed.)
Mel.
Terry Reedy
Recursion only happens through runtime name resolution, not through
direct access to the function or code object from within the code.
--
Terry Jan Reedy
Eric Snow
<steve+comp....@pearwood.info> wrote:
> Eric Snow wrote:
>
>> On Fri, Aug 5, 2011 at 8:36 AM, Steven D'Aprano
>> <steve+comp....@pearwood.info> wrote:
>>> Eric Snow wrote:
>>>
>>>> In Python, three types of objects have special syntax and mechanics
>>>> for their instantiation, during which a code object is generated:
>>>> modules, classes, and functions.
>>>
>>> I believe you are labouring under a misapprehension. Modules and classes
>>> don't generate code objects.
>>
>> Sorry for any confusion. I was talking specifically about the their
>> instantiation through their respective special syntax. During that
>> process a code object is generated for each.
>
>
> Do you believe that this process of generating a code object and throwing it
> away is a part of the Python language specification, which any compiler
> must do in order to call itself "Python", or a mere implementation detail?
That's a great point which I hadn't considered. Honestly, I only used
my experience with CPython in making these observations. After
reviewing the language reference I see that I missed out on a bunch of
nomenclature that would have made things more clear, and I got a few
points wrong, which you pointed out. :)
Regarding code objects and classes, your are right. The language
reference indicates the following:
"The class’s suite is then executed in a new execution frame...When
the class’s suite finishes execution, its execution frame is discarded
but its local namespace is saved." [1]
So the use of code objects for execution is an implementation detail.
Instead of "code object" I should have referred to the code executed
in the execution frame or just to the frame. Incidently, I had not
realized that the builtin __build_class__() is also an implementation
detail[3].
For modules, the language reference doesn't say anything about how
execution is accomplished, which only matters when execution is
involved in the creation of the module object. It does refer to
importlib as a reference implementation[4]. The order-of-operations
observations I made are based on that reference implementation.
>
> Is this documented somewhere? If it is not documented, what makes you think
> that it happens at all? You are writing as if it were self-evidently true,
> but I don't believe it is self-evident at all. I think you need to
> demonstrate the truth of two of those three pillars, not just take them for
> granted.
>
>
>> For classes and modules
>> the code object is executed and then discarded. I covered this
>> explicitly in one of the observations.
>
> I think your definition of "explicitly" and mine differ here.
>
>
>> Agreed that [non-builtin] functions are the only objects that have a
>> code object attribute. However, they are not the only objects for
>> which a code object is generated and executed.
>
> Are you talking about the fact that importing a module, class statements and
> function statements all involve executing a block of code?
>
> How does that differ from executing any other fragment of code?
The difference is that modules, classes, and functions (really the
function body) are code blocks tied to syntax that results in module,
type, and function objects. There are other code blocks but none of
them have a unique syntax, much less one that results in an object of
the corresponding type[5]. This is relevant for trying to find the
object that corresponds to an execution frame, which is what led me to
my original post and drove the direction of the observations I made.
Anyway, I appreciate the feedback! I'm going to have to revisit my
observations with the language definition in hand. You've been really
insightful, as usual.
-eric
[1] http://docs.python.org/dev/reference/compound_stmts.html#class-definitions
[2] http://docs.python.org/dev/reference/datamodel.html#metaclasses
[3] http://mail.python.org/pipermail/python-3000/2007-March/006338.html
[4] http://docs.python.org/dev/reference/simple_stmts.html#the-import-statement
[5] http://docs.python.org/dev/reference/executionmodel.html#naming
>
> Modules are special, of course, because they can get compiled to byte-code,
> but I trust you're not talking about .pyc files.
>
>
>
> --
> Steven
>
Steven D'Aprano
> Steven D'Aprano wrote:
>
>> There may be some other obscure built-in type that includes code objects,
>> but I can't imagine what it would be. I feel confident in saying that
>> functions, and functions alone, contain code. Even methods are just
>> wrappers around functions. Even built-in functions like len don't contain
>> code! (Or at least, their code isn't accessible from Python.) Which makes
>> sense, if you think about it: their code is part of the Python virtual
>> machine, not the object.
>
> Interesting question. Iterators seem to have code objects:
[...]
Generators. But nice catch, thank you!
Iterators are *any* object which obeys the iterator protocol, that is, have
a next() method and an __iter__() method which behave in the expected way.
Iterators are a duck-type. Generators, whether created from a generator
expression or a generator function, are an actual type.
>>> type(x for x in (1,2))
<class 'generator'>
--
Steven
Steven D'Aprano
> On Fri, Aug 5, 2011 at 11:29 AM, Steven D'Aprano
> <steve+comp....@pearwood.info> wrote:
[...]
>> Do you believe that this process of generating a code object and throwing
>> it away is a part of the Python language specification, which any
>> compiler must do in order to call itself "Python", or a mere
>> implementation detail?
>
> That's a great point which I hadn't considered. Honestly, I only used
> my experience with CPython in making these observations. After
> reviewing the language reference I see that I missed out on a bunch of
> nomenclature that would have made things more clear, and I got a few
> points wrong, which you pointed out. :)
>
> Regarding code objects and classes, your are right. The language
> reference indicates the following:
>
> "The class’s suite is then executed in a new execution frame...When
> the class’s suite finishes execution, its execution frame is discarded
> but its local namespace is saved." [1]
It turns out that in CPython 2.5 at least, I'm strictly wrong and you got it
right, at least for classes:
>>> code = compile("""class K: pass""", '', 'exec')
>>> dis.dis(code)
1 0 LOAD_CONST 0 ('K')
3 LOAD_CONST 3 (())
6 LOAD_CONST 1 (<code object K at 0xb7e8ad10,
file "", line 1>)
9 MAKE_FUNCTION 0
12 CALL_FUNCTION 0
15 BUILD_CLASS
16 STORE_NAME 0 (K)
19 LOAD_CONST 2 (None)
22 RETURN_VALUE
So a code object is compiled, turned into a function, executed, the results
turned into a class, and the code object and function thrown away.
Is this an implementation detail? I would say so. The semantics of Python
the language are different from the details of it's virtual machine. Surely
we would be allowed to call something Python if it executed the body of the
class statement *without* creating a code object first? The important part
is *execute the body of the class statement*, not building the code object.
The later is merely a means to an end.
> So the use of code objects for execution is an implementation detail.
> Instead of "code object" I should have referred to the code executed
> in the execution frame or just to the frame.
Unless you really intend to talk about implementation details, I think you
should keep the discussion as high-level as possible. I'd talk about
executing blocks of code, and not even mention execution frames unless you
need to understand the role of frames during execution.
--
Steven