metaclass & __slots__
Holden Caulfield
I am trying to get a hang of the metaclass and some new features in
v2.2. But, I am running into a problem or it is a limitation, I guess.
I just need clarification. Here is the code, it is an extension of
Guido's autoprop example.
Basically, I am trying to set "property" attributes AND also limit the
attributes set by using "__slots__".
It looks like because the Metalcass statement is executed in the end
of the class statement, hence the class methods seems to have a
"early" binding to the __slots__ variable. Is this behaviour normal?
Thanks
M
-----
class MX(type):
def __init__(cls,name,bases,dict):
super(MX,cls).__init__(name,bases,dict)
props = {}
slots = getattr(cls,'__slots__', [])
print slots
for v in dict.keys():
vs = v.startswith
if vs("_get_") or vs("_set_"):
props[v[5:]] = 1
for v in props.keys():
fget = getattr(cls,"_get_%s" % v, None)
fset = getattr(cls,"_set_%s" % v, None)
setattr(cls,v,property(fget,fset))
slots.append("_%s__%s" % (name,v))
setattr(cls,'__slots__',slots)
class X(object):
__metaclass__ = MX
__slots__ = ['z']
def __init__(self):
self.x = 1
self.z = 2
def _get_x(self):
return self.__x
def _set_x(self,v):
self.__x = v
def Test():
y = X()
y.x = 4
if __name__ == "__main__": Test()
--------------
The above code fails with:
Traceback (most recent call last):
File "__test.py", line 50, in ?
if __name__ == "__main__": Test()
File "__test.py", line 43, in Test
y = X()
File "__test.py", line 34, in __init__
self.x = 1
File "__test.py", line 39, in _set_x
self.__x = v
AttributeError: 'X' object has no attribute '_X__x'
Jonathan Hogg
c2595393.02070...@posting.google.com, "Holden Caulfield"
<phoe...@att.net> wrote:
> It looks like because the Metalcass statement is executed in the end
> of the class statement, hence the class methods seems to have a
> "early" binding to the __slots__ variable. Is this behaviour normal?
The __slots__ stuff works at allocation rather than initialisation (along
with lots of other magic). So you need to override __new__ instead of
__init__ in your metaclass.
I made the following changes to your code:
-----
class MX(type):
def __new__(cls,name,bases,dict):
props = {}
slots = dict.get('__slots__', [])
print slots
for v in dict.keys():
vs = v.startswith
if vs("_get_") or vs("_set_"):
props[v[5:]] = 1
for v in props.keys():
fget = dict.get("_get_%s" % v, None)
fset = dict.get("_set_%s" % v, None)
dict[v] = property(fget,fset)
slots.append("_%s__%s" % (name,v))
dict['__slots__'] = slots
return super(MX, cls).__new__(cls,name,bases,dict)
class X(object):
__metaclass__ = MX
__slots__ = ['z']
def __init__(self):
self.x = 1
self.z = 2
def _get_x(self):
return self.__x
def _set_x(self,v):
self.__x = v
def Test():
y = X()
y.x = 4
print y.x
if __name__ == "__main__": Test()
-----
which gives me the desired result. Note that you do all your work on the
dict in __new__ as the class hasn't been created yet (the 'cls' in the
__new__ method is 'MX'), then you chain the type.__new__ to do the actual
class creation.
Jonathan
Mark McEahern
[snip]
> But, let's get into details:
[snip]
Alex, thank you for this very patient explanation of the fine details of
Python class creation. This helps me tremendously.
// mark
-
Alex Martelli
...
"""
I am trying to get a hang of the metaclass and some new features in
"""
"""
v2.2. But, I am running into a problem or it is a limitation, I guess.
I just need clarification. Here is the code, it is an extension of
"""
Yes, I fully agree you just need a little bit of clarification -- you
seem to be very, very close to grasping the metaclass issues
fully, and only need to look at them from a slightly different
angle for them all to click into place and make sense to you, I
suspect. No limitation involved, and, to give the punch line away,
all the problem is that your class MX is calling its superclass's
__init__ too early -- it just needs to delay that a little bit. But,
let's get into details:
"""
Basically, I am trying to set "property" attributes AND also limit the
attributes set by using "__slots__".
It looks like because the Metalcass statement is executed in the end
of the class statement, hence the class methods seems to have a
"early" binding to the __slots__ variable. Is this behaviour normal?
"""
There is no such thing as a 'Metaclass statement' . Rather, when your class
statement's body binds an attribute named __metaclass__, as you do here,
Python knows to use the value of that attribute as your new class
object's type, i.e., metaclass. (Otherwise, Python uses other ways
to decide what metaclass to use, but they don't matter here).
There's no issue of early or late. Statements in a class body are
executed one after the other as usual, when the class statement
itself executes, *before* the metaclass comes into play: i.e., in your
example:
class X(object):
__metaclass__ = MX
__slots__ = ['z']
def __init__(self):
self.x = 1
self.z = 2
def _get_x(self):
return self.__x
def _set_x(self,v):
self.__x = v
two bindings, to names __metaclass__ and __slots__, then three def
statements (which bind function object to names __init__, _get_x,
_set_x). The results of these bindings go into a dict, so it really makes
no difference in the end which binding was executed earlier and which
one was executed later, unless you bind more than one value to the
same name (in which case the later binding "overrides" the earlier
one) or similar second-order effects, none of which applies here.
All the statements in class body boil down to constructing a dict and
then the metaclass gets to work on that dict.
So, what your MX's __init__ gets as its dict argument:
class MX(type):
def __init__(cls,name,bases,dict):
is a dictionary with five items, as bound in the class's body. Add a
"print dict" statement here to double check -- no rocket science
involved, really, although it may SEEM there's some!-)
So, now it's entirely up to your code in the body of MX.__init__ what
happens. So let's examine what you actually do there to initialize
object cls, which is the class X you're creating:
super(MX,cls).__init__(name,bases,dict)
so you first delegate everything to type, which is MX's superclass,
with the dict exactly as build by the class statement. type.__init__
is responsible among other things for building the descriptors as
specified by __slots__, so of course what rules is the value of
dict[__slots__] at this point -- when you call type.__init__ -- not
any changes that you might further make to that entry, right?
props = {}
slots = getattr(cls,'__slots__', [])
print slots
for v in dict.keys():
vs = v.startswith
if vs("_get_") or vs("_set_"):
props[v[5:]] = 1
for v in props.keys():
fget = getattr(cls,"_get_%s" % v, None)
fset = getattr(cls,"_set_%s" % v, None)
setattr(cls,v,property(fget,fset))
slots.append("_%s__%s" % (name,v))
setattr(cls,'__slots__',slots)
So all of these statements may affect X.__slots__ in some ways,
but since type.__init__ has already worked with its original value,
that in turn has no effect on the descriptors of X.
You do have some effect on the descriptors of X by the setattr
in the for loop in which you instantiate the property built-in, but
that's another issue.
So, if you've followed so far, I hope the solution is reasonably
obvious: what you want to do is alter your dict argument (don't
bother getting or setting attributes on cls -- work with the dict
arguments directly, possibly taking the bases into account if
you need to) and THEN call type.__init__ with the _modified_
dictionary object.
More generally, in your __init__ (whether of a metaclass or
any other class): call the superclass's __init__ AFTER you have
performed modifications to the arguments, if you need to change
the arguments and the superclass's __init__ is going to take
some action based on them. Metaclasses are no different (in
this or most other respects) from any other class.
Alex
Alex Martelli
> [Alex Martelli]
> [snip]
>
> > But, let's get into details:
>
>
> Alex, thank you for this very patient explanation of the fine details of
> Python class creation. This helps me tremendously.
You're most welcome! And, thanks right back for the feedback -- finding
out which explanatory "takes" on hard subjects work best in written form
is very helpful to me in determine how to write articles and book chapters
about those subjects.
I think the part that most needs to be fully explained and explored today
is WHY one would want to write customized metaclasses, rather than
HOW to write them (although the HOW is of course also necessary). There
are, I think, few and somewhat marginal things that can ONLY be done with
a customized metaclass (e.g., determining what comes out when the class
object itself, rather than an instance, is printed). Most metaclass tasks
could alternatively be handled with suitable __init__, __getattr__ &c special
methods. But metaclasses can be simpler AND offer better performance.
Moreover, what complexity remains inevitable can better be moved into a
"framework", in such a way that a class-author does not need to worry all
the time about niggling details, but can just rely on framework-supplied
metaclasses to build the resulting class objects "just right", with whatever
tweaked semantics are necessary.
Still, until we get a good, solid tutorial set of examples comparing and
contrasting the metaclass-way and the classic-way to perform some such
set of tasks, I think this will remain rather hard to grasp in practice. And
I'm still thinking about WHAT "good, solid tutorial set of examples" might
be most appropriate for this...
Alex
Jonathan Hogg
mailman.1025801286...@python.org, "Alex Martelli"
<al...@aleax.it> wrote:
> More generally, in your __init__ (whether of a metaclass or
> any other class): call the superclass's __init__ AFTER you have
> performed modifications to the arguments, if you need to change
> the arguments and the superclass's __init__ is going to take
> some action based on them. Metaclasses are no different (in
> this or most other respects) from any other class.
Unfortunately, you can't set or modify __slots__ in the __init__ method of a
metaclass. The damage has already been done by the time you get to the
__init__ method:
>>> class foo( type ):
... def __init__( cls, name, bases, dict ):
... dict['__slots__'] = ['x','y']
... super(foo,cls).__init__( name, bases, dict )
...
>>> class bar:
... __metaclass__ = foo
...
>>> b = bar()
>>> b.x = 10
>>> b.foo = 'heck'
>>> b.__dict__
{'x': 10, 'foo': 'heck'}
>>>
You need to make the change in the __new__ method before the class is
allocated:
>>> class foo( type ):
... def __new__( cls, name, bases, dict ):
... dict['__slots__'] = ['x','y']
... return type.__new__( cls, name, bases, dict )
...
>>> class bar:
... __metaclass__ = foo
...
>>> b = bar()
>>> b.x = 10
>>> b.foo = 'heck'
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'bar' object has no attribute 'foo'
>>> b.__dict__
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'bar' object has no attribute '__dict__'
>>>
See my other post for more on this.
Jonathan
Holden Caulfield
Thanks for good explanations and clarifications. This was just an
exercise to hurt my brain, I guess :)
But somehow I feel the addition of the "new" style classes and the
"automatic" call of "__new__" and "__init__" is really confusing. The
"__init__" seems redundant, since we actually have a true
"constructor". I guess, what I am trying to say is, with the scarce
documentation (and if you are lazy to browse through the source) it
seems like you need to know the distinction on what to override
(__new__ or __init__) for a class derived from a builtin.
Thanks again
M
Thomas Heller
http://www.python.org/2.2.1/descrintro.html#__new__
Thomas
Teja Sastry
I want to delete a file in the current directory from inside a python
program. How do i go about doing it.
Thanks,
Teja Sastry
Alex Martelli
...
> Unfortunately, you can't set or modify __slots__ in the __init__ method of
> a metaclass. The damage has already been done by the time you get to the
> __init__ method:
> You need to make the change in the __new__ method before the class is
Perfectly right! I should have checked whether __slots__ was taken into
consideration in type's __new__ or __init__, but I was lazy or hurried and
just focused on the obvious issue that the original poster's code called
type.__init__ at the start of the custom metaclass's own __init__, so it
wouldn't have worked "even if" it was type.__init__'s job, rather than
type.__new__'s, to implement __slots__. Thanks for the correction!
Alex
Bryan Olson
>
>
> I want to delete a file in the current directory from inside a python
> program. How do i go about doing it.
See: os.remove(path)
--Bryan
Steve Tregidgo
> class MX(type):
> def __init__(cls,name,bases,dict):
> super(MX,cls).__init__(name,bases,dict)
> props = {}
> slots = getattr(cls,'__slots__', [])
> print slots
> for v in dict.keys():
> vs = v.startswith
> if vs("_get_") or vs("_set_"):
> props[v[5:]] = 1
> for v in props.keys():
> fget = getattr(cls,"_get_%s" % v, None)
> fset = getattr(cls,"_set_%s" % v, None)
> setattr(cls,v,property(fget,fset))
> slots.append("_%s__%s" % (name,v))
> setattr(cls,'__slots__',slots)
Holden,
Others have commented on all the groovy metaclass stuff (which I am at
last coming to terms with); I just wanted to point out something I
presume to be a typo in the above code (maybe not in your original,
but in the version I see posted, and which therefore others may be
using).
The indentation of the 'slots.append' line is such that the statement
falls outside of the preceding 'for' loop. In the example that you
gave, this would not have shown up whilst running the code.
It so happens that in the case where get/set methods exist for exactly
one property, the above code will work as expected (since the 'for'
loop is iterated over just once, and therefore upon finishing the
local name 'v' will be set to the only property). However, if two or
more properties are defined then only one of them will have the munged
name added to the slots list (whichever ended up being last in
'prop.items()'). If no properties are defined, I would expect the
code to fail with a NameError (probably an UnboundLocalError
actually).
Apologies if the indentation error is an artifact of copying from
editor to mail package; otherwise, consider this a pre-emptive attempt
to answer your next question <wink>.
Cheers,
Steve
--
Steve Tregidgo
Aahz
Holden Caulfield <phoe...@att.net> wrote:
>
> But somehow I feel the addition of the "new" style classes and the
>"automatic" call of "__new__" and "__init__" is really confusing. The
>"__init__" seems redundant, since we actually have a true
>"constructor". I guess, what I am trying to say is, with the scarce
>documentation (and if you are lazy to browse through the source) it
>seems like you need to know the distinction on what to override
>(__new__ or __init__) for a class derived from a builtin.
Thing is, historically Python has not had a constructor, just an
initializer. Now that we have new-style classes with constructors, it's
still necessary to bring forward idioms that worked with older versions
of Python. It's possible that Python 3.0 might collapse this into a
single method, but you're the first person I've seen complaining.
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/
Project Vote Smart: http://www.vote-smart.org/
holger krekel
> phoe...@att.net (Holden Caulfield) wrote in message news:<c2595393.02070...@posting.google.com>...
> > class MX(type):
> > def __init__(cls,name,bases,dict):
> > super(MX,cls).__init__(name,bases,dict)
> > props = {}
> > slots = getattr(cls,'__slots__', [])
> > print slots
> > for v in dict.keys():
> > vs = v.startswith
> > if vs("_get_") or vs("_set_"):
> > props[v[5:]] = 1
> > for v in props.keys():
> > fget = getattr(cls,"_get_%s" % v, None)
> > fset = getattr(cls,"_set_%s" % v, None)
> > setattr(cls,v,property(fget,fset))
> > slots.append("_%s__%s" % (name,v))
> > setattr(cls,'__slots__',slots)
>
> falls outside of the preceding 'for' loop. In the example that you
> gave, this would not have shown up whilst running the code.
Wasn't python supposed to be much safer against off-by-one errors?
:-) holger