[rfc] An object that creates (nested) attributes automatically on assignment
Edd
I'd like to use Python itself as the configuration language for my
Python application. I'd like the user to be able to write something
like this in their config file(s):
cfg.laser.on = True
cfg.laser.colour = 'blue'
cfg.discombobulated.vegetables = ['carrots', 'broccoli']
# ...
To this end, I've created a class that appears to allow instance
variables to be created on the fly. In other words, I can to the
following to read a config file:
cfg = Config()
execfile(filename, {'cfg', cfg}, {})
However, I think my implementation of the Config class is a little
crappy. I'd really appreciate the critical eye of a pro. Here's the
sauce:
class Config(object):
def __init__(self, sealed=False):
def seal():
for v in self._attribs.values():
if isinstance(v, self.__class__): v.seal()
del self.__dict__['seal']
d = {'_attribs': {}, '_a2p': None}
if not sealed: d['seal'] = seal
self.__dict__.update(d)
def __getattr__(self, key):
if not key in self._attribs:
d = Config(sealed='seal' not in self.__dict__)
def add2parent():
self._attribs[key] = d
if self._a2p:
self._a2p()
self._a2p = None
# if anything is assigned to an attribute of d,
# make sure that d is recorded as an attribute of this
Config
d._a2p = add2parent
return d
else:
return self._attribs[key]
def __setattr__(self, key, value):
if key in self.__dict__:
self.__dict__[key] = value
else:
if not 'seal' in self.__dict__:
clsname = self.__class__.__name__
raise AttributeError("'%s' object attribute '%s'
is read-only (object is sealed)" % (clsname, key))
self.__dict__['_attribs'][key] = value
if self._a2p:
self._a2p()
self._a2p = None
def __delattr__(self, key):
if key in self.__dict__:
clsname = self.__class__.__name__
raise AttributeError("can't delete '%s' object
attribute '%s' as it is used for book-keeping!" % (clsname, key))
else:
if key in self._attribs:
del self._attribs[key]
def __bool__(self):
return bool(self._attribs)
def __nonzero__(self):
return bool(self._attribs)
if __name__ == '__main__':
cfg = Config()
cfg.a = 1
cfg.b.c = 2
cfg.d.e.f.g.h = [1, 2, 3]
print cfg.a
print cfg.b.c
print cfg.d.e.f.g.h
del cfg.b.c
print cfg.b.c
try:
del cfg.d.e._attribs
except AttributeError, ex:
print ex
cfg.seal()
try:
cfg.k.l.z = []
except AttributeError, ex:
print ex
Once the config is loaded, it will be passed down to other user-
written scripts and it's important that these scripts don't
accidentally change the config. So the idea is that I'll call cfg.seal
() to prevent any further changes before passing it on to these other
scripts. Beyond the general fiddliness of the code, I think the way
seal() currently works is particularly pants.
I considered using a simpler approach:
def mkdd(): return defaultdict(mkdd)
cfg = mkdd()
execfile(filename, {'cfg': cfg}, {})
But I quite like the way the '.' separators quite naturally (IMO)
indicate a hierarchy of settings.
Comments and suggestions welcome!
Kind regards,
Edd
Steven D'Aprano
> Hi folks,
>
> I'd like to use Python itself as the configuration language for my
> Python application. I'd like the user to be able to write something like
> this in their config file(s):
>
> cfg.laser.on = True
> cfg.laser.colour = 'blue'
> cfg.discombobulated.vegetables = ['carrots', 'broccoli'] # ...
>
> To this end, I've created a class that appears to allow instance
> variables to be created on the fly.
Um, don't all classes allow that?
>>> class MyClass(): pass
...
>>> instance = MyClass()
>>>
Or do you mean instance *attributes*? Again, apart from built-in types
and classes that use __slots__, all classes allow that.
>>> instance.parrot = 'Norwegian Blue'
>>>
> In other words, I can to the
> following to read a config file:
>
> cfg = Config()
> execfile(filename, {'cfg', cfg}, {})
That's okay so long as you trust the user not to put malicious, or buggy,
code in your config file. Personally, I think config files should be more
tolerant of errors than a programming language.
> However, I think my implementation of the Config class is a little
> crappy. I'd really appreciate the critical eye of a pro. Here's the
> sauce:
For starters, where is your documentation? No doc strings, not even any
comments! No, I tell a lie... *one* obscure comment that doesn't really
explain much.
> class Config(object):
> def __init__(self, sealed=False):
> def seal():
> for v in self._attribs.values():
> if isinstance(v, self.__class__): v.seal()
> del self.__dict__['seal']
>
> d = {'_attribs': {}, '_a2p': None}
> if not sealed: d['seal'] = seal
>
> self.__dict__.update(d)
I'm going to try to guess what the above does. When you initialise an
instance, you can tell the instance to be "sealed" or unsealed. I'm not
sure what the difference is, or why you would choose one over the other.
Sealed instances seem to be exactly the same as unsealed instances,
except they have a seal() method (actually a closure). The seal method,
when called, recursively seals any embedded Config instances inside the
current instance, then deletes itself.
Arghhh!!! Self-modifying code!!! Unclean, unclean!!!
I'm not sure why seal() is necessary -- it seems to me that if present,
all it does is delete itself. So why not just leave it out altogether?
You also have a rather complicated way of adding instance attributes.
Instead of
d = {'_attribs': {}, '_a2p': None}
self.__dict__.update(d)
why not just do the more obvious:
self._attribs = {}
self._a2p = None
?
> def __getattr__(self, key):
> if not key in self._attribs:
> d = Config(sealed='seal' not in self.__dict__) def
> add2parent():
> self._attribs[key] = d
> if self._a2p:
> self._a2p()
> self._a2p = None
It looks like you are just re-inventing the normal attribute mechanism of
Python. I'm not sure why you feel this is necessary. And it contains MORE
self-modifying code! Yuck! Frankly I don't care enough to dig into your
code to understand how it works in detail.
> # if anything is assigned to an attribute of d, # make
> sure that d is recorded as an attribute of this
> Config
> d._a2p = add2parent
> return d
> else:
> return self._attribs[key]
>
> def __setattr__(self, key, value):
> if key in self.__dict__:
> self.__dict__[key] = value
> else:
> if not 'seal' in self.__dict__:
> clsname = self.__class__.__name__
> raise AttributeError("'%s' object attribute '%s'
> is read-only (object is sealed)" % (clsname, key))
> self.__dict__['_attribs'][key] = value if self._a2p:
> self._a2p()
> self._a2p = None
Does "sealed" mean that the instance is read-only? If so, and if I'm
reading this correctly, I think it is buggy. You allow modifications to
attributes inside __dict__ *without* checking to see if the instance is
read-only. Then you get the test backwards: surely the existence, not the
absence, of a 'seal' attribute should mean it is sealed?
> def __delattr__(self, key):
> if key in self.__dict__:
> clsname = self.__class__.__name__
> raise AttributeError("can't delete '%s' object
> attribute '%s' as it is used for book-keeping!" % (clsname, key))
> else:
> if key in self._attribs:
> del self._attribs[key]
Nothing much to say here, except that you're doing more work re-inventing
the wheel, storing attributes inside _attribs instead of using the
general attribute mechanism. Seems unnecessary to me, but perhaps I don't
understand your use-case.
> Once the config is loaded, it will be passed down to other user- written
> scripts and it's important that these scripts don't accidentally change
> the config. So the idea is that I'll call cfg.seal () to prevent any
> further changes before passing it on to these other scripts.
Or you could pass a *copy* of the config, and let them change it to their
heart's content, it won't matter.
Or you could say "we're all adults here", simply document that any
changes will have consequences, and let user scripts change the config.
And why not?
> Beyond the
> general fiddliness of the code, I think the way seal() currently works
> is particularly pants.
Is "pants" slang for "fragile, hard to understand and difficult to debug"?
--
Steven
John Machin
cybersource.com.au> wrote:
> Is "pants" slang for "fragile, hard to understand and difficult to debug"?
pommy slang for "sucks intensely, like the Deathstar's tractor
beam" ... I think we agree with him.
Edd
Thank you for your response!
On Apr 11, 4:22 am, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.au> wrote:
> On Fri, 10 Apr 2009 19:04:38 -0700, Edd wrote:
> > Hi folks,
>
> > I'd like to use Python itself as the configuration language for my
> > Python application. I'd like the user to be able to write something like
> > this in their config file(s):
>
> > cfg.laser.on = True
> > cfg.laser.colour = 'blue'
> > cfg.discombobulated.vegetables = ['carrots', 'broccoli'] # ...
>
> > To this end, I've created a class that appears to allow instance
> > variables to be created on the fly.
>
> Um, don't all classes allow that?
>
> >>> class MyClass(): pass
> ...
> >>> instance = MyClass()
>
> Or do you mean instance *attributes*? Again, apart from built-in types
> and classes that use __slots__, all classes allow that.
>
> >>> instance.parrot = 'Norwegian Blue'
Yes I probably mean instance attributes. Forgive me, I am not
particularly sure of the terminology. But your MyClass example, won't
quite do what I want, as I'd like to be able to define instance
attributes on top of instance attributes by assignment:
>>> class MyClass(): pass
...
>>> instance = MyClass()
>>> instance.lasers.armed = True
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: MyClass instance has no attribute 'laser'
>>>
> > In other words, I can to the
> > following to read a config file:
>
> > cfg = Config()
> > execfile(filename, {'cfg', cfg}, {})
>
> That's okay so long as you trust the user not to put malicious, or buggy,
> code in your config file. Personally, I think config files should be more
> tolerant of errors than a programming language.
That's certainly a valid remark, but this will be a tool for
programmers. I am hoping that the user will make use of the power in
moderation. Often, it really will be useful to allow functions to be
defined in the config files, for example.
> > However, I think my implementation of the Config class is a little
> > crappy. I'd really appreciate the critical eye of a pro. Here's the
> > sauce:
>
> For starters, where is your documentation? No doc strings, not even any
> comments! No, I tell a lie... *one* obscure comment that doesn't really
> explain much.
Yes, you're quite right. I was about to add some doc strings, but I
didn't think the implementation was good enough. That's somewhat
backwards, though, right?! Especially considering I'm asking for
improvements. Anyway, I had hoped that the example usage at the end
would show what the purpose of the class is.
>
> > class Config(object):
> > def __init__(self, sealed=False):
> > def seal():
> > for v in self._attribs.values():
> > if isinstance(v, self.__class__): v.seal()
> > del self.__dict__['seal']
>
> > d = {'_attribs': {}, '_a2p': None}
> > if not sealed: d['seal'] = seal
>
> > self.__dict__.update(d)
>
> I'm going to try to guess what the above does. When you initialise an
> instance, you can tell the instance to be "sealed" or unsealed. I'm not
> sure what the difference is, or why you would choose one over the other.
> Sealed instances seem to be exactly the same as unsealed instances,
> except they have a seal() method (actually a closure). The seal method,
> when called, recursively seals any embedded Config instances inside the
> current instance, then deletes itself.
>
> Arghhh!!! Self-modifying code!!! Unclean, unclean!!!
Quite!
> I'm not sure why seal() is necessary -- it seems to me that if present,
> all it does is delete itself. So why not just leave it out altogether?
As I said in the original post, such Config objects will be made
available to other kinds of user-written script and it's important
that the Config not change between the execution of one script and the
next. The seal() mechanism was an attempt to help the user from
*accidentally* doing this and then having to try to diagnose the
problem and understand how changing the config might have broken the
invariants of the software. I guess a big "DON'T CHANGE THE CONFIG IN
YOUR SCRIPTS" message in the manual, might be sufficient, though :)
> You also have a rather complicated way of adding instance attributes.
> Instead of
>
> d = {'_attribs': {}, '_a2p': None}
> self.__dict__.update(d)
>
> why not just do the more obvious:
>
> self._attribs = {}
> self._a2p = None
Because that would go through __setattr__(), which does something else
(which is the whole point of the class). At least, that was my
understanding, which certainly could be at fault.
This might be nicer I guess:
self.__dict__['_attribs'] = {}
self.__dict__['_a2p'] = None
There used to be more instance attributes than just two so it was
easier to put them in a dict and use update. I agree that it's rather
obfuscated, though.
[Edd's horrendous code snipped]
> It looks like you are just re-inventing the normal attribute mechanism of
> Python. I'm not sure why you feel this is necessary. And it contains MORE
> self-modifying code! Yuck! Frankly I don't care enough to dig into your
> code to understand how it works in detail.
Ok. But is there a quick-and-easy way of creating an object, cfg, such
that I can write:
cfg.hovercraft.full.of = 'eels'
without knowing in advance that the user will want a .hovercraft
instance attribute, or a .full attribute inside that, or a .of inside
that, or ... ?
> > def __setattr__(self, key, value):
> > if key in self.__dict__:
> > self.__dict__[key] = value
> > else:
> > if not 'seal' in self.__dict__:
> > clsname = self.__class__.__name__
> > raise AttributeError("'%s' object attribute '%s'
> > is read-only (object is sealed)" % (clsname, key))
> > self.__dict__['_attribs'][key] = value if self._a2p:
> > self._a2p()
> > self._a2p = None
>
> Does "sealed" mean that the instance is read-only? If so, and if I'm
> reading this correctly, I think it is buggy. You allow modifications to
> attributes inside __dict__ *without* checking to see if the instance is
> read-only. Then you get the test backwards: surely the existence, not the
> absence, of a 'seal' attribute should mean it is sealed?
The absence of the seal() 'method' means that it has already been
called i.e. the object has already been sealed. I agree it's somewhat
fishy, which is why I'm asking for suggestions for improvements.
Perhaps I should just forget the seal() idea.
> > Once the config is loaded, it will be passed down to other user- written
> > scripts and it's important that these scripts don't accidentally change
> > the config. So the idea is that I'll call cfg.seal () to prevent any
> > further changes before passing it on to these other scripts.
>
> Or you could pass a *copy* of the config, and let them change it to their
> heart's content, it won't matter.
I think that's probably the best thing. I was worried about some parts
not being deep-copyable, but I think I'm happy to put that concern
aside. Besides, right now, even though you can't add/delete attributes
in a Config, you can still change existing ones:
cfg = Config()
cfg.a.b.c = [1, 2]
cfg.seal()
cfg.a.b.c[1] = 3 # pffff
Yes, you've helped convince me that it's just a bad idea.
> Or you could say "we're all adults here", simply document that any
> changes will have consequences, and let user scripts change the config.
> And why not?
Even adults make the occasional tiny mistake which has confusing
consequences in a larger system. It was an attempt to help prevent
this. I probably worry too much about that.
> > Beyond the
> > general fiddliness of the code, I think the way seal() currently works
> > is particularly pants.
>
> Is "pants" slang for "fragile, hard to understand and difficult to debug"?
Yes! Rest assured that I am under no illusion that what I have written
is good!
Steven, I greatly appreciate your taking the time to understand the
aforementioned horrors. Assuming that the seal() stuff is no longer a
requirement, is there a cleaner way of creating an object where
(nested) instance attributes can be defined by simple assignment?
Perhaps it would have been better if I had left out my awful attempt
altogether. It seems like it only made my question more confusing than
it needed to be :( If it would help, I'd be happy to add some doc
strings and tests but I was getting ready to throw this code away when
a cleaner 5-line alternative was presented!
Kind regards,
Edd
Steven D'Aprano
> Yes I probably mean instance attributes. Forgive me, I am not
> particularly sure of the terminology. But your MyClass example, won't
> quite do what I want, as I'd like to be able to define instance
> attributes on top of instance attributes by assignment:
>
>>>> class MyClass(): pass
> ...
>>>> instance = MyClass()
>>>> instance.lasers.armed = True
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> AttributeError: MyClass instance has no attribute 'laser'
Ah, now it is more clear.
Okay, let's try this:
>>> class C(object):
... def __getattr__(self, name):
... # Only called if self.name doesn't exist.
... inst = self.__class__()
... setattr(self, name, inst)
... return inst
...
>>> c = C()
>>> c.x.y.z = 45
>>> c.__dict__
{'x': <__main__.C object at 0xb7c3b78c>}
>>> c.x.__dict__
{'y': <__main__.C object at 0xb7c3b7ec>}
>>> c.x.y.z
45
--
Steven
Edd
cybersource.com.au> wrote:
> Ah, now it is more clear.
>
> Okay, let's try this:
>
> >>> class C(object):
>
> ... def __getattr__(self, name):
> ... # Only called if self.name doesn't exist.
> ... inst = self.__class__()
> ... setattr(self, name, inst)
> ... return inst
Ha! Perfect! I knew it should be simpler. Thanks very much!
Kind regards,
Edd