Overriding "__setattr__" of a module - possible?
John Nagle
want to capture changes to global variables for debug purposes.
None of the following seem to have any effect.
modu.__setattr__ = myfn
setattr(modu, "__setattr__", myfn)
delattr(modu, "__setattr__")
John Nagle
Michele Simionato
There is a dirty trick which involves fiddling with sys.modules.
For instance:
$ cat x.py
import sys
class FakeModule(object):
def __init__(self, dic):
vars(self).update(dic)
def __setattr__(self, name, value):
print "setting %s=%s" % (name, value)
object.__setattr__(self, name, value)
a = 1
def f():
print 'called f'
sys.modules[__name__] = FakeModule(globals())
Here is an ipython session:
In [1]: import x
In [2]: x.a
Out[2]: 1
In [3]: x.f
Out[3]: <function f at 0x93f5614>
In [4]: x.f()
called f
In [5]: x.a=2
setting a=2
In [6]: x.a
Out[6]: 2
John Nagle
> On Jun 16, 4:43 am, John Nagle<na...@animats.com> wrote:
>> Is it possible to override "__setattr__" of a module? I
>> want to capture changes to global variables for debug purposes.
>>
>> None of the following seem to have any effect.
>>
>> modu.__setattr__ = myfn
>>
>> setattr(modu, "__setattr__", myfn)
>>
>> delattr(modu, "__setattr__")
>>
>> John Nagle
>
> There is a dirty trick which involves fiddling with sys.modules.
> For instance:
>
> $ cat x.py
> import sys
>
> class FakeModule(object):
Cute, but it doesn't work in general. Faking a module as a
class fails when you simply call
x()
within the module.
I also tried
modu.__dict__['__setattr__'] = myfun
...
modu.foo = 1 # "myfun" does not get called.
Any more ideas?
John Nagle
Stephen Hansen
> Cute, but it doesn't work in general. Faking a module as a
> class fails when you simply call
>
> x()
>
> within the module.
Huh? Explain how it doesn't work? I've done it at least twice
(shamefully do I admit this) and it works fine. Real life code.
> Any more ideas?
There aren't any; modules do not follow the class object protocol. They
are simple types with a __dict__ (which you can't change, either, so no
replacing it with a dict that implements __setattr__).
Replacing the module with a class in sys.modules is really the only way
to fake the behavior.
--
Stephen Hansen
... Also: Ixokai
... Mail: me+list/python (AT) ixokai (DOT) io
... Blog: http://meh.ixokai.io/
John Nagle
> On 6/15/10 9:16 PM, John Nagle wrote:
>> Cute, but it doesn't work in general. Faking a module as a
>> class fails when you simply call
>>
>> x()
>>
>> within the module.
>
> Huh? Explain how it doesn't work? I've done it at least twice
> (shamefully do I admit this) and it works fine. Real life code.
>
>> Any more ideas?
>
> There aren't any; modules do not follow the class object protocol. They
> are simple types with a __dict__ (which you can't change, either, so no
> replacing it with a dict that implements __setattr__).
>
> Replacing the module with a class in sys.modules is really the only way
> to fake the behavior.
OK, working on this. I can make a module make itself into a
fake class, but can't yet do it to other modules from outside.
John Nagle
Stephen Hansen
Can you show a bit of code that you're doing? I've never had much
trouble with it: but I'm not sure what you mean by that final statement.
You're attempting to turn "other" modules into faux-modules-as-classes?
I don't think you can do that successfully. The whole faux-module
strategy only works when the module is designed around that principle, I
think.
Michele Simionato
> OK, working on this. I can make a module make itself into a
> fake class, but can't yet do it to other modules from outside.
>
> John Nagle
I think you can with something like
import module
sys.modules[module.__name__] = FakeModule(vars(module))
Christian Heimes
> are simple types with a __dict__ (which you can't change, either, so no
> replacing it with a dict that implements __setattr__).
You are wrong, my friend. :)
Modules follow the new style class and instance protocol. Modules aren't
classes but instances of the internal module type. Since you can't
overwrite magic methods on instances of new style classes you can't
overwrite __setattr__ on module level, too.
Christian
Lie Ryan
I doubt this is what you wanted, but...
import itertools
module = type(itertools)
class FakeModule(module):
def __init__(self, module):
super(FakeModule, self).__init__(module.__name__, module.__doc__)
self.__module = module
def __getattr__(self, attr):
return getattr(self.__module, attr)
def __setattr__(self, attr, val):
print self, attr, val
super(FakeModule, self).__setattr__(attr, val)
Steven D'Aprano
> On Jun 16, 4:43 am, John Nagle <na...@animats.com> wrote:
>> Is it possible to override "__setattr__" of a module? I
>> want to capture changes to global variables for debug purposes.
[...]
> There is a dirty trick which involves fiddling with sys.modules. For
> instance:
[snip example of installing a class in sys.modules]
I'm not sure that this is a dirty trick. Isn't it officially supported?
If I recall correctly, Python used to insist that modules in sys.modules
were actual module objects, and that restriction was lifted deliberately
to allow clever tricks like this. Or am I thinking of something else?
--
Steven
John Nagle
>> Is it possible to override "__setattr__" of a module? I
>> want to capture changes to global variables for debug purposes.
>>
>> None of the following seem to have any effect.
>>
>> modu.__setattr__ = myfn
>>
>> setattr(modu, "__setattr__", myfn)
>>
>> delattr(modu, "__setattr__")
>>
>> John Nagle
>
That doesn't do quite what you'd expect. Here's a version with
more debug output:
import sys
class FakeModule(object):
def __init__(self, dic):
vars(self).update(dic)
def __setattr__(self, name, value):
print "setting %s=%s" % (name, value)
object.__setattr__(self, name, value)
a = 1
def f():
print ('called f, a = %s' % (a,))
print("Patching " + __name__)
sys.modules[__name__] = FakeModule(globals())
---
C:\projects\newthreading>\python26\python
ActivePython 2.6.5.12 (ActiveState Software Inc.) based on
Python 2.6.5 (r265:79063, Mar 20 2010, 14:22:52) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import x
Patching x
>>> x.f()
called f, a = None
>>> x.a
1
>>> x.a = 2
setting a=2
>>> x.f()
called f, a = None
---
Note that there are now two copies of "a", one bound to the module and
referenced in "f", and a second bound to the class, and referenced by
"x.a". Uh oh.
The problem here is that when "def f..." was defined, its reference
to "a" was bound to the object containing "a" at the time, which is
the module-level copy of the variable.
I don't think turning a module into a class is going to work.
Binding the functions into the class won't work, because Python
doesn't have class-level (as opposed to instance-level) functions.
If we move x.f into class FakeModule, then "x.f()" is called as
"f(x)", with an unexpected argument.
Any better ideas? All I really want to do is to override "__setattr__"
for a module.
John Nagle
Ian Kelly
> Note that there are now two copies of "a", one bound to the module and
> referenced in "f", and a second bound to the class, and referenced by
> "x.a". Uh oh.
>
> The problem here is that when "def f..." was defined, its reference
> to "a" was bound to the object containing "a" at the time, which is
> the module-level copy of the variable.
>
> I don't think turning a module into a class is going to work.
> Binding the functions into the class won't work, because Python
> doesn't have class-level (as opposed to instance-level) functions.
> If we move x.f into class FakeModule, then "x.f()" is called as
> "f(x)", with an unexpected argument.
>
> Any better ideas? All I really want to do is to override "__setattr__"
> for a module.
It seems to me that a proxy for the actual module would work better.
import sys
class ModuleProxy(object):
def __init__(self, module):
object.__setattr__(self, 'module', module)
def __getattribute__(self, name):
module = object.__getattribute__(self, 'module')
return getattr(module, name)
def __setattr__(self, name, value):
module = object.__getattribute__(self, 'module')
print "setting %s=%s" % (name, value)
setattr(module, name, value)
a = 1
def f():
print a
sys.modules[__name__] = ModuleProxy(__import__(__name__))
Cheers,
Ian
John Nagle
That just leaves things in a state where even "sys" and "import"
are undefined.
John Nagle
Ian Kelly
> That just leaves things in a state where even "sys" and "import"
> are undefined.
Say what? It works fine for me.
>>> import proxy_mod
>>> proxy_mod.f()
1
>>> proxy_mod.a = 2
setting a=2
>>> proxy_mod.f()
2
>>> proxy_mod.sys
<module 'sys' (built-in)>
Ian
Gabriel Genellina
escribi�:
> On Wed, Jun 16, 2010 at 3:38 PM, John Nagle <na...@animats.com> wrote:
>> That just leaves things in a state where even "sys" and "import"
>> are undefined.
>
> Say what? It works fine for me.
>
>>>> import proxy_mod
>>>> proxy_mod.f()
> 1
>>>> proxy_mod.a = 2
> setting a=2
>>>> proxy_mod.f()
> 2
>>>> proxy_mod.sys
> <module 'sys' (built-in)>
It *mostly* works, but not always. Try this function:
def g(x):
global a
print 'global a -> ', x
a = x
py> import fake # ModuleProxy stuff
py> fake.f()
1
py> fake.a = 3
setting a=3
py> fake.f()
3
py> fake.g(8)
global a -> 8
py> fake.f()
8
py> fake.a
8
Note the fake.g(8) call: __setattr__ wasn't called.
If the OP wants to trace assignments to global variables, this becomes a
problem.
A function defined in a module holds a reference to the module's __dict__
in its func_globals attribute. Getting and setting global variables goes
directly to this dictionary, and does not use the module object at all.
Even worse, the LOAD_GLOBAL/STORE_GLOBAL opcodes (which implement getting
and setting global variables) assume func_globals is a true dictionary and
bypass any overriden __getitem__/__setitem__ methods (an optimization,
surely). I'm afraid it will be hard to intercept global variable usage in
these circumstances.
--
Gabriel Genellina
Alf P. Steinbach
Great exposition.
But who would have thunk that Python *isn't dynamic enough*? :-)
Cheers,
- Alf
--
blog at <url: http://alfps.wordpress.com>
Gabriel Genellina
escribió:
> * Gabriel Genellina, on 17.06.2010 09:25:
>> En Wed, 16 Jun 2010 19:56:39 -0300, Ian Kelly <ian.g...@gmail.com>
>> escribió:
Yep... There are other examples too (e.g. the print statement in 2.x
bypasses sys.stdout.write; see also a recent thread "Which objects are
expanded by double-star ** operator?")
Most of them seem to be speed optimizations, some might be considered
subtle bugs. But in this case (global variable references) speed is so
critical than even the dict lookup is inlined; the code in ceval.c says:
/* Inline the PyDict_GetItem() calls.
WARNING: this is an extreme speed hack.
Do not try this at home. */
Python is dynamic but not so much as to make it crawl like a snail...
--
Gabriel Genellina
Fuzzyman
wrote:
>
> Yep... There are other examples too (e.g. the print statement in 2.x
> bypasses sys.stdout.write;
What do you mean by this? The print statement in 2.x does *not* bypass
sys.stdout. It may use other methods besides write (writeln perhaps)
but you can *definitely* override sys.stdout to capture the output
from print statements.
Michael Foord
> see also a recent thread "Which objects are
> expanded by double-star ** operator?")
>
> Most of them seem to be speed optimizations, some might be considered
> subtle bugs. But in this case (global variable references) speed is so
> critical than even the dict lookup is inlined; the code in ceval.c says:
>
> /* Inline the PyDict_GetItem() calls.
> WARNING: this is an extreme speed hack.
> Do not try this at home. */
>
> Python is dynamic but not so much as to make it crawl like a snail...
>
> --
> Gabriel Genellina
John Nagle
> Note the fake.g(8) call: __setattr__ wasn't called.
> If the OP wants to trace assignments to global variables, this becomes a
> problem.
>
> A function defined in a module holds a reference to the module's
> __dict__ in its func_globals attribute. Getting and setting global
> variables goes directly to this dictionary, and does not use the module
> object at all.
>
> Even worse, the LOAD_GLOBAL/STORE_GLOBAL opcodes (which implement
> getting and setting global variables) assume func_globals is a true
> dictionary and bypass any overriden __getitem__/__setitem__ methods (an
> optimization, surely). I'm afraid it will be hard to intercept global
> variable usage in these circumstances.
OK, thanks. You can't actually replace "__setattr__" for
a module, and the "module as class" hack is iffy. I didn't
really think this would work, but it was worth a try.
I'm trying out a proof of concept implementation for a new
approach to safe threading. It's somewhat similar in concept
to Alan Olsen's scheme. The basic difference is that once
the program goes multi-thread, code objects and some other
bindings are locked down and become unchangeable. Olsen
was climbing the walls trying to get the locking right for
the awful cases like redefining a function while another thread
is inside it. I'm trying to lock out some of those cases.
If you do that, removing the GIL requires less pain than
Olsen experienced.
The key idea is that the use cases for most of Python's
code dynamism are during setup and initialization. You usually
don't change code once the program has gone into its heavy
parallel processing phase. This suggests a practical compromise.
More on this once I have something people can download and try.
I'm doing a test implementation in Python so people can try the
concept and see if it works in practice. It won't go fast;
it's just to give a feel for what it would be like.
John Nagle
Gabriel Genellina
> On Jun 17, 10:29 am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
> wrote:
>> En Thu, 17 Jun 2010 04:52:48 -0300, Alf P. Steinbach <al...@start.no>
>>
>> > But who would have thunk that Python *isn't dynamic enough*? :-)
>>
>> Yep... There are other examples too (e.g. the print statement in 2.x
>> bypasses sys.stdout.write;
>
> What do you mean by this? The print statement in 2.x does *not* bypass
> sys.stdout. It may use other methods besides write (writeln perhaps)
> but you can *definitely* override sys.stdout to capture the output
> from print statements.
Suppose you want to implement a "tee" variant in Python: print output
should go to stdout and also to some file (with timestamp added, just to
be fancy). First attempt:
py> import sys
py> import time
py>
py> class tee(file):
... def write(self, data):
... file.write(self, '%s: %r\n' % (time.ctime(), data))
... sys.__stdout__.write(data)
...
py> sys.stdout = tee('test.txt', 'w')
py> print "Hello world"
py> print "Bye"
py> ^Z
D:\TEMP>type test.txt
Hello world
Bye
Note:
- no output to stdout inside the interpreter
- no timestamp in the file
This modified version works fine:
py> class tee():
... def __init__(self, filename, mode):
... self.file = open(filename, mode)
... def write(self, data):
... self.file.write('%s: %r\n' % (time.ctime(), data))
... sys.__stdout__.write(data)
What happened? When sys.stdout is an instance of some class inheriting
from file (that is, isinstance(sys.stdout, file) is true) then the print
statement ignores sys.stdout.write() completely -- instead it calls
directly some C stdio functions (fwrite).
The only way to influence 'print' is *not* to inherit from file in the
first place.
It's an optimization, sure. I guess it is there before inheriting from
builtin types was allowed (in such scenario, it's a perfectly valid
optimization). Now, perhaps the test for 'file' should be more strict,
only taking the C shortcut when using an actual file instance, not a
subclass of it. This would allow the example above to work correctly.
--
Gabriel Genellina
Gabriel Genellina
escribi�:
> I'm trying out a proof of concept implementation for a new
> approach to safe threading. It's somewhat similar in concept
> to Alan Olsen's scheme. The basic difference is that once
> the program goes multi-thread, code objects and some other
> bindings are locked down and become unchangeable. Olsen
> was climbing the walls trying to get the locking right for
> the awful cases like redefining a function while another thread
> is inside it. I'm trying to lock out some of those cases.
> If you do that, removing the GIL requires less pain than
> Olsen experienced.
>
> The key idea is that the use cases for most of Python's
> code dynamism are during setup and initialization. You usually
> don't change code once the program has gone into its heavy
> parallel processing phase. This suggests a practical compromise.
Seems interesting...!
--
Gabriel Genellina
Fuzzyman
wrote:
> En Thu, 17 Jun 2010 07:12:23 -0300, Fuzzyman <fuzzy...@gmail.com> escribi�:
>
> > On Jun 17, 10:29 am, "Gabriel Genellina" <gagsl-...@yahoo.com.ar>
> > wrote:
> >> En Thu, 17 Jun 2010 04:52:48 -0300, Alf P. Steinbach <al...@start.no>
Ah, so by "bypasses" you mean "under certain specific circumstances
bypasses". By all means file a bug report on this, I agree that
bypassing the optimization for file subclasses (assuming your
diagnosis is correct) would be a sensible approach.
All the best,
Michael
> --
> Gabriel Genellina