Extending sqlalchemy.schema.Column and metaprogramming traps
Angri
I had a problem described in subject. Here is the testcase:
import sqlalchemy
engine = sqlalchemy.create_engine('sqlite:///:memory:')
metadata = sqlalchemy.MetaData(engine)
class ForeignKey(sqlalchemy.Column):
def __init__(self, name, foreign_column, *args, **kwargs):
fk = sqlalchemy.ForeignKey(foreign_column)
super(ForeignKey, self).__init__(name, fk, *args, **kwargs)
table1 = sqlalchemy.Table('table1', metadata,
sqlalchemy.Column('id', sqlalchemy.Integer)
)
table2 = sqlalchemy.Table('table2', metadata,
ForeignKey('fk', 'table1.id')
)
metadata.create_all()
It fails with really strange exception: "AttributeError: 'ForeignKey'
object has no attribute 'use_alter'". It was really hard to me to
track down the error's nature and I found that it came from here:
http://www.sqlalchemy.org/trac/browser/sqlalchemy/trunk/lib/sqlalchemy/sql/visitors.py#L29
I solved my problem by adding class property __visit_name__ = 'column'
to ForeignKey class. But I am a bit confused now and I have two
questions about the code in
sqlalchemy.sql.visitors.VisitableType.__init__()
1. What about another side-effects depending on clsname? Is it
actually safe to extend sqlalchemy.schema.Column, or it may have
unpredictable behavior similar to that i've encountered?
2. (almost offtopic) Is 'exec' really need there? What's wrong with
closures?
3. Maybe I should send it to developers mailing list?
Thanks.
--
Angri
Michael Bayer
On Dec 6, 2008, at 4:27 PM, Angri wrote:
> 1. What about another side-effects depending on clsname? Is it
> actually safe to extend sqlalchemy.schema.Column, or it may have
> unpredictable behavior similar to that i've encountered?
The Column object is one of the most key classes in all of SQLAlchemy
and we do put it through some fairly intricate copy/proxy patterns
particularly when using the ORM. Extending it should be "safe",
although this is usually not needed. For custom creation patterns as
you're seeking here its more straightforward to build a creation
function, so that the resulting object is returned unchanged in its
type.
>
> 2. (almost offtopic) Is 'exec' really need there? What's wrong with
> closures?
the visit step is called very intensively during statement compilation
so exec'ing the direct code instead of relying upon getattr() with a
composed name at runtime is an optimization to reduce function-call
and attribute-retrieval overhead. Just as a point of reference I
tried rewriting our "visit" dispatcher in C, and my experiments showed
that using the exec approach you see there performs just as well -
though the time savings compared to a basic getattr() approach are
very small.
Since you raised the issue, I went to try a different approach which
is probably the best possible approach without using exec, which is
this:
visit_name = cls.__dict__["__visit_name__"]
if isinstance(visit_name, str):
getter = operator.attrgetter("visit_%s" % visit_name)
def _compiler_dispatch(self, visitor, **kw):
return getter(visitor)(self, **kw)
else:
def _compiler_dispatch(self, visitor, **kw):
return getattr(visitor, 'visit_%s' %
self.__visit_name__)(self, **kw)
Above, we use operator.attrgetter so that the string composition is
already handled, and the attrgetter itself is a native object which
performs as fast as direct access. This change still adds a few
function calls per compile. Our bench of a single select() compile
goes from 183 to 187, and two of the zoomark_orm tests fail past the
5% threshhold, with tests three and four moving from 6623 to 6723 and
23345 to 23861 function calls, respectively. Which is an extremely
small amount, so its not a terribly big deal either way. So the exec
approach is saving a tiny amount of call counts, but not necessarily
any actual time. I'd be willing to scrap it if it continues to scare
other developers.
The reason I'm at all comfortable with exec is that we're already
using 'exec' for decorators - its a technique used by the "decorators"
module (which I'd like to transition to at some point) to faithfully
represent the calling signature of a decorated function.
>
> 3. Maybe I should send it to developers mailing list?
either....devel is not very active. though this is a pretty develop-y
subject...
Angri
problem with custom class name:
http://www.sqlalchemy.org/trac/browser/sqlalchemy/trunk/lib/sqlalchemy/sql/util.py#L145
And I guess it is not the last :(
I see, you propose not to extend class Column and write "a function
that creates instances" but it seems to me that this approach is not
good because it is not "pythonic", it is not "object-oriented"ly, it
is hard to write, support and extend.
Continuing experiments I created the following (ugly) code. Maybe it
will help someone with similar needs.
class CustomColumnMetaclass(sqlalchemy.Column.__metaclass__):
def __init__(cls, clsname, bases, dct):
cls.__name__ = sqlalchemy.Column.__name__
cls.__visit_name__ = sqlalchemy.Column.__visit_name__
super(CustomColumnMetaclass, cls).__init__(clsname, bases, dct)
class CustomColumn(sqlalchemy.Column):
__metaclass__ = CustomColumnMetaclass
Btw, I still think that relying on class name is a bad way to do
things. What do you think Michael, how difficult it can be to rewrite
those pieces of code to use more OOP-like technics, particularly
explicitly define class properties (inheritable class properties!)
instead of doing things like eval("Annotated%s" %
element.__class__.__name__)?
Michael Bayer
On Dec 9, 2008, at 4:09 PM, Angri wrote:
>
> I can not agree that extending is "safe" as I've encountered another
> problem with custom class name:
> http://www.sqlalchemy.org/trac/browser/sqlalchemy/trunk/lib/sqlalchemy/sql/util.py#L145
> And I guess it is not the last :(
It probably is the last. That's something which has been added
recently. It previously worked in a dynamic fashion but was changed
to what you see there for pickling purposes. However it can be
expanded to support classes which aren't present, so that you wouldn't
notice it. This technique is also something I picked up from a very
well known and respected Python developer.
> I see, you propose not to extend class Column and write "a function
> that creates instances" but it seems to me that this approach is not
> good because it is not "pythonic", it is not "object-oriented"ly, it
> is hard to write, support and extend.
It's how most of SQLA clause elements are invoked, in fact. Python's
philosophy is defnitely not in the vein of "use objects and
inheritance for everything", you're thinking of Java, which does not
provide standalone functions in the language. Using functions to
return objects provides a convenient layer of variability between the
composed structure of what the function returns and the public
signature of that function.
Subclassing is definitely not the only way to extend something and the
problem you're experiencing is a variant of the so-called "fragile
base" problem. What strikes me as most "unpythonic" here is that
you think you need a rewrite of the library in a Java-like GOF style
when a simple, one line function will solve your problem.
>
> Btw, I still think that relying on class name is a bad way to do
> things.
Here is where the technique is derived from, from the "visitor.py"
module in Python's own compiler module:
> What do you think Michael, how difficult it can be to rewrite
> those pieces of code to use more OOP-like technics, particularly
> explicitly define class properties (inheritable class properties!)
> instead of doing things like eval("Annotated%s" %
> element.__class__.__name__)?
When I first started Python, I used a traditional GOF-visitor pattern,
with explicit visit_XXX and dispatcher methods. It's extremely
verbose and tedious to refactor. It didn't take too long for me to
realize I was porting Java methodologies that are entirely
inappropriate for a dynamic language like Python. Subclassing Column
in any case requires knowledge of the visitation logic - you either
need to implement visit_XXX() in the GOF style, or __visit_name__ =
'whatever' in the Pythonic style.
Michael Bayer
On Dec 9, 2008, at 4:09 PM, Angri wrote:
>
> I can not agree that extending is "safe" as I've encountered another
> problem with custom class name:
> http://www.sqlalchemy.org/trac/browser/sqlalchemy/trunk/lib/sqlalchemy/sql/util.py#L145
rev 5454 removes AnnotatedColumn's reliance upon names within the
expression package. To subclass Column, this is the current recipe:
from sqlalchemy.sql.util import Annotated, annotated_classes
class MyColumn(Column):
__visit_name__ = "column"
class AnnotatedMyColumn(Annotated, MyColumn):
__visit_name__ = "column"
annotated_classes[MyColumn] = AnnotatedMyColumn
There are of course metaclass methods of making this automatic. But
here we have pure "traditional OOP" style - make all the classes
explicitly, register them, etc.
Michael Bayer
On Dec 9, 2008, at 4:09 PM, Angri wrote:
> particularly
> explicitly define class properties (inheritable class properties!)
if you'd like to submit a patch which defines __visit_name__ for all
ClauseElements and removes the logic from VisitableType to guess the
name, it will be accepted. The second half of VisitableType still may
be needed since it improves performance.
> instead of doing things like eval("Annotated%s" %
> element.__class__.__name__)?
The __name__ based logic is gone.
The code that used to be there, which is the code i was mentioning,
was this:
return object.__new__(
type.__new__(type, "Annotated%s" %
element.__class__.__name__, (Annotated, element.__class__), {})
)
This generates a new type dynamically and is the preferred way to do
this. The __name__ is only used there to give the resultling class a
name and not any kind of lookup semantics.
Unfortunately the resulting class is not pickleable since it is not
declared in the module. So there are two options I'm aware of.
Either do this:
class AnnotatedClauseElement(Annotated, ClauseElement):
....
class AnnotatedFromClause(Annotated, FromClause):
...
class AnnotatedColumnElement(Annotated, ColumnElement):
...
< continue for 100 more classes in sqlalchemy.expression>
Or do the "Exec" thing we have. A third alternative would be great if
you have one in mind.
a...@svilendobrev.com
one can inject things in module's locals() via locals().update( ...)
but i'm not sure if this is a feature-to-stay.
Angri
> ClauseElements and removes the logic from VisitableType to guess the
> name, it will be accepted. The second half of VisitableType still may
> be needed since it improves performance.
google groups, so I put it in paste bin: http://paste.org/index.php?id=4463
I made some automated tests to make sure that nothing will break.
Existing Visitable's descendants after applying the patch will have
exactly the same value of __visit_name__ property. I also put small
test from first message in the topic. It fails with vanila trunk and
pass ok with patched.
Take a look, please.
Michael Bayer
guest/guest and attach it there: http://www.sqlalchemy.org/trac/newticket
Angri
Maybe it is good idea to drop some new lines in faq? Something like
this:
Q: How should I extend sqlalchemy.schema.Column?
A: You surely dont need it. Recommended way to achive your possible
needs is to write instance-factory function which decorates creation
of sqlalchemy.schema.Column instances.
Q: But I'm really need it!
A: Ok. To subclass Column, this is the current recipe:
from sqlalchemy.sql.util import Annotated, annotated_classes
class MyColumn(Column):
class AnnotatedMyColumn(Annotated, MyColumn):
pass
annotated_classes[MyColumn] = AnnotatedMyColumn
Do not forget to put AnnotatedMyColumn in the module namespace, or
your schema will not be pickleable!
Correct me please if I am wrong somewhere and excuse me for my
English.
Michael Bayer
On Dec 11, 2008, at 3:37 AM, Angri wrote:
>
> Here it is: http://www.sqlalchemy.org/trac/ticket/1244
>
> Maybe it is good idea to drop some new lines in faq? Something like
> this:
>
> Q: How should I extend sqlalchemy.schema.Column?
> A: You surely dont need it. Recommended way to achive your possible
> needs is to write instance-factory function which decorates creation
> of sqlalchemy.schema.Column instances.
>
> Q: But I'm really need it!
> A: Ok. To subclass Column, this is the current recipe:
>
> from sqlalchemy.sql.util import Annotated, annotated_classes
>
> class MyColumn(Column):
> ...
>
> class AnnotatedMyColumn(Annotated, MyColumn):
> pass
>
> annotated_classes[MyColumn] = AnnotatedMyColumn
>
> Do not forget to put AnnotatedMyColumn in the module namespace, or
> your schema will not be pickleable!
>
> Correct me please if I am wrong somewhere and excuse me for my
> English.
Well the AnnotatedMyColumn part is less than ideal since its an
internal. the way that works could very likely change. Creating an
AnnotatedXXX class *can* be automated. the pickle thing just might be
a caveat we'd document or arrange for an exception to occur (like
putting a throw in a __getstate__ method).
Martijn Moeling
I know this is an "OLD" threat but I was searching the group to see If I was not the first one doing this.
I am not sure I understand very well what this threat is all about, but I want to extend the Column class for a different reason.
I want to add extra functionality to the Column class which is absolutely NOT SA related. SA functionality should not be effected though.
say I want to add a config value and some methods for rendering and validating screens:
def MyColumn(Column):
def __init():
dosomething to init
def ExtraInfo(self):
do_something_not_sa_related
validation = 'someregex'
and use MyColumn in places where I normally use Column(......)
What do I need to take into account, I've done some tests and "Error hell" broke loose, where the errors are hidden deep inside SA so hard to overcome.
Martijn
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
> To post to this group, send email to sqlal...@googlegroups.com
> To unsubscribe from this group, send email to sqlalchemy+...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en
> -~----------~----~----~----~------~----~------~--~---
>
Sean Devlin
class ValidatorAspect:
validators = {}
@classmethod
def add_validator():
@classmethod
def del_validator():
from UserDict import DictMixin
class MyDataObject(Base,ValidatorAspect,DictMixin):
def __getitem__():
def __setitem__():
def __delitem__():
def keys():
def update():
#Loop over inputs
#Apply validator if present
My $.02
Sean
--
You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
To post to this group, send email to sqlal...@googlegroups.com.
To unsubscribe from this group, send email to sqlalchemy+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
Michael Bayer
There are two scenarios where Column objects are copied, and in one case copied into an altered class, so the "copying" of Column uses an attribute called _constructor to point to which class should be used when creating this copy. Usually setting that to Column:
class MyColumn(Column):
_constructor = Column
# .... go nuts
is all you need.
> --
> You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
> To post to this group, send email to sqlal...@googlegroups.com.
> To unsubscribe from this group, send email to sqlalchemy+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
>
Martijn Moeling
On Feb 28, 2011, at 18:21 , Michael Bayer wrote:
> Column can be subclassed but because they are intensively used in complex expression transformations, your custom class may be used in more scenarios than you first anticipate.
>
> There are two scenarios where Column objects are copied, and in one case copied into an altered class, so the "copying" of Column uses an attribute called _constructor to point to which class should be used when creating this copy. Usually setting that to Column:
>
> class MyColumn(Column):
> _constructor = Column
>
> # .... go nuts
LOL!! Thanks
Martijn Moeling
sorry I have to get back on this. I renamed all Columns in my application definitions to MyColumn a while back. and everything worked.
Now that I'm starting to use functionality of MyColumn. (The reason I needed this) I run into some trouble.
What do I want:
I want to add properties to the Column definition, properties which vane NOTHING to do with SA, but should not break the inner workings of SA.
The Properties should be initialized and dealt with by MyColumn
Michael pointed at setting _constructor to Column (See mail history below)
class MyColumn(Column):
_constructor = Column
X = 0
Y = 0
class Test(Base, GeneratorClass):
__tablename__ = …..
etc...
Name = MyColumn(Uncicode(100), Index=True, Last_of_SQLAlchemy_properties, X = 100, Y = 200)
Note the X and Y (Which will be the position on screen, as I will generate screens directly using the GeneratorClass.__subclasses__ and a lots of Introspection from within SA (reflection.Inspector.from_engine(engine) )and python.
Te above, you mentioned is not working for me. The X and Y are passed to the _contructor.__init__ directly and class property Test.Name looses track of them
I need something like:
class MyColumn(Column)
def __init__(self, Type, **options)
#filter all options to set to this column and have nothing to do with SA (Like X and Y)
FilteredOptions = {}
For option in options:
if hasattr(self, option.name)
#init self
else:
#add to Filteredoptions
#Now Init the Superclass:Column with the correct properties
Column.__init__(self,Type, *Filteredoptions)
At the other end, I might need to introspect the objects, I have tried but I have trouble in relating to X or Y since the Name Column is an InstrumentedAttribute.
One other thing. I can get Columns by iterating over self.__table__.Columns. I can get Foreign keys using:
inspector.get_foreign_keys(SomeClass.__tablename__)
But Now I need to access the relation objects defined in my classes and Introspect them.
There just does not seem to be a Class.__table__.relations, neither see I something in the documentation but I might overlook something, not sure…
Martijn
On Feb 28, 2011, at 18:21 , Michael Bayer wrote:
Michael Bayer
On Jan 30, 2012, at 9:49 AM, Martijn Moeling wrote:
> sorry I have to get back on this. I renamed all Columns in my application definitions to MyColumn a while back. and everything worked.
> Now that I'm starting to use functionality of MyColumn. (The reason I needed this) I run into some trouble.
so we can skip the easy "_constructor" thing and implement the rest of Column. If you read the first error message you get with a straight subclass:
TypeError: Could not create a copy of this <class '__main__.MySpecialColumn'> object. Ensure the class includes a _constructor() attribute or method which accepts the standard Column constructor arguments, or references the Column class itself. Original error: __init__() takes exactly 2 arguments (8 given)
So Column goes through a lot of trouble to try to diagnose what's going on. It's telling us to create a method called _constructor(), that accepts the standard arguments that Column does. The return value is our modified column:
class MySpecialColumn(Column):
x = 0
y = 0
def __init__(self, type_, **options):
filtered_options = {}
for name, option in options.items():
if hasattr(self, name):
setattr(self, name, option)
else:
filtered_options[name] = option
Column.__init__(self, type_, **filtered_options)
def _constructor(self, name, type_, **kw):
kw['x'] = self.x
kw['y'] = self.y
col = MySpecialColumn(type_, **kw)
col.name = name
return col
>
> At the other end, I might need to introspect the objects, I have tried but I have trouble in relating to X or Y since the Name Column is an InstrumentedAttribute.
Yeah to get at them directly you'd need to say:
assert MyClass.value.property.columns[0].x == 5
if you want MyClass.value.x == 5, you need to tack that on when it gets instrumented:
@event.listens_for(Base, 'attribute_instrument')
def configure_listener(class_, key, inst):
if isinstance(inst.property, ColumnProperty) and \
isinstance(inst.property.columns[0], MySpecialColumn):
inst.x = inst.property.columns[0].x
inst.y = inst.property.columns[0].y
>
>
> One other thing. I can get Columns by iterating over self.__table__.Columns. I can get Foreign keys using:
> inspector.get_foreign_keys(SomeClass.__tablename__)
I'd note those are not at all equivalent operations, in that inspector is going to go out to the database in order to get the FK information. Assuming your table metadata has it configured, you can get it locally by iterating through SomeClass.__table__.foreign_keys:
for fk in MyClass.__table__.foreign_keys:
print fk.parent, fk.column
if you want the full constraint object, which I'd recommend if you have composite FKs in use:
for const in MyClass.__table__.constraints:
if isinstance(const, ForeignKeyConstraint):
for element in const.elements:
print element.parent, element.column
>
> But Now I need to access the relation objects defined in my classes and Introspect them.
relationship is not the same as a foreign key, see below...
There's a ticket for 0.8 that would attempt to provide more accessors for these things, including things like mapper.relationships and stuff like that. Feel free to add in things we should consider on http://www.sqlalchemy.org/trac/ticket/2208.
an example with everything happening, including in a tricky from_self() query:
from sqlalchemy import create_engine, Column, Integer, ForeignKey, ForeignKeyConstraint
from sqlalchemy.orm import Session, configure_mappers, relationship
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import event
from sqlalchemy.orm.properties import ColumnProperty, RelationshipProperty
class MySpecialColumn(Column):
x = 0
y = 0
def __init__(self, type_, **options):
filtered_options = {}
for name, option in options.items():
if hasattr(self, name):
setattr(self, name, option)
else:
filtered_options[name] = option
Column.__init__(self, type_, **filtered_options)
def _constructor(self, name, type_, **kw):
kw['x'] = self.x
kw['y'] = self.y
col = MySpecialColumn(type_, **kw)
col.name = name
return col
Base= declarative_base()
@event.listens_for(Base, 'attribute_instrument')
def configure_listener(class_, key, inst):
if isinstance(inst.property, ColumnProperty) and \
isinstance(inst.property.columns[0], MySpecialColumn):
inst.x = inst.property.columns[0].x
inst.y = inst.property.columns[0].y
class MyClass(Base):
__tablename__ = "sometable"
id = Column(Integer, primary_key=True)
foo_id = Column(Integer, ForeignKey('foo.id'))
value = MySpecialColumn(Integer, nullable=False, x=5, y=12)
myfoo = relationship("MyFoo")
class MyFoo(Base):
__tablename__ = 'foo'
id = Column(Integer, primary_key=True)
assert MyClass.value.property.columns[0].x == 5
# configure mappers now to fire off
# the instrumentation event ahead
# of time, just so we can see value.x
configure_mappers()
assert MyClass.value.x == 5
# foreign keys
for fk in MyClass.__table__.foreign_keys:
print fk.parent, fk.column
for const in MyClass.__table__.constraints:
if isinstance(const, ForeignKeyConstraint):
for element in const.elements:
print element.parent, element.column
# relationships
for prop in MyClass.__mapper__.iterate_properties:
if isinstance(prop, RelationshipProperty):
print "relationship:", prop.key, prop.mapper.class_,
e = create_engine("sqlite://", echo=True)
Base.metadata.create_all(e)
s = Session(e)
s.add_all([
MyClass(value=12), MyClass(value=18)
])
s.commit()
# from_self() here creates a subquery and exercises the
# column copying
print s.query(MyClass).filter(
MyClass.value.between(MyClass.value.x, MyClass.value.y)
).from_self().all()
Martijn Moeling
Interesting stuff, The first part I had "almost" covered, I did not have the _constructor part.
It wil be part of something more complex…
Thankx,
Martijn
Martijn Moeling
The below example works, except if a Foreign key is given. On those columns -e.g. orderId = MySpecialColumn(Integer, ForeignKey('Order.Id'))-
To fix this I have added stuff, see comments in code below:
As long as the Non SA Column arguments are named it is OK.
On Jan 30, 2012, at 17:46 , Michael Bayer wrote:
>
> On Jan 30, 2012, at 9:49 AM, Martijn Moeling wrote:
>
>> sorry I have to get back on this. I renamed all Columns in my application definitions to MyColumn a while back. and everything worked.
>> Now that I'm starting to use functionality of MyColumn. (The reason I needed this) I run into some trouble.
>
>
> so we can skip the easy "_constructor" thing and implement the rest of Column. If you read the first error message you get with a straight subclass:
>
> TypeError: Could not create a copy of this <class '__main__.MySpecialColumn'> object. Ensure the class includes a _constructor() attribute or method which accepts the standard Column constructor arguments, or references the Column class itself. Original error: __init__() takes exactly 2 arguments (8 given)
>
>
> So Column goes through a lot of trouble to try to diagnose what's going on. It's telling us to create a method called _constructor(), that accepts the standard arguments that Column does. The return value is our modified column:
>
> class MySpecialColumn(Column):
> x = 0
> y = 0
> def __init__(self, type_, **options):
def __init__(self, type_, *arg, **options):
> filtered_options = {}
> for name, option in options.items():
> if hasattr(self, name):
> setattr(self, name, option)
> else:
> filtered_options[name] = option
>
> Column.__init__(self, type_, **filtered_options)
Column.__init__(self, type_, *arg, **filtered_options)
>
> def _constructor(self, name, type_, **kw):
def _constructor(self, name, type_,*arg, **kw):
> kw['x'] = self.x
> kw['y'] = self.y
> col = MySpecialColumn(type_, **kw)
col = MySpecialColumn(type_, *arg, **kw)