Michael Bayer wrote:
> The Column objects you declare within declarative become members of
> the underlying Table object, so its not as simple as just having
> those members present on a mixin - what would really be needed would
> be some sort of copying of each column object when declarative comes
> across them.
Indeed, how about a marker on "abstract" base classes like the Django ORM?
> The mechanism of declarative is also to look at the "dict" of only
> the endpoint class itself when picking up the attributes to add to
> the Table/mapper, so at the moment any mixin would be ignored in any
> case.
That's a shame, are there plans to change that approach and iterate over
dir() of the endpoint class instead?
> There is an alternative approach to the "place these columns on
> every class" problem, which involves making a subclass of the
> metaclass that adds the desired columns when a new class is created.
What about the cls parameter to declarative_base? Someone on #sqlalchemy
recommended that and linked to some working code:
http://progetti.arstecnica.it/sol/browser/sol/model/base.py
Is there a problem with having more than one Base in a project?
> I don't prefer this method since we're usually talking about just a
> few columns, the metaclass itself is slightly awkward, and at the
> end of the day I do like to see the columns listed out on each class
> (hence "declarative"...).
Yeah, but duplicate typing is always bad... That's why python allows
inheritance...
> def created_at():
> return Column(DateTime, default=func.now)
>
> so you only need to add a little bit to each class:
>
> class Foo(Base):
> ...
>
> created_at = created_at()
> updated_at = updated_at()
Yeah, I've used this approach too, but when it's several columns, and a
few utility methods, a class really seems like the right (dare I say
pythonic?) place for it to be?
How about something like:
class Foo(Base):
__abstract__ = True
...
...which would provide the desired functionality here?
> The "metaclass subclass" approach is somewhere in the list archives
> if you want to search for it.
Advising end users to write metaclasses is always bad in my books :-(
(In fact, this all came up 'cos a colleague of mine was reaching for one
of our python books, I asked why, she replied "to remind myself how
metaclasses work", to which my reply was "you *never* want to do that" ;-) )
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
I took the approach with creating a Mixin with 4 staticmethods and
call them in each of my models, which works quite good. But now i face
another, i think related, problem.
i get the following columns produced:
created_by_id = Column(Integer, ForeignKey('worker.id',
onupdate="cascade", ondelete="restrict"))
updated_by_id = Column(Integer, ForeignKey('worker.id',
onupdate="cascade", ondelete="restrict"))
created_by = relation('worker')
updated_by = relation('worker')
Now sqlalchemy is throwing an exception, because the join-conditions
are not clear. Ok, so i have to add a primaryjoin-expression on the
relations. But how?
They should be specified like relation('worker',
primaryjoin=worker.c.id==???.c.created_by_id), but in my Mixin-Class i
dont know what ??? is, because it changes depending on the class using
the mixin. I could specify a parameter for these two static methods,
but thats another step making things more complicated, so i want to
avoid that if possible. Any advice?
Thanks!
Daishy
> (sorry, missed this earlier)
>
> Michael Bayer wrote:
>> The Column objects you declare within declarative become members of
>> the underlying Table object, so its not as simple as just having
>> those members present on a mixin - what would really be needed would
>> be some sort of copying of each column object when declarative comes
>> across them.
>
> Indeed, how about a marker on "abstract" base classes like the Django ORM?
im sorry, did you say you were feeling slightly sick earlier ? :)
>
>> The mechanism of declarative is also to look at the "dict" of only
>> the endpoint class itself when picking up the attributes to add to
>> the Table/mapper, so at the moment any mixin would be ignored in any
>> case.
>
> That's a shame, are there plans to change that approach and iterate over
> dir() of the endpoint class instead?
probably not. I can see that causing more problems than it solves.
I noticed you cut out the part of my reply with the likely fix here, to provide a special class that is specifically for declarative mixins. It would only allow elements on it that make sense for copying to many subclasses.
> Hi again,
>
> I took the approach with creating a Mixin with 4 staticmethods and
> call them in each of my models, which works quite good. But now i face
> another, i think related, problem.
> i get the following columns produced:
>
> created_by_id = Column(Integer, ForeignKey('worker.id',
> onupdate="cascade", ondelete="restrict"))
> updated_by_id = Column(Integer, ForeignKey('worker.id',
> onupdate="cascade", ondelete="restrict"))
> created_by = relation('worker')
> updated_by = relation('worker')
>
> Now sqlalchemy is throwing an exception, because the join-conditions
> are not clear. Ok, so i have to add a primaryjoin-expression on the
> relations. But how?
> They should be specified like relation('worker',
> primaryjoin=worker.c.id==???.c.created_by_id), but in my Mixin-Class i
> dont know what ??? is, because it changes depending on the class using
> the mixin. I could specify a parameter for these two static methods,
> but thats another step making things more complicated, so i want to
> avoid that if possible. Any advice?
right. see why I recommend just placing the Column objects on each class explicitly ?
* If I setup 2 sessions, that means I will have 2 connections to the database, meaning the database is the one that will handle threading conflicts per what is described here: http://www.postgresql.org/docs/8.4/static/mvcc.html
So, I have 1 app with multiple processes and threads. Each thread and/or process may end up trying to do something to the database at the same time. What is the solution to threading? How do web frameworks solve it? Is there some inherent design in databases and/or SQLAlchemy that makes this not a actual problem? What do I do if I have 2 separate apps? 3?
I really don't like relying on that.
However, the SQLAlchemy docs suggest making many small Sessions for short interaction with the database. The problem is if I have many threads/processes accessing the DB simultaneously (through separate sessions\connections), then I could get into threading problems working with the database, right?
Is the solution to use one single session for the entire application (the application consisting of many processes and threads) and making all my processes communicate with this single session when it needs to do database stuff?
Somehow, I have my doubts that it how it should be done -- and even the SQLAlchemy docs say otherwise (although again, very vague about what it is talking about).
--
it really is that - simultaneous transactions are isolated from one another, and their results are only made visible to other transactions after they're committed. as far as locking, you generally choose between an "optimistic" and a "pessimistic" approach:
--
I forgot about the really obvious way to do this - class decorators:
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
def common_columns(cls):
cls.created_at = Column(DateTime, default=func.now)
cls.udpated_at = Column(DateTime, default=func.now, onupdate=func.now)
return cls
@common_columns
class MyClass(Base):
__tablename__ = 'mytable'
id = Column(Integer, primary_key=True)
data = Column(String)
@common_columns
class MyOtherClass(Base):
__tablename__ = 'myothertable'
id = Column(Integer, primary_key=True)
data = Column(String)
engine = create_engine('sqlite://', echo=True)
Base.metadata.create_all(engine)
An abstract marker on a base class to say "actually, this is just here
to provide information and methods, it's not table-mapped class" isn't
such a bad thing, is it? Especially given how horrible the resulting
code is...
> I noticed you cut out the part of my reply with the likely fix here, to provide a special class that is specifically for declarative mixins. It would only allow elements on it that make sense for copying to many subclasses.
I did indeed miss this, but I don't see how it's fundamentally different
from marking a base class as abstract ;-)
Anyway, here's the thought process that got us where we are (which feels
a bit icky to me), any ways to make things "nicer" would be very
gratefully received.
Okay, so, the naive approach that a normal python user might expect to
work would be:
Base = declarative_base()
class VersionedBase(Base):
id = Column(Integer, primary_key=True)
version = Column(Integer, nullable=False, index=True)
a_constant = ('some','tuple')
def aMethod(self):
do_some_stuff()
class Employee(VersionedBase):
__tablename__ = 'employee'
name = Column(String, nullable=False, index=True)
...but this results in:
sqlalchemy.exc.InvalidRequestError: Class <class
'__main__.VersionedBase'> does not have a __table__ or __tablename__
specified and does not inherit from an existing table-mapped class.
Fair enough, so why not use multi-table inheritance? Because we really
want those columns in each table:
- no overhead of joins when selecting a particular set of versions
- ability to bypass the orm layer more easilly if emergency debugging is
needed
Okay, so why not concrete or single table inheritance? Because we
*really* want one table per subclass but we don't ever want to select
from more than one table at once...
So, moving on, we tried:
Base = declarative_base()
class VersionedBase(object):
id = Column(Integer, primary_key=True)
version = Column(Integer, nullable=False, index=True)
a_constant = ('some','tuple')
def aMethod(self):
do_some_stuff(self.a_constant)
class Employee(Base, VersionedBase):
__tablename__ = 'employee'
name = Column(String, nullable=False, index=True)
...which gave a more cryptic error:
sqlalchemy.exc.ArgumentError: Mapper Mapper|Employee|employee could not
assemble any primary key columns for mapped table 'employee'.
I'm guessing this is because VersionedBase is basically ignored by the
declarative metaclass?
Okay, so metaclass time... First attempt failed:
def aMethod(self):
do_some_stuff(self.a_constant)
class VersionedMeta(DeclarativeMeta):
def __init__(cls, name, bases, dict_):
cls.id = Column(Integer, primary_key=True)
cls.version = Column(Integer, nullable=False, index=True)
cls.a_constant = ('some','tuple')
cls.aMethod = aMethod
return DeclarativeMeta.__init__(cls, name, bases, dict_)
...because DeclarativeMeta's __init__ ignores columns already set on cls
by the time it's called, which is fair enough, but means we end up with
a (working) metaclass method that's a bit ugly:
def aMethod(self):
do_some_stuff(self.a_constant)
class VersionedMeta(DeclarativeMeta):
def __init__(cls, name, bases, dict_):
dict_.update(dict(
id = Column(Integer, primary_key=True),
version = Column(Integer, nullable=False, index=True),
))
cls.a_constant = ('some','tuple')
cls.aMethod = aMethod
return DeclarativeMeta.__init__(cls, name, bases, dict_)
def versioned_declarative_base():
return declarative_base(metaclass=VersionedMeta)
Base = version_declarative_base()
class Employee(Base):
__tablename__ = 'employee'
name = Column(String, nullable=False, index=True)
Is it okay to have multiple declarative bases floating around like this?
Will they all end up using the same metadata collection or are we in for
problems down the line?
So anyway, on to your next suggestion: class decorator...
def aMethod(self):
do_some_stuff(self.a_constant)
def versioned(cls):
cls.id = Column(Integer, primary_key=True),
cls.version = Column(Integer, nullable=False, index=True),
cls.a_constant = ('some','tuple')
cls.aMethod = aMethod
@versioned
class Employee(Base):
__tablename__ = 'employee'
name = Column(String, nullable=False, index=True)
Much nicer! Except... we're on Python 2.5, so no class decorators. No
problems, thinks I, we'll just do it the old fashioned way:
class Employee(Base):
__tablename__ = 'employee'
name = Column(String, nullable=False, index=True)
Employee = versioned(Employee)
Uh oh:
sqlalchemy.exc.ArgumentError: Mapper Mapper|Employee|employee could not
assemble any primary key columns for mapped table 'employee'
...which makes me wonder if the class decorator would actually work at
all. Surely it'll only kick in after the DeclarativeMeta has already
done its thing and got upset?
Ideas welcome...
Chris
its not. I was still recovering from seeing the word "django" :)
> Uh oh:
>
> sqlalchemy.exc.ArgumentError: Mapper Mapper|Employee|employee could not
> assemble any primary key columns for mapped table 'employee'
>
> ...which makes me wonder if the class decorator would actually work at
> all. Surely it'll only kick in after the DeclarativeMeta has already
> done its thing and got upset?
the mapper is mostly configured by the time the class is done, and the
primary key columns are required to be there. The overwhelming use case
we're talking about here are columns like "created_at", "updated_at",
etc., not primary keys.
But agreed, make your mixin implement a marker class from the declarative
module (MixinsDeclarative, i dunno), and as_declarative() will issue
copy() for the Column objects within. It will raise errors for relation()
since it doesn't make sense to copy those.
Hmm, well, the primary key column is, in this case, one that wants to be
shared across all classes...
> But agreed, make your mixin implement a marker class from the declarative
> module (MixinsDeclarative, i dunno),
How about just a marker attribute? How about something in
__mapper__args__, which already exists?
class MyMixin:
__mapper_args__ = {'abstract':True}
> and as_declarative() will issue
> copy() for the Column objects within. It will raise errors for relation()
> since it doesn't make sense to copy those.
I assume this is a feature waiting to be implemented?
What to do in the meantime? The class decorator felt really quite neat
except that it won't work where primary key columns are added by the
decorator, right?
well mapper_args is args that go to the mapper. Its a little unclean
IMHO to reuse that same dict for declarative purposes. Id rather have
another attribute like __declarative_abstract__ = True or something like
that (ideas welcome).
>
> I assume this is a feature waiting to be implemented?
>
> What to do in the meantime? The class decorator felt really quite neat
> except that it won't work where primary key columns are added by the
> decorator, right?
yeah the class decorator can't really handle the PKs. the mapper wants
those when it starts up. as far as implementation this feature isn't on
my personal radar, so if you really want anytime soon you should provide a
patch, which I can of course help with.
>
> Chris
>
> --
> Simplistix - Content Management, Batch Processing & Python Consulting
> - http://www.simplistix.co.uk
>
__declaratice_abstract__ = True works for me :-)
>> What to do in the meantime? The class decorator felt really quite neat
>> except that it won't work where primary key columns are added by the
>> decorator, right?
>
> yeah the class decorator can't really handle the PKs. the mapper wants
> those when it starts up.
Metaclass it is then. I asked a while back but I think it got lost in
the melee: is it okay to have several declarative bases knocking around
or will they each end up with their own metadata collection causing
problems when models from one have relations to models from another?
> as far as implementation this feature isn't on
> my personal radar, so if you really want anytime soon you should provide a
> patch, which I can of course help with.
Cool, it is on my radar, but sadly pretty far away :-(
you know, doing the "metaclass" approach is probably the first part of doing the "demarcated mixin" approach.
here it is: http://www.sqlalchemy.org/trac/wiki/UsageRecipes/DeclarativeMixins
Indeed, it'd just be great if the default declarative base mixin handled
this demarcation package already ;)
I argue about the comment on relations on that page, if the relation is
identical, it might well make sense to abstract it out and put it in the
mixin...
maybe someday. not until users actually use the recipes first and provide
feedback.
>
> I argue about the comment on relations on that page, if the relation is
> identical, it might well make sense to abstract it out and put it in the
> mixin...
just another complicated and error prone core function I'd rather not
support.
>
> Chris
>
> --
> Simplistix - Content Management, Batch Processing & Python Consulting
> - http://www.simplistix.co.uk
>