automating inheritance

[lars van gemerden]

I am trying to my my joined inheritance code clearer, for the dynamic
generation of sa classes and tried to do something like this:


class InheritMixin(object):

@declared_attr
def __tablename__(cls):
return cls.__name__
@declared_attr
def id(cls):
if cls.__name__ == 'Object':
return Column(Integer, primary_key = True)
else:
print 'in id: ', cls.__name__, cls.__bases__[0].__name__
return Column(Integer,
ForeignKey(cls.__bases__[0].__name__ + '.id'), primary_key = True)
@declared_attr
def __mapper_args__(cls):
if cls.__name__ == 'Object':
return {'polymorphic_on': 'discriminator'}
else:
print 'in mapper_args: ', cls.__name__,
cls.__bases__[0].__name__
return {'polymorphic_identity': cls.__name__,
'inherit_condition': (cls.id ==
cls.__bases__[0].id)}

Object = type('Object', (Base, InheritMixin), clsdict)

Where Object should be the (not necessarily direct) baseclass of all
inheriting classes. However I get errors: "Mapper Mapper|person|person
could not assemble any primary key columns for mapped table 'Join
object on Object(65389120) and person(65428224)' " etc ..

I noticed that the method __mapper_args__(cls) is always called before
id(cls) (which is never called, probably due to the error.

Is there some way to fix this, while keeping the inheritance code in a
mixin?

Also, is there a way to add the discriminator column to the mixin (if
i just directly add it to the declaration, this gave another maybe
related error)?

Cheers, Lars

[Michael Bayer]

im not sure of the cause of that error, can you attach a full test case which illustrates this message being generated ?

>
> I noticed that the method __mapper_args__(cls) is always called before
> id(cls) (which is never called, probably due to the error.

the __mapper_args__(cls) method here directly calls upon .id, so if you see .id() not being called it suggests some other form of .id is being used.

Is it possible that Base or something else has a conflicting "id" attribute?

>
> Also, is there a way to add the discriminator column to the mixin (if
> i just directly add it to the declaration, this gave another maybe
> related error)?

maybe, let's start with the general idea of the mixin you're going to send me as a working script.

[lars van gemerden]

this is the testcase:

from sqlalchemy import *
from sqlalchemy.orm.session import sessionmaker
from sqlalchemy.ext.declarative import declarative_base, declared_attr

engine = create_engine('sqlite:///:memory:', echo=False)
Base = declarative_base(bind = engine)
Session = sessionmaker(bind = engine)

def setup(engine):
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
return Session()

class InheritMixin(object):

@declared_attr
def __tablename__(cls):
return cls.__name__
@declared_attr
def id(cls):

if Base in cls.__bases__:
print 'base in id(cls): ', cls

return Column(Integer, primary_key = True)
else:

print 'in id(cls): ', cls, cls.__bases__[0]
return Column(Integer, ForeignKey(cls.__bases__[0].id),

primary_key = True)
@declared_attr
def __mapper_args__(cls):

if Base in cls.__bases__:
print 'base in __mapper_args__(cls): ', cls
return {'polymorphic_on': 'discriminator'}
else:
print 'in __mapper_args__(cls): ', cls, cls.__bases__[0],
cls.id, (cls.id is cls.__bases__[0].id)
return {'polymorphic_identity': cls.__name__}

class Person(Base, InheritMixin):
discriminator = Column(String(50))
name = Column(String(50))

class Engineer(Person):
job = Column(String(50))

if __name__ == '__main__':

session = setup(engine)
a = Person(name = 'ann')
b = Engineer(name = 'bob', job = 'car repair')
session.add_all([a, b])
session.commit()
people = session.query(Person).all()
print people

Note that i left out the 'inherit_condition', because without there is
already a problem:

base in id(cls): <class '__main__.Person'>
base in mapper_args(cls): <class '__main__.Person'>
in mapper_args(cls): <class '__main__.Engineer'> <class
'__main__.Person'> Person.id True
Traceback (most recent call last):
File "D:\Documents\Code\Eclipse\workspace\process_data3\src
\little_tests2.py", line 40, in <module>
class Engineer(Person):
File "C:\Python27\lib\site-packages\sqlalchemy\ext\declarative.py",
line 1336, in __init__
_as_declarative(cls, classname, cls.__dict__)
File "C:\Python27\lib\site-packages\sqlalchemy\ext\declarative.py",
line 1329, in _as_declarative
**mapper_args)
File "C:\Python27\lib\site-packages\sqlalchemy\orm\__init__.py",
line 1116, in mapper
return Mapper(class_, local_table, *args, **params)
File "C:\Python27\lib\site-packages\sqlalchemy\orm\mapper.py", line
197, in __init__
self._configure_inheritance()
File "C:\Python27\lib\site-packages\sqlalchemy\orm\mapper.py", line
473, in _configure_inheritance
self.local_table)
File "C:\Python27\lib\site-packages\sqlalchemy\sql\util.py", line
303, in join_condition
"between '%s' and '%s'.%s" % (a.description, b.description, hint))
sqlalchemy.exc.ArgumentError: Can't find any foreign key relationships
between 'Person' and 'Engineer'.

What am i missing?

Cheers, Lars

[Michael Bayer]

On Apr 20, 2012, at 4:59 AM, lars van gemerden wrote:

> this is the testcase:
>
>
> What am i missing?

the issue here is one of Python inheritance mechanics. Declarative calls upon @declared_attr in terms of the class, that is, we look through the class to find each @declared_attr, but when we find one, we invoke it by just calling it as a method, that is, getattr(cls, name). This works for things like __mapper_args__ which remain as callable methods on classes like Person, Engineer. But "id", when that is declared on Person is immediately replaced with a mapping. By the time you get to Engineer, the id() method is gone.

So for inheriting cases you need to build a mixin that is applied to every subclass. This makes sense because a mixin with a column on it implies that the column is being associated only with that immediate class - if you wanted a subclass to act as single table inheritance, you'd omit this class. In this case you want the same column on all subclasses. So you can do it like this (note also using declarative.has_inherited_table helper):

class InheritMixin(object):

@declared_attr
def __tablename__(cls):
return cls.__name__

@declared_attr
def id(cls):

return Column(Integer, primary_key = True)

@declared_attr
def __mapper_args__(cls):
if not has_inherited_table(cls):
return {'polymorphic_on': 'discriminator'}
else:
return {'polymorphic_identity': cls.__name__}

class Inherits(InheritMixin):
@declared_attr
def id(cls):
super_id = super(Inherits, cls).id
return Column(Integer, ForeignKey(super_id),primary_key = True)

class Person(InheritMixin, Base):

discriminator = Column(String(50))
name = Column(String(50))

class Engineer(Inherits, Person):
job = Column(String(50))

this should be in the docs so I've added ticket #2471 to handle this.

> --
> You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
> To post to this group, send email to sqlal...@googlegroups.com.
> To unsubscribe from this group, send email to sqlalchemy+...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
>

[lars van gemerden]

Ok, thank you, that helps, but now i cannot inherit from Engineer, as in:

class BaseMixin(object):

    discriminator = Column(String(50))

    @declared_attr

    def __tablename__(cls):

        return cls.__name__

    @declared_attr

    def id(cls):

        return Column(Integer, primary_key = True)

    @declared_attr

    def __mapper_args__(cls):

        if not has_inherited_table(cls):

            return {'polymorphic_on': 'discriminator'}

        else:

            return {'polymorphic_identity': cls.__name__}

class InheritMixin(BaseMixin):

    @declared_attr

    def id(cls):

        super_id = super(InheritMixin, cls).id

        return Column(Integer, ForeignKey(super_id), primary_key = True)

    

class Person(BaseMixin, Base):

    name = Column(String(50))

   

class Engineer(InheritMixin, Person):

    job = Column(String(50))

class MasterEngineer(InheritMixin, Engineer):

    specialty = Column(String(50))

Gives an MRO() error and if i would reverse the baseclasses (like class Engineer(Person, InheritMixin):  ... ), the inheriting classes pick up the wrong id.

Do you see any solution for this? 

BTW: could i just move

    @declared_attr

    def __mapper_args__(cls):

        return {'polymorphic_identity': cls.__name__}

to InheritMixin instead of doing the 'has_inherited_table' if-statement in BaseMixin?

Cheers, Lars

> To unsubscribe from this group, send email to sqlalchemy+unsubscribe@googlegroups.com.

[lars van gemerden]

Hi Mike,

How about this approach with a custom metaclass; so far it works and seems the cleanest to me:

from sqlalchemy import *

from sqlalchemy.orm.session import sessionmaker

from sqlalchemy.ext.declarative import declarative_base, has_inherited_table, DeclarativeMeta

engine = create_engine('sqlite:///:memory:', echo=False)

class InheritMeta(DeclarativeMeta):

    

    def __init__(cls, name, bases, clsdict):

        cls.__tablename__ = cls.__name__

        if not has_inherited_table(cls):

            cls.id = Column(Integer, primary_key = True)

            cls.discriminator = Column(String(50))

            cls.__mapper_args__ = {'polymorphic_on': 'discriminator'}

        else:

            cls.id = Column(Integer, ForeignKey(bases[0].id), primary_key = True)

            cls.__mapper_args__ = {'polymorphic_identity': cls.__name__}

        super(InheritMeta, cls).__init__(name, bases, clsdict)

InheritableBase = declarative_base(bind = engine, metaclass = InheritMeta)

class Person(InheritableBase):

    name = Column(String(50))

   

class Engineer(Person):

    job = Column(String(50))

class MasterEngineer(Engineer):

    specialty = Column(String(50))

    

if __name__ == '__main__':

    InheritableBase.metadata.create_all(engine)

    session = sessionmaker(bind=engine)()

    

    a = Person(name = 'ann')

    b = Engineer(name = 'bob', job = 'car repair')

    c = MasterEngineer(name = 'carl', job = 'car repair', specialty = 'tires')

    session.add_all([a, b, c])

    session.commit()

    people = session.query(Person).all()

    print people

Do you see any drawbacks, gotchas for later on?

Regards, Lars

On Friday, April 20, 2012 12:41:28 PM UTC+2, Michael Bayer wrote:

> To unsubscribe from this group, send email to sqlalchemy+unsubscribe@googlegroups.com.

[Michael Bayer]

yeah I suppose if you're building out joined inheritance more than one level then this becomes awkward. I never use joined inh more than one level because it has too much of an impact on queries.

the metaclass as you mention is always the last resort when the various declarative trickery reaches its limit. I'm not thrilled about the metaclass approach because it quickly gets confusing and shouldn't be necessary. though in this case without some extra mechanism on declarative, such as a __pre_declare__() method of some kind, it might be the only approach.

[lars van gemerden]

Ok, so speed might become an issue for me as well; 

Do you think a similar metaclass approach would work for concrete inheritance would work without major drawbacks (before i do a major overhaul)?

Is there any indication about how much faster concrete inheritance is, compared to joined inheritance?

Cheers, Lars

[Michael Bayer]

concrete inheritance is very challenging overall, if you expect there to be any kind of polymorphic interaction between the classes.     if you want to query polymorphically then speed will be probably worse.   If you can do without polymorphic and stick to each subclass directly it wont have an issue.

Usually though if I'm inheriting more than two levels, I'll use joined inheritance for level one->two then single table for all levels beyond that.   Depends on what you're trying to do.

--
You received this message because you are subscribed to the Google Groups "sqlalchemy" group.

To view this discussion on the web visit https://groups.google.com/d/msg/sqlalchemy/-/MH4tZazKT0EJ.

To post to this group, send email to sqlal...@googlegroups.com.

To unsubscribe from this group, send email to sqlalchemy+...@googlegroups.com.

[lars van gemerden]

Well thats the thing, my users will determine the data structure
(graphically) and it is hard to predict what they will come up with.
On the other hand, I am only building a prototype at the moment, so
speed issues (if not easily solved) will have to wait.

I'll stick with joined inheritance for now (though I'll probalby take
out the unique base class for all classes).

Thank you again for all the help,

Lars

> > To view this discussion on the web visithttps://groups.google.com/d/msg/sqlalchemy/-/MH4tZazKT0EJ.

[Michael Bayer]

building SQLAlchemy apps graphically, that's extremely challenging good luck !

[Gerald Thibault]

The code referenced in the above post:

@declared_attr
   def __mapper_args__(cls):
       if not has_inherited_table(cls):
           return {'polymorphic_on': 'discriminator'}
       else:
           return {'polymorphic_identity': cls.__name__}

 has broken between 0.7.9 and 0.8.0, now that __mapper_args__ is a lambda. The call to has_inherited_table will always return true because mapper_args are evaluated after everything is configured, so the first class in cls__mro__ (which will be class) will now have a table attached to it, while in 0.7.9, the mapper_args would be evaluated during as_declarative and the table would not be present.

Is there a new way to accomplish the same thing?

[Michael Bayer]

the script below is my best guess what we're talking about, it's my original test from April 20 2012, and includes an assertion that both Person and Engineer get their own tables.  The script runs identically in 0.7 and 0.8, and __mapper_args__ is called in the same way in both versions.  If you could provide a complete and specific test script that refers to the behavior you're talking about that would be helpful.

from sqlalchemy import *

from sqlalchemy.orm import *

from sqlalchemy.ext.declarative import declarative_base, declared_attr, has_inherited_table

Base = declarative_base()

class InheritMixin(object):

    @declared_attr

    def __tablename__(cls):

        return cls.__name__

    @declared_attr

    def id(cls):

        return Column(Integer, primary_key = True)

    @declared_attr

    def __mapper_args__(cls):

        if not has_inherited_table(cls):

            return {'polymorphic_on': 'discriminator'}

        else:

            return {'polymorphic_identity': cls.__name__}

class Inherits(InheritMixin):

    @declared_attr

    def id(cls):

        super_id = super(Inherits, cls).id

        return Column(Integer, ForeignKey(super_id), primary_key = True)

class Person(InheritMixin, Base):

    discriminator = Column(String(50))

    name = Column(String(50))

class Engineer(Inherits, Person):

    job = Column(String(50))

assert Engineer.__table__.name == 'Engineer'

assert Person.__table__.name == 'Person'

To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+...@googlegroups.com.

To post to this group, send email to sqlal...@googlegroups.com.

Visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Gerald Thibault]

Here is a test. It's possible the failure is because the provided example uses a mixin, while i am declaring directly on the base class.

from sqlalchemy import *

from sqlalchemy.ext.declarative import (declarative_base,

    declared_attr, has_inherited_table)

Base = declarative_base()

class Test(Base):

    __tablename__ = 'test'

    id = Column(Integer, primary_key=True)

    type = Column(String(20))

    @declared_attr

    def __mapper_args__(cls):

        if not has_inherited_table(cls):

            ret =  {

                'polymorphic_identity': 'default',

                'polymorphic_on': cls.type,

                }

        else:

            ret = {'polymorphic_identity': cls.__name__}

        print '%s.__mapper_args__:' % cls

        print ret

        return ret

        

class PolyTest(Test):

    __tablename__ = 'poly_test'

    id = Column(Integer, ForeignKey(Test.id), primary_key=True)

Running this under 0.7.9 produces:

<class '__main__.Test'>.__mapper_args__:

{'polymorphic_identity': 'default', 'polymorphic_on': Column(None, String(length=20), table=None)}

<class '__main__.Test'>.__mapper_args__:

{'polymorphic_identity': 'default', 'polymorphic_on': Column(None, String(length=20), table=None)}

<class '__main__.PolyTest'>.__mapper_args__:

{'polymorphic_identity': 'PolyTest'}

Running under 0.8.0 produces:

<class '__main__.Test'>.__mapper_args__:

{'polymorphic_identity': 'Test'}

<class '__main__.PolyTest'>.__mapper_args__:

{'polymorphic_identity': 'PolyTest'}

in 7.9, the polymorphic_on was set correctly, in 8.0 the table already exists when the mapper_args are processed (because it's a lambda now), so has_inherited_table returns True. Is it intended behavior to have has_inherited_table return true when the only "inherited" table is that on the class itself? I am working around this by using my own has_inherited_table method, which first checks if the base is equal to the active class, and skipping it if they're the same. It seems this should be default behavior, unless i am misunderstanding what 'inherited' is representing here.

Is there a way to accomplish what I'm trying to do without setting up my own has_inherited_table function?

[Michael Bayer]

thanks, now I know what you mean by "the class will now have a table attached to it".  This behavior is a side effect of the issue raised in http://www.sqlalchemy.org/trac/ticket/2656 (which also applies to 0.7), but since __mapper_args__() is evaluated later, this turns it into more or less of a regression, issue is fixed in r4ddeed7a3248.

[Gerald Thibault]

Awesome, thanks a ton for the update and fix.