SQLAlchemy : declarative_base and metaclass conflict

[Sven Dumay]

Hello,

I am actually working on an existing project (an online-game) which is already advanced and pretty consequent.

My goal is to replace the save system actually based on Pickle by SQLAlchemy. Not so easy because I have to deal with the existing classes and there is a lot of work to do (about 400 classes to persist).

I'm not sure to know what is the best way to proceed and I think that I require some help.

Let's look at the classes organization of the project :

Every class which should be persistent has to be inherited from Stockable. It is already designed this way and I think that it would be too complicated to change that. Below Stockable, there is hundred of classes with their own hierarchy. For example, Character is inherited from Stockable and Player and NPC (Non-player Character) are inherited from Character.

My problem today is that I don't know how to proceed regarding the metaclass "MetaBase". I am not able to use declarative_base() and MetaBase at the same time. There is a metabase conflict. I found some other topics about this problem on the internet and I tried several solutions, but still, it never works for me. 

To resume, here is how it basically works without SQLAlchemy :

class MetaBase(type):

    def __init__(cls, nom, bases, contenu):
        type.__init__(cls, nom, bases, contenu)
        pass

class Stockable(metaclass = MetaBase):

    def __init__(self):
        pass


class Character(Stockable):
 
    def __init__(self):
        pass

Here is what I would like to do with SQLAlchemy:

from sqlalchemy.ext.declarative import declarative_base, DeclarativeMeta
from sqlalchemy import Column, Integer

Base = declarative_base()

class MetaBase(DeclarativeMeta):

    def __init__(cls, nom, bases, contenu):
        super(MetaBase, cls).__init__(nom, bases, contenu)
        print("Init MetaBase")


class Stockable(metaclass = MetaBase):

    def __init__(self):
        print("Init Stockable")

class Character(Stockable, Base):

    __tablename__ = 'characters'
    id = Column(Integer, primary_key=True)
 
    def __init__(self, name):
        self.name = name
        print("Init character")


jean = Character("Jean")
print(jean.name)

Here is what I get :

>>> 

Traceback (most recent call last):

  File "C:\Users\Sven\Desktop\SQL Alchemy Tests\test2.py", line 10, in <module>

    class Stockable(metaclass = MetaBase):

  File "C:\Users\Sven\Desktop\SQL Alchemy Tests\test2.py", line 7, in __init__

    super(MetaBase, cls).__init__(nom, bases, contenu)

  File "C:\Python34\lib\site-packages\sqlalchemy\ext\declarative\api.py", line 64, in __init__

    _as_declarative(cls, classname, cls.__dict__)

  File "C:\Python34\lib\site-packages\sqlalchemy\ext\declarative\base.py", line 88, in _as_declarative

    _MapperConfig.setup_mapping(cls, classname, dict_)

  File "C:\Python34\lib\site-packages\sqlalchemy\ext\declarative\base.py", line 103, in setup_mapping

    cfg_cls(cls_, classname, dict_)

  File "C:\Python34\lib\site-packages\sqlalchemy\ext\declarative\base.py", line 125, in __init__

    clsregistry.add_class(self.classname, self.cls)

  File "C:\Python34\lib\site-packages\sqlalchemy\ext\declarative\clsregistry.py", line 34, in add_class

    if classname in cls._decl_class_registry:

AttributeError: type object 'Stockable' has no attribute '_decl_class_registry'

>>>

Does someone knows what it means and how it can be resolved ?

I tried other things and I found the following solution :

from sqlalchemy.ext.declarative import declarative_base, DeclarativeMeta
from sqlalchemy import Column, Integer

class MetaBase(DeclarativeMeta):

    def __init__(cls, nom, bases, contenu):
        super(MetaBase, cls).__init__(nom, bases, contenu)
        print("Init MetaBase")

Base = declarative_base(metaclass = MetaBase)

class Stockable(Base):

    __abstract__ = True

    def __init__(self):
        print("Init Stockable")

class Character(Stockable):

    __tablename__ = 'characters'
    id = Column(Integer, primary_key=True)
 
    def __init__(self, name):
        self.name = name
        print("Init character")


jean = Character("Jean")
print(jean.name)

It seems to work. I get the following result :

>>> 

Init MetaBase

Init MetaBase

Init MetaBase

Init compte

Jean

>>>

However, the problem with this method is that I have to add "__abstract__ = True" to every class which is inherited by Stockable... so, about 400 classes. It is not very clean. Is it possible to avoid that by using something similar to my first code ? It would be great !

Thank you very much.

Sven

[Sven Dumay]

Version of Python : 3.4.0 

Version of SQLAlchemy : 1.2.0b2

[Mike Bayer]

On Fri, Oct 20, 2017 at 11:55 AM, Sven Dumay <sven....@gmail.com> wrote:

Hello,

I am actually working on an existing project (an online-game) which is already advanced and pretty consequent.

My goal is to replace the save system actually based on Pickle by SQLAlchemy. Not so easy because I have to deal with the existing classes and there is a lot of work to do (about 400 classes to persist).

I'm not sure to know what is the best way to proceed and I think that I require some help.

Let's look at the classes organization of the project :

Every class which should be persistent has to be inherited from Stockable. It is already designed this way and I think that it would be too complicated to change that. Below Stockable, there is hundred of classes with their own hierarchy. For example, Character is inherited from Stockable and Player and NPC (Non-player Character) are inherited from Character.

this is going to work very poorly.   SQLAlchemy supports class inheritance mapped to databases but it's not really something that scales well to deep hierarchies.     You can actually stretch this a lot by emphasizing single-table inheritance so that you aren't hobbled with dozens of joins, but this seems like it is still a very deep hierarchy even for that approach.

What you need to do here is forget about your whole class hierarchy, and first design the database schema.   You want to persist this data in a relational database.   How?   What do the tables look like?   For any non-trivial application, this is where you need to design things from.  

you need to pass your metaclass to declarative_base() so that the _decl_class_registry and other things are established:

http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative/api.html?highlight=declarative_base#sqlalchemy.ext.declarative.declarative_base.params.metaclass

 Stockable then just descends from Base normally.

--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
 
http://www.sqlalchemy.org/
 
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description.
---
You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscribe@googlegroups.com.
To post to this group, send email to sqlal...@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[Mike Bayer]

CONTINUING !  sorry

On Fri, Oct 20, 2017 at 11:55 AM, Sven Dumay <sven....@gmail.com> wrote:

I tried other things and I found the following solution :

from sqlalchemy.ext.declarative import declarative_base, DeclarativeMeta
from sqlalchemy import Column, Integer

class MetaBase(DeclarativeMeta):

    def __init__(cls, nom, bases, contenu):
        super(MetaBase, cls).__init__(nom, bases, contenu)
        print("Init MetaBase")

Base = declarative_base(metaclass = MetaBase)

class Stockable(Base):

    __abstract__ = True

    def __init__(self):
        print("Init Stockable")

class Character(Stockable):

    __tablename__ = 'characters'
    id = Column(Integer, primary_key=True)
 
    def __init__(self, name):
        self.name = name
        print("Init character")


jean = Character("Jean")
print(jean.name)

this seems like roughly the correct approach.

 

It seems to work. I get the following result :

>>> 

Init MetaBase

Init MetaBase

Init MetaBase

Init compte

Jean

>>>

However, the problem with this method is that I have to add "__abstract__ = True" to every class which is inherited by Stockable... so, about 400 classes.

I don't see why that is.  If these classes are mapped to tables (which, if they are persisted, they are), then there is no reason to add "__abstract__".        As in my previous email, how these 400 classes link to tables is what needs to be answered and then we can formulate the correct calling style.

 

It is not very clean. Is it possible to avoid that by using something similar to my first code ? It would be great !

Thank you very much.

Sven

[Sven]

Thank you for your answer.

I'm sorry, I have not been clear enough regarding the "__abstract__ = True".

I suppose that I will have to add this to a lot of classes for the following reasons :

1) because it will allow me to persist the classes one by one and still be able to run and test the project (i will not have all the errors because there is no table name and no primary key). So, I would have to delete the "__abstract__" when I begin to work on the persistence of a new class.

2) because I thought that the best solution in this case is to map only the concrete classes. So, in my example, I would have to map "Player" and "NPC", but not "Character". So only the classes at the bottom of the hierarchy would have to be mapped. That's still a lot of classes but probably easier to implement.

But I have to say that this is not absolutely clear for me for now. This is the first time I use SQLAlchemy. Do you think that this method is possible and is the right way to proceed ?

CONTINUING !  sorry

To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+...@googlegroups.com.

[Mike Bayer]

On Fri, Oct 20, 2017 at 2:40 PM, Sven <sven....@gmail.com> wrote:
> Thank you for your answer.
>
> I'm sorry, I have not been clear enough regarding the "__abstract__ = True".
>
> I suppose that I will have to add this to a lot of classes for the following
> reasons :
> 1) because it will allow me to persist the classes one by one and still be
> able to run and test the project (i will not have all the errors because
> there is no table name and no primary key). So, I would have to delete the
> "__abstract__" when I begin to work on the persistence of a new class.
> 2) because I thought that the best solution in this case is to map only the
> concrete classes. So, in my example, I would have to map "Player" and "NPC",
> but not "Character". So only the classes at the bottom of the hierarchy
> would have to be mapped. That's still a lot of classes but probably easier
> to implement.

OK, so I see you are looking to add something like "Mappable" or
similar only to those classes that are concrete.

how about this?

from sqlalchemy.ext.declarative import declarative_base, DeclarativeMeta
from sqlalchemy import Column, Integer

class MetaBase(type):

def __init__(cls, nom, bases, contenu):

type.__init__(cls, nom, bases, contenu)
print("Init MetaBase")
pass


class DeclMetaBase(MetaBase, DeclarativeMeta):

def __init__(cls, nom, bases, contenu):
super(MetaBase, cls).__init__(nom, bases, contenu)

print("Init DeclMetaBase")

Base = declarative_base(metaclass=DeclMetaBase)


class Stockable(metaclass=MetaBase):

def __init__(self):
print("Init Stockable")

class Character(Stockable, Base):

__tablename__ = 'characters'
id = Column(Integer, primary_key=True)

def __init__(self, name):
self.name = name
print("Init character")

print(repr(Character.__table__))

jean = Character("Jean")
print(jean.name)



>

> But I have to say that this is not absolutely clear for me for now. This is
> the first time I use SQLAlchemy. Do you think that this method is possible
> and is the right way to proceed ?

Whether or not it "works" involves mostly the database schema that
gets created and how reasonable this schema is from a relational /
database performance / complexity perspective. This depends a lot on
how these class hierarchies are organized and how they interact with
each other, so it's difficult to say.

[Mike Bayer]

actually like this:

class DeclMetaBase(MetaBase, DeclarativeMeta):

def __init__(cls, nom, bases, contenu):

MetaBase.__init__(cls, nom, bases, contenu)
DeclarativeMeta.__init__(cls, nom, bases, contenu)

[Sven]

Exactly what I was looking for and it works (even applied to my project).

I tried so many things these last days and the solution now looks so simple.

Thank you very much !