how to organise files in model package using sqlachemy?

[Chung WONG]

I have a models.py that contains 10 classes. And I am trying to move all classes out of that file and put each class into its own file under a models package.

I am using the alternative(at bottom of page) method mentioned here, but I don't know how  it works for initialise db script as it is using config.scan()

However, when i ran bin/initialize_bug_db development.in to initialise the database, it threw such as :

sqlalchemy.exc.InvalidRequestError: When initializing mapper Mapper|User|users, expression 'TopicUser' failed to locate a name ("name 'TopicUser' is not defined"). If this is a class name, consider adding this relationship() to the <class 'bug.models.user.User'> class after both dependent classes have been defined.

which was caused by  something like topics = relationship('TopicUser', backref="user", lazy='dynamic')

and

sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column 'notifications.topic_id' could not find table 'topics' with which to generate a foreign key to target column 'id'

which was caused by something like topic_id=Column(Integer, ForeignKey('topics.id'))

I am wondering how you guys organise the model files. Maybe there is a better way?

Thanks.

[Vincent Catalano]

Chung,

I believe you are running into a similar problem I had recently when I wanted to change the structure of my models folders for my application. What I ended up doing is moving each model into its own file and importing each of the files into packages __init__.py file.

Something like this:

application

|-- models

     |--- __init__.py

     |--- meta.py

     |--- foo

     |--- bar

     |--- baz

and in your init file, you would have the following imports:

from .foo import Foo

from .bar import Bar

from .baz import Baz

At this point, if you are still receiving the same exception when you are initializing the database, make sure the order of the imports in the __init__.py file matches the order of which the dependent tables should be created. In your case, make sure you are importing User before you import TopicUser, for example.

Hope this gives you some direction!

-Vincent

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.
Visit this group at http://groups.google.com/group/pylons-discuss.
For more options, visit https://groups.google.com/groups/opt_out.

--
Vincent Catalano
Software Engineer and Web Ninja,
(520).603.8944

[Wyatt Baldwin]

In general: I would keep the model completely separate from the Pyramid Web application. In simpler projects, this would just mean creating a subpackage (like you're already doing) and being sure to not use any Pyramid stuff in the model (in particular, the request and registry thread locals). For more complex projects, I usually create an entirely separate package for the model that is entirely unaware of the Web app it's used in.

In the configuration you linked, I think the problem is with the line `Base.metadata.create_all(engine)`. The database tables shouldn't be created every time the application is started. In a real production application, you almost certainly wouldn't do that (because you'd create the database up front and then use migrations when you change your schema).

If you're going to use the `config.scan()` trick they show (and personally, I wouldn't), it'd probably be better to wrap it up in a `create_db()` script and call it only when you need to.

[Jonathan Vanasco]

In addition to what Vincent noted, I also like to do the following:

|-- models

   |-- __init__.py

   |-- _core.py

   |-- api.py

_core.py -- anything that is shared across the other files in the package.  i.e. a base class, mixins, etc

api.py -- although I still import everything into __init__.py , I like to have an 'api' namespace to contain any of the various functions for the package that other packages might interact with.

I do this in-addition-to what Vincent recommended.  It's just ended up easiest for me.

In terms of Wyatt's recommendation.  i'm lazy and don't keep the model completely separate -- one day I will.  BUT, I do keep pyramid about of the model, so it can be used by non-pyramid packages (and it often is).  I STRONGLY recommend that you keep anything that has to do with Pyramid out of the /model namespace.  

[Chung WONG]

thanks Vincent and Wyatt.

Can I say that even though importing every model into __init__.py is tedious, it is the best approach available at this moment?

[AM]

On 02/28/2014 11:58 PM, Chung WONG wrote:
> I have a *models.py* that contains 10 classes. And I am trying to move

> all classes out of that file and put each class into its own file
> under a models package.
>
> I am using the alternative(at bottom of page) method mentioned here

> <https://docs.pylonsproject.org/projects/pyramid_cookbook/en/latest/database/sqlalchemy.html#importing-all-sqlalchemy-models>,

> but I don't know how it works for initialise db script as it is using

> *config.scan()*
>
> However, when i ran *bin/initialize_bug_db development.in *to

> initialise the database, it threw such as :
>
> sqlalchemy.exc.InvalidRequestError: When initializing mapper
> Mapper|User|users, expression 'TopicUser' failed to locate a name
> ("name 'TopicUser' is not defined"). If this is a class name, consider
> adding this relationship() to the <class 'bug.models.user.User'> class
> after both dependent classes have been defined.
>

> which was caused by something like *topics =
> relationship('TopicUser', backref="user", lazy='dynamic')*

>
> and
>
> sqlalchemy.exc.NoReferencedTableError: Foreign key associated with
> column 'notifications.topic_id' could not find table 'topics' with
> which to generate a foreign key to target column 'id'

> which was caused by something like *topic_id=Column(Integer,
> ForeignKey('topics.id'))*

>
>
> I am wondering how you guys organise the model files. Maybe there is a
> better way?
>
> Thanks.

> --
> You received this message because you are subscribed to the Google
> Groups "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pylons-discus...@googlegroups.com

> <mailto:pylons-discus...@googlegroups.com>.

> To post to this group, send email to pylons-...@googlegroups.com

> <mailto:pylons-...@googlegroups.com>.

> Visit this group at http://groups.google.com/group/pylons-discuss.
> For more options, visit https://groups.google.com/groups/opt_out.

One that has worked for me in the past is having the fs structure as this:

models/
__init__.py
....

and then in the models/__init__.py:


def includeme(config):
settings = config.get_settings()
engine = sa.engine_from_config(settings, 'sqlalchemy.')
DB.configure(bind=engine)
BASE.metadata.bind = engine


and in your app main:

config = Configurator(settings=settings)
config.include('foo.models')

HTH.
AM

[Mike Orr]

On Mon, Mar 3, 2014 at 10:38 AM, AM <ams...@gmail.com> wrote:
> On 02/28/2014 11:58 PM, Chung WONG wrote:
>>
>> I have a *models.py* that contains 10 classes. And I am trying to move all
>> classes out of that file and put each class into its own file under a models
>> package.
>>
>> I am using the alternative(at bottom of page) method mentioned here
>> <https://docs.pylonsproject.org/projects/pyramid_cookbook/en/latest/database/sqlalchemy.html#importing-all-sqlalchemy-models>,
>> but I don't know how it works for initialise db script as it is using
>> *config.scan()*
>>
>> However, when i ran *bin/initialize_bug_db development.in *to initialise
>> the database, it threw such as :
>>
>> sqlalchemy.exc.InvalidRequestError: When initializing mapper
>> Mapper|User|users, expression 'TopicUser' failed to locate a name ("name
>> 'TopicUser' is not defined"). If this is a class name, consider adding this
>> relationship() to the <class 'bug.models.user.User'> class after both
>> dependent classes have been defined.
>>
>> which was caused by something like *topics = relationship('TopicUser',
>> backref="user", lazy='dynamic')*
>>
>> and
>>
>> sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column
>> 'notifications.topic_id' could not find table 'topics' with which to
>> generate a foreign key to target column 'id'
>> which was caused by something like *topic_id=Column(Integer,
>> ForeignKey('topics.id'))*

It sounds like config.scan() is importing the modules in the wrong
order. There's not much you can do about that. If your DeclarativeBase
is in one module, the child model is in a second module, and the
module with the relationship is in a third module, you have to import
them in that order, and config.scan() can't guess that. I think you
can define the models out of order by using a string in the
relationship, but I've only done that within a module so I'm not sure
it works across modules.


AM's includeme() would be one way to address this, but the primary
issue is what's *in* the includeme() and what's above it. You could
have the includeme() body import the modules in the right order and
configure everything. You would probably not want any unnecessary
global-level imports above it, which might throw things off.

I don't use includeme() in models but I do this instead:

def init_models(engine):
DBSession.bind = engine
Base.metadata.bind = engine

This is nice because it's one-stop initialization for both the
application and any standalone scripts. It would also be the place to
do any reflection or preloading memory indexes.

You *can* do circular imports this way:

# .__init__.py
import sqlalchemy.orm as orm
DBSession = orm.scoped_session(orm.sessionmaker(extension=...))
import submodule_a

# submodule_a.py
from . import DBSession

The trick is that the "import submodule_a" must *below* anything that
submodule_a imports from the parent module, because when submodule_a
is initializing only the top part of the parent's module has executed.
I didn't recommend it in the Cookbook article because I considered it
failure-prone, but several people have successfully used it since
then. I still prefer to avoid circular imports whenever possible.