New Elixir Tutorial - Split a Turbogears Model File Across Multiple Files

[DanL]

Hey guys,

I've been working on a large Turbogears project that required me to
split an Elixir model across multiple files. As a programmer new to
Python and Elixir, I had some troubles (a good tutorial for this has
noticeably missing from the docs).

There were a few subtle (and frustrating) tricks involved, so I wrote
up in the wiki the technique and patterns I ultimately used, along
with some common pitfalls I encountered. It is written with a
Turbogears project in mind, but it should be abstract enough that it
will apply to Pylons projects as well.

Check it out under Recipes: http://elixir.ematia.de/trac/wiki/Recipes/SplittingAModelAcrossMultipleFiles

Please comment or fix if I did anything architecturally wrong or bad
-- my background is mostly in C++, so I may not be grasping certain
Python concepts properly.

[Vortexmind]

Thank you
I'll bookmark this as I'm newbie too and still I'll have to face this
issue soon :)

Paolo

[Gaetan de Menten]

On Sun, Apr 20, 2008 at 8:49 PM, DanL <dlo...@mit.edu> wrote:
>
> Hey guys,
>
> I've been working on a large Turbogears project that required me to
> split an Elixir model across multiple files. As a programmer new to
> Python and Elixir, I had some troubles (a good tutorial for this has
> noticeably missing from the docs).
>
> There were a few subtle (and frustrating) tricks involved, so I wrote
> up in the wiki the technique and patterns I ultimately used, along
> with some common pitfalls I encountered. It is written with a
> Turbogears project in mind, but it should be abstract enough that it
> will apply to Pylons projects as well.
>
> Check it out under Recipes: http://elixir.ematia.de/trac/wiki/Recipes/SplittingAModelAcrossMultipleFiles

Thanks *a lot* for this. I've been wanting to add an entry in the FAQ
for this for quite a while but didn't find (or take) the time yet.

>
> Please comment or fix if I did anything architecturally wrong or bad
> -- my background is mostly in C++, so I may not be grasping certain
> Python concepts properly.

There are a few mistakes and oddities. Some of which rather important
(I fail to see how it works at all the way you do it) but I have to
investigate it further to correct and comment properly what's wrong.

The most minor one is that in the following line:
using_options(shortnames=True, tablename="my_man_table")

the shortnames=True is totally useless as it's an option which is only
used when you don't specify the table name manually.

Thanks again,
--
Gaëtan de Menten
http://openhex.org

[DanL]

You're right... what I posted completely does not work. I apologize
for that... it was a combination of it being really late and me not
keeping track of what hackish additions did make it work and which
didn't. Anyways, I would like to figure this out, and I need you
guys' help.


Lets step back from Turbogears for a second and go to the most simple
possible multi-file model with a dependency between models. I'll be
using the Man/Dog classes I wrote up in that wiki. Lets start with
putting everything into one file: the model.py file. A second file
(script_create_tables.py) creates the metadata and calls
create_tables. So our two-file project looks like this:


== script_create_tables.py ==
=============================

from elixir import metadata, create_all
metadata.bind = "mysql://multimodel:secretpw@localhost:3306/
multifilemodel"
metadata.bind.echo = True


from model import *


create_all()
=============================


== model.py ==
=============================
from elixir import Entity, setup_all

from elixir import Field, Unicode, ManyToOne, OneToMany

class Dog(Entity):
name = Field(Unicode(10))
owner = ManyToOne('Man')
def __repr__(self):
return '<Dog \'%s\'>' % (self.name)

class Man(Entity):
name = Field(Unicode(10))
pets = OneToMany('Dog')
def __repr__(self):
return '<Man \'%s\'>' % (self.name)



## Debugging Outputs
print "Man's metaclass's entities: "
print Man.__metaclass__._entities


## Setup the Tables
setup_all()
=============================

This works great.



Now, lets say we want to move just Man into a separate file,
humans.py:

== new model.py ==
=============================
from elixir import Entity, setup_all

from elixir import Field, Unicode, ManyToOne, OneToMany

class Dog(Entity):
name = Field(Unicode(10))
owner = ManyToOne('Man')
def __repr__(self):
return '<Dog \'%s\'>' % (self.name)

from humans import *


## Debugging Outputs
print "Man's metaclass' entities: "
print Man.__metaclass__._entities


## Setup the Tables
setup_all()
=============================


== new humans.py ==
=============================
from elixir import Entity, Field, Unicode, OneToMany

class Man(Entity):
name = Field(Unicode(10))
pets = OneToMany('Dog')
def __repr__(self):
return '<Man \'%s\'>' % (self.name)
=============================


Now we get an error tracing up to setup_all(). The error is:
KeyError: 'Dog'

(Note: in my testing these three files, humans.py actually resides in
a module called "submodels". This is to reflect the ultimate goal of
putting this in the scope of a Turbogears project, but the results
appear to be the same as above)


Now comes where I need the community's help. I believe the problem
lies in the construction of the Elixir MetaEntity metaclass object. I
don't know exactly how the Elixir MetaEntity is constructed, but I can
guess its function. I need someone who understands the MetaEntity to
help me figure this out:

In the first case (all Entities in model.py, things worked), the
printout of the metaclass's entities was:

{24620536: {'Man': <class 'tgmultifilemodel.model.Man'>,
'Dog': <class 'tgmultifilemodel.model.Dog'>,
'Entity': <class 'elixir.entity.Entity'>}}

This is what works (and what we want to get). In the second case (Man-
class moved to humans.py, KeyError), the printout of the metaclass's
entites was:
{24630320: {'Entity': <class 'elixir.entity.Entity'>,
'Man': <class 'tgmultifilemodel.submodels.humans.Man'>},
24633600: {'Man': <class 'tgmultifilemodel.submodels.humans.Man'>,
'Dog': <class 'tgmultifilemodel.model.Dog'>,
'Entity': <class 'elixir.entity.Entity'>}}

This explains the KeyError: 'Dog': in the 24630320 domain/set/whatever-
it-is, the Dog entity is not listed, and so when we try to setup_all
that set, we understandably get the KeyError because of Man's
reference to (the missing) Dog class.

Here's where my understanding of Elixir ends and I hope someone can
fill in the blanks. How is this metaentity class constructed? Why
the different sets when we split across multiple files? And the
million-dollar question: how can we split dependant entities across
multiple files and make elixir's metaentity not create that pesky
partially-filled domain that breaks stuff?


On Apr 21, 6:11 am, "Gaetan de Menten" <gdemen...@gmail.com> wrote:

> On Sun, Apr 20, 2008 at 8:49 PM, DanL <dlop...@mit.edu> wrote:
>
> > Hey guys,
>
> > I've been working on a large Turbogears project that required me to
> > split an Elixir model across multiple files. As a programmer new to
> > Python and Elixir, I had some troubles (a good tutorial for this has
> > noticeably missing from the docs).
>
> > There were a few subtle (and frustrating) tricks involved, so I wrote
> > up in the wiki the technique and patterns I ultimately used, along
> > with some common pitfalls I encountered. It is written with a
> > Turbogears project in mind, but it should be abstract enough that it
> > will apply to Pylons projects as well.
>

> > Check it out under Recipes:http://elixir.ematia.de/trac/wiki/Recipes/SplittingAModelAcrossMultip...

[alex23]

Hey Dan,

I've had no problems splitting TG models across multiple files in the
past. I generally set 'model' up as a package, so for your code above
I'd structure it like so:

mmtest/
create_tables.py
model/
__init__.py
dog.py
man.py

dog.py:
-------
from elixir import Entity, Field, Unicode, ManyToOne

class Dog(Entity):
name = Field(Unicode(10))
owner = ManyToOne('Man')
def __repr__(self):
return '<Dog \'%s\'>' % (self.name)

man.py:
-------

from elixir import Entity, Field, Unicode, OneToMany

class Man(Entity):
name = Field(Unicode(10))
pets = OneToMany('Dog')
def __repr__(self):
return '<Man \'%s\'>' % (self.name)

__init.py__:
------------
from elixir import setup_all

from dog import Dog
from man import Man

print "Man: %s" % Man.__metaclass__._entities
print "Dog: %s" % Dog.__metaclass__._entities

create_tables.py:
-----------------
from elixir import metadata, setup_all
from model import *

metadata.bind = "sqlite:///multifilemodel.db"

setup_all(True)

Running create_tables.py gives me:

Man: {20702984: {'Man': <class 'model.man.Man'>,
'Dog': <class 'model.dog.Dog'>,
'Entity': <class 'elixir.entity.Entity'>}}
Dog: {20702984: {'Man': <class 'model.man.Man'>,
'Dog': <class 'model.dog.Dog'>,
'Entity': <class 'elixir.entity.Entity'>}}

And more pertinently:

>sqlite3 multifilemodel.db
SQLite version 3.5.7
Enter ".help" for instructions
sqlite> .schema
CREATE TABLE model_dog_dog (
id INTEGER NOT NULL,
name VARCHAR(10),
owner_id INTEGER,
PRIMARY KEY (id),
CONSTRAINT model_dog_dog_owner_id_fk FOREIGN
KEY(owner_id) REFERENCES model_man_man (id)
);
CREATE TABLE model_man_man (
id INTEGER NOT NULL,
name VARCHAR(10),
PRIMARY KEY (id)
);
CREATE INDEX ix_model_dog_dog_owner_id ON model_dog_dog
(owner_id);

This is all tested & working under Elixir 0.5.1.

Hope this helps.

- alex23

[DanL]

Hi Alex,

Thanks for that. I confirmed it works, kind of.

It seems the key was to put all the models into one package, do the
imports in the __init__.py, and import the package (my approach was
import the files in the package from a model.py file outside the
package). Your approach works if you run the script, but it doesn't
quite work inside a turbogears environment.

Putting it into a turbogears project, there seems to be a problem
regarding where you put the setup_all. Here's the problem:
The way it "should work" is in your turbogears root controller, you
import the model package and use it:

controllers/root.py:
-------
...
from models_package import *
setup_all()

class Root(controllers.RootController):
....
-------

The problem, however, seems to be that when you launch the tg project
using $python start-tgproject.py, the metadata engine object created
by turbogears is not bound at the time when the root controller is
loaded, and this somehow causes the KeyError in the previous post (ie
the above root.py causes the previously-described KeyError with the
same symptoms). At this point, the problem seems to be Turbogears
related, but I'd appreciate some Elixir-people help (see below) before
turning to the Turbogears board.

Anyways, there are two solutions I've found, but both seem hack-ish:
1) Bind the metadata object in the root controller:

controllers/root.py:
-------
...
from turbogears.database import metadata #Wrapper for Elixir's
metadata object

metadata.bind = "mysql://multimodel:secretpw@localhost:3306/
multifilemodel"

from models_package import *
setup_all()

class Root(controllers.RootController):
....
-------
but this is very hackish because we're overriding Turbogear's metadata
engine, which in general is a bad idea.


2) Import the models inside whichever root controller method needs
them:

controllers/root.py:
-------
...

class Root(controllers.RootController):
....

@expose()
def controller1(self):
from models_package import *
setup_all()
....

@expose()
def controller2(self):
from models_package import *
setup_all()
....
....
-------

But this also seems really hackish and just obnoxious to use.

So, any suggestions on:
A) Elixir: Why do we get the previously-described KeyError when there
is no bound metadata engine but don't get it when there IS a bound
metadata engine?
or
B) TurboGears: Whats the proper way of using the models in a
TurboGears project, given that turbogears does not seem to bind its
metadata object until after the root controller file is loaded?

-Dan

[Yap Sok Ann]

The model below doesn't seem to work with any of the import methods
laid out in this discussion and in the wiki. It has an "odd many-to-
many relationship" which is modeled following Gaeten's suggestion in
http://www.mail-archive.com/sqlal...@googlegroups.com/msg02616.html

man.py
------
from elixir import Entity, Field, Unicode, has_many

class Man(Entity):
name = Field(Unicode(10))

has_many('relationships', of_kind='Relationship')

def __repr__(self):
return '<Man \'%s\'>' % (self.name)

woman.py
--------
from elixir import Entity, Field, Unicode, has_many

class Woman(Entity):
name = Field(Unicode(10))
has_many('relationships', of_kind='Relationship')
def __repr__(self):
return '<Woman \'%s\'>' % (self.name)

relationship.py
---------------
from elixir import Entity, Field, Unicode, belongs_to

class Relationship(Entity):
name = Field(Unicode(10))
belongs_to('man', of_kind='Man', required=True)
belongs_to('woman', of_kind='Woman', required=True)
def __repr__(self):
return '<Relationship \'%s\'>' % (self.name)

Tested under Elixir 0.5.2

[Yap Sok Ann]

On Apr 30, 11:32 am, Yap Sok Ann <sok...@gmail.com> wrote:
> The model below doesn't seem to work with any of the import methods
> laid out in this discussion and in the wiki. It has an "odd many-to-

> many relationship" which is modeled following Gaeten's suggestion inhttp://www.mail-archive.com/sqlal...@googlegroups.com/msg02616.html

I got it to work using full path addressing as mentioned by Gaeten in
http://groups.google.com/group​/sqlelixir/browse_thread/thread​/bc8b23a1ae8bd9ec

Dan, can you try if it works for you? Should work fine regardless of
how to structure the files or where to do the import.

FWIW, here's what I have:

model/__init__.py
-----------------


model/man.py
------------

from elixir import Entity, Field, Unicode, has_many

c​lass Man(Entity):
name = Field(Unicode(10))

has_many('re​lationships', of_kind='model.relationship.Re​
lationship')
has_many('women', through='relationships', via='woman')

def __repr__(self):
return '<Man \'%s\'>' % (self.name)

model/woman.py
------​--------

from elixir import Entity, Field, Unicode, has_many

class Woman(Entity):
name = Field(Unicode(10))
has_many('relationships',

of_kind='model.relationship.Relationship')
has_many('men', through='relationships', via='man')

def __repr__(self):
return '<Woman \'%s\'>' % (self.name)

model/relationship.py
---------------------

from elixir import Entity, Field, Unicode, belongs_to

class Relationship(Entity):
name = Field(Unicode(10))

belongs_to('man', of_kind='model.man.Man',
inverse='relationships', required=True)
belongs_to('woman', of_kind='model.woman.Woman',
inverse='relationships', required=True)

def __repr__(self):
return '<Relationship \'%s\'>' % (self.name)

test.py
-------
from elixir import Entity, setup_all

from model.man import Man
from model.woman import Woman
from model.relationship import Relationship

print Entity.__metaclass__._entities

setup_all()

[DanL]

Hi Yap,

Thanks for that full addressing trick. I've confirmed it works in the
scope of a turbogears project, but as Ben Bangert wrote in the post
you linked:
> This works, though its rather verbose

So full addressing is a solution better than anything else discussed
here, but it's awkward and I feel like Elixir should do a better job
at knowing the right thing.

For reference, the whole metaclass issue with Yap's solution looks
like this:
{23981776:
{'Dog': <class 'tgmultifilemodel.submodels.animals.Dog'>,
'Entity': <class 'elixir.entity.Entity'>},
22361496:

{'Entity': <class 'elixir.entity.Entity'>,
'Man': <class 'tgmultifilemodel.submodels.humans.Man'>}
}

so it looks like we still have the same initial problem described
above (the different files' Entities are in different 'domains'), but
it works because they're fully addressed. I guess this is how you
make the metaclass issue work, but again, I feel like Elixir should
know how to do this without the code being so annoyingly verbose.


For more reference, the final solution I used (as a turbogears
project) looks like this:
tgmultifilemodel/
controllers.py
model.py
submodels/
__init__.py
animals.py
humans.py


model.py
--------------
from elixir import setup_all
from submodels import *

## Debugging Outputs
print "Man's metaclass' entities: "
print Man.__metaclass__._entities

print "Dog's metaclass' entities: "
print Dog.__metaclass__._entities #Same as Man's metaclass


setup_all()


submodels/__init__.py:
--------------
from animals import *
from humans import *


submodels/animals.py
--------------
from elixir import Field, Unicode, ManyToOne, Entity
#from tgmultifilemodel.model import Man #Won't work: Circular
imports!

class Dog(Entity):

name = Field(Unicode(10))

owner = ManyToOne('tgmultifilemodel.model.Man')
# NOTE: You could have done
# owner = ManyToOne('tgmultifilemodel.submodels.humans.Man')
# but the above works due to the way we structured model.py and is
slightly less verbose

def __repr__(self):
return '<Dog \'%s\'>' % (self.name)

submodels/humans.py
--------------
from elixir import Field, Unicode, OneToMany, Entity
#from tgmultifilemodel.model import Dog #Won't work: Circular
imports!

class Man(Entity):

name = Field(Unicode(10))

pets = OneToMany('tgmultifilemodel.model.Dog')

def __repr__(self):
return '<Man \'%s\'>' % (self.name)

controllers.py
--------------
....
from tgmultifilemodel.model import *

class Root(controllers.RootController):
...


I'll update the Elixir wiki in the next few days to include this
solution.
If Elixir is ever updated in the future to make this issue less
awkward, please make a note of it in this thread.


-Dan

> I got it to work using full path addressing as mentioned by Gaeten inhttp://groups.google.com/group​/sqlelixir/browse_thread/thread​/bc8b23a1ae8bd9ec

[Gaetan de Menten]

Yeah, that's the only reliable method currently. Thanks for digging in
the mailing list archive for the answer!
I've had an answer to DanL message in my draft folder for quite a
while now (no idea why I didn't send it when I wrote it :-/).

I'll definitely need to address this in a FAQ or even the tutorial
given the number of people who trip on this...

--

[alex bodnaru]

to avoid unnatural full address, which will further complicate after each
import, use from model import *.

[DanL]

So do you mean for, say, humans.py:

humans.py:
-----------
from elixir import Field, Unicode, OneToMany, Entity
from tgmultifilemodel.model import *

class Man(Entity):

name = Field(Unicode(10))

pets = OneToMany('Dog')

def __repr__(self):
return '<Man \'%s\'>' % (self.name)

This doesn't seem to work... it creates the key error I described
above.

-Dan

[alex bodnaru]

every class knows about classes defined in the same model, or in models imported
before it's definition.

call setup_all at the end of the model that knows everything.

i'll try your model hierarchy tomorrow, but i have my working hierarchy for some
time :) .

best regards,

alex

[kremlan]

The following setup is partially working for me.

This is in a pylons project.

erp/model/user.py
--------------------------
from elixir import *

class User(Entity):
using_options(tablename='users', autoload=True)

preferences = OneToMany('Preference')


erp/model/pref.py
-------------------------
from elixir import *

class Preference(Entity):
using_options(tablename='user_preferences', autoload=True)

user = ManyToOne('User', colname='user_id')


erp/model/__init__.py
-------------------------------
from pylons import config
from elixir import *
metadata.bind = config['sqlalchemy.url']

from prefs import *
from users import *

setup_all()


The above setup works fine. The trouble is when I add additional
imports
NEW erp/model/user.py
----------------------------------
from sets import Set # <---- new line, only change.
from elixir import *

class User(Entity):
using_options(tablename='users', autoload=True)

preferences = OneToMany('Preference')


This setup works fine as well. The trouble starts when I add an
additional import.

NEW erp/model/user.py
----------------------------------
import datetime # <---- new line, only change
from sets import Set
from elixir import *

class User(Entity):
using_options(tablename='users', autoload=True)

preferences = OneToMany('Preference')

With this change I get KeyError: 'Preference'. I have tried many
variations on what the
additional imports are (different modules, * vs explicit import, etc)
and the only pattern
that has emerged is I can't import more than one additional item other
than elixir.
(I have also tried importing elixir classes/functions explicitly)

anyone have any ideas?

-brad

[Jonathan LaCour]

kremlan wrote:

> With this change I get KeyError: 'Preference'. I have tried many
> variations on what the additional imports are (different modules,
> * vs explicit import, etc) and the only pattern that has emerged
> is I can't import more than one additional item other than elixir.
> (I have also tried importing elixir classes/functions explicitly)
>
> anyone have any ideas?

We badly need to update our documentation, and remove the incorrect
tutorial on the wiki to reflect this, but you need to make sure to
specify the full importable path to the entity you are referencing
in your relationships. In your case:

erp/model/user.py
--------------------------
from elixir import *

class User(Entity):
using_options(tablename='users', autoload=True)

preferences = OneToMany('erp.model.pref.Preference')

erp/model/pref.py
-------------------------
from elixir import *

class Preference(Entity):
using_options(tablename='user_preferences', autoload=True)

user = ManyToOne('erp.model.user.User', colname='user_id')

Good luck.

--
Jonathan LaCour
http://cleverdevil.org

[kremlan]

This is the method we've been using for the past few months. The
problem we're running into now is that we're going to have to start
using these models across multiple projects. We'd obviously prefer to
not have to fork them just to change the class reference. We'll keep
using the full path for now. Thanks for the help and thanks for the
great project.

-brad


On May 20, 4:44 pm, Jonathan LaCour <jonathan-li...@cleverdevil.org>
wrote: