Advice on multiple ORM classes in multiple files and imports

[Martino Io]

Hello, I've been busy writing an application which now has grown considerably, it spans across several packages/modules totalling 200K lines of python code.

Recently I made some changes to the structure and decided to split several classes into separate modules for greater flexibility; the problem I'm facing now

is quite strange, before I was using mainly 1 file for all the ORM definitions, but since I split that into several files, I've started to get exceptions due to multiple

Base objects being defined and no shared mapper to store the metadata. I've changed approach and started to import the base from a single module,

however given the high number of modules and cross imports I get exceptions like:

sqlalchemy.exc.InvalidRequestError: Table 'fabrics' is already defined for this MetaData instance.  Specify 'extend_existing=True' to redefine options and columns on an existing Table object.

I  thought that by adding "use_existing" would fix the issue (the mapper will ignore further imports which redefine the objects) but instead got:

InvalidRequestError: Multiple classes found for path "Group" in the registry of this declarative base. Please use a fully module-qualified path.

So going back to the question, if I have a cross dependant package (which imports classes which are ORM classes), how do I avoid the mapper "redefining" a class? 

Also any advice on best way to structure packages/modules in an ORM use case would be appreciated.

--

Marcin

[Simon King]

extend_existing is almost certainly not what you want here. Each
mapped class should only be defined in a single place. If you've done
that, and are still getting this error, it suggests that perhaps your
Python modules are being imported more than once. Try putting "print
'importing module %s' % __name__" before your class definition. If you
see the output more than once, then the module is being imported
multiple times under different names.

Hope that helps,

Simon

[Martino Io]

Hi Simon, definitely multiple imports is the issue here, however I don't know how to get rid of it,

let's say I have a module which i named "ormstuff" inside it I have a class named "mybase", I do have 2 other modules which

are using this class, and to do so I import it in both places. Now this might just be me skipping the basics, however

if I don't import the mybase in both modules I cannot use it, since it should throw an undefined variable exception.

--
You received this message because you are subscribed to a topic in the Google Groups "sqlalchemy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sqlalchemy/gExzW1RNF7A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sqlalchemy+...@googlegroups.com.
To post to this group, send email to sqlal...@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[Simon King]

Hi Martin,

Normally, Python caches modules after they are imported, so when you
import a module for the second time it doesn't actually execute the
code again. If you have 3 modules, "one", "two" and "common", and
"one" and "two" both import "common", "common" should only be loaded
once.

However, there are a few situations where this can break. One is when
the initial script for your application (the one that you run from the
command line) is also imported by other modules. Typically the script
will be loaded twice, once with the name __main__, and once with its
"real" name.

Other possibilities might be import loops, certain uses of "from
<module> import <something>", manipulation of PYTHONPATH/sys.path etc.
It's difficult to know what the cause is without seeing the code. See
also http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#the-double-import-trap.

By looking at the value of __name__ and the stack traces when you get
exceptions, you might be able to figure it out.
http://stackoverflow.com/a/4897479/395053 suggests inserting "import
pdb; pdb.set_trace()" above the class definition and using bt to
display backtraces.

In general, the solution is to ensure that your imports are structured
as simply as possible and avoid circular dependencies.

Hope that helps,

Simon

> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an

[Martino Io]

Thanks for your help Simon, I've found the issue to be related to mixed source imports and duplicate paths set in eclipse, which both lead to a double import situation;

again thanks for the useful links.

--

Marcin