Documenting SQLAlchemy models with Sphinx fails (table already defined)

[Christoph Haas]

Dear list...

I'm on a documentation frenzy now in my current Pylons project. And I'm
using Sphinx for that purpose. All my controllers and helper functions are
documented automatically. Great. Unfortunately Sphinx fails to create
module documentation for my models. I'm staying close to the "Using
SQLAlchemy with Pylons" [0] way as documented on the Pylons wiki. This is
the output on my console:

=================================
$> make html
mkdir -p _build/html _build/doctrees
sphinx-build -b html -d _build/doctrees   . _build/html
Sphinx v0.4.2, building html
trying to load pickled env... not found
building [html]: targets for 4 source files that are out of date
updating environment: 4 added, 0 changed, 0 removed
reading... antrag-allgemein antrag-erstellen api Exception occurred:
  File "/var/lib/python-support/python2.5/sqlalchemy/schema.py", line 101,
in __call__
    "columns on an existing Table object." % key)
InvalidRequestError: Table 'applications' is already defined for this
MetaData instance.  Specify 'useexisting=True' to redefine options and
columns on an existing Table object.
The full traceback has been saved in /tmp/sphinx-err-lBwkYm.log, if you
want to report the issue to the author.
Please also report this if it was a user error, so that a better error
message can be provided next time.
Send reports to sphin...@googlegroups.com. Thanks!
make: *** [html] Error 1
=================================

I have a table "applications" here - but it's only defined once. My
model/__init__.py looks roughly like (less interesting parts removed):

=================================
# -*- coding: utf-8 -*-

import sqlalchemy as sql
import sqlalchemy.orm as orm
from myapp.model import meta

def init_model(engine):
    sm = orm.sessionmaker(autoflush=True, transactional=True, bind=engine)
    meta.engine = engine
    meta.Session = orm.scoped_session(sm)

applications_table = sql.Table(
    'applications', meta.metadata,
    sql.Column('id', sql.Integer, primary_key=True),
    sql.Column('applicant', sql.Unicode(8), nullable=False),
    sql.Column('created_date', sql.DateTime(), default=sql.func.now()),
    sql.Column('valid_until', sql.DateTime()),
    sql.Column('description', sql.Unicode(100)),
    sql.Column('reason', sql.Unicode(4000)),
    sql.Column('status', sql.Integer, nullable=False),
)

class Application(object): pass

orm.mapper(Application, applications_table)
=================================

Does anyone have an idea what's going wrong? And whether it needs to fixed
in Sphinx, SQLAlchemy (I'm on 0.4.6) or my code? Mike Orr assumed that it
could be that Sphinx loads the module twice to tear out docstrings.

Cheers
 Christoph

[0]
http://wiki.pylonshq.com/display/pylonsdocs/Using+SQLAlchemy+with+Pylons

[Michael Bayer]

my initial impression is that Sphinx is loading modules artificially
(and executing them) despite their already being present in
sys.modules. The error indicates that the same MetaData instance is
being used to create "applications_table" twice. Definitely ask the
Sphinx folks about this one.

[Georg Brandl]

Christoph Haas schrieb:

> Dear list...
>
> I'm on a documentation frenzy now in my current Pylons project. And I'm
> using Sphinx for that purpose. All my controllers and helper functions are
> documented automatically. Great. Unfortunately Sphinx fails to create
> module documentation for my models. I'm staying close to the "Using
> SQLAlchemy with Pylons" [0] way as documented on the Pylons wiki. This is
> the output on my console:

[...]

> Does anyone have an idea what's going wrong? And whether it needs to fixed
> in Sphinx, SQLAlchemy (I'm on 0.4.6) or my code? Mike Orr assumed that it
> could be that Sphinx loads the module twice to tear out docstrings.

I agree that Sphinx is likely importing the module twice. Since it's using
Python's standard import mechanism, this shouldn't happen.

Problems can arise, however, if the module is imported under different names,
e.g. once as "myapp.application" and once as "application".

You can try to start sphinx with "python -v /path/to/sphinx-build.py" to
see which imports are done.

Georg

[tloc...@gmail.com]

On Aug 30, 7:35 am, Georg Brandl <ge...@python.org> wrote:

> Problems can arise, however, if the module is imported under different names,
> e.g. once as "myapp.application" and once as "application".
>
> You can try to start sphinx with "python -v /path/to/sphinx-build.py" to
> see which imports are done.

I have been bitten by this same problem in an application of mine,
too. The result of running sphinx-build under "python -v" shows:

import myapp.model # directory /home/tholo/src/MyApp/myapp/model
# /home/tholo/src/MyApp/myapp/model/__init__.pyc matches /home/tholo/
src/MyApp/myapp/model/__init__.py
import myapp.model # precompiled from /home/tholo/src/MyApp/myapp/
model/__init__.pyc
[...]
import myapp.model. # directory /home/tholo/src/MyApp/myapp/model/
# /home/tholo/src/MyApp/myapp/model/__init__.pyc matches /home/tholo/
src/MyApp/myapp/model/__init__.py
import myapp.model. # precompiled from /home/tholo/src/MyApp/myapp/
model/__init__.pyc

Note the trailing dot in the second occurrence of the import.
Immediately after than second import SQLAlchemy throws "ArgumentError:
Table 'sometable' is already defined for this MetaData instance."

Thorsten

[Georg Brandl]

tloc...@gmail.com schrieb:

That's not good and smells like a bug in autodoc. What version of Sphinx
are you using? Can you show me the snippet of reST that contains the
autodoc directives in question?

Thanks,
Georg

[Thorsten Lockert]

On Sep 11, 2008, at 15:40 , Georg Brandl wrote:
>
> tloc...@gmail.com schrieb:

>> Note the trailing dot in the second occurrence of the import.
>> Immediately after than second import SQLAlchemy throws
>> "ArgumentError:
>> Table 'sometable' is already defined for this MetaData instance."
>
> That's not good and smells like a bug in autodoc. What version of
> Sphinx
> are you using? Can you show me the snippet of reST that contains the
> autodoc directives in question?

This is with Sphinx 0.4.2, tho it also happened with trunk at the end
of last week.

This reST will do it:

:mod:`myapp.model` Module
=================================

.. automodule:: myapp.model

Module Contents
---------------

.. autofunction:: init_model

.. autoclass:: User

One sample myapp/model/__init__.py to go with that would be:

import sqlalchemy as sa
from sqlalchemy import orm
from myapp.model import meta

def init_model(engine):
sm = orm.sessionmaker(autoflush=True, transactional=True,
bind=engine)

meta.engine = engine
meta.Session = orm.scoped_session(sm)

user_table = sa.Table("users", meta.metadata,
sa.Column("username", sa.types.String(32),
primary_key=True),
sa.Column("password", sa.types.String(32),
nullable=False)
)

class User(object):
"""
User object.
"""
def __init__(self, user, password):
self.id = None
self.username = unicode(user)
self.password = unicode(password)

def __repr__(self):
myclass = self.__class__.__name__
return "%s(%r, %r)" % (myclass, self.username, self.password)

orm.mapper(User, user_table)

And finally myapp/model/meta.py:

from sqlalchemy import MetaData

__all__ = ['Session', 'metadata']

engine = None
Session = None
metadata = MetaData()

[Georg Brandl]

Thorsten Lockert schrieb:

> On Sep 11, 2008, at 15:40 , Georg Brandl wrote:
>>
>> tloc...@gmail.com schrieb:
>>> Note the trailing dot in the second occurrence of the import.
>>> Immediately after than second import SQLAlchemy throws
>>> "ArgumentError:
>>> Table 'sometable' is already defined for this MetaData instance."
>>
>> That's not good and smells like a bug in autodoc. What version of
>> Sphinx
>> are you using? Can you show me the snippet of reST that contains the
>> autodoc directives in question?
>
>
> This is with Sphinx 0.4.2, tho it also happened with trunk at the end
> of last week.
>
> This reST will do it:

Thanks to your test case, I found the problem quickly. Turns out that
__import__ doesn't like to be called with a fromlist=['']...

Should be fixed in trunk and 0.4.x branch!

Georg

[Thorsten Lockert]

On Sep 12, 2008, at 8:48 , Georg Brandl wrote:

>
> Thanks to your test case, I found the problem quickly. Turns out that
> __import__ doesn't like to be called with a fromlist=['']...
>
> Should be fixed in trunk and 0.4.x branch!

Cool! Wonderful!

Thank you! Thank you! Thank you! :-)

Thorsten