SQLAlchemy error when autogenerating documentation for database module using pdoc3

[Yaakov Bressler]

Stack Overflow Link: https://stackoverflow.com/questions/67393499/sqlalchemy-error-when-autogenerating-documentation-for-database-module-using-pdo

I'm creating documentation for a project using pydoc3 but am encountering an odd error from SQLAlchemy when running against my `database` module.

Project layout:

PROJECT_NAME/

|-- __init__.py

|

|-- api

|   |-- main.py

|

|-- database

|   |-- __init__.py

|   |-- metadata.py

|

|   |-- methods

|       |-- __init__.py

|       |-- foo.py

|

|   |-- models

|       |-- __init__.py

|       |-- base.py

|       |-- models.py

|

| - tests

|   |-- test_this.py

Command to generate documentation:

Where `MODULE` is the name of the module...

python3 venv/bin/pdoc3 --html --output-dir docs $MODULE

I'm able to generate documentation for each module, one at a time with no issues. 

(Ex: `python3 venv/bin/pdoc3 --html --output-dir docs database`)

But when I try generating documentation for the entire project (`MODULE='.'`), I get the following error:

ImportError: Error importing 'PROJECT_NAME.database':

InvalidRequestError: Table 'FooBar' is already defined for this MetaData instance.  Specify 'extend_existing=True' to redefine options and columns on an existing Table object.

What makes this error odd is that I have a full testing suite where there are no issues at all. Further, the error occurs in database.__init__.py which just doesn't make sense to me at all.

As far as a possible solution:

My thought is that my issue is coming from how I've configured my declarative base

Content of database.__init__.py

from .metadata import metadata

from . import methods

from . import models

Content of database.metadata.py

from sqlalchemy import MetaData

metadata = MetaData()

Content of database.models.base.py

from database import metadata

from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base(metadata=metadata)

Content of database.models.model.py

from database.models import Base

from sqlalchemy import Column, Integer, String

class FooBar(Base):

    __tablename__ = 'foo_bar'

    __table_args__ = {

        'comment': 'Example...'

    }

    id = Column(Integer, primary_key=True, autoincrement=True)

    name = Column(String(80), nullable=False, unique=True)

Is it possible that my models are attempted to be created twice by pydoc when called from the root?

A short term workaround (hack):

I was able to get pdoc to work for my database module by explicitly skipping documentation generation of database.models and changing my database.__init__.py to the following:

__pdoc__ = {'models': False}

from database.metadata import metadata

from database import methods from database import models

Additional thoughts:

• I am in no way attached to pdoc/pdoc3 as an autodoc tool. If ya'll have a tool you enjoy working with more / has fewer issues, please recommend.
• I'd like to think my configuration of database.models is done cleverly -> given my structure, the database module can be loaded throughout my project, will will not transact with the actual database until a transaction is initiated / session is opened. (Testing proves this as true – unless I am mistaken?) 
• I'd also like to think that others have autogenerated documentation for the database modules. If so, what are some best practices? 

Much appreciated! I look forward to any assistance ya'll can provide!

-Yaakov Bressler

[Mike Bayer]

On Wed, May 5, 2021, at 11:18 AM, Yaakov Bressler wrote:

But when I try generating documentation for the entire project (`MODULE='.'`), I get the following error:

ImportError: Error importing 'PROJECT_NAME.database':

InvalidRequestError: Table 'FooBar' is already defined for this MetaData instance.  Specify 'extend_existing=True' to redefine options and columns on an existing Table object.

What makes this error odd is that I have a full testing suite where there are no issues at all. Further, the error occurs in database.__init__.py which just doesn't make sense to me at all.

Is it possible that my models are attempted to be created twice by pydoc when called from the root?

it certainly seems that way.   Try putting a pdb.set_trace() inside of database.models.model.py and see if it's called twice.

[Yaakov Bressler]

Will do, will follow up with results.