Issue with dynamically loaded modules and SQLAlchemy

[Jeffrey McLarty]

Hello,

I've got a very peculiar situation, where SQLA fails to create an engine, because it can't find the installed module, only when I do some relatively straightforward package discovery & import logic using imp.find_module and imp.load_module.

I'm wondering if SQLAlchemy does some magic which might be impacted if I've used find_module and load_module upstream, despite being in different name spaces(?)

I'm working with Python 2.7.6, SQLAlchemy 0.9.2, and Postgres 9.4, and psycopg2

Here's the traceback...

Traceback (most recent call last):
  File "C:\WinPython-32bit-2.7.6.3-20140407\python-2.7.6\lib\pdb.py", line 1314,
 in main
    pdb._runscript(mainpyfile)
  File "C:\WinPython-32bit-2.7.6.3-20140407\python-2.7.6\lib\pdb.py", line 1233,
 in _runscript
    self.run(statement)
  File "C:\WinPython-32bit-2.7.6.3-20140407\python-2.7.6\lib\bdb.py", line 400,
in run
    exec cmd in globals, locals
  File "<string>", line 1, in <module>
  File "install.py", line 2, in <module>
    from equitable.db.psyw import DBpicker, SQLAeng, SmartDB
  File "C:\WinPython-32bit-2.7.6.3-20140407\python-2.7.6\lib\site-packages\trump
\orm.py", line 2287, in SetupTrump
    engine = create_engine(engine_str)
  File "C:\WinPython-32bit-2.7.6.3-20140407\python-2.7.6\lib\site-packages\sqlal
chemy\engine\__init__.py", line 344, in create_engine
    return strategy.create(*args, **kwargs)
  File "C:\WinPython-32bit-2.7.6.3-20140407\python-2.7.6\lib\site-packages\sqlal
chemy\engine\strategies.py", line 50, in create
    dialect_cls = u.get_dialect()
  File "C:\WinPython-32bit-2.7.6.3-20140407\python-2.7.6\lib\site-packages\sqlal
chemy\engine\url.py", line 116, in get_dialect
    cls = registry.load(name)
  File "C:\WinPython-32bit-2.7.6.3-20140407\python-2.7.6\lib\site-packages\sqlal
chemy\util\langhelpers.py", line 186, in load
    (self.group, name))
NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:postgresql.psycopg2

SQLAlchemy works as expected, (ie, it finds psycopg2) if I comment out line 60 of https://github.com/Equitable/trump/blob/master/trump/orm.py (and replace the variable "sources" with an empty dictionary).  Line 60, is an import to sub-module which dynamically loads modules.

[Mike Bayer]

On 7/17/15 1:21 PM, Jeffrey McLarty wrote:

Hello,

I've got a very peculiar situation, where SQLA fails to create an engine, because it can't find the installed module, only when I do some relatively straightforward package discovery & import logic using imp.find_module and imp.load_module.

feel free to share that logic.  Otherwise, no clue.

I'm wondering if SQLAlchemy does some magic

only if you consider this to be magic:

module = __import__('sqlalchemy.dialects.%s' % (dialect, )).dialects

--
You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+...@googlegroups.com.
To post to this group, send email to sqlal...@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[Mike Bayer]

On 7/17/15 1:39 PM, Mike Bayer wrote:

On 7/17/15 1:21 PM, Jeffrey McLarty wrote:

Hello,

I've got a very peculiar situation, where SQLA fails to create an engine, because it can't find the installed module, only when I do some relatively straightforward package discovery & import logic using imp.find_module and imp.load_module.

feel free to share that logic.  Otherwise, no clue.

oh sorry, I see you did.     this is a little more code than I can follow, but you want to make sure you don't prohibit the __import__ line below from working.

[Jeffrey McLarty]

Thanks for the pointers; that was all I needed. Plunking around SQLA's PluginLoader and dialect.__init__.py with pdb...allowed me to figure out the problem.

__import__ doesn't reload packages if they are already loaded, but the check to determine "already loaded",  obviously can't include a comparison with folder location, since __import__ is given only a name.  I had an extension, **isolated** from both my package and site-packages, named "psycopg2", loaded via an explicit folder+name call using load_module()...but after I loaded it, SQLA couldn't find the original "psycopg2" package.

I just renamed my project's extension, to something other than an exact match for psycopg2; all worked.

Thanks SO much, for your help Mike, and making SQLA one the most awesome libraries ever.

Jeffrey McLarty

--
You received this message because you are subscribed to a topic in the Google Groups "sqlalchemy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sqlalchemy/AbPSrqH0kb4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sqlalchemy+...@googlegroups.com.