Using metadata built from parent process's engine

[Zach]

Our application uses gunicorn and we are looking to enable preload_app. In preparation, we moved the creation of the sqlalchemy engine into a post-fork hook for each gunicorn worker, ensuring that DB connections are created separately within each process.

Unfortunately our sqlalchemy declarative base class, which is imperative during app preload for import dependency reasons, needs a MetaData object with specific naming conventions. Because MetaData requires an engine, we initially create an engine, instantiate the MetaData object, then after application preload, dispose of the engine’s connection. After this, the post-fork hook kicks in and all processes’ engines and sessions are created.

Our question is: Could this implementation become problematic, and how might we test it?

Preload:

from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base

engine = create_engine(conn_string)
metadata = MetaData(engine, naming_convention=convention)
Base = declarative_base(cls=Base, metadata=metadata)

# Multiple model classes inherit from Base

After preload:

engine.dispose()

Post-fork hooks:

# Runs once for every process
# All model classes are still based on the original declarative_base metadata

engine = create_engine(conn_string)
# Session is created from engine

Thank you.

[Mike Bayer]

On Wed, Jul 10, 2019, at 5:04 PM, Zach wrote:

Our application uses gunicorn and we are looking to enable preload_app. In preparation, we moved the creation of the sqlalchemy engine into a post-fork hook for each gunicorn worker, ensuring that DB connections are created separately within each process.

that's good

Unfortunately our sqlalchemy declarative base class, which is imperative during app preload for import dependency reasons, needs a MetaData object with specific naming conventions. Because MetaData requires an engine,

MetaData does not require an engine, and in fact, the whole ability to associate an engine with a MetaData is going to be deprecated soon, because it has been confusing people for years, and despite my taking virtually all mentions of it out of the docs and putting a big dragon warning in that section (https://docs.sqlalchemy.org/en/13/core/connections.html#connectionless-execution-implicit-execution) everyone still seems to think it needs it to be there (I am mystified why this is but a deprecation warning should hopefully do the trick).

does that help?

we initially create an engine, instantiate the MetaData object, then after application preload, dispose of the engine’s connection. After this, the post-fork hook kicks in and all processes’ engines and sessions are created.

Our question is: Could this implementation become problematic, and how might we test it?

Preload:

from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base

engine = create_engine(conn_string)
metadata = MetaData(engine, naming_convention=convention)
Base = declarative_base(cls=Base, metadata=metadata)

# Multiple model classes inherit from Base

After preload:

engine.dispose()

Post-fork hooks:

# Runs once for every process
# All model classes are still based on the original declarative_base metadata

engine = create_engine(conn_string)
# Session is created from engine

Thank you.

--

SQLAlchemy -

The Python SQL Toolkit and Object Relational Mapper

 

http://www.sqlalchemy.org/

 

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description.

---

You received this message because you are subscribed to the Google Groups "sqlalchemy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+...@googlegroups.com.

To post to this group, send email to sqlal...@googlegroups.com.

Visit this group at https://groups.google.com/group/sqlalchemy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/6c892370-1f0e-4e05-98ab-5f256b21fb0d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.