Update Username and Password from vault

[Steven Schierholz]

So I have seen some chats here about cred refresh from vault and some suggestions have been to use @event.listens_for(engine, "do_connect") to update creds when the connection is established. My understanding of this is that connecting to the database should only happen once when my flask application starts up, but I need to update the creds without restarting my application so I'm not sure that the event listener will work in my case.

Am I understanding that correctly? If so, is there a way to get the right creds to pass to the engine for sqlalchemy every 24 hours when the creds from vault get updated without restarting my application?

[Mike Bayer]

the documentation for this pattern is at https://docs.sqlalchemy.org/en/20/core/engines.html#generating-dynamic-authentication-tokens , and a completely specific example is at https://docs.sqlalchemy.org/en/20/dialects/mssql.html#mssql-pyodbc-access-tokens .   Basically your application needs to have some way to retrieve the correct credentials as it runs, and you hook that into the event to populate the connect arguments with the correct credentials.

On Thu, Sep 7, 2023, at 1:54 PM, Steven Schierholz wrote:

So I have seen some chats here about cred refresh from vault and some suggestions have been to use @event.listens_for(engine, "do_connect") to update creds when the connection is established. My understanding of this is that connecting to the database should only happen once when my flask application starts up, but I need to update the creds without restarting my application so I'm not sure that the event listener will work in my case.

Am I understanding that correctly? If so, is there a way to get the right creds to pass to the engine for sqlalchemy every 24 hours when the creds from vault get updated without restarting my application?

--

SQLAlchemy -

The Python SQL Toolkit and Object Relational Mapper

 

http://www.sqlalchemy.org/

 

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description.

---

You received this message because you are subscribed to the Google Groups "sqlalchemy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/2de2e3fb-526c-49f7-8780-9ef55a9ad8bcn%40googlegroups.com.

[Steven Schierholz]

That makes sense but doesn't connect only happen once when create_engine() is called?

[Mike Bayer]

no, create_engine() does not connect at all.     connections occur when you first call `engine.connect()`.   From that point, the behavior of subsequent `engine.connect()` calls depends on connection pool configuration.   all connection pools have points at which they continue to establish new connections as the application proceeds, it's just a question of how often and under what circumstances.    The default QueuePool will make new connections when it goes into "overflow", as well as when existing connections are invalidated due to connectivity problems or if the pool_recycle timeout is reached.

On Thu, Sep 7, 2023, at 2:34 PM, Steven Schierholz wrote:

That makes sense but doesn't connect only happen once when create_engine() is called?

On Thursday, September 7, 2023 at 12:00:35 PM UTC-6 Mike Bayer wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/5c9c5b5f-a84b-41b8-8eb2-c3117fbf50f5n%40googlegroups.com.

[Steven Schierholz]

Ok that makes sense and clarifies some stuff for me. I have tried your implementation but it doesn't seem like its getting new connections. We are using sessionmaker(). So basically this is what we are doing. Can you help me understand if we are doing this right and if any changes need to happen to make this work? Sorry the tabbing is not right after paste. Thanks for your help!

# Create the engine to connect to the database
engine = create_engine(
  f"postgresql+psycopg2://test:password@{pg_host}:5432/{pg_database}",
  # connect_args=ssl_args,
  connect_args={"options": "-c timezone=utc"},
  pool_pre_ping=True,
  encoding="utf8",
)

@event.listens_for(engine, "do_connect")
def receive_do_connect(dialect, conn_rec, cargs, cparams):

# Getting the postgres details
try:
# Get the configs
configs = Properties()

# Open the file to get the values needed
with open("/var/secrets/pgsql/vault-credentials.properties", "rb") as config_file:
configs.load(config_file)

# Get each of the properties, hopefully
pg_user = configs.get("username").data
pg_password = configs.get("password").data

except FileNotFoundError:

# Use whats in the environment
pg_user = os.getenv("pg_user")
pg_password = os.getenv("pg_password")

print("Connecting to db with username: ", pg_user)
print("Connecting to db with password: ", pg_password)

cparams["user"] = pg_user
cparams["password"] = pg_password

session_factory = sessionmaker(bind=engine)
sqla_session = session_factory()

# Then using the sqla_session to execute queries and make modifications to the database

[Mike Bayer]

assuming proper indentation it looks fine.  are your print statements being seen ?

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/a2aecc6b-7a9b-418a-a41f-80d7366a8d7bn%40googlegroups.com.

[Steven Schierholz]

Yes but only once when the app starts up.

[Mike Bayer]

makes sense, the connection is pooled.  if you make lots of connections, like more than five simultaneous connections, you'd see more of it, if you call engine.dispose() then engine.connect(), you would see it again also, etc.   Also try using NullPool, then you'd see the hook run every time the engine is used.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/90a985d8-ef5e-4107-be95-feb530ddefbcn%40googlegroups.com.