Event listener for when query results get matched with pre-existing objects on the session

[Tony Cosentini]

Hi,

Is there any way to listen for an event for when a query result gets merged into a pre-existing object in the session?

I'm working on disabling autoflush for some of our codebase (mostly to cut down on queries on a high QPS path), but before doing that I want to write a utility to detect when we read data that was flushed via autoflush to cut down on any potential bugs.

What I'd like to do is this (while autoflush is still enabled):

table_one = TableOne(name='test')
session.add(table_one) # Track this object as pending a flush via the before_* listeners, this is working as expected.

table_two_instances = session.query(TableTwo).all() # All good, doesn't do anything with the table_one instance created earlier

table_one_instances = session.query(TableOne).all() # I would like to log a warning here as the results of this query depend on a flush happening. What I'm hoping to do is detect that one of the rows coming back is the same object that was flagged earlier, but I can't see to find the correct event to use.

Is this possible? I can't seem to find the appropriate event that would trigger when the results from the .all() query get merged back in with the existing objects in the session.

Thanks,

Tony

[Mike Bayer]

On Fri, Oct 20, 2023, at 8:08 AM, 'Tony Cosentini' via sqlalchemy wrote:

Hi,

Is there any way to listen for an event for when a query result gets merged into a pre-existing object in the session?

this is the refresh event:

https://docs.sqlalchemy.org/en/20/orm/events.html#sqlalchemy.orm.InstanceEvents.refresh

I'm working on disabling autoflush for some of our codebase (mostly to cut down on queries on a high QPS path), but before doing that I want to write a utility to detect when we read data that was flushed via autoflush to cut down on any potential bugs.

What I'd like to do is this (while autoflush is still enabled):

table_one = TableOne(name='test')
session.add(table_one) # Track this object as pending a flush via the before_* listeners, this is working as expected.

table_two_instances = session.query(TableTwo).all() # All good, doesn't do anything with the table_one instance created earlier

table_one_instances = session.query(TableOne).all() # I would like to log a warning here as the results of this query depend on a flush happening. What I'm hoping to do is detect that one of the rows coming back is the same object that was flagged earlier, but I can't see to find the correct event to use.

you probably want to also use refresh_flush also, which will be invoked for column defaults that are populated on the object

https://docs.sqlalchemy.org/en/20/orm/events.html#sqlalchemy.orm.InstanceEvents.refresh_flush

Is this possible? I can't seem to find the appropriate event that would trigger when the results from the .all() query get merged back in with the existing objects in the session.

that's the refresh event

[Tony Cosentini]

Weird, I did try that but I can't seem to trigger it.

Here's a self-contained test: https://gist.github.com/tonycosentini/4dee3478695d032ca67707b5e26739b6

I have the event listener set to just crash, but it never triggers.

--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
 
http://www.sqlalchemy.org/
 
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description.
---
You received this message because you are subscribed to a topic in the Google Groups "sqlalchemy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sqlalchemy/9bevy3Kpql4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sqlalchemy+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/c5b89f67-e3fa-45f2-8774-463d154514f3%40app.fastmail.com.

[Mike Bayer]

On Fri, Oct 20, 2023, at 9:50 AM, 'Tony Cosentini' via sqlalchemy wrote:

Weird, I did try that but I can't seem to trigger it.

Here's a self-contained test: https://gist.github.com/tonycosentini/4dee3478695d032ca67707b5e26739b6

the object was not affected in that query (that is, not mutated).  if you change the query to this:

     table_one_instances = session.query(TableOne).populate_existing().all()

then the refresh event is triggered.

Im not really following what kind of bug you are trying to detect.

You received this message because you are subscribed to the Google Groups "sqlalchemy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/CAEx_o%2BAhhUSPDYuGL4TFkTvxjHYfKEWUnBMxh2PzCDw3GJPaGg%40mail.gmail.com.

[Tony Cosentini]

Oh I see, thanks for clarifying.

I'm trying to detect cases where we depend on the autoflush behavior. For example, in the sample above, when the query runs with no_autoflush, we won't get back any results (when previously we would get back one row). It's a fairly large codebase, so I was trying to automate finding these cases in order to add explicit flush calls when needed or just pass in the pending object instead of running a query.

Tony

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/1047d2d1-6964-4d03-a978-dbac1b3e57ec%40app.fastmail.com.

[Mike Bayer]

On Fri, Oct 20, 2023, at 10:46 AM, 'Tony Cosentini' via sqlalchemy wrote:

Oh I see, thanks for clarifying.

I'm trying to detect cases where we depend on the autoflush behavior. For example, in the sample above, when the query runs with no_autoflush, we won't get back any results (when previously we would get back one row). It's a fairly large codebase, so I was trying to automate finding these cases in order to add explicit flush calls when needed or just pass in the pending object instead of running a query.

there's not really an event for an object that just gets returned by a query from the identity map.    you would need to do something more drastic like a do_orm_execute() hook that runs queries internally and then looks at all the returned objects.  i wouldnt put that in production.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/CAEx_o%2BDfRUDP6rmNUrN8pNpLvo4EQeWrUdhz9-7cm%2BKOS0Gf5g%40mail.gmail.com.

[Tony Cosentini]

do_orm_execute() (and freezing the results) totally allows for what I'm trying to do! Also I forgot to mention earlier, but this is only going to run in tests.

Thanks again for all the help,

Tony

[Mike Bayer]

great, glad to know that new API is finding itself useful.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/8f2d4fe1-5fff-4a1b-86b6-7307e500e725n%40googlegroups.com.