Callback for when orm data is load

[Mike Bernson]

I am moving from version 1.0.14 to version 1.4.11 of sqlalchemy.

I need to get a callback when all of the data from orm query is loaded not just single instance.

In the 1.0 version I created the session using query_cls:

self.session = Session(bind=engine, autoflush=False,
       _enable_transaction_accounting=False, autocommit=True,
        query_cls=my_query)

and the query class
class my_query(Query):

    def _execute_and_instances(self, context):
        results = list(super(my_query, self)._execute_and_instances(context))
        rdbm_events.run_created()
        for instance in results:
            yield instance

    def instances(self, cusor, __context=None):
        results = list(super(my_query, self).instances(cusor, __context))
        rdbm_events.run_created()
        for instance in results:
            yield instance

This does not work for 1.4.11.
What is the best way to get a callback when all
the data from query or lazy/deferred  loaded into instances ?

[Mike Bayer]

the event that corresponds most directly here is the do_orm_execute() event:

https://docs.sqlalchemy.org/en/14/orm/session_events.html#session-execute-events

you would want to use the "re-execute statement" feature described at:

https://docs.sqlalchemy.org/en/14/orm/session_events.html#re-executing-statements

the example there is a little more complex than what you need.  you would have something more like:

@event.listens_for(Session, "do_orm_execute")

def _do_orm_execute(orm_execute_state):

    if orm_execute_state.is_orm_statement:

        result = orm_execute_state.invoke_statement()

        rdbm_events.run_created()

        return result

-- 

SQLAlchemy - 

The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example.  See  http://stackoverflow.com/help/mcve for a full description.

--- 

You received this message because you are subscribed to the Google Groups "sqlalchemy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/164552c2-f11a-526b-abae-a8287afdf3e6%40mlb.org.

[Mike Bernson]

That callback look like it happening before orm instances are created. I am looking for a
callback that is after all the instances are created. I am using the events to get a list of instances
that need work done when they are loaded from the database.

I need the instance data and relationship data filled in before run_created is called.

 listen(mapper, 'load', self.load_instance)
 listen(mapper, 'refresh', self.refresh_instance)

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/1422b5e7-e15f-49ed-894c-9ba718a6f0f3%40www.fastmail.com.

[Mike Bayer]

they're created as the result object is iterated, which is up to how the end user handles the result.

there's a not-yet documented execution option you can use to prebuffer the rows:

    orm_execute_state.update_execution_options(prebuffer_rows=True)

    result = orm_execute_state.invoke_statement()

    # objects are loaded when we get here

    return result

note that in your code example, you dont seem to be actually passing the loaded ORM objects to your custm methods so I was going off of that.

or you could do what's in the example at https://docs.sqlalchemy.org/en/14/orm/session_events.html#re-executing-statements and freeze the result, then return a new one

    frozen_result = orm_execute_state.invoke_statement().freeze()

    # objects are loaded when we get here

    return frozen_result()

you can use that freeze thing to get the results more than once, if you want to actually iterate the objects and then return a result:

    frozen_result = orm_execute_state.invoke_statement().freeze()

    process_my_objects(frozen_result())

    # then return it

    return frozen_result()

also for a long time there's been the ORM load() event which is fired for every object as it is loaded:  https://docs.sqlalchemy.org/en/14/orm/events.html?highlight=instanceevents%20load#sqlalchemy.orm.InstanceEvents.load

doesn't play as well with eager loaders though.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/90bb5eae-a20d-228d-a720-78ae07790d7a%40mlb.org.

[Mike Bernson]

The load and refresh events is close to what I am looking for.

I need to be able to modify attribute of object/instance that have been
load or refreshed. I can not do the work in load or refresh event because
any changes made in the events are not track and written out on a flush/commit.
.
I am using the load event and refresh event to build build a list of object/instance that
have been loaded or refreshed and my query class does the callback to instance that
have been loaded or refresh. This work for query that I do but does not handle the case
where lazy loading or deferred/refresh attribute.

What I looking for is way to get callback for any data that loaded or refreshed once it is safe to modify
attribute. I can the use the list of instance that need to be worked based on load and refresh events.

I only need this for ORM. I am filter stuff in event

instance_new = []

def _load_event(self, target, context):
    global instance_new

    if '_sa_orm_load_options' in context.execution_options:
       instance_new.append(target)

       

On 4/27/21 4:51 PM, Mike Bayer wrote:'

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/a71cc745-9e30-4424-8308-0422d75fef49%40www.fastmail.com.

[Mike Bernson]

Looking at the code in loading I see where instance  state is
getting set to unmodified.

attr.py in _instance at line 975:
                if effective_populate_existing or state.modified:
                    if refresh_state and only_load_props:
                        state._commit(dict_, only_load_props)
                    else:
                        state._commit_all(dict_, session_identity_map)

are there any events after this point in the loading/refresh process ?