deferred column_properties should probably not be expired unless they were already loaded
10 views
Skip to first unread message
Kent
May 10, 2017, 11:35:30 AM5/10/17
to sqlalchemy
deferred column_properties may be less-efficient subquery selects (and thus marked deferred). When a flush occurs that updates an object, any read-only column_properties are marked as expired, even if they weren't even loaded. This means if the object needs to be refreshed, all these deferred column properties are loaded.
We probably want the behavior to only expire read-only attributes that were actually loaded, right?
See attached script. This behavior is as of 1.1.1
Thoughts?
Kent
May 10, 2017, 12:32:45 PM5/10/17
to sqlalchemy
Another question surrounding this:
in persistence.py:
def _finalize_insert_update_commands(...)
....
if mapper._readonly_props:
readonly = state.unmodified_intersection(
[p.key for p in mapper._readonly_props
if p.expire_on_flush or p.key not in state.dict]
)
if readonly:
state._expire_attributes(state.dict, readonly)
I'm confused by "or p.key not in state.dict"... wouldn't we want to expire the ones that are in state.dict?
Wouldn't we want
"or p.key in state.dict"
?
Just wanted someone to look that code over again.
Kent
May 10, 2017, 12:37:43 PM5/10/17
to sqlalchemy
I'm thinking that should be
"if p.expire_on_flush and p.key in state.dict"
On Wednesday, May 10, 2017 at 11:35:30 AM UTC-4, Kent wrote:
mike bayer
May 10, 2017, 12:55:45 PM5/10/17
to sqlal...@googlegroups.com
so you can confirm this is only for custom SQL + column_property(), not
a regular column right? definitely a bug for 1.2 if you can post it up
On 05/10/2017 12:37 PM, Kent wrote:
> I'm thinking that should be
> *"if p.expire_on_flush and p.key in state.dict"*
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
> description.
> ---
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlalchemy+...@googlegroups.com
> <mailto:sqlalchemy+...@googlegroups.com>.
> To post to this group, send email to sqlal...@googlegroups.com
> <mailto:sqlal...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.
a regular column right? definitely a bug for 1.2 if you can post it up
On 05/10/2017 12:37 PM, Kent wrote:
> I'm thinking that should be
>
>
> On Wednesday, May 10, 2017 at 11:35:30 AM UTC-4, Kent wrote:
>
> deferred column_properties may be less-efficient subquery selects
> (and thus marked deferred). When a flush occurs that updates an
> object, any read-only column_properties are marked as expired, even
> if they weren't even loaded. This means if the object needs to be
> refreshed, all these deferred column properties are loaded.
>
> We probably want the behavior to only expire read-only attributes
> that were actually loaded, right?
>
> See attached script. This behavior is as of 1.1.1
>
> Thoughts?
>
>
> --
>
> On Wednesday, May 10, 2017 at 11:35:30 AM UTC-4, Kent wrote:
>
> deferred column_properties may be less-efficient subquery selects
> (and thus marked deferred). When a flush occurs that updates an
> object, any read-only column_properties are marked as expired, even
> if they weren't even loaded. This means if the object needs to be
> refreshed, all these deferred column properties are loaded.
>
> We probably want the behavior to only expire read-only attributes
> that were actually loaded, right?
>
> See attached script. This behavior is as of 1.1.1
>
> Thoughts?
>
>
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
> description.
> ---
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlalchemy+...@googlegroups.com
> <mailto:sqlalchemy+...@googlegroups.com>.
> To post to this group, send email to sqlal...@googlegroups.com
> <mailto:sqlal...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.
Kent Bower
May 10, 2017, 1:36:21 PM5/10/17
to sqlal...@googlegroups.com
The regular columns seem to expire and reload properly without issue. (Is that what you're asking?)
You want me to submit a PR changing:
if p.expire_on_flush or p.key not in state.dict
to
if p.expire_on_flush and p.key in state.dict
?
On Wed, May 10, 2017 at 12:55 PM, mike bayer <mik...@zzzcomputing.com> wrote:
so you can confirm this is only for custom SQL + column_property(), not a regular column right? definitely a bug for 1.2 if you can post it up
On 05/10/2017 12:37 PM, Kent wrote:
I'm thinking that should be
*"if p.expire_on_flush and p.key in state.dict"*
On Wednesday, May 10, 2017 at 11:35:30 AM UTC-4, Kent wrote:
deferred column_properties may be less-efficient subquery selects
(and thus marked deferred). When a flush occurs that updates an
object, any read-only column_properties are marked as expired, even
if they weren't even loaded. This means if the object needs to be
refreshed, all these deferred column properties are loaded.
We probably want the behavior to only expire read-only attributes
that were actually loaded, right?
See attached script. This behavior is as of 1.1.1
Thoughts?
--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description.
---
You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscribe@googlegroups.com <mailto:sqlalchemy+unsubscribe@googlegroups.com>.
To post to this group, send email to sqlal...@googlegroups.com <mailto:sqlalchemy@googlegroups.com>.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description.
--- You received this message because you are subscribed to a topic in the Google Groups "sqlalchemy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sqlalchemy/g0QVQ6RbxNU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sqlalchemy+unsubscribe@googlegroups.com.
To post to this group, send email to sqlal...@googlegroups.com.
Kent
May 10, 2017, 1:38:41 PM5/10/17
to sqlalchemy
The regular columns seem to expire and reload properly without issue. (Is that what you're asking?)
You want me to submit a PR changing:
if p.expire_on_flush or p.key not in state.dict
to
if p.expire_on_flush and p.key in state.dict
?
(If so, which branch?)
> <mailto:sqlalchemy+unsub...@googlegroups.com>.
mike bayer
May 10, 2017, 1:48:42 PM5/10/17
to sqlal...@googlegroups.com
nevermind, the issue is at
https://bitbucket.org/zzzeek/sqlalchemy/issues/3984/deferred-column_property-gets-set-to
the fix is not as obvious as that, that particular check is assuming a
column_property() where its value was never present in __dict__ in the
first place, so it needs to be marked "expired".
On 05/10/2017 01:38 PM, Kent wrote:
> The regular columns seem to expire and reload properly without issue.
> (Is that what you're asking?)
>
> You want me to submit a PR changing:
>
> if p.expire_on_flush or p.key*not*in state.dict
>
> to
>
> if p.expire_on_flush*and*p.key in state.dict*
> *
> > an email to sqlalchemy+...@googlegroups.com <javascript:>
> > <mailto:sqlalchemy+...@googlegroups.com <javascript:>>.
> > <mailto:sqlal...@googlegroups.com <javascript:>>.
https://bitbucket.org/zzzeek/sqlalchemy/issues/3984/deferred-column_property-gets-set-to
the fix is not as obvious as that, that particular check is assuming a
column_property() where its value was never present in __dict__ in the
first place, so it needs to be marked "expired".
On 05/10/2017 01:38 PM, Kent wrote:
> The regular columns seem to expire and reload properly without issue.
> (Is that what you're asking?)
>
> You want me to submit a PR changing:
>
>
> to
>
> if p.expire_on_flush*and*p.key in state.dict*
> *
> > <mailto:sqlalchemy+...@googlegroups.com <javascript:>>.
> > To post to this group, send email to sqlal...@googlegroups.com
> <javascript:>
> > <mailto:sqlal...@googlegroups.com <javascript:>>.
> > Visit this group at https://groups.google.com/group/sqlalchemy
> <https://groups.google.com/group/sqlalchemy>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
> description.
> ---
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlalchemy+...@googlegroups.com
> <mailto:sqlalchemy+...@googlegroups.com>.
> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
> description.
> ---
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlalchemy+...@googlegroups.com
Kent Bower
May 10, 2017, 2:00:46 PM5/10/17
to sqlal...@googlegroups.com
If never present in __dict__, why does it need to be marked as expired after an insert or update? If not in __dict__ and referenced, isn't won't it load as whether or not it is marked as expired?
> <mailto:sqlalchemy+unsubscribe@googlegroups.com <javascript:>>.
> To post to this group, send email to sqlal...@googlegroups.com
<javascript:>
> <mailto:sqlal...@googlegroups.com <javascript:>>.
> Visit this group at https://groups.google.com/group/sqlalchemy
<https://groups.google.com/group/sqlalchemy>.
> For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description.
---
You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscribe@googlegroups.com <mailto:sqlalchemy+unsubscribe@googlegroups.com>.
To post to this group, send email to sqlal...@googlegroups.com <mailto:sqlalchemy@googlegroups.com>.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description.
--- You received this message because you are subscribed to a topic in the Google Groups "sqlalchemy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sqlalchemy/g0QVQ6RbxNU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sqlalchemy+unsubscribe@googlegroups.com.
To post to this group, send email to sqlal...@googlegroups.com.
mike bayer
May 10, 2017, 2:07:56 PM5/10/17
to sqlal...@googlegroups.com
On 05/10/2017 02:00 PM, Kent Bower wrote:
> If never present in __dict__, why does it need to be marked as expired
> after an insert or update? If not in __dict__ and referenced, isn't
> won't it load as whether or not it is marked as expired?
1. an attribute is a normal column-oriented attribute
2. the object is "persistent", meaning, it refers to a row in the DB
3. the attribute is not present in __dict__
then:
when you go to access the attribute, the decision made is as follows:
1. if the attribute has a lazy loader associated with it, use that.
this is only present if the attribute is "deferred" at the configuration
level or via a query option.
2. if the attribute has an expiry callable associated with it, use that.
this is only present if the attribute was explicitly marked as expired
3. otherwise, the attribute has no value, return None
The logic you see with "key not in dict" is making sure we hit #2 and
not #3 when the object goes from pending to persistent and the column
attribute wasn't loaded.
Run test/orm/test_unitofwork.py to see one test fail if you make just
the simple change, where we hit #3 instead of #2 for a simple
column_property.
> > <mailto:sqlalchemy+...@googlegroups.com
> <mailto:sqlalchemy%2Bunsu...@googlegroups.com> <javascript:>>.
> > To post to this group, send email to
> sqlal...@googlegroups.com <mailto:sqlal...@googlegroups.com>
> <javascript:>
> > <mailto:sqlal...@googlegroups.com
> <mailto:sqlal...@googlegroups.com> <javascript:>>.
> > Visit this group at
> https://groups.google.com/group/sqlalchemy
> <https://groups.google.com/group/sqlalchemy>
> <https://groups.google.com/group/sqlalchemy
> <https://groups.google.com/group/sqlalchemy>>.
> > For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
>
> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete,
> and Verifiable Example. See http://stackoverflow.com/help/mcve
> > Visit this group at
> https://groups.google.com/group/sqlalchemy
> <https://groups.google.com/group/sqlalchemy>
> <https://groups.google.com/group/sqlalchemy
> <https://groups.google.com/group/sqlalchemy>>.
> > For more options, visit
> https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
>
> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete,
> and Verifiable Example. See http://stackoverflow.com/help/mcve
> <http://stackoverflow.com/help/mcve> for a full description.
> ---
> You received this message because you are subscribed to the
> Google Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to sqlalchemy+...@googlegroups.com
> <mailto:sqlalchemy%2Bunsu...@googlegroups.com>
> ---
> You received this message because you are subscribed to the
> Google Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to sqlalchemy+...@googlegroups.com
> <mailto:sqlalchemy+...@googlegroups.com
> <mailto:sqlalchemy%2Bunsu...@googlegroups.com>>.
> To post to this group, send email to sqlal...@googlegroups.com
> <mailto:sqlal...@googlegroups.com>
> <mailto:sqlal...@googlegroups.com
> <mailto:sqlal...@googlegroups.com>>.
> Visit this group at https://groups.google.com/group/sqlalchemy
> <https://groups.google.com/group/sqlalchemy>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve
> <https://groups.google.com/group/sqlalchemy>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve
> <http://stackoverflow.com/help/mcve> for a full description.
> --- You received this message because you are subscribed to a topic
> in the Google Groups "sqlalchemy" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/sqlalchemy/g0QVQ6RbxNU/unsubscribe
> <https://groups.google.com/d/topic/sqlalchemy/g0QVQ6RbxNU/unsubscribe>.
> --- You received this message because you are subscribed to a topic
> in the Google Groups "sqlalchemy" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/sqlalchemy/g0QVQ6RbxNU/unsubscribe
> To unsubscribe from this group and all its topics, send an email to
> sqlalchemy+...@googlegroups.com
> <mailto:sqlalchemy%2Bunsu...@googlegroups.com>.
> To post to this group, send email to sqlal...@googlegroups.com
> <mailto:sqlal...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/sqlalchemy
> <https://groups.google.com/group/sqlalchemy>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
> description.
> ---
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sqlalchemy+...@googlegroups.com
> <https://groups.google.com/group/sqlalchemy>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
> description.
> ---
> You received this message because you are subscribed to the Google
> Groups "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send
> <mailto:sqlalchemy+...@googlegroups.com>.
> To post to this group, send email to sqlal...@googlegroups.com
> <mailto:sqlal...@googlegroups.com>.
mike bayer
May 10, 2017, 2:24:08 PM5/10/17
to sqlal...@googlegroups.com
this is all patched in 1.2, your original test works too.
The fix here is a little too intricate for 1.1 right now as this is a
very long-standing bug (goes back to 0.7 at least and probably further)
and 1.1 is getting near maintenance mode.
The fix here is a little too intricate for 1.1 right now as this is a
very long-standing bug (goes back to 0.7 at least and probably further)
and 1.1 is getting near maintenance mode.
Kent Bower
May 10, 2017, 2:27:57 PM5/10/17
to sqlal...@googlegroups.com
Thanks very much!
> <mailto:sqlalchemy+unsubscribe@googlegroups.com <javascript:>>.
> To post to this group, send email to sqlal...@googlegroups.com
<javascript:>
> <mailto:sqlal...@googlegroups.com <javascript:>>.
> Visit this group at https://groups.google.com/group/sqlalchemy
<https://groups.google.com/group/sqlalchemy>.
> For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description.
---
You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscribe@googlegroups.com <mailto:sqlalchemy+unsubscribe@googlegroups.com>.
To post to this group, send email to sqlal...@googlegroups.com <mailto:sqlalchemy@googlegroups.com>.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description.
--- You received this message because you are subscribed to a topic in the Google Groups "sqlalchemy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sqlalchemy/g0QVQ6RbxNU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sqlalchemy+unsubscribe@googlegroups.com.
To post to this group, send email to sqlal...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages