Integrating the ORM with Trellis
Phillip J. Eby
mappers with Trellis components, so that Trellis applications can use
SQLAlchemy in a relatively-transparent way.
The catch is that Trellis components use custom descriptors (like the
property builtin, only completely different) so that the Trellis
system can see when attributes are read or written, so that
inter-component dependencies can be automatically detected, and
dependencies automatically updated when values change.
I've been digging through the docs and the sqlalchemy.orm package
trying to figure out a clean way to integrate the two, but it appears
that the only documented ways to deal with something like this are to
either 1) create alternate attributes like '_foo' to be mapped, and
then delegate to those attributes from the descriptor, or 2) create
an dummy class to which the mapping is applied, and then create
linkages between the mapped object and the "real" object.
...and all that's just for the non-collection attributes. I'd really
like to use Trellis collections (which also detect and auto-notify
their dependencies), too.
Before diving even deeper into this, and possibly end up relying on
undocumented aspects of the ORM package for all this, I thought it
might be a good idea to see if anybody has any pointers on
accomplishing this, especially if there are standard interfaces or
things I can subclass to add this stuff in. My ideal outcome would
be if I could set things up so that when somebody defines mappings on
a Trellis component class, it "just works", even if that means I have
to have a whole bunch of logic behind the scenes plugging into
SQLAlchemy-supplied hooks.
If that isn't possible, however, I could provide an alternative
mapping interface, but I'd still like to be able to have SQLAlchemy
returning trellis components, rather than dummy objects that then
have to be re-wrapped. (Of course, there's yet another layer of
workaround possible, but...)
Anyway, the documentation for the mapping framework seems a bit slim,
apart from a few bits about doing collections. I thought I'd check
to make sure I'm looking in the right place(s), and check in to see
if any of the SQLAlchemy developers have any interest in this sort of
integration.
The primary use case for this Trellis-SQLAlchemy integration is
database-backed GUI applications that use the Trellis to
automatically update displays from model changes, and to update model
objects from user input. The Trellis makes this kind of code very
simple because it doesn't require explicit publish-subscribe
operations; inter-object dependencies are automatically detected, and
dependent operations are automatically re-run when their dependencies change.
The specific application that will be used for first, is the
experimental re-architecture branch of the Chandler open-source
PIM. Right now, there are some simple Trellis-based GUI and model
components, that we'd like to hook up to a database backend.
Anyway, if anyone has any pointers, ideas, etc., it would be most
helpful. Thanks.
sdo...@sistechnology.com
use trellis one day somehow coupled with a DB, inside my projects...
there sooo many dependencies to solve there. Now it might get
easier ;-).
Recent year i have done some things like what u need - transparently
wrapping SA on declaration side with descriptors et al. On the
runtime side (e.g. collections etc), i dont have much. i do have some
idea how to do those but not much experience.
on the point:
1. check out dbcook.sf.net/
svn co https://dbcook.svn.sf.net/svnroot/dbcook/trunk
2. check out my static_type library
svn co https://dbcook.svn.sf.net/svnroot/dbcook/static_type
then, see dbcook/dbcook/usage/static_type/ which concocts both, namely
u have staticaly_typed structs/attributes (validation etc) which are
also DB-aware i.e. SA-instrumented.
implementation details: i didnot want to overload SA-instrumented
descriptors, and i didnot want to copycat all attribute abc as _abc,
as this would break one of my goals - not to be able to workaround
what have been declared (validation, type etc). All value-access in
SA goes via direct __dict__ ... hence i hacked that one. Recently, in
0.4.x i needed to split the value-storage from autoset etc
defaultvalue behavior, as SA tracking of changes has evolved. Now the
value-storage part is "under" SA, and the autoset behaviour
is "above" SA.
yes it looks a bit like a quick and dirty hack, but it works (and i
maintain it, more than year already, both 0.4 and 0.3 supported). i
didnt have time to niceify it, and i haven't advertised all of it too
much as i am sort-of disappointed about how many people are
interested in really abstract programming (abstract not in pure
mathematical sence, but to be able to "talk" in app.field terms).
Anyway, it is there, and i am willing to work together to get
something on either side - or a new one.
if this gets too offtopic, u can take it off the list.
use either this email or the new a...@svilendobrev.com.
ciao
svilen dobrev
Michael Bayer
I've taken a look at Trellis and it seems like a SQLAlchemy/Trellis
integration layer would be a really cool thing to have. The first
thing I've noticed is that usage involves subclassing a Trellis base
class, which implies that when the object is later mapped via
SQLAlchemy, SQLAlchemy would somehow have to know to not step on
whatever instrumentation Trellis has set up.
Ill comment briefly on the "_foo" thing, thats the way we normally
recommend to have one's own Python property descriptors coexist with
those that SQLAlchemy provides; only because its easy to understand
and requires only very simplistic interaction between the user-defined
descriptor and SQLAlchemy's, which remains hidden behind the regular
getattr()/setattr() API.
In this case, I can certainly see that we probably want to do
something more integrated, if for no other reason than to prevent an
explosion of attribute names. If you took a look at attributes.py in
the ORM package, you'd see that the primary property descriptor is
called InstrumentedAttribute - its a very short class which delegates
the __get__(), __set__() and __del__() over to an "impl", which is
where the actual work is done. For a scalar attribute, attributes.py
handles the whole interaction. For a collection, it delegates some
responsibility over to collections.py, which is Jason Kirtland's area
of expertise.
So at one level, if we could get a TrellisInstrumentedAttribute to be
used instead, Trellis and SA could have a reasonably small overlapping
point on just that one object, with some additional instrumentation on
the collection side. We would probably have to add some SA hooks for
the InstrumentedAttribute override to happen, and I'd have to think
about the best way to provide that hook.
Theres also another level within the attributes package which is the
event system we have - you can add your own event receivers to
InstrumentedAttribute as well, which are subclasses of
sqlalchemy.orm.interfaces.AttributeExtension . Though it seems that
Trellis does more than just catch events so I'm not sure if that area
is appropriate...but if you wanted to consider riding on that, it just
receives the events append(), remove() and set().
So those are my initial thoughts. I want to think some more about how
user-defined InstrumentedAttributes could be cleanly embedded
throughout all attribute instrumentation....but let me know if any of
this seems doable so far.
Looking forward to your involvement with the project !
- mike
Phillip J. Eby
>Hi Phillip -
>
>I've taken a look at Trellis and it seems like a SQLAlchemy/Trellis
>integration layer would be a really cool thing to have. The first
>thing I've noticed is that usage involves subclassing a Trellis base
>class, which implies that when the object is later mapped via
>SQLAlchemy, SQLAlchemy would somehow have to know to not step on
>whatever instrumentation Trellis has set up.
Right. Ideally, SQLAlchemy wouldn't touch the class at all, but
instead touch the *instance*. Trellis attributes are stored in
"cell" objects with a '.value' property that does all the
instrumentation. So, mapping a Trellis object should consist of
replacing its cells with ones that talk to SQLAlchemy. These might
be a cell subclass, or just regular cells with appropriate rule(s).
>So at one level, if we could get a TrellisInstrumentedAttribute to be
>used instead, Trellis and SA could have a reasonably small overlapping
>point on just that one object, with some additional instrumentation on
>the collection side. We would probably have to add some SA hooks for
>the InstrumentedAttribute override to happen, and I'd have to think
>about the best way to provide that hook.
What I'm looking for now is how to decouple the massive amount of
stuff that happens when you declare a mapping with SQLAlchemy, separating out:
* recording the data about what/how to map
* modifying the class (i.e., so I can skip this entirely)
* setting up an instance (so I can hook this to replace the cells)
Failing that, I think I would need to provide an alternative mapping
declaration system that "compiles down" to SQLAlchemy and creates a
dummy class. The catch then is that SQLAlchemy's going to return
instances of the dummy class, so I'll end up having to wrap the APIs
that return instances... and then deal with queries, and so on,
until I'm duplicating every SQLAlchemy API.
However, if there are well-defined hook points where an object can
either handle some of this stuff on its own, or you can register
handlers for a type, etc., then maybe we could make this pretty transparent.
>Theres also another level within the attributes package which is the
>event system we have - you can add your own event receivers to
>InstrumentedAttribute as well, which are subclasses of
>sqlalchemy.orm.interfaces.AttributeExtension . Though it seems that
>Trellis does more than just catch events so I'm not sure if that area
>is appropriate...but if you wanted to consider riding on that, it just
>receives the events append(), remove() and set().
Yeah, I think I'd just as soon leave that to the existing cell
system, as that instrumentation is per-cell, not
per-attribute. (Cells are independent of what attribute they're used
in; Trellis descriptors are just there so you can do ob.attr =
someVal instead of ob.attr.value = someVal.)
>So those are my initial thoughts. I want to think some more about how
>user-defined InstrumentedAttributes could be cleanly embedded
>throughout all attribute instrumentation....
What I would suggest is that any place where right now you do stuff
directly on a mapped class, you delegate the operation to the class
instead of having the operation be directly in SQLAlchemy code. For
example, instead of directly instrumenting the class, you call
cls._sqlalchemy_instrument_attributes(...).
For any of these hook methods you don't find on the class... you
simply add them via monkeypatching (which you're already doing for
the descriptors). Thus, classes that defined their own versions of
the hooks would get to control the instrumentation, while classes
that did not, would get the existing behavior.
>but let me know if any of
>this seems doable so far.
Not so much, I'm afraid, due to the instance vs. class distinction.
sdo...@sistechnology.com
> > be used instead, Trellis and SA could have a reasonably small
> > overlapping point on just that one object, with some additional
> > instrumentation on the collection side. We would probably have
> > to add some SA hooks for the InstrumentedAttribute override to
> > happen, and I'd have to think about the best way to provide that
> > hook.
>
> What I'm looking for now is how to decouple the massive amount of
> stuff that happens when you declare a mapping with SQLAlchemy,
> separating out:
>
> * recording the data about what/how to map
> * modifying the class (i.e., so I can skip this entirely)
> * setting up an instance (so I can hook this to replace the cells)
>
> Failing that, I think I would need to provide an alternative
> mapping declaration system that "compiles down" to SQLAlchemy and
> creates a dummy class. The catch then is that SQLAlchemy's going
> to return instances of the dummy class, so I'll end up having to
> wrap the APIs that return instances... and then deal with queries,
> and so on, until I'm duplicating every SQLAlchemy API.
TrellisInstrumentedAttribute... Michael near slapped me last year when
i was talking about user-friendly (= user-defined)
instrumentations...
here some ideas... good bad or ugly.
IMO The declaration (class-level) stuff can be done in several ways.
Trellis class-stuff is separate from SA-class-stuff, they express 2
different aspects of an object type declaration. So they can be put
one over another, or side by side one to another (and have another
composite object encapsulate both). The first case is plain
inheritance+overloading, and the order of inheritance would be
important. It would be clear what happens but the approach may not be
scalable to more aspects. In the latter case one can have 3rd, 4th
etc n-th aspect of type declaration, linearly (and not by
inclusion/inheritance). There would be hooks on ctor, set/get, etc
level in both approaches. The order of these would be important as in
the inheritance case, as e.g. DB is the lowest level - it just stores
data.
svil
Michael Bayer
On Feb 26, 2008, at 2:43 PM, Phillip J. Eby wrote:
>
> Right. Ideally, SQLAlchemy wouldn't touch the class at all, but
> instead touch the *instance*. Trellis attributes are stored in
> "cell" objects with a '.value' property that does all the
> instrumentation. So, mapping a Trellis object should consist of
> replacing its cells with ones that talk to SQLAlchemy. These might
> be a cell subclass, or just regular cells with appropriate rule(s).
Well these days, the attributes package deals primarily with an object
attached to the instance called _state. _state mirrors (well, by
default directly references) the __dict__ of the object so its
possible that using a Trellis-specific _state implementation could be
the way to go.
We do have a little bit of stuff on the class, namely a variable
_class_state, which is the holding zone for things to do with class-
level instrumentation and also the link to all the mappers for a
class. If we were going to hookify class-level instrumentation in a
fine grained way I'd want it to stay within that namespace instead of
cluttering up the class with a bunch of _sqlalchemy_XXX methods.
_class_state is actually where we'd get at class-level
InstrumentedAttributes (they are there now). The ultimate goal with
SA instrumentation has been so that _class_state and _state are the
only two "magic" attributes you'd see on the instance. We have some
others like _instance_key and _entity_name but these are slated to be
accessed via _state as well.
Grepping around, we do have a lot of things like this:
return getattr(state.class_, self.key).impl.get(state)
Where "state" is an InstanceState.
So we'd probably have to replace those with higher level methods on
InstanceState such as:
return state.get(self.key)
To me this is like a cleanup thing, this seems like cleanup which
should be done in any case.
InstanceState itself would get at "impl" via ClassState instead of
getattr() (its there already in this way):
return self.class_._class_state.attrs[key]
ClassState and InstanceState would be the two classes that you'd
either have to decorate, subclass, or otherwise customize to talk to
Trellis' system.
Also we have a hook already where you could be setting up your own
_class_state and _state objects, within the MapperExtension API. Its
true that we could also have SA start looking for magic attributes on
the class itself but so far this has not been our style.
> What I'm looking for now is how to decouple the massive amount of
> stuff that happens when you declare a mapping with SQLAlchemy,
> separating out:
>
> * recording the data about what/how to map
> * modifying the class (i.e., so I can skip this entirely)
> * setting up an instance (so I can hook this to replace the cells)
So what I wonder is, how much of SA instrumentation are you looking to
re-implement ? This would be a big, big job depending on how much
you're looking to do. We have things like get_history(), commit(),
our existing AttributeExtension event system would have to still be
working in some way, etc. We have pretty comprehensive test coverage
for it as well so you could adapt our tests to exercise your Trellis
versions of these extensions.
> Failing that, I think I would need to provide an alternative mapping
> declaration system that "compiles down" to SQLAlchemy and creates a
> dummy class. The catch then is that SQLAlchemy's going to return
> instances of the dummy class, so I'll end up having to wrap the APIs
> that return instances... and then deal with queries, and so on,
> until I'm duplicating every SQLAlchemy API.
yeah that wouldn't be so ideal.
> However, if there are well-defined hook points where an object can
> either handle some of this stuff on its own, or you can register
> handlers for a type, etc., then maybe we could make this pretty
> transparent.
Currently, the MapperExtension provides hooks for instrument_class as
well as init_instance and/or create_instance. This is where you'd
dig in and fiddle around with ClassState and InstanceState. I'd
imagine that we'd turn ClassState into a factory for objects like
InstrumentedAttribute, InstanceState etc. so that the types of objects
generated in those cases can be modified to be Trellis-specific.
MapperExtension rides on the mapper() function and you could provide
it largely seamlessly by just wrapping mapper().
If you *really* want SA to just look for magic attributes on classes,
we can look into that, but it means that SA is now doing things in an
entirely different way than has historically. Keeping this to a
minimum, it might even be something like "__sqlalchemy_extension__ =
MyMapperExtension" which is picked up by mapper().
>>
> What I would suggest is that any place where right now you do stuff
> directly on a mapped class, you delegate the operation to the class
> instead of having the operation be directly in SQLAlchemy code. For
> example, instead of directly instrumenting the class, you call
> cls._sqlalchemy_instrument_attributes(...).
>
> For any of these hook methods you don't find on the class... you
> simply add them via monkeypatching (which you're already doing for
> the descriptors). Thus, classes that defined their own versions of
> the hooks would get to control the instrumentation, while classes
> that did not, would get the existing behavior.
let me know if you're willing to deal with the _class_state and _state
magic variables. On the SA side we should be able to get all class-
based as well as instance-based access through these methods...but the
hard part is that behavior will have to be maintained (and obviously
theres a lot of it).
Michael Bayer
On Feb 26, 2008, at 3:35 PM, sdo...@sistechnology.com wrote:
>
> TrellisInstrumentedAttribute... Michael near slapped me last year when
> i was talking about user-friendly (= user-defined)
> instrumentations...
a year ago was an eternity..... but also I wouldnt want to popularize
user-defined InstrumentedAttributes as the default way to do
something. Its a big job and I can see us needing to add some more
test suite just to ensure that 3rd party attribute refactorings
continue to function.
sdo...@sistechnology.com
(or whatever, can be a multiple things), all that SA test has to do
is to ensure this delegation does happen everywhere correctly -
workflow-wise. This would require actual definition of the workflow
and sticking to it (as a systematical protocol) - good thing to have
anyway (remember that __init__.__init thing?).
And to have a default implementation doing what it does now. Whether
the 3rd party is following the rules is up to it then... it could
replace/inherit/chain or make some Distributor that handles any
number of such extensions in a generic way...
jason kirtland
> At 02:06 PM 2/26/2008 -0500, Michael Bayer wrote:
>> Hi Phillip -
>>
>> I've taken a look at Trellis and it seems like a SQLAlchemy/Trellis
>> integration layer would be a really cool thing to have. The first
>> thing I've noticed is that usage involves subclassing a Trellis base
>> class, which implies that when the object is later mapped via
>> SQLAlchemy, SQLAlchemy would somehow have to know to not step on
>> whatever instrumentation Trellis has set up.
>
> Right. Ideally, SQLAlchemy wouldn't touch the class at all, but
> instead touch the *instance*. Trellis attributes are stored in
> "cell" objects with a '.value' property that does all the
> instrumentation. So, mapping a Trellis object should consist of
> replacing its cells with ones that talk to SQLAlchemy. These might
> be a cell subclass, or just regular cells with appropriate rule(s).
One of the services the SA descriptors provide is query construction via
column/relation mappings:
query(Foo).filter(Foo.bar == 2).filter(Foo.children.any(Bar.age > 2))
Not having those would diminish the utility of the ORM in user code-
querying based on higher-level relations would be lost. It looks like
the CellProperty descriptor provides __eq__ as well?
If combining descriptor functions is acceptable and the __eq__/__ne__
conflict can be resolved, I like Mike's suggestion about providing some
get/set routing logic in the InstanceState. Any Trellis-managed
attribute the ORM wanted to get or set could be directed to
ob.__cells__[...].value; similarly, the combined Trellis/SA descriptor
would maintain Trellis's __cell__ logic.
The collections may able to bolt on almost as-is. The collections
package provides decorators that emit add/remove events for the ORM. If
mapped instances enter and leave the Trellis collections directly, then
it's pretty trivial. If they go in and out wrapped in something, then
some custom code may be needed. The collection descriptor also has some
logic in its __set__ that would need to be preserved. It does the same
thing I would imagine Trellis does for collections: orchestrate events
on foo.attr = [b1, b2, b3] or foo.attr = foo.attr[2:].
Phillip J. Eby
>ClassState and InstanceState would be the two classes that you'd
>either have to decorate, subclass, or otherwise customize to talk to
>Trellis' system.
Yep, that seems reasonable.
>Also we have a hook already where you could be setting up your own
>_class_state and _state objects, within the MapperExtension API. Its
>true that we could also have SA start looking for magic attributes on
>the class itself but so far this has not been our style.
So, should I be looking at append_result(), create_instance(), or what?
> > What I'm looking for now is how to decouple the massive amount of
> > stuff that happens when you declare a mapping with SQLAlchemy,
> > separating out:
> >
> > * recording the data about what/how to map
> > * modifying the class (i.e., so I can skip this entirely)
> > * setting up an instance (so I can hook this to replace the cells)
>
>So what I wonder is, how much of SA instrumentation are you looking to
>re-implement ?
I don't want to re-implement any of it, if I can avoid it. :) I
just want to make it so that:
1. Trellis components can have cells and collections that delegate
their read-write operations to the corresponding SA mapping machinery.
2. SA loads Trellis components, can query for them, etc.
#1 could be implemented entirely outside of SA, using standard
Trellis methods. It's #2 that's the tricky bit; so far I don't see
any way to do it without basically putting another API layer in.
> > Failing that, I think I would need to provide an alternative mapping
> > declaration system that "compiles down" to SQLAlchemy and creates a
> > dummy class. The catch then is that SQLAlchemy's going to return
> > instances of the dummy class, so I'll end up having to wrap the APIs
> > that return instances... and then deal with queries, and so on,
> > until I'm duplicating every SQLAlchemy API.
>
>yeah that wouldn't be so ideal.
Right. So, if the MapperExtension approach allows me to
create/initialize instances that can delegate to a dummy class or
something, that would be awesome.
>If you *really* want SA to just look for magic attributes on classes,
>we can look into that, but it means that SA is now doing things in an
>entirely different way than has historically. Keeping this to a
>minimum, it might even be something like "__sqlalchemy_extension__ =
>MyMapperExtension" which is picked up by mapper().
It's probably a bit early to worry about this one way or the
other. Once it's *possible* to map Trellis objects, we can always
worry later about making it more transparent. :)
>let me know if you're willing to deal with the _class_state and _state
>magic variables.
My ideal would be that those things didn't live on the class or
instance at all, but instead were managed by the mapper and/or
session. Eg. mapper.class_state(cls) or session.instance_state(ob),
with the ability to be handled by mapper or session extensions. That
would probably do the trick, I think.
Phillip J. Eby
>One of the services the SA descriptors provide is query construction via
>column/relation mappings:
>
> query(Foo).filter(Foo.bar == 2).filter(Foo.children.any(Bar.age > 2))
>
>Not having those would diminish the utility of the ORM in user code-
>querying based on higher-level relations would be lost.
Good point. I forgot about this because my long-term plan was to
implement a similar capability for querying on Trellis objects (that
are not stored in a DB). Integrating the two might be... interesting.
>If combining descriptor functions is acceptable and the __eq__/__ne__
>conflict can be resolved, I like Mike's suggestion about providing some
>get/set routing logic in the InstanceState. Any Trellis-managed
>attribute the ORM wanted to get or set could be directed to
>ob.__cells__[...].value; similarly, the combined Trellis/SA descriptor
>would maintain Trellis's __cell__ logic.
Yep.
>The collections may able to bolt on almost as-is. The collections
>package provides decorators that emit add/remove events for the ORM. If
>mapped instances enter and leave the Trellis collections directly, then
>it's pretty trivial. If they go in and out wrapped in something, then
>some custom code may be needed. The collection descriptor also has some
>logic in its __set__ that would need to be preserved. It does the same
>thing I would imagine Trellis does for collections: orchestrate events
>on foo.attr = [b1, b2, b3] or foo.attr = foo.attr[2:].
Actually, CellProperty knows nothing about an attribute but its name,
so it doesn't actually do anything special for cell
attributes. However, a specialized cell could be used to do the same
thing, so that's only a minor detail.
Michael Bayer
On Feb 26, 2008, at 5:28 PM, Phillip J. Eby wrote:
>
> At 03:27 PM 2/26/2008 -0500, Michael Bayer wrote:
>> ClassState and InstanceState would be the two classes that you'd
>> either have to decorate, subclass, or otherwise customize to talk to
>> Trellis' system.
>
> Yep, that seems reasonable.
>
>
>> Also we have a hook already where you could be setting up your own
>> _class_state and _state objects, within the MapperExtension API. Its
>> true that we could also have SA start looking for magic attributes on
>> the class itself but so far this has not been our style.
>
> So, should I be looking at append_result(), create_instance(), or
> what?
on MapperExtension it would be instrument_class(). We have an
init_instance() hook but this only gets invoked for
class.__init__()..there is also create_instance() which is used when
rows are being returned to return new instances. The two instance-
level hooks may very likely need some modifications to do what you're
trying to do here since theres still some hardwiring of
"InstanceState()" going on even if you override those. We havent
yet had the "I want to modify InstanceState directly" use case yet so
I think I want to work up some tests so that you have a clean hook to
futz with ClassState and InstanceState.
>>>
> I don't want to re-implement any of it, if I can avoid it. :) I
> just want to make it so that:
>
> 1. Trellis components can have cells and collections that delegate
> their read-write operations to the corresponding SA mapping machinery.
I think this is doable if we make it easy for you to screw with
InstanceState (and route all access to classes/instances to ClassState/
InstanceState).
> 2. SA loads Trellis components, can query for them, etc.
if #1 is done, this comes for free, if im understanding things
correctly. (perhaps I'm not).
>
> Right. So, if the MapperExtension approach allows me to
> create/initialize instances that can delegate to a dummy class or
> something, that would be awesome.
explain to me what you mean by "dummy class", when you used that term
earlier I think you were referring to having the Trellis instance
delegate to an entirely different class that is mapped. What I'm
talking about here wouldn't require that - the class which subclasses
Trellis would be the mapped class - SQLAlchemy would get information
about this class/instances of it via modified ClassState/InstanceState
objects that have been swapped onto the class/instance.
>
>> let me know if you're willing to deal with the _class_state and
>> _state
>> magic variables.
>
> My ideal would be that those things didn't live on the class or
> instance at all, but instead were managed by the mapper and/or
> session. Eg. mapper.class_state(cls) or session.instance_state(ob),
> with the ability to be handled by mapper or session extensions. That
> would probably do the trick, I think.
OK, I think a long time ago we tried doing it this way; using a
WeakValueDictionary to associate mappers with classes, attribute
history information with instances. Suffice to say that the issues
with garbage collection, dangling references, as well as plain latency
(WVDs use mutexes and other slowish things) made it pretty
cumbersome. We still have some weakref things going on and wow, are
those weakrefs picky.
the bottom line is that there is behavior attached to class
descriptors that is part of the public API. Particularly what Jason
mentioned; people are going to want to say User.name.like('foo'),
User.firstname + User.lastname == "john smith", etc. It doesnt
matter much how that is implemented but if Trellis' API gets in the
way of that, you'd have to tell your users that they can't use SA's
documented API exactly as is; they'd have to use the Table constructs
directly (which is the old way) or youd have to provide some other
magic place to get those attributes (perhaps like SQLObject's ".q."
attribute, User.q.name=='foo').
Michael Bayer
getattr() I spoke about, and I moved things around so that you can
define your own ClassState and InstanceState implementations, and you
can also redefine (or disable) all of the class-level instrumentation
SQLAlchemy sets up. You can check it out at:
http://svn.sqlalchemy.org/sqlalchemy/branches/user_defined_state
I only spent 30 minutes on this to get the most basic example working
so YMMV applies.
Theres a proof of concept example in the root called
custom_management.py which illustrates a barebones create-save-load
cycle, using a class that relies entirely on __getattr__() and
__setattr_() to get/set attributes; there are no class-level
InstrumentedAttributes used, and all of the class's state is shuttled
away from __dict__ and into another member called "_goofy_dict"
instead. The custom InstanceState references the "_goofy_dict" and
shuttles get/set/del events into SA's existing attribute managers,
which in turn populate the InstanceState and back into _goofy_dict.
Now I dont really know much about what Trellis is doing, but the point
of this demo is that we've defined a completely custom way for data to
get in/out of an instance, so you'd substitute with whatever scheme
you want.
Also it turns out we didn't need to use any MapperExtensions at all;
the _class_state attribute already existing on the class before
attribute instrumentation is applied is the only entry point needed;
the implementation rides totally within the attributes package. If we
can keep it out of ME, that would be best since class/instance
attribute instrumentation is not its focus and I could tell the
existing methods weren't going to be stellar.
Collections would be the next step here and I'd ask Jason to take a
crack at it.
Assuming we can get this all working at a higher level, we should work
it into a suite of unit tests to be part of SA so that the "attribute
extension" API is subject to continuous integration.
Phillip J. Eby
>I've got a new branch in SVN where I've applied the cleanup of
>getattr() I spoke about, and I moved things around so that you can
>define your own ClassState and InstanceState implementations, and you
>can also redefine (or disable) all of the class-level instrumentation
>SQLAlchemy sets up. You can check it out at:
>
>http://svn.sqlalchemy.org/sqlalchemy/branches/user_defined_state
You rock! That looks awesome. :)
>I only spent 30 minutes on this to get the most basic example working
>so YMMV applies.
>
>Theres a proof of concept example in the root called
>custom_management.py which illustrates a barebones create-save-load
>cycle, using a class that relies entirely on __getattr__() and
>__setattr_() to get/set attributes; there are no class-level
>InstrumentedAttributes used, and all of the class's state is shuttled
>away from __dict__ and into another member called "_goofy_dict"
>instead. The custom InstanceState references the "_goofy_dict" and
>shuttles get/set/del events into SA's existing attribute managers,
>which in turn populate the InstanceState and back into _goofy_dict.
So, let me see if I get this... for ClassState, it appears that I do
not have to have an 'attrs' dict, right? Everything is accessed
through the defined methods?
And for the InstanceState, could I just use a plain InstanceState, if
the effective logic of the various 'custom_*' methods was done
elsewhere? (like say, in a SQLAlchemyCell instance...) That is, I
don't need a '.dict' attribute or anything like that?
>Now I dont really know much about what Trellis is doing, but the point
>of this demo is that we've defined a completely custom way for data to
>get in/out of an instance, so you'd substitute with whatever scheme
>you want.
Yep. It seems like it should be possible to do this by just making
sure that every trellis.Component subclass has a _class_state, which
should be pretty easy to do in __class_init__.
>Collections would be the next step here and I'd ask Jason to take a
>crack at it.
That's a good point; it's easy to see how this works for
non-collection attributes, but I haven't wrapped my head around the
other stuff very deeply yet.
Phillip J. Eby
>So, let me see if I get this... for ClassState, it appears that I do
>not have to have an 'attrs' dict, right? Everything is accessed
>through the defined methods?
Whoops... apparently not (this code is from
sqlalchemy.orm.attributes in the branch):
def _managed_attributes(class_):
"""return all InstrumentedAttributes associated with the given
class_ and its superclasses."""
return chain(*[cl._class_state.attrs.values() for cl in
class_.__mro__[:-1] if hasattr(cl, '_class_state')])
Still, it occurs to me I probably *want* an attrs dict anyway (gotta
store those darn things somewhere), so this probably isn't a big deal.
Michael Bayer
On Feb 26, 2008, at 8:08 PM, Phillip J. Eby wrote:
>
> So, let me see if I get this... for ClassState, it appears that I do
> not have to have an 'attrs' dict, right? Everything is accessed
> through the defined methods?
theres some access through "attrs" right now but not much. The tricky
thing about ClassState and "attrs" is when we deal with class
hierarchies we have to look in the ClassState for each class in
__mro__ sometimes. so theres one method like this:
def _managed_attributes(class_):
"""return all InstrumentedAttributes associated with the given
class_ and its superclasses."""
return chain(*[cl._class_state.attrs.values() for cl in
class_.__mro__[:-1] if hasattr(cl, '_class_state')])
so in that case we might want to do something like
_class_state.all_attributes or something. The other access to attrs
seems to be in the "un-instrument" step, which is not going to work
yet with this scheme (since it blows away _class_state)...."un-
instrument" is mostly used for unit test suites that setup/teardown
different mappings.
But also the get_impl() and get_inst() methods need to look up their
value within the super/__mro__ hierarchy, which also is not happening
for the proof of concept. The default implementation of those methods
uses getattr(cls, key) which does the superclass thing for us.
> And for the InstanceState, could I just use a plain InstanceState, if
> the effective logic of the various 'custom_*' methods was done
> elsewhere? (like say, in a SQLAlchemyCell instance...) That is, I
> don't need a '.dict' attribute or anything like that?
the only requirement in that area is that InstanceState.dict points to
a dictionary-like object that affects attributes on the instance, i.e.
it behaves just like instance.__dict__ would. When the attribute
instrumentation ultimately sets the value after all the events have
occured, it sets the value in that dict. When a list of instances are
loaded from DB rows, the objects responsible for populating the
attributes stick the value of each column in that dict directly and no
instrumentation/events are involved (its where speed is most
critical). Then when you go to access an attribute, the
instrumentation looks in that dict, and if the key is not present, we
fall back onto callables which might issue additional queries to the
database, or if there arent any callables we initialize to None/empty
collection.
Theres additionally some stuff going on with direct instance.__dict__
access when InstanceState gets pickled in its __getstate__() and
__setstate__() methods, those would have to be overridden as well if
you wanted your instances to be pickleable.
>> Collections would be the next step here and I'd ask Jason to take a
>> crack at it.
>
> That's a good point; it's easy to see how this works for
> non-collection attributes, but I haven't wrapped my head around the
> other stuff very deeply yet.
yes im very fortunate that Jason totally re-implemented how we do
collections some time ago, not only because we have a world-class
implementation now but also I dont need to be the primary understander
of it :).
sdo...@sistechnology.com
> I've got a new branch in SVN where I've applied the cleanup of
> getattr() I spoke about, and I moved things around so that you can
> define your own ClassState and InstanceState implementations, and
> you can also redefine (or disable) all of the class-level
> instrumentation SQLAlchemy sets up. You can check it out at:
>
> http://svn.sqlalchemy.org/sqlalchemy/branches/user_defined_state
>
> I only spent 30 minutes on this to get the most basic example
> working so YMMV applies.
>
> Theres a proof of concept example in the root called
> custom_management.py which illustrates a barebones create-save-load
> cycle, using a class that relies entirely on __getattr__() and
> __setattr_() to get/set attributes; there are no class-level
> InstrumentedAttributes used, and all of the class's state is
> shuttled away from __dict__ and into another member called
> "_goofy_dict" instead. The custom InstanceState references the
> "_goofy_dict" and shuttles get/set/del events into SA's existing
> attribute managers, which in turn populate the InstanceState and
> back into _goofy_dict.
>
> Now I dont really know much about what Trellis is doing, but the
> point of this demo is that we've defined a completely custom way
> for data to get in/out of an instance, so you'd substitute with
> whatever scheme you want.
from my experience with the way u do this stuff (attributes.py), there
are 2 levels of access to some attr: one is the user-one, external
(getattr/setattr), and the other is SA-internal one, done by
accessing/populating the state.dict.
as i see it now, the external one is redefined, but the internal one
is still same.
so the state.dict still has to be smart to do things at set/get, e.g.
in trellis' example raise a flag about the attribute being changed or
whatever. which isnt any much further from what i have in
dbcook.usage.static_type.sa2static.dict_via_attr class.
Or am i missing something?
sdo...@sistechnology.com
wrote:
one difference though is that now i can create the dict-replacement
just once, and not at each and every getattr, which at least would be
faster than before. i'll re-wheel my stuff on the new rails and will
tell how it behaves.
Here's a list of what i have needed from such attr-Extension API (bear
with me, the attr-access story has been a pain ever since i started
tampering with SA :-):
- i have no dict - everything is descriptors. A substitute dict is
okay, but SA should not rely too much on dict-like behaviours, or at
least rely/stick only on certain subset of the dict-ness (e.g. there
were usages of dict.get instead of dict.has_key and similar
dict-specific things). The subset should be well-defined and the
tests should ensure nothing else is being used.
- i am storing the attr-values myself, which lives underneath SA, and
SA has no idea of it, it goes where the dict is. Thus, pickling has
to also know about attr-Extension - or the other way around.
- i have defaultvalue/autoset behaviour, which is above SA - so SA
becomes aware of the changes coming from that. Such autoset applies
to atomic values as well as to references (autocreate). i also have
other such behaviour, a master-slave link, but it is on collections
and i havent yet alchemized those at all. Example is a collection
x.people and an attribute x.number_of_people, which can be related
either way master-slave ot both ways, so resizing first changes the
second, or setting second resizes first, or both. Seems trellis-like
dependencies may come here handy... (like
x.half_the_number_of_people) - but i'm not sure if that sort of
things are supported.
- the above autoset/autocreate used to conflict with SA lazy-set and
expire behaviours, and so far i have fixed that by hack-replacing the
triggering and chaining the whatever (short-lived) SA-trigger before
my own stuff. This seems to work but i havent tested it properly, and
there's no API way to do it - i'm guessing on availability of
state.trigger/state.expire_attributes etc completely out-of-my scope
stuff, and all that within the dict.get method. Definitely this stuff
has to have separate way of setting up, and here ordering would
matter.
Without such clearly defined protocol(s), it would be again just
hacking around the dict and *current* internal SA-methods (without SA
knowing anything of it) as i did - be it ugly or not. and Mantaining
those hacks is a pain... i tell you.
Eeeeh...
i'll see if i can think of something else...
collections might have something, e.g. master-slave composition - x.A
is declared as x.B|x.C|x.D, and putting something in x.B automaticaly
adds it into x.A. So the master-slave dependencies can be on same
dimension/level - atomic-atomic or collection-collection - as well as
inter-level - atomic-collection.
ciao
svilen
Michael Bayer
On Feb 27, 2008, at 2:50 AM, sdo...@sistechnology.com wrote:
> Without such clearly defined protocol(s), it would be again just
> hacking around the dict and *current* internal SA-methods (without SA
> knowing anything of it) as i did - be it ugly or not. and Mantaining
> those hacks is a pain... i tell you.
> Eeeeh...
>
> i'll see if i can think of something else...
> collections might have something, e.g. master-slave composition - x.A
> is declared as x.B|x.C|x.D, and putting something in x.B automaticaly
> adds it into x.A. So the master-slave dependencies can be on same
> dimension/level - atomic-atomic or collection-collection - as well as
> inter-level - atomic-collection.
>
the thing to keep in mind with _state is....*this is all brand new
code*. its the internals of the attribute system, it uses a lot of
direct attribute access instead of methods to be as fast as possible,
and its very much in flux as there are even more things I'm going to
need to do to it soon. until PJE's suggestion, it didn't even occur
to me that the example I wrote up tonight was even possible - i.e. to
change the way a class is instrumented such that the descriptors
aren't even needed. I was almost ready to put out my usual "well
*that* seems really impossible, we'd have to rewrite the whole thing,
etc" until the lightbulb went on. It seems like we'd add a little
more hookness to allow collection implementations to be affected by
ClassState and we're just about there; at that point we can look into
smoothing over the when-do-i-talk-to-dict-when-do-i-talk-to-instrument-
get/set/del.
With just the very slight changes I made, adding a teeny bit more
abstraction to how we get at __dict__ and class-level attrs, the
masseagerload.py test gained 500 method calls. So building this out
in such a way that doesnt bloat the whole thing out is going to be a
pain.
This is why we dont have "one magic dict to rule them all" right now;
I could build a big thick API that is all things to all external
toolsets but then we've built a big and slow system, which also would
be too rigid in structure to quickly adapt to the level of development
going on. If we can get this thing to work, it might be presented as
a set of extension classes (like a UserClassState/UserInstanceDict
type of API maybe) which present a nicer front end, but aren't used in
default SQLAlchemy which is where things need to stay raw.
svilen
well... i didn't say "make a AttrExt class and put methods etc"... i
said "make a protocol", or actualy, *recoginize* the existing
protocol and clean a little bit, then stick to it as much as
possible. Whether the protocol consists of certain calls to a dict
(or dict-like thing) or brand new obj, whatever, doesnot matter - as
long it is recognized as such and complied with...
i can help with testing/specification of the protocol/ separation of
concerns if u want.
Judah De Paula
> well... i didn't say "make a AttrExt class and put methods etc"... i
> said "make a protocol", or actualy, *recoginize* the existing
> protocol and clean a little bit, then stick to it as much as
> possible. Whether the protocol consists of certain calls to a dict
> (or dict-like thing) or brand new obj, whatever, doesnot matter - as
> long it is recognized as such and complied with...
> i can help with testing/specification of the protocol/ separation of
> concerns if u want.
>
>
I have a strong urge to say "me too", but I'm not going to do it. :-)
The "Integrating the ORM with Trellis" thread showed up a day before I
was going to compose an e-mail entitled "Integrating the ORM with
Traits." Many of Phillip's concerns with getting Trellis working with
SQLAlchemy also directly applies to what I'm trying to do with Traits
and it's own event-driven programming environment.
Over the last month I have used SQLAlchemy 0.4.1 to create a generic
Traits class that would automatically persist defined Traits (special
class attributes that support events and callbacks) into dynamically
generated mapped tables. I just upgraded from 0.4.1 to 0.4.3 and am
frustrated to find that the newest version of SA no longer detects
changes to my objects when I do a commit.
Since this whole area of code appears to still be in flux I was going to
stop development on my package until an API was provided, or the
consensus is that the interfaces are stable. So I am very interested in
seeing a public interface that I can work with. I understand that it's
not clear what is needed, but I too will be willing to help where I can.
Cheers,
Judah De Paula
ju...@enthought.com
Michael Bayer
On Feb 27, 2008, at 9:47 AM, Judah De Paula wrote:
>
> I have a strong urge to say "me too", but I'm not going to do it. :-)
you just did :)
> Over the last month I have used SQLAlchemy 0.4.1 to create a generic
> Traits class that would automatically persist defined Traits (special
> class attributes that support events and callbacks) into dynamically
> generated mapped tables. I just upgraded from 0.4.1 to 0.4.3 and am
> frustrated to find that the newest version of SA no longer detects
> changes to my objects when I do a commit.
I was googling around the other day and I believe I saw this code, and
I was going to write you since I could see what you were doing wasn't
going to work. I think you were flipping on ._state.modified to
establish the instance as "dirty". "modified" is a hint flag that
will lead to an object being placed in the unit of work, but when the
INSERT or UPDATE actually happens, the history of each attribute is
checked for actual changes before deciding whether or not to include
it (and if no columns changed, no statement).
the change from 0.4.1 to 0.4.3 is one that shaved off 20% of the
loading time of instances - namely that the "committed" verision of an
attribute is calculated when an attribute change event is detected,
and not across the board when instances are loaded. Its basically
copy-on-write. At the expense of, writing to the instance.__dict__
directly would no longer be detected as an attribute change at flush
time.
The _state has a dictionary called "committed_state" on it. The way
we tell if an attribute has changed since it was loaded is (in
simplified terms):
instance.foobar != instance._state.committed_state['foobar']
So in the 0.3 and earlier 0.4 series, loading an instance looked like
this:
fetch row
instance.foobar = row['foobar'] # repeat for all columns in the
row
instance._state.committed_state['foobar'] = instance.foobar #
after all objects are loaded, do this for every instance/attribute
the implication here is after that load is complete, you can put data
directly into instance.__dict__ and it will be different against
what's present in committed_state, thus being detected as a change as
long as state.modified is flipped on.
in 0.4.2, we took out that third step from the loading process:
fetch row
instance.foobar = row['foobar'] # repeat for all columns in the
row
and thats it. huge performance increase. then when you change
instance.foobar to something else, the attribute instrumentation
detects your change:
instance._state.committed_state['foobar'] = instance.foobar #
copy old value
instance.foobar = 'something new' # populate new value
Thereby only copying those attributes which need it....the overall
savings in processing time for a typical conversation is pretty
dramatic for a large load of objects with just a few changes.
But now, writing to __dict__ directly without notifying the attribute
system will not write to the "committed state" dictionary. I can see
that while I mentioned this change in the CHANGES I didn't add a note
that writing to __dict__ was not going to work going forward. Its
always been the official policy, admitted lack of documentation
notwithstanding, that folks should be using setattr(instance, key,
value) to set attribues dynamically and actually in older versions
like 0.1 and 0.2 it was also needed. I'm not sure if your particular
issue is as simple as a __dict__ write becoming a setattr(), though.
> Since this whole area of code appears to still be in flux I was
> going to
> stop development on my package until an API was provided, or the
> consensus is that the interfaces are stable. So I am very
> interested in
> seeing a public interface that I can work with. I understand that
> it's
> not clear what is needed, but I too will be willing to help where I
> can.
Well, I guess we're going to play with that branch for a bit, and
since so far the changes in it are very minor we can merge to trunk
very soon. If we create something without underscores and add some
test cases to keep it running then we'll be good. I desperately need
to work on my pycon presentations but I'll try to see if I can propose
something stable in that regard so at least you have something to code
to. But one thing happening here is that what we're doing for PJE is
pretty dramatic - he doesnt even want class-level attributes to be
present, we're dropping you way in there where you are required to
redefine attribute access totally across the board. So far it seems
like an API thats appropriate for when you really need to completely
change the flow of data from instance to SQLAlchemy.....it remains to
be seen how "easy" we can make it.
Michael Bayer
great....so, heres the next pass at the API in rev 4199. I've hidden
away the underlying messiness and added a hook to provide a collection
class - so this is most of the idea, and you don't have to deal with
the internals of ClassState or InstanceState anymore. You decide if
attributes get set on the class or not, and you also return a
dictionary-like object that SA will use to store the data it gets from
the database and from its events (or leave out that method, and it
returns .__dict__). You also get a hook that receives the
collection_class argument in the case of a collection-based attribute
and you get to return your own class, in which case all the
collections API stuff can kick in on that side. Your method of
getting and setting attributes delegates SA-instrumented events to
some functions available in the attributes package, where they will
populate your dict-like object after doing SA's events, or if an
attribute is not instrumented (and you have to check that), you
populate your dict-like object (or whatever) directly.
script attached.
Judah De Paula
>
> I desperately need
> to work on my pycon presentations but I'll try to see if I can propose
> something stable in that regard so at least you have something to code
> to.
>
>
If I was going to PyCon this year, I'd definitely come by and say 'hi.'
I have a coworker going to one of the PyCon SQLAlchemy sessions, but
he's not involved in this project. ;-) Maybe I can convince him when
he gets back.
Judah De Paula
ju...@enthought.com
jason kirtland
> You also get a hook that receives the
> collection_class argument in the case of a collection-based attribute
> and you get to return your own class, in which case all the
> collections API stuff can kick in on that side.
This can be opened up a bit to allow custom collection event systems as
well. We'd move the
'collections._prepare_instrumentation(typecallable)'
out of the Attribute and into SA's default implementation of
'instrument_collection_class'. If custom extenders want SA
instrumentation, they can call that themselves. The
_prepare_instrumentation itself can become a public function, it's stable.
Judah De Paula
> On Feb 27, 2008, at 9:47 AM, Judah De Paula wrote:
>
>
>> Over the last month I have used SQLAlchemy 0.4.1 to create a generic
>> Traits class that would automatically persist defined Traits (special
>> class attributes that support events and callbacks) into dynamically
>> generated mapped tables. I just upgraded from 0.4.1 to 0.4.3 and am
>> frustrated to find that the newest version of SA no longer detects
>> changes to my objects when I do a commit.
>>
>
> I was googling around the other day and I believe I saw this code, and
> I was going to write you since I could see what you were doing wasn't
> going to work. I think you were flipping on ._state.modified to
> establish the instance as "dirty". "modified" is a hint flag that
> will lead to an object being placed in the unit of work, but when the
> INSERT or UPDATE actually happens, the history of each attribute is
> checked for actual changes before deciding whether or not to include
> it (and if no columns changed, no statement).
>
>
Yes, that's the code, and that's the problem. Proper OOP says I should
not use your protected attribute ._state.modified, but I couldn't see an
obvious work-around to get my objects to update using only the public
interface. I have not tried yet, but it sounds like I can get things
working again if I use ._state.committed_state[attr] instead?
> But now, writing to __dict__ directly without notifying the attribute
> system will not write to the "committed state" dictionary. I can see
> that while I mentioned this change in the CHANGES I didn't add a note
> that writing to __dict__ was not going to work going forward. Its
> always been the official policy, admitted lack of documentation
> notwithstanding, that folks should be using setattr(instance, key,
> value) to set attribues dynamically and actually in older versions
> like 0.1 and 0.2 it was also needed. I'm not sure if your particular
> issue is as simple as a __dict__ write becoming a setattr(), though.
>
>
Thank you for the explanation, it was extremely useful. When I tried
0.4.3 I looked through the CHANGES and had a suspicion that SQLAlchemy
interacting with __dict__ differently was my problem. I know that
Traits uses C code that modifies __dict__ directly, so if I redefined my
class's setattr() I might detect my first object change, but not
necessarily any additional trait changes that were triggered by events.
> since so far the changes in it are very minor we can merge to trunk
> very soon. If we create something without underscores and add some
> test cases to keep it running then we'll be good. I desperately need
> to work on my pycon presentations but I'll try to see if I can propose
> something stable in that regard so at least you have something to code
> to. But one thing happening here is that what we're doing for PJE is
> pretty dramatic - he doesnt even want class-level attributes to be
> present, we're dropping you way in there where you are required to
> redefine attribute access totally across the board. So far it seems
> like an API thats appropriate for when you really need to completely
> change the flow of data from instance to SQLAlchemy.....it remains to
> be seen how "easy" we can make it.
>
>
>
I look forward to seeing what happens.
Thank you,
Judah De Paula
ju...@enthought.com
Michael Bayer
go for it
Michael Bayer
On Feb 27, 2008, at 12:00 PM, Judah De Paula wrote:
>
> Yes, that's the code, and that's the problem. Proper OOP says I
> should
> not use your protected attribute ._state.modified, but I couldn't
> see an
> obvious work-around to get my objects to update using only the public
> interface. I have not tried yet, but it sounds like I can get things
> working again if I use ._state.committed_state[attr] instead?
>
err you can *try* but that might be a hard road to travel. when i did
the copy-on-write refactoring i added a huge amount of unit tests to
that whole system since it became more particular about how things are
done..what decisions to make when theres no key present in
committed_state, when theres no key in __dict__ but present in
committed_state, etc. Also you're still dealing with internal
things so you're not going to be stable.
Another issue with allowing direct __dict__ access is that you arent
issuing any of the other attribute events SA needs. The "cascade"
functionality, which does things like saves "B" when you say A.b = B,
also drives off of events, as well as bidirectional relationships
(i.e. backrefs) working themselves out. On the get side the whole
ability to lazy load expired/deferred/unloaded attributes and
collections drives off of the event system.
If you could take a look at the most recent thing I've done in the new
branch and let me know if that meets your needs, that would be
helpful. It seems like not, since you are dealing with C code that
unconditionally writes straight to __dict__ and you cant use the
normal event routes. Its possible you could fire off the normal SA
events in the method illustrated in the example, which ultimately
write to __dict__ anyway and it would just be a "double write"
scenario, but not ultimately harmful.
Phillip J. Eby
>great....so, heres the next pass at the API in rev 4199.
Even more awesome!
> I've hidden
>away the underlying messiness and added a hook to provide a collection
>class - so this is most of the idea, and you don't have to deal with
>the internals of ClassState or InstanceState anymore. You decide if
>attributes get set on the class or not, and you also return a
>dictionary-like object
What methods must this object support, if I were writing a completely
new object from scratch? (Because I just might be.) Are
__getitem__, __setitem__, and __contains__ sufficient? Or is there
more needed? (E.g. __delitem__?)
>You also get a hook that receives the
>collection_class argument in the case of a collection-based attribute
>and you get to return your own class, in which case all the
>collections API stuff can kick in on that side.
Hm. Is that for the collection SA will use, or the collection my
object will expose? (Those might need to be different objects.)
> Your method of
>getting and setting attributes delegates SA-instrumented events to
>some functions available in the attributes package, where they will
>populate your dict-like object after doing SA's events, or if an
>attribute is not instrumented (and you have to check that), you
>populate your dict-like object (or whatever) directly.
By the way, if you're worried about the lookup performance of
is_instrumented() and get_impl(), I have a straightforward way to fix
that. Since SA is only supporting new-style types, you can use the
__subclasses__() method to notify subclass states when a parent class
is changed. So in instrument_attribute(), you can do something like:
for sc in class_.__subclasses__():
sc._class_state.instrument(...)
And, in ClassState.__init__, you can have it build its initial
'attrs' from the states of the class' __mro__. The result will be
that attribute implementation lookups can be a single dictionary lookup.
In fact, you could go a bit further, and make ClassState a dictionary
subclass, whose 'get_inst()' is just a plain
__getitem__. (e.g. ClassState.get_inst = dict.__getitem__). That
would be awesomely fast -- perhaps faster even than your original
getattr() approach. (Seriously -- especially since it also skips the
call to the descriptor __get__, as well.)
get_impl() would then be just "return self[key].impl", and
ClassState.is_instrumented = dict.__contains__.
I also wonder if one might just combine initialize_instance_dict()
and get_instance_dict(), since they are being called at basically the
same time and in only one place, AFAICT. Similarly, I'm not sure
whether the custom class state needs to have separate hooks for
pre_instrument_attribute vs. instrument_attribute. It seems like it
could be just one hook - effectively the pre_instrument_attribute,
which would be responsible for modifying the class as needed.
Which means that ClassState.instrument_attribute could look like this:
def instrument_attribute(self, key, inst):
if key not in self:
self[key] = inst
self.pre_instrument_attribute(key, inst)
for cls in self.class_.__subclasses__():
_init_class_state(cls)
cls._class_state.instrument_attribute(key, inst)
And _ClassStateAdapter would then only override
pre_instrument_attribute() to do the delegation to the custom state.
By the way, ISTM that '_init_class_state' could be renamed to
'get_class_state' and return the class state, which would make the
above code a bit cleaner, e.g.:
def instrument_attribute(self, key, inst):
if key not in self:
self[key] = inst
self.pre_instrument_attribute(key, inst)
for cls in self.class_.__subclasses__():
get_class_state(cls).instrument_attribute(key, inst)
And _managed_attributes() could be simplified to:
return iter(get_class_state(class_))
or replaced with a loop directly on get_class_state().
>class MyClass(object):
> __sa_instrument_class__ = MyClassState
I assume I could do this instead:
@classmethod
def __sa_instrument_class__(cls):
# Don't bother importing this module unless somebody
# actually defines an SA mapping for this class...
from sql_support import TrellisClassState
return TrellisClassState(cls)
i.e., there's no requirement that __sa_instrument_class__ actually be
a class? (In which case, perhaps a better name might be
__sa_instrument_factory__?)
Michael Bayer
On Feb 27, 2008, at 3:10 PM, Phillip J. Eby wrote:
>> I've hidden
>> away the underlying messiness and added a hook to provide a
>> collection
>> class - so this is most of the idea, and you don't have to deal with
>> the internals of ClassState or InstanceState anymore. You decide
>> if
>> attributes get set on the class or not, and you also return a
>> dictionary-like object
>
> What methods must this object support, if I were writing a completely
> new object from scratch? (Because I just might be.) Are
> __getitem__, __setitem__, and __contains__ sufficient? Or is there
> more needed? (E.g. __delitem__?)
it'll need __delitem__ for sure since "del myinstance.foo" comes down
to "del state.dict[key]". we also call dict.get() on a lot of
occasions. I dont think you need any aggreagte methods like __iter__
or keys() though.
>> You also get a hook that receives the
>> collection_class argument in the case of a collection-based attribute
>> and you get to return your own class, in which case all the
>> collections API stuff can kick in on that side.
>
> Hm. Is that for the collection SA will use, or the collection my
> object will expose? (Those might need to be different objects.)
the class you return is what will be exposed. Currently, the class
will need to be compatible with the spec in the collections
docs....i.e. its list/set/dictlike, or otherwise has decorators
marking its append and remove methods.
>>
> By the way, if you're worried about the lookup performance of
> is_instrumented() and get_impl(), I have a straightforward way to fix
> that. Since SA is only supporting new-style types, you can use the
> __subclasses__() method to notify subclass states when a parent class
> is changed. So in instrument_attribute(), you can do something like:
>
> for sc in class_.__subclasses__():
> sc._class_state.instrument(...)
>
> And, in ClassState.__init__, you can have it build its initial
> 'attrs' from the states of the class' __mro__. The result will be
> that attribute implementation lookups can be a single dictionary
> lookup.
oh....you mean that the InstrumentedAttribute will be present in
multiple ClassState dictionaries all the way down, thats interesting.
OK I might try that.
> In fact, you could go a bit further, and make ClassState a dictionary
> subclass, whose 'get_inst()' is just a plain
> __getitem__. (e.g. ClassState.get_inst = dict.__getitem__). That
> would be awesomely fast -- perhaps faster even than your original
> getattr() approach. (Seriously -- especially since it also skips the
> call to the descriptor __get__, as well.)
>
> get_impl() would then be just "return self[key].impl", and
> ClassState.is_instrumented = dict.__contains__.
Thats interesting too. Im a little antsy about ClassState being
singularly minded around "I am a dict of attribute instrumentors" but
give me time on that.
> I also wonder if one might just combine initialize_instance_dict()
> and get_instance_dict(), since they are being called at basically the
> same time and in only one place, AFAICT.
yeah at the moment they are....i need to add one more hook which is
that get_instance_dict() is called within InstanceState.__getstate__
though (or otherwise change how __getstate__ works perhaps), so that
would be a place that get_instance_dict() is called again.
> Similarly, I'm not sure
> whether the custom class state needs to have separate hooks for
> pre_instrument_attribute vs. instrument_attribute. It seems like it
> could be just one hook - effectively the pre_instrument_attribute,
> which would be responsible for modifying the class as needed.
OK the pre_instrument_attribute hook, which I agree looks silly, is
because we have a little weirdness about how mappers configure
themselves.
When you say mapper(), it does everything it can to initialize itself,
*except* for initializing the relation()s that are associated with it;
this because the order of mapper construction is not determined, and
you might say mapper(User, users,
properties={'addresses':relation(Address)}) before the Address class
has a mapper() set up. The dependency resolution gets even weirder
when we do things like mapper(MySubClass, table, inherits=SuperClass,
properties={'related':relation(SuperClass, backref='parent')}).
So we have this step called compile() which, besides being another one
of my mediocre name choices, is where all the mappers that are set up
go through and complete their links to each other. That stage is also
currently where the instrument_attribute hook takes place (which
perhaps could be moved upwards to be pre-compile....its just been the
way it is for a long time).
So the pre_instrument_attribute hook is actually putting on this
quickie descriptor called _CompileOnAttr, which catches people doing
things like this:
mycriterion = User.name == 'foo'
the User.name access says, "hey he's doing mapper stuff, lets compile"
and then the whole compile phase and instrumentation occurs, the
_CompileOnAttrs get thrown away. The whole "User.name=='foo'" syntax
was new in 0.4, and we suddenly realized users were creating criterion
before mappers had a chance to compile through other means, so the
"pre compile" step was done for expediency...its just a current
implementation detail until we get around to organizing that better.
> Which means that ClassState.instrument_attribute could look like this:
>
> def instrument_attribute(self, key, inst):
> if key not in self:
> self[key] = inst
> self.pre_instrument_attribute(key, inst)
> for cls in self.class_.__subclasses__():
> _init_class_state(cls)
> cls._class_state.instrument_attribute(key, inst)
>
> And _ClassStateAdapter would then only override
> pre_instrument_attribute() to do the delegation to the custom state.
>
> By the way, ISTM that '_init_class_state' could be renamed to
> 'get_class_state' and return the class state, which would make the
> above code a bit cleaner, e.g.:
>
> def instrument_attribute(self, key, inst):
> if key not in self:
> self[key] = inst
> self.pre_instrument_attribute(key, inst)
> for cls in self.class_.__subclasses__():
> get_class_state(cls).instrument_attribute(key, inst)
>
> And _managed_attributes() could be simplified to:
>
> return iter(get_class_state(class_))
>
> or replaced with a loop directly on get_class_state().
yeah this is all great stuff !
>
>> class MyClass(object):
>> __sa_instrument_class__ = MyClassState
>
> I assume I could do this instead:
>
> @classmethod
> def __sa_instrument_class__(cls):
> # Don't bother importing this module unless somebody
> # actually defines an SA mapping for this class...
> from sql_support import TrellisClassState
> return TrellisClassState(cls)
>
> i.e., there's no requirement that __sa_instrument_class__ actually be
> a class? (In which case, perhaps a better name might be
> __sa_instrument_factory__?)
yes thats fine with me.... I usually can get something to "just work"
but naming and silliness reduction often needs to be inspired by
others....
do you have any interest in committing changes to the branch
yourself ? as long as the unit tests keep running whatever you'd want
is most likely fine with me....otherwise I will at least experiment
with doing away with __mro__ searching and possibly doing away with
pre_instrument_attribute.
Michael Bayer
On Feb 27, 2008, at 12:00 PM, Judah De Paula wrote:
> Thank you for the explanation, it was extremely useful. When I tried
> 0.4.3 I looked through the CHANGES and had a suspicion that SQLAlchemy
> interacting with __dict__ differently was my problem. I know that
> Traits uses C code that modifies __dict__ directly, so if I
> redefined my
> class's setattr() I might detect my first object change, but not
> necessarily any additional trait changes that were triggered by
> events.
>
>>
I thought of another way to maybe work around this. Depending on how
Traits hits __dict__, you could do something like:
instance.__raw_dict__ = instance.__dict__
instance.__dict__ = DelegatingDict(instance)
where DelegatingDict sends __getitem__ and __setitem__ events to the
get_attribute and set_attribute methods I've just provided, and
get_instance_dict() returns __raw_dict__. So this means Traits
populates __dict__ which is your DelegatingDict, the attribute
operation gets sent to SQLalchemy events which do the requisite
bookkeeping and then store the actual information in the __raw_dict__
collection. All this would need is that Traits would have to be OK
with getting a replaced, dict-like object for instance.__dict__
instead of what Python natively puts there.
sdo...@sistechnology.com
i have been dealing myself with this stuff for an year+ now, from
0.3.6 onwards, and keep it working all the way through (and that
clutters it a lot). My code is here
http://dbcook.svn.sourceforge.net/viewvc/dbcook/trunk/dbcook/usage/static_type/sa2static.py?view=log
have a look, its not that dangerous... class dict_via_attr.
For the dict methods u need getitem, setitem, delitem, contains and
has_key, get and pop.
There are several catches in the way Mike uses that dict though..
- if there is a trigger scheduled (e.g. from session.expire), getitem
must fail.
- any of the special SA attrs (see Base.__slots__ :_sa_session_id
_sa_insert_order _instance_key _entity_name) should go direct
- any changes going into dict are not SA-trackable since 0.4.2+. if u
need trackable changes, either do them via plain setattr, or replace
the trigger callable_, and fix the state.modified yourself. see class
Autosetter. Again this is not negotiated with mike, it's what i have
found to work.
This is all experience from my SA-usage, and i'm not sure if it covers
more than 60-70%. There might be surprises.
ciao
sdo...@sistechnology.com
> dangerous... class dict_via_attr.
ahh, sourceforge' svn browser isnt very usable.. here proper link
http://dbcook.svn.sourceforge.net/viewvc/*checkout*/dbcook/trunk/dbcook/usage/static_type/sa2static.py?revision=217&content-type=text%2Fplain
Michael Bayer
On Feb 28, 2008, at 1:50 AM, sdo...@sistechnology.com wrote:
>
> i see you are all inventing the hot-water...
> i have been dealing myself with this stuff for an year+ now, from
> 0.3.6 onwards, and keep it working all the way through (and that
> clutters it a lot). My code is here
> http://dbcook.svn.sourceforge.net/viewvc/dbcook/trunk/dbcook/usage/static_type/sa2static.py?view=log
> have a look, its not that dangerous... class dict_via_attr.
i've no doubt what you're doing works but your style there is very,
very dense. Is this specifically a strategy to replace
instance.__dict__ ?
>
> There are several catches in the way Mike uses that dict though..
> - if there is a trigger scheduled (e.g. from session.expire), getitem
> must fail.
> - any of the special SA attrs (see Base.__slots__ :_sa_session_id
> _sa_insert_order _instance_key _entity_name) should go direct
I dont really understand the __slots__ thing there.
> - any changes going into dict are not SA-trackable since 0.4.2+. if u
> need trackable changes, either do them via plain setattr, or replace
> the trigger callable_, and fix the state.modified yourself. see class
> Autosetter. Again this is not negotiated with mike, it's what i have
> found to work.
have you tried the API I just proposed in the branch ? seems a lot
simpler.
> This is all experience from my SA-usage, and i'm not sure if it covers
> more than 60-70%. There might be surprises.
I'm sorry we werent ready for this stuff back in 0.3. Now that it
seems like we are, time to take a fresh look at it.
sdo...@sistechnology.com
> On Feb 28, 2008, at 1:50 AM, sdo...@sistechnology.com wrote:
> > i see you are all inventing the hot-water...
> > i have been dealing myself with this stuff for an year+ now, from
> > 0.3.6 onwards, and keep it working all the way through (and that
> > clutters it a lot). My code is here
> > http://dbcook.svn.sourceforge.net/viewvc/dbcook/trunk/dbcook/usag
> > dangerous... class dict_via_attr.
>
> i've no doubt what you're doing works but your style there is very,
> very dense. Is this specifically a strategy to replace
> instance.__dict__ ?
the dict on *each* x.__dict__ call, and which then routes the
dict-protocol into an attr-protocol over some other hidden object.
ignore the actual routing, its specific to my lib.
i suppose in new branch it should be possible to do the creation and
replacement only once-per-initialization, and not each and every
time. But the object as such stays same.
> > There are several catches in the way Mike uses that dict though..
> > - if there is a trigger scheduled (e.g. from session.expire),
> > getitem must fail.
> > - any of the special SA attrs (see Base.__slots__ :_sa_session_id
> > _sa_insert_order _instance_key _entity_name) should go direct
>
> I dont really understand the __slots__ thing there.
ignore the __slots__ as such, look at the contents of the list.
maybe u have other such special attrs, that need some special
attention, i have found these so far. In my case i am explicitly
allowing these attributes under an instance, as by default i have
only the real DB-attributes there (or even a subset of), and nothing
else is allowed: myobj.whatever_wrong_attr=5 raises AttrError.
> > - any changes going into dict are not SA-trackable since 0.4.2+.
> > if u need trackable changes, either do them via plain setattr, or
> > replace the trigger callable_, and fix the state.modified
> > yourself. see class Autosetter. Again this is not negotiated with
> > mike, it's what i have found to work.
>
> have you tried the API I just proposed in the branch ? seems a lot
> simpler.
mmm i dont see where i can attach my own setter.
The case is that i want an autocreating getattr, so a.x.y =1
autocreates a.x on the fly and assigns its y =1. i did complain about
it to you some time ago. i guess any auto-calculating engine will
have similar requirement about setting certain values when *it*
decides to (which might not be at setattr at all - usualy when u need
something just then u calculate it).
i'm looking at custom_management.py right now... by API u mean whats
exposed in there, right? what i see is a way to replace/extend the
instrumentation. Which is fine, it might be useful to replace the
instance-attr-value-storage with whatever in less hackish way; but i
dont have a ready use case for it right now.
What i'm talking about is a protocol of how SA is using that
attr-value-storage. i.e. that dict-like thing must have these and
these methods (like, why u need pop?), and expect them do behave so
and so if not really obvious (like that getitem failing when a
trigger is pending), etc. i dont expect u to do it straight away,
from my experience with SA code these things are scattered all around
and sometimes in quite weird occasions.
do u understand me now? sorry about being a bit, er, intrusive..
svilen
sdo...@sistechnology.com
> > I dont really understand the __slots__ thing there.
>
> ignore the __slots__ as such, look at the contents of the list.
> maybe u have other such special attrs, that need some special
> attention, i have found these so far. In my case i am explicitly
> allowing these attributes under an instance, as by default i have
> only the real DB-attributes there (or even a subset of), and
> nothing else is allowed: myobj.whatever_wrong_attr=5 raises
> AttrError.
live under the _state and not under instance itself.
Michael Bayer
On Feb 28, 2008, at 4:38 PM, sdo...@sistechnology.com wrote:
>> ignore the __slots__ as such, look at the contents of the list.
>> maybe u have other such special attrs, that need some special
>> attention, i have found these so far. In my case i am explicitly
>> allowing these attributes under an instance, as by default i have
>> only the real DB-attributes there (or even a subset of), and
>> nothing else is allowed: myobj.whatever_wrong_attr=5 raises
>> AttrError.
> my idea is these special attrs has to be minimized and/or moved to
> live under the _state and not under instance itself.
yes I mentioned that will happen eventually, _class_state and _state
would be the only things we add. _instance_key and ".c." cant be
removed for the rest of 0.4 though since people code to those.
Michael Bayer
On Feb 28, 2008, at 4:25 PM, sdo...@sistechnology.com wrote:
> What i'm talking about is a protocol of how SA is using that
> attr-value-storage. i.e. that dict-like thing must have these and
> these methods (like, why u need pop?), and expect them do behave so
> and so if not really obvious (like that getitem failing when a
> trigger is pending), etc. i dont expect u to do it straight away,
> from my experience with SA code these things are scattered all around
> and sometimes in quite weird occasions.
> do u understand me now? sorry about being a bit, er, intrusive..
what dict-like thing, you mean __dict__ ? the contract between
__dict__ and a trigger firing is pretty straightforward. key is in
dict, no trigger fires. key is not there, trigger fires. that logic
takes place entirely within attributes.py in the get() method starting
on line 208. we use pop() sometimes to remove an item if its
present, else ignore. you can make any old dict that acts like
__dict__ and that should be pretty easy to do.
sdo...@sistechnology.com
thats ok, as long as it is known, and theres a good reason to keep
them there (e.g. avoid redundant getattrs or whatever).
what i really mean in this thread is: to me anything is okay as long
it is well-defined and known (and kept as rule). To you this probably
means we better make it in a way so u dont have to twist yourself
around it later...
thats why i'm throwing in requirements, throw in yours too, let others
throw theirs, and then lets shape something.
sdo...@sistechnology.com
> On Feb 28, 2008, at 4:25 PM, sdo...@sistechnology.com wrote:
> > What i'm talking about is a protocol of how SA is using that
> > attr-value-storage. i.e. that dict-like thing must have these and
> > these methods (like, why u need pop?), and expect them do behave
> > so and so if not really obvious (like that getitem failing when a
> > trigger is pending), etc. i dont expect u to do it straight away,
> > from my experience with SA code these things are scattered all
> > around and sometimes in quite weird occasions.
> > do u understand me now? sorry about being a bit, er, intrusive..
>
> what dict-like thing, you mean __dict__ ? the contract between
> __dict__ and a trigger firing is pretty straightforward. key is in
> dict, no trigger fires. key is not there, trigger fires. that
> logic takes place entirely within attributes.py in the get() method
> starting on line 208.
(therefore in my case i have to know if there's a trigger pending to
simulate a missing value. not very clean..)
any other similar contracts?
> we use pop() sometimes to remove an item if
> its present, else ignore. you can make any old dict that acts like
> __dict__ and that should be pretty easy to do.
ok, so u need the method pop() to be there, and do exactly what plain
dict does.
let me do a draft:
class dict4attr_storage_protocol:
'required methods of the state.dict's usage-protocol'
def has_key( self, sk):
'return True if present, else False'
raise NotImplementedError
__contains__ = has_key
def __getitem__( self, key, *defaultvalue):
'''return value of that key if found
else if defaultvalue present, return that
else raise KeyError
'''
raise NotImplementedError
def get( self, key, defaultvalue =None):
return self.__getitem__( key, defaultvalue)
def __setitem__( self, key, value):
'''store value for key'''
raise NotImplementedError
def __delitem__( self, key):
'''delete value for key'''
raise NotImplementedError
def pop( self, key, *defaultvalue):
'''if key found, remove it and return its value
else if defaultvalue present, return that
else raise KeyError
'''
raise NotImplementedError
for example having has_key AND __contains__ being same, is redundant.
contract for triggerring:
trigger checks for key in state.dict via
(contains?haskey?getitem?get?) and fires if not found.
contract for ... any other specific behaviours that may have some
influence into the dict itself...
etc.....
is this making sense?
Phillip J. Eby
If I'm understanding this correctly, it sounds like it would be best
if, for Trellis integration, I just give SA an ordinary dict (or the
object __dict__), and ignore its contents entirely! Instead, I would
set up the object's cells to entirely delegate the attribute storage
to SA. Which is nice, in that I then wouldn't have to make a custom
dict type. I guess that'd work.
Interestingly, this'd mean that I'd only be using the instance_dict
hooks to set up the cell delegation, not to actually create an
instance dict. It's starting to look like it'll be really simple to
set up - I just need a cell type that delegates reads and writes to a
descriptor (given a target object).
One minor complication is that I'll need to know if the value being
set is a modification of the current value, but doing a __get__
before each __set__ seems likely to be a problem if an attribute is
lazily loaded. I suppose I could check the __dict__ instead, but it
probably makes more sense to cache the last __get__ value in the cell
and use that for comparison instead. If there's no such value, then
I can assume that a write is causing a change. That approach would
let me make the delegating cell class generic -- i.e., it could be
used to delegate to any descriptor, not just SA ones.
So, assuming I have a DescriptorCell(ob, descr) type, and the
proposed "dict of descriptors" API for SA, I could do something like:
def initialize_instance_dict(self, ob):
c = trellis.Cells(ob)
for attr, descr in get_class_state(type(ob)).items():
c[attr] = DescriptorCell(ob, descr)
In truth, it would need to be a bit more sophisticated than this, so
as not to replace existing DescriptorCell objects in the cells
dict. But so far it's looking like this is going to be a pretty
sweet'n'easy hookup.
Phillip J. Eby
>On Feb 27, 2008, at 3:10 PM, Phillip J. Eby wrote:
> > What methods must this object support, if I were writing a completely
> > new object from scratch? (Because I just might be.) Are
> > __getitem__, __setitem__, and __contains__ sufficient? Or is there
> > more needed? (E.g. __delitem__?)
>
>it'll need __delitem__ for sure since "del myinstance.foo" comes down
>to "del state.dict[key]". we also call dict.get() on a lot of
>occasions. I dont think you need any aggreagte methods like __iter__
>or keys() though.
After more reading, it seems like this is moot. I'll use a plain
dict (probably just the instance __dict__) and let SA handle the
storage management, as I don't need the Trellis to capture SA events
and broadcast them to the rest of the app. It's sufficient that
Trellis read/writes get pushed down to SA, without notices needing to
go the other way.
> > In fact, you could go a bit further, and make ClassState a dictionary
> > subclass, whose 'get_inst()' is just a plain
> > __getitem__. (e.g. ClassState.get_inst = dict.__getitem__). That
> > would be awesomely fast -- perhaps faster even than your original
> > getattr() approach. (Seriously -- especially since it also skips the
> > call to the descriptor __get__, as well.)
> >
> > get_impl() would then be just "return self[key].impl", and
> > ClassState.is_instrumented = dict.__contains__.
>
>Thats interesting too. Im a little antsy about ClassState being
>singularly minded around "I am a dict of attribute instrumentors" but
>give me time on that.
Well, it should probably be called something more like ClassMetadata
or ClassMappingInfo anyway. :)
>do you have any interest in committing changes to the branch
>yourself ? as long as the unit tests keep running whatever you'd want
>is most likely fine with me....
Well, I am getting some test failures on the branch, without having
made any changes:
======================================================================
FAIL: test_orderby (orm.eager_relations.EagerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"C:\cygwin\home\pje\user_defined_state\test\orm\eager_relations.py",
line 55, in test_orderby
] == q.all()
AssertionError
======================================================================
FAIL: test_orderby_desc (orm.eager_relations.EagerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"C:\cygwin\home\pje\user_defined_state\test\orm\eager_relations.py",
line 123, in test_orderby_desc
] == sess.query(User).all()
AssertionError
======================================================================
FAIL: test_orderby_multi (orm.eager_relations.EagerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"C:\cygwin\home\pje\user_defined_state\test\orm\eager_relations.py",
line 75, in test_orderby_multi
] == q.all()
AssertionError
----------------------------------------------------------------------
Ran 1191 tests in 139.547s
FAILED (failures=3)
Dunno what's up with that. But yes, I'd be happy to help with the
changes I suggested, assuming I can start from a version that passes
all the tests. :)
(By the way, I'm using a Windows Python 2.5, not a Cygwin one,
despite using Cygwin as my command shell.)
Michael Bayer
what SQLite are you running there ? SQLite had some ordering issues
at some point, perhaps the windows builds of py2.5/sqlite3 are still
carrying around those errors (unless you have an explicit pysqlite/
sqlite installation external to your Python 2.5).
Michael Bayer
http://www.sqlite.org/cvstrac/tktview?tn=2211
was introduced between 3.3.8 and 3.3.9 and was fixed in 3.3.13.
sdo...@sistechnology.com
like plain setattr but going through your descriptor, and let SA keep
the values wherever. u'll need to know how to get them back though...
Another thing, when SA-object loads, how are u going to know what
events of yours to fire -- or you just fire anything aka setup_all()?
some values may have not been filled in (lazy, deferred, missing,
whatever).
btw it would be interestng ifu can chain the sa/trellis descriptors
behavour somehow, i.e. for some lazy-load-attr, fire the SA-stuff
first, then fire yours.
sdo...@sistechnology.com
> > best if, for Trellis integration, I just give SA an ordinary dict
> > (or the object __dict__), and ignore its contents entirely!
> > Instead, I would set up the object's cells to entirely delegate
> > the attribute storage to SA. Which is nice, in that I then
> > wouldn't have to make a custom dict type. I guess that'd work.
>
> yes, IMO in your case u only need the "above"-SA attr-access, i.e.
> like plain setattr but going through your descriptor, and let SA
> keep the values wherever. u'll need to know how to get them back
> though... Another thing, when SA-object loads, how are u going to
> know what events of yours to fire -- or you just fire anything aka
> setup_all()? some values may have not been filled in (lazy,
> deferred, missing, whatever).
>
> btw it would be interestng ifu can chain the sa/trellis descriptors
> behavour somehow, i.e. for some lazy-load-attr, fire the SA-stuff
> first, then fire yours.
y... or maybe this i not correct example?
also check how session.expire() and similar should work.
Phillip J. Eby
>yes, IMO in your case u only need the "above"-SA attr-access, i.e.
>like plain setattr but going through your descriptor, and let SA keep
>the values wherever. u'll need to know how to get them back though...
I'll just ask the SA descriptor to __get__ them.
>Another thing, when SA-object loads, how are u going to know what
>events of yours to fire
There's nothing to fire, unless we're talking about reloading an
object that's been changed in the DB by another process, which is not
a notification scenario I intend to support at the moment.
>-- or you just fire anything aka setup_all()?
>some values may have not been filled in (lazy, deferred, missing,
>whatever).
If they haven't been read yet, then nothing can depend on them. The
Trellis creates dependency links only when a value is read - the rule
that reads the value will then be re-run if the value changes
later. If you never read a value, there's nothing to re-run, so
nothing special happens when it changes.
>btw it would be interestng ifu can chain the sa/trellis descriptors
>behavour somehow, i.e. for some lazy-load-attr, fire the SA-stuff
>first, then fire yours.
More precisely, it'll be wrapped. The cell, when read, will simply
do the normal trellis "i'm being read" dance, and then ask SA for the
value to return. When written, it will ask SA to do the write, then
tell the trellis the value changed.
The tricky part is that normally, cells only indicate a change when
their new value is not equal to their old value. Right now I'm not
sure what sort of transformations might be taking place on values
managed by SA, so I'm not sure of how to do a clean change detect at
this level. But in the simplest case, it could work by caching the
last-read value.
Phillip J. Eby
>what SQLite are you running there ? SQLite had some ordering issues
>at some point, perhaps the windows builds of py2.5/sqlite3 are still
>carrying around those errors (unless you have an explicit pysqlite/
>sqlite installation external to your Python 2.5).
I do, but it still has the problem, and I can't seem to find any
binary releases of pysqlite for Python 2.5 on Windows at the moment;
pysqlite.org and initrd.org appear to be down right now.
sdo...@sistechnology.com
> >events of yours to fire
>
> There's nothing to fire, unless we're talking about reloading an
> object that's been changed in the DB by another process, which is
> not a notification scenario I intend to support at the moment.
it will be calculated? lazily when someone gets it or?
is there some sort of calculate-all() which can be called to ensure
that the massive calculation part (massive=slow - imagine 100 rules
on one object) happens around load'ing, and not later?
on another tune... one of my usecases was about inside-db dependencies
and how they can be automated. For a simple example, see the
dbcook/misc/aggregator (or http://dev.gafol.net/t/aggregator). may be
not the actual calculations / writes, they will be managed by
aggregator-like thing, maybe mapper-extension, but about solving the
actual graph of dependencies into a (recipe-like) sequence of
calculations. some topological sort maybe...
> >-- or you just fire anything aka setup_all()?
> >some values may have not been filled in (lazy, deferred, missing,
> >whatever).
>
> If they haven't been read yet, then nothing can depend on them.
> The Trellis creates dependency links only when a value is read -
> the rule that reads the value will then be re-run if the value
> changes later. If you never read a value, there's nothing to
> re-run, so nothing special happens when it changes.
>
> >btw it would be interestng ifu can chain the sa/trellis
> > descriptors behavour somehow, i.e. for some lazy-load-attr, fire
> > the SA-stuff first, then fire yours.
>
> More precisely, it'll be wrapped. The cell, when read, will simply
> do the normal trellis "i'm being read" dance, and then ask SA for
> the value to return. When written, it will ask SA to do the write,
> then tell the trellis the value changed.
>
> The tricky part is that normally, cells only indicate a change when
> their new value is not equal to their old value. Right now I'm not
> sure what sort of transformations might be taking place on values
> managed by SA, so I'm not sure of how to do a clean change detect
> at this level. But in the simplest case, it could work by caching
> the last-read value.
u have to find out when/how in the workflow to pass the new value down
to SA..
Phillip J. Eby
> > >Another thing, when SA-object loads, how are u going to know what
> > >events of yours to fire
> >
> > There's nothing to fire, unless we're talking about reloading an
> > object that's been changed in the DB by another process, which is
> > not a notification scenario I intend to support at the moment.
>how come... if x=y+z, where y and z stay in DB, and x does not, when
>it will be calculated? lazily when someone gets it or?
That depends on how the rule is defined. The current plan is that
when initialize_instance_dict is called, the y and z cells will be
replaced, and any previous readers of those cells will be scheduled
for recalculation.
There is, however, a twist that might be a problem, because SA's
loading process might not be finished before the recalculation occurs
-- and that would be A Bad Thing.
So, maybe the SA instance initialization protocol should have a
callback to indicate when it's safe to actually *use* the instance
that's just been initialized.
>is there some sort of calculate-all() which can be called to ensure
>that the massive calculation part (massive=slow - imagine 100 rules
>on one object) happens around load'ing, and not later?
That's not how the trellis does recalculation; it's specific to the
rules that actually *looked* at a particular cell value. Also, rules
can be defined as "optional" (i.e. don't calculate until needed), and
rules that depend on DB values -- especially lazy-loaded ones --
should definitely be defined that way.
> > More precisely, it'll be wrapped. The cell, when read, will simply
> > do the normal trellis "i'm being read" dance, and then ask SA for
> > the value to return. When written, it will ask SA to do the write,
> > then tell the trellis the value changed.
> >
> > The tricky part is that normally, cells only indicate a change when
> > their new value is not equal to their old value. Right now I'm not
> > sure what sort of transformations might be taking place on values
> > managed by SA, so I'm not sure of how to do a clean change detect
> > at this level. But in the simplest case, it could work by caching
> > the last-read value.
>u have to find out when/how in the workflow to pass the new value down
>to SA..
Actually, no, that's not the question. I'll *always* pass the new
value down to SA when someone sets an attribute. It's deciding
whether that assignment constitutes a change from the Trellis POV
that's the open question. A normal Trellis cell just compares the
assigned value to its previous stored value for that attribute. But
with SA, there are two potential hitches. Getting the old value to
do a comparison on it might trigger a DB load of some kind -- which
would be unnecessary if we weren't trying to compare against
it. Second, we'd need to also get the value *after* the assignment,
since SA might have done some type conversion on it. (At least, if I
understand correctly.)
Michael Bayer
On Feb 29, 2008, at 10:38 AM, Phillip J. Eby wrote:
>
> So, maybe the SA instance initialization protocol should have a
> callback to indicate when it's safe to actually *use* the instance
> that's just been initialized.
I can lay out for you the full initialization sequence for an
instance. What I dont know here is if "safe" means, the instance is
good to go on the mapper side but its __init__ has not been called
yet, or if its __init__ has been called already. I find this to be
a very sticky area since everyone has their own things they want to do
in their __init__ methods.
On attributes.py line 1099 is register_class, where we decorate
__init__ on the mapped class. If the class doesnt have __init__ (i.e.
its that of object()), we just add our own __init__ instead.
so our decorated init, including with the new plugin setup, does:
if no '_state' attribute present:
instance._class_state.manage(instance)
your_plugin.initialize_instance_dict(instance)
instance._state = InstanceState()
instance._state.dict =
your_plugin.get_instance_dict(instance)
"extra_init()":
mapper.compile() # compiles the mapper if not already compiled
mapper.extension.init_instance(cls, oldinit, instance, ...) # this
is a MapperExtension hook. we have one extension using it
if decorated "__init__" method present:
__init__()
So we do have a hook thats called when the instance is "safe" to use
but hasnt yet had its real __init__() method called, which is
init_instance() attached to MapperExtension. The "extra_init()"
callable, which is what calls the MapperExtension is passed in when
the mapper first contacts the attributes package to register_class.
Since it seems like here we're trying to avoid dealing with
MapperExtension we can give InstrumentClass either the chance to
decorate extra_init(), or we can give it the chance to add its own
extra_init() callable to a collection of callables...although
decorating gives you more options. A patch is attached which adds
instrument_oninit() and instrument_onexception() methods to
InstrumentClass.
Phillip J. Eby
>On attributes.py line 1099 is register_class, where we decorate
>__init__ on the mapped class. If the class doesnt have __init__ (i.e.
>its that of object()), we just add our own __init__ instead.
Hm. So all of SA's setup occurs in the decorated
__init__? Excellent - because all of Trellis' special wrapping
occurs in the metaclass __call__, *after* __init__ returns.
So, initialize_instance_dict() (aka manage()) only gets called once
per object, ever?
>So we do have a hook thats called when the instance is "safe" to use
>but hasnt yet had its real __init__() method called, which is
>init_instance() attached to MapperExtension. The "extra_init()"
>callable, which is what calls the MapperExtension is passed in when
>the mapper first contacts the attributes package to register_class.
>Since it seems like here we're trying to avoid dealing with
>MapperExtension we can give InstrumentClass either the chance to
>decorate extra_init(), or we can give it the chance to add its own
>extra_init() callable to a collection of callables...although
>decorating gives you more options. A patch is attached which adds
>instrument_oninit() and instrument_onexception() methods to
>InstrumentClass.
I don't need another hook, as long as __init__ is the only place from
which initialize_instance_dict() will be called, and __init__ is
being called via the normal 'SomeClass()' constructor. Trellis
initialization takes place inside the 'SomeClass()' constructor
*after* __init__ returns, which is perfect for this setup. Among
other things, it means I won't have to worry about re-running rules
that have already read existing cells on the object... because they
won't have existed at the time initialize_instance_dict() is
called. That would be perfect.
By the way, I managed to get an up-to-date pysqlite, so I can now run
the unit tests. How do we get me set up with write access so I can
start on the ClassState refactoring?
Judah De Paula
In the short term I'm accessing the _state.committed_state to make my
code work again.
In the long run I think the mechanism described above, and the code you
gave in the other e-mail, provide me with enough support so that I could
write a more permanent solution.
Thank you,
Judah
Phillip J. Eby
>do you have any interest in committing changes to the branch
>yourself ? as long as the unit tests keep running whatever you'd want
>is most likely fine with me....otherwise I will at least experiment
>with doing away with __mro__ searching and possibly doing away with
>pre_instrument_attribute.
Okay, I've run into an interesting hitch on the branch -- I think
it's a test coverage issue. The definition of
'ClassState.is_instrumented' is different from that of
_ClassStateAdapter, in a way that appears to *matter*. That is, if
you change ClassState.is_instrumented to work in roughly the same way
as _ClassStateAdapter.is_instrumented, the standard tests
fail. Specifically, with this rather unhelpful error... :)
Traceback (most recent call last):
File "test/orm/attributes.py", line 397, in test_collectionclasses
assert False
AssertionError
This would appear to indicate that there is some kind of hidden
dependency between collections support, and the presence of SA
descriptors on a class.
Further investigation reveals that the problem comes from
register_attribute() checking to see if an attribute is already
instrumented, and if so, returns immediately. It would appear that
this really wants to know if there is actually an instrumented
descriptor, rather than whether one has merely been registered with
the class state, per these comments::
# this currently only occurs if two primary mappers are made for the
same class.
# TODO: possibly have InstrumentedAttribute check "entity_name" when
searching for impl.
# raise an error if two attrs attached simultaneously otherwise
I'm pretty much stumped at this point. Essentially, even though the
tests don't cover this point for custom instrumentation, it means
that this problem *will* occur with custom instrumentation, even
without my changes.
Now, the *good* news is, the other 1190 tests all pass with my changes. :)
I've attached my current diff against the user_defined_state
branch. Note that this bit is the part that makes the one test break:
def is_instrumented(self, key, search=False):
if search:
- return hasattr(self.class_, key) and
isinstance(getattr(self.class_, key), InstrumentedAttribute)
+ return key in self
else:
- return key in self.class_.__dict__ and
isinstance(self.class_.__dict__[key], InstrumentedAttribute)
+ return key in self.local_attrs
In other words, it's only the change to ClassState.is_instrumented
that breaks the test, showing that there's a meaningful difference
here between what gets registered in the state, and what's actually
on the class. Which is going to be a problem for anybody rerouting
the descriptors (the way the Trellis library will be).
Michael Bayer
suite is using to remove and re-register new instrumentation:
class Foo(object):
pass
attributes.register_attribute(Foo, "collection",
uselist=True, typecallable=set, useobject=True)
assert
attributes.get_class_state(Foo).is_instrumented("collection")
attributes.unregister_attribute(Foo, "collection")
assert not
attributes.get_class_state(Foo).is_instrumented("collection")
seems like unregister_attribute() is still doing the old thing of just
"del class.attr", so we'd just need to stick a hook similar to
"instrument_attribute" for this on ClassState which will take over the
job.
looks good !
> <classstate_dict.patch>
Phillip J. Eby
>the bug is that unregister_attribute() is not working, which the test
>suite is using to remove and re-register new instrumentation:
>
> class Foo(object):
> pass
>
> attributes.register_attribute(Foo, "collection",
>uselist=True, typecallable=set, useobject=True)
> assert
>attributes.get_class_state(Foo).is_instrumented("collection")
> attributes.unregister_attribute(Foo, "collection")
> assert not
>attributes.get_class_state(Foo).is_instrumented("collection")
>
>seems like unregister_attribute() is still doing the old thing of just
>"del class.attr", so we'd just need to stick a hook similar to
>"instrument_attribute" for this on ClassState which will take over the
>job.
>
>looks good !
Okay, so I did a matching uninstrument_attribute and
pre_uninstrument_attribute, which look like really dumb names at this
point. I propose the following name changes (in addition to the ones
in my previous patch):
pre_instrument_attribute -> install_descriptor
pre_uninstrument_attribute -> uninstall_descriptor
That okay? instrument_attribute will then call install_descriptor to
do the actual installing. And of course the hooks on the adapted
thingy from the class would work the same way.
If that's okay with you, then after I'm done I'll post a patch for
review before checkin. After that, I'll start work on the
Trellis-side support for this, and then eventually dig into collections stuff.
Michael Bayer
On Mar 4, 2008, at 7:27 PM, Phillip J. Eby wrote:
>
> Okay, so I did a matching uninstrument_attribute and
> pre_uninstrument_attribute, which look like really dumb names at this
> point. I propose the following name changes (in addition to the ones
> in my previous patch):
>
> pre_instrument_attribute -> install_descriptor
> pre_uninstrument_attribute -> uninstall_descriptor
>
> That okay? instrument_attribute will then call install_descriptor to
> do the actual installing. And of course the hooks on the adapted
> thingy from the class would work the same way.
>
> If that's okay with you, then after I'm done I'll post a patch for
> review before checkin. After that, I'll start work on the
> Trellis-side support for this, and then eventually dig into
> collections stuff.
hey phillip -
It all sounds great to me ! I think its been established in the past
several months that my names are generally pretty bad so I usually
defer to others at this point.
- mike
jason kirtland
>
> On Feb 27, 2008, at 12:19 PM, jason kirtland wrote:
>
>> Michael Bayer wrote:
>>> You also get a hook that receives the
>>> collection_class argument in the case of a collection-based attribute
>>> and you get to return your own class, in which case all the
>>> collections API stuff can kick in on that side.
>> This can be opened up a bit to allow custom collection event systems
>> as
>> well. We'd move the
>>
>> 'collections._prepare_instrumentation(typecallable)'
>>
>> out of the Attribute and into SA's default implementation of
>> 'instrument_collection_class'. If custom extenders want SA
>> instrumentation, they can call that themselves. The
>> _prepare_instrumentation itself can become a public function, it's
>> stable.
>>
> go for it
Ok, that's in. At the extreme end, collections can route events to
anything that quacks like a CollectionAdapter. The moderate path is to
make the collection's events compatible with the built-in
CollectionAdapter. And of course the easy path is to just use the
existing collections instrumentation toolkit, it's already plenty flexible.
The sample script has an example of the extreme route, and the moderate
is in test/.