I'm weighing whether to use traversal in a limited way, and whether the benefits would outweigh the costs.
I'm converting a Pylons application that has two database-related routes for read-only access: /incidents/{id} => Incident ORM class /entries/{id} => Entry ORM class
I can use URL dispatch in Pyramid, but then the views have to do the work of converting the ID into an int, and aborting 404 if it's invalid or the record doesn't exist, and check whether the user's permissions are compatible with the record's properties. (I don't have to check perms in this application because the database views hide non-public records, but I would in another application so I'm thinking ahead.) On the other hand, it's easy to generate a URL to any incident: request.route_url("incident", id=incident_id)
Using traversal, the resource tree would take care of fetching the database record and reporting if it doesn't exist, and it could also tie in with ACL permissions. I've verified that SQLAlchemy ORM instances don't have .__name__ or .__parent__ attributes so I could set these for location awareness. I'm thinking of a resource structure like this:
=== class Resource(dict): def __init__(self, name, parent): self.name = name self.parent = parent
class RecordGetter(dict): """Traversal component tied to a SQLAlchemy ORM class.
Calling .__getitem__ calls ``orm_class.get``, adds certain attributes to the object, and returns it. (My classes define the .get method as a wrapper for query get.) """ def __init__(self, name, parent, request, orm_class): self.__name__ = name self.__parent__ = parent self.request = request self.orm_class = orm_class
So first, is this the right way to to it? Second, how can I generate URLs to resources without loading the objects through this interface? The home page queries the 30 most recent incidents and displays links to them. request.resource_url() requires a resource object, but the home page or browse pages query the database directly and get plain ORM objects rather than location-aware ones. I could make the URLs manually but that sucks. I could somehow get the "incident" resource and append the ID, or get the root resource and append "incident" and the ID, but that's not much better.
Creating a traversal path ("/incident/123"), asking for the resource, and then generating the URL from it (which would be the same "/incident/123"), would be equally silly, and would also trigger a redundant query.
And that raises another issue, the "incident" resource doesn't really exist. I.e., "/incident" should return an error, not connect to a regular view or default view. How do I do that? Or how do I make the default view be "not found"?
On Thu, Feb 16, 2012 at 12:09:36PM -0800, Mike Orr wrote: > I'm weighing whether to use traversal in a limited way, and whether > the benefits would outweigh the costs.
> I'm converting a Pylons application that has two database-related > routes for read-only access: > /incidents/{id} => Incident ORM class > /entries/{id} => Entry ORM class
> I can use URL dispatch in Pyramid, but then the views have to do the > work of converting the ID into an int, and aborting 404 if it's > invalid or the record doesn't exist, and check whether the user's > permissions are compatible with the record's properties. (I don't have > to check perms in this application because the database views hide > non-public records, but I would in another application so I'm thinking > ahead.) On the other hand, it's easy to generate a URL to any > incident: request.route_url("incident", id=incident_id)
> Using traversal, the resource tree would take care of fetching the > database record and reporting if it doesn't exist, and it could also > tie in with ACL permissions. I've verified that SQLAlchemy ORM > instances don't have .__name__ or .__parent__ attributes so I could > set these for location awareness.
As a side note, you could also use pyramid_traversalwrapper[1] "which wraps each traversed object in a location-aware proxy", so you don't need to manage __parent__ and __name__ attributes by yourself, I've found it pretty useful.
> So first, is this the right way to to it?
I think it's fine.
> Second, how can I generate URLs to resources without loading the objects > through this interface? The home page queries the 30 most recent incidents > and displays links to them. request.resource_url() requires a resource > object, but the home page or browse pages query the database directly and get > plain ORM objects rather than location-aware ones. I could make the URLs > manually but that sucks. I could somehow get the "incident" resource and > append the ID, or get the root resource and append "incident" and the ID, but > that's not much better.
That's interesting question. I'm thinking of the following construct:
1. Make your resource graph lazy, e.g. producing only proxies which only knows its location (pk in database in case of SQLAlchemy) and how to load itself (table and parent resources, which impose some query constraints or JOIN clauses).
2. Reuse your resource graph in pyramid's traversal engine and in your home page view:
def homepage(): entries_ids = db.entries_ids_for_homepage() entries = [root["entries"][id] for id in entries_ids]
So you get ``entries`` list of location-aware Entry objects which are loaded only if you access their fields, but not __name__ or __parent__ which allows you to generate URL for them w/o any overhead.
> Creating a traversal path ("/incident/123"), asking for the resource, > and then generating the URL from it (which would be the same > "/incident/123"), would be equally silly, and would also trigger a > redundant query.
Why not memoize results of traversing? This kind of caching seems pretty natural to me.
> And that raises another issue, the "incident" resource doesn't really > exist. I.e., "/incident" should return an error, not connect to a > regular view or default view. How do I do that? Or how do I make the > default view be "not found"?
I'm sure it should exist -- it's a container of incidents! Any reasons you don't want to expose it? If yes -- just adapt it to HTTPForbidden or something like this.
On Thu, Feb 16, 2012 at 12:09:36PM -0800, Mike Orr wrote: > I'm weighing whether to use traversal in a limited way, and whether > the benefits would outweigh the costs.
> I'm converting a Pylons application that has two database-related > routes for read-only access: > /incidents/{id} => Incident ORM class > /entries/{id} => Entry ORM class
> I can use URL dispatch in Pyramid, but then the views have to do the > work of converting the ID into an int, and aborting 404 if it's > invalid or the record doesn't exist, and check whether the user's > permissions are compatible with the record's properties. (I don't have > to check perms in this application because the database views hide > non-public records, but I would in another application so I'm thinking > ahead.) On the other hand, it's easy to generate a URL to any > incident: request.route_url("incident", id=incident_id)
> Using traversal, the resource tree would take care of fetching the > database record and reporting if it doesn't exist, and it could also > tie in with ACL permissions. I've verified that SQLAlchemy ORM > instances don't have .__name__ or .__parent__ attributes so I could > set these for location awareness.
As a side note, you could also use pyramid_traversalwrapper[1] "which wraps each traversed object in a location-aware proxy", so you don't need to manage __parent__ and __name__ attributes by yourself, I've found it pretty useful.
> So first, is this the right way to to it?
I think it's fine.
> Second, how can I generate URLs to resources without loading the objects > through this interface? The home page queries the 30 most recent incidents > and displays links to them. request.resource_url() requires a resource > object, but the home page or browse pages query the database directly and get > plain ORM objects rather than location-aware ones. I could make the URLs > manually but that sucks. I could somehow get the "incident" resource and > append the ID, or get the root resource and append "incident" and the ID, but > that's not much better.
That's interesting question. I'm thinking of the following construct:
1. Make your resource graph lazy, e.g. producing only proxies which only knows its location (pk in database in case of SQLAlchemy) and how to load itself (table and parent resources, which impose some query constraints or JOIN clauses).
2. Reuse your resource graph in pyramid's traversal engine and in your home page view:
def homepage(): entries_ids = db.entries_ids_for_homepage() entries = [root["entries"][id] for id in entries_ids]
So you get ``entries`` list of location-aware Entry objects which are loaded only if you access their fields, but not __name__ or __parent__ which allows you to generate URL for them w/o any overhead.
> Creating a traversal path ("/incident/123"), asking for the resource, > and then generating the URL from it (which would be the same > "/incident/123"), would be equally silly, and would also trigger a > redundant query.
Why not memoize results of traversing? This kind of caching seems pretty natural to me.
> And that raises another issue, the "incident" resource doesn't really > exist. I.e., "/incident" should return an error, not connect to a > regular view or default view. How do I do that? Or how do I make the > default view be "not found"?
I'm sure it should exist -- it's a container of incidents! Any reasons you don't want to expose it? If yes -- just adapt it to HTTPForbidden or something like this.
On Thu, Feb 16, 2012 at 1:42 PM, Andrey Popp <8may...@gmail.com> wrote: >> And that raises another issue, the "incident" resource doesn't really >> exist. I.e., "/incident" should return an error, not connect to a >> regular view or default view. How do I do that? Or how do I make the >> default view be "not found"?
> I'm sure it should exist -- it's a container of incidents! Any reasons you > don't want to expose it? If yes -- just adapt it to HTTPForbidden or something > like this.
In a full REST application I would, but in this case the home page has the most recent incidents (which is all most people care about), and there are separate browse pages if somebody want to page through all 5000 incidents by date or name. That's also why the URL is "/incident/1234" instead of "/incidents/1234", because there is no (valid) "/incidents" page. I suppose that in itself may be an argument against traversal.
On Thu, 2012-02-16 at 19:13 -0800, Mike Orr wrote: > On Thu, Feb 16, 2012 at 1:42 PM, Andrey Popp <8may...@gmail.com> wrote: > >> And that raises another issue, the "incident" resource doesn't really > >> exist. I.e., "/incident" should return an error, not connect to a > >> regular view or default view. How do I do that? Or how do I make the > >> default view be "not found"?
> > I'm sure it should exist -- it's a container of incidents! Any reasons you > > don't want to expose it? If yes -- just adapt it to HTTPForbidden or something > > like this.
> In a full REST application I would, but in this case the home page has > the most recent incidents (which is all most people care about), and > there are separate browse pages if somebody want to page through all > 5000 incidents by date or name. That's also why the URL is > "/incident/1234" instead of "/incidents/1234", because there is no > (valid) "/incidents" page. I suppose that in itself may be an argument > against traversal.
Coming to this discussion late but I'm not sure I understand the problem. If there's no view registered for the type of object implied by "/incident", it will 404. I don't think there's any subtlety here.
On Thu, Feb 16, 2012 at 7:35 PM, Chris McDonough <chr...@plope.com> wrote: > On Thu, 2012-02-16 at 19:13 -0800, Mike Orr wrote: >> On Thu, Feb 16, 2012 at 1:42 PM, Andrey Popp <8may...@gmail.com> wrote: >> >> And that raises another issue, the "incident" resource doesn't really >> >> exist. I.e., "/incident" should return an error, not connect to a >> >> regular view or default view. How do I do that? Or how do I make the >> >> default view be "not found"?
>> > I'm sure it should exist -- it's a container of incidents! Any reasons you >> > don't want to expose it? If yes -- just adapt it to HTTPForbidden or something >> > like this.
>> In a full REST application I would, but in this case the home page has >> the most recent incidents (which is all most people care about), and >> there are separate browse pages if somebody want to page through all >> 5000 incidents by date or name. That's also why the URL is >> "/incident/1234" instead of "/incidents/1234", because there is no >> (valid) "/incidents" page. I suppose that in itself may be an argument >> against traversal.
> Coming to this discussion late but I'm not sure I understand the > problem. If there's no view registered for the type of object implied > by "/incident", it will 404. I don't think there's any subtlety here.
What about the default view? How do I know which of my views is the default? Is it the first one registered?
On Thu, 2012-02-16 at 20:22 -0800, Mike Orr wrote: > On Thu, Feb 16, 2012 at 7:35 PM, Chris McDonough <chr...@plope.com> wrote: > > On Thu, 2012-02-16 at 19:13 -0800, Mike Orr wrote: > >> On Thu, Feb 16, 2012 at 1:42 PM, Andrey Popp <8may...@gmail.com> wrote: > >> >> And that raises another issue, the "incident" resource doesn't really > >> >> exist. I.e., "/incident" should return an error, not connect to a > >> >> regular view or default view. How do I do that? Or how do I make the > >> >> default view be "not found"?
> >> > I'm sure it should exist -- it's a container of incidents! Any reasons you > >> > don't want to expose it? If yes -- just adapt it to HTTPForbidden or something > >> > like this.
> >> In a full REST application I would, but in this case the home page has > >> the most recent incidents (which is all most people care about), and > >> there are separate browse pages if somebody want to page through all > >> 5000 incidents by date or name. That's also why the URL is > >> "/incident/1234" instead of "/incidents/1234", because there is no > >> (valid) "/incidents" page. I suppose that in itself may be an argument > >> against traversal.
> > Coming to this discussion late but I'm not sure I understand the > > problem. If there's no view registered for the type of object implied > > by "/incident", it will 404. I don't think there's any subtlety here.
> What about the default view? How do I know which of my views is the > default? Is it the first one registered?
It's the one that names the object type represented "/incident" as its "context" argument and which has no "name" argument. If there isn't one of those, it's a 404.
On Thu, Feb 16, 2012 at 8:25 PM, Chris McDonough <chr...@plope.com> wrote: > On Thu, 2012-02-16 at 20:22 -0800, Mike Orr wrote: >> On Thu, Feb 16, 2012 at 7:35 PM, Chris McDonough <chr...@plope.com> wrote: >> > On Thu, 2012-02-16 at 19:13 -0800, Mike Orr wrote: >> >> On Thu, Feb 16, 2012 at 1:42 PM, Andrey Popp <8may...@gmail.com> wrote: >> >> >> And that raises another issue, the "incident" resource doesn't really >> >> >> exist. I.e., "/incident" should return an error, not connect to a >> >> >> regular view or default view. How do I do that? Or how do I make the >> >> >> default view be "not found"?
>> >> > I'm sure it should exist -- it's a container of incidents! Any reasons you >> >> > don't want to expose it? If yes -- just adapt it to HTTPForbidden or something >> >> > like this.
>> >> In a full REST application I would, but in this case the home page has >> >> the most recent incidents (which is all most people care about), and >> >> there are separate browse pages if somebody want to page through all >> >> 5000 incidents by date or name. That's also why the URL is >> >> "/incident/1234" instead of "/incidents/1234", because there is no >> >> (valid) "/incidents" page. I suppose that in itself may be an argument >> >> against traversal.
>> > Coming to this discussion late but I'm not sure I understand the >> > problem. If there's no view registered for the type of object implied >> > by "/incident", it will 404. I don't think there's any subtlety here.
>> What about the default view? How do I know which of my views is the >> default? Is it the first one registered?
> It's the one that names the object type represented "/incident" as its > "context" argument and which has no "name" argument. If there isn't one > of those, it's a 404.
Ah, so there's one default view per 'context' argument, rather than just one global default view. That makes more sense now. I'm intending to register all my views with a 'context' argument and without a name, at least for the views I've needed so far. So in that sense, all of them are the default views for their respective contexts.
On Fri, Feb 17, 2012 at 01:42:04AM +0400, Andrey Popp wrote: > On Thu, Feb 16, 2012 at 12:09:36PM -0800, Mike Orr wrote: > > Second, how can I generate URLs to resources without loading the objects > > through this interface? The home page queries the 30 most recent incidents > > and displays links to them. request.resource_url() requires a resource > > object, but the home page or browse pages query the database directly and get > > plain ORM objects rather than location-aware ones. I could make the URLs > > manually but that sucks. I could somehow get the "incident" resource and > > append the ID, or get the root resource and append "incident" and the ID, but > > that's not much better.
> That's interesting question. I'm thinking of the following construct:
> 1. Make your resource graph lazy, e.g. producing only proxies which only > knows its location (pk in database in case of SQLAlchemy) and how to load > itself (table and parent resources, which impose some query constraints or > JOIN clauses).
> 2. Reuse your resource graph in pyramid's traversal engine and in your home > page view:
> def homepage(): > entries_ids = db.entries_ids_for_homepage() > entries = [root["entries"][id] for id in entries_ids]
> So you get ``entries`` list of location-aware Entry objects which are loaded > only if you access their fields, but not __name__ or __parent__ which allows > you to generate URL for them w/o any overhead.
If someone is still interested in this approach, I've implemented[1] sqlalchemy.Query.lazy_get(ident) method which works that way. My use-case is different -- to allow User object to be attached on request without touching database (user id is encoded in authc cookie), but I think it should work for constructing lazy resource graph as well.
Implementation based on SQLAlchemy mailing list post[2].
On Sat, Feb 18, 2012 at 3:53 PM, Andrey Popp <8may...@gmail.com> wrote: > My use-case is > different -- to allow User object to be attached on request without > touching > database (user id is encoded in authc cookie), but I think it should work > for > constructing lazy resource graph as well.
In Pyramid 1.3 I've updated the cookbook to directly support this idea of a "lazy get" via attaching lazily evaluated properties to the request object. You may consider it as an alternative to a custom Query class.
On Sun, Feb 19, 2012 at 12:44:24AM -0600, Michael Merickel wrote: >On Sat, Feb 18, 2012 at 3:53 PM, Andrey Popp <8may...@gmail.com> wrote: >> My use-case is >> different -- to allow User object to be attached on request without >> touching >> database (user id is encoded in authc cookie), but I think it should >> work for >> constructing lazy resource graph as well.
>In Pyramid 1.3 I've updated the cookbook to directly support this idea of a >"lazy get" via attaching lazily evaluated properties to the request object. >You may consider it as an alternative to a custom Query class. >http://docs.pylonsproject.org/projects/pyramid_cookbook/en/latest/aut...
Yeah, thank you, that would work too, but already has custom Query class with othe goodies and also want to access user's id as request.user.id.
this simplify a lot the concept and the design I use with my applications with user data.
Do you think I can write a class to retrieve data from the beaker session if it is loaded instead of dbconn? In case of None I can proceed with dbconn..is this the best approach to minimize queries to the db and using the beaker session in an elegant and efficient way?
On Sun, Feb 19, 2012 at 06:44, Michael Merickel <mmeri...@gmail.com> wrote: > On Sat, Feb 18, 2012 at 3:53 PM, Andrey Popp <8may...@gmail.com> wrote:
>> My use-case is >> different -- to allow User object to be attached on request without >> touching >> database (user id is encoded in authc cookie), but I think it should work >> for >> constructing lazy resource graph as well.
> In Pyramid 1.3 I've updated the cookbook to directly support this idea of > a "lazy get" via attaching lazily evaluated properties to the request > object. You may consider it as an alternative to a custom Query class.
> -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To post to this group, send email to pylons-discuss@googlegroups.com. > To unsubscribe from this group, send email to > pylons-discuss+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/pylons-discuss?hl=en.
On Sun, Feb 19, 2012 at 3:20 AM, Andrey Popp <8may...@gmail.com> wrote: > Yeah, thank you, that would work too, but already has custom Query class > with > othe goodies and also want to access user's id as request.user.id.
That's fine of course. request.user.id works with this pattern though, just to clarify, because the User object is returned as request.user.
On Sun, Feb 19, 2012 at 12:09:19PM -0600, Michael Merickel wrote: > On Sun, Feb 19, 2012 at 3:20 AM, Andrey Popp wrote: > > Yeah, thank you, that would work too, but already has custom Query class > > with othe goodies and also want to access user's id as request.user.id.
> That's fine of course. request.user.id works with this pattern though, just > to clarify, because the User object is returned as request.user.
Yeah, but it touches the database if dbconn['users'].query({'id':userid}) isn't implemented as Query.lazy_get I've posted in gist, so both patterns are useful and orthogonal for each other :-).
On Sat, Feb 18, 2012 at 10:44 PM, Michael Merickel <mmeri...@gmail.com> wrote: > On Sat, Feb 18, 2012 at 3:53 PM, Andrey Popp <8may...@gmail.com> wrote:
>> My use-case is >> different -- to allow User object to be attached on request without >> touching >> database (user id is encoded in authc cookie), but I think it should work >> for >> constructing lazy resource graph as well.
> In Pyramid 1.3 I've updated the cookbook to directly support this idea of a > "lazy get" via attaching lazily evaluated properties to the request object. > You may consider it as an alternative to a custom Query class.
There is one issue with this approach. If the record doesn't exist, the tree shouldn't be retturning a resource at all, the caller should raise KeyError. But it doesn't know if the record exists without querying the database. So it's kind of a catch-22, unless you assume the resource object means "syntactically valid ID" rather than "record that definitely exists".
On Sun, 2012-02-19 at 11:13 -0800, Mike Orr wrote: > On Sat, Feb 18, 2012 at 10:44 PM, Michael Merickel <mmeri...@gmail.com> wrote: > > On Sat, Feb 18, 2012 at 3:53 PM, Andrey Popp <8may...@gmail.com> wrote:
> >> My use-case is > >> different -- to allow User object to be attached on request without > >> touching > >> database (user id is encoded in authc cookie), but I think it should work > >> for > >> constructing lazy resource graph as well.
> > In Pyramid 1.3 I've updated the cookbook to directly support this idea of a > > "lazy get" via attaching lazily evaluated properties to the request object. > > You may consider it as an alternative to a custom Query class.
> There is one issue with this approach. If the record doesn't exist, > the tree shouldn't be retturning a resource at all, the caller should > raise KeyError.
You're applying traversal terminology to this cookbook entry, but the cookbook entry is unrelated to traversal. Can you clarify?
On Sun, Feb 19, 2012 at 11:21 AM, Chris McDonough <chr...@plope.com> wrote: > On Sun, 2012-02-19 at 11:13 -0800, Mike Orr wrote: >> On Sat, Feb 18, 2012 at 10:44 PM, Michael Merickel <mmeri...@gmail.com> wrote: >> > On Sat, Feb 18, 2012 at 3:53 PM, Andrey Popp <8may...@gmail.com> wrote:
>> >> My use-case is >> >> different -- to allow User object to be attached on request without >> >> touching >> >> database (user id is encoded in authc cookie), but I think it should work >> >> for >> >> constructing lazy resource graph as well.
>> > In Pyramid 1.3 I've updated the cookbook to directly support this idea of a >> > "lazy get" via attaching lazily evaluated properties to the request object. >> > You may consider it as an alternative to a custom Query class.
>> There is one issue with this approach. If the record doesn't exist, >> the tree shouldn't be retturning a resource at all, the caller should >> raise KeyError.
> You're applying traversal terminology to this cookbook entry, but the > cookbook entry is unrelated to traversal. Can you clarify?
My original question in this thread was using traversal to fetch a record from a certain table, and the tradeoffs of doing so vs using a route. The suggestion of using a lazy evaluator was brought up in this context, and the cookbook example was shown as an example of a lazy evaluator.
