Generic routes for API functions

[Ross Vandegrift]

Hi everyone,

I'm adding API functionality to an exiting app whose controller
actions aren't close to being in shape for direct exposure. This
pretty much precludes me from turning them into REST controllers and
accomplishing this task that way.

So I've created a new api controller that will house the actions for
programmatic access. This has an unfortunate side effect of making my
routing kind of difficult.

For example, suppose I have model objects X and Y and the following API:

class ApiController(BaseController):
@jsonify
def getXbyid(self, id):
q = meta.session.query(model.X)
result = q.get(id)
return {"result": result.__json__()}

@jsonify
def getXbyacct(self, acct):
q = meta.session.query(model.X)
result = q.filter_by(acct=acct).one()
return {"result": result.__json__()}

@jsonify
def getYbyname(self, name):
q = meta.session.query(model.Y)
result = q.filter_by(name=name).one()
return {"result": result.__json__()}

How can I effectively map these actions without listing each action in
my routes? The best I have come up with is to include a route for
each parameter name I use:

map.connect("/api/{action}/{id:[0-9]+}", controller="api")
map.connect("/api/{action}/{acct:[0-9]+}", controller="api")
map.connect("/api/{action}/{name:[a-zA-Z0-9+}", controller="api")

--
Ross Vandegrift
ro...@kallisti.us

"If the fight gets hot, the songs get hotter. If the going gets tough,
the songs get tougher."
--Woody Guthrie

[Mike Orr]

The answer is to use a generic 'id' variable in the route, and to
rename the variable in the action. This is the philosophy behind the
default "/{controller}/{action}/{id}" route.

map.connect("/api/{action}/{id}", controller="api")

#### Actions
def by_id(self, id):
id = self._get_int_id(id)
...

def by_acct(self, id):
acct = self._get_int_id(id)
...

def by_name(self, id):
name = self._get_alphanumeric_id(id)
...

#### Private methods, maybe in base controller
def _get_int_id(self, id, errmsg="Invalid numeric ID."):
try:
return int(id)
except ValueError:
abort(404, errmsg)

NAME_RX = re.compile(R"^[a-zA-Z0-9]+$")

def _get_alphanumeric_id(self, id, errmsg="ID may contain only letters
and digits."):
if not NAME_RX.match(id):
abort(404, errmsg)
return id

--
Mike Orr <slugg...@gmail.com>

[Ross Vandegrift]

On Mon, Nov 23, 2009 at 11:59:03AM -0800, Mike Orr wrote:
> The answer is to use a generic 'id' variable in the route, and to
> rename the variable in the action. This is the philosophy behind the
> default "/{controller}/{action}/{id}" route.
>
> map.connect("/api/{action}/{id}", controller="api")

I thought about doing something like this, but this will break
documentation generation: the parameters will all be generated in the
docs as "id". While I might be able to get over that, I have to
publish this API to programmers that will (fairly) get confused when
they see a funtion that takes an id, but the documentation claims it's
something like a name.

I guess I may have to live with a bunch of routes. That's not really
the end of the world, since I can write things like:

map.connect(r"/api/{action:get.*byid}/{id:[0-9]+}", controller="api")
map.connect(r"/api/{action:get.*byacct}/{acct:[0-9]+}", controller="api")
map.connect(r"/api/{action:get.*bystr}/{searchstr}", controller="api")

Ross

[Wyatt Baldwin]

On Nov 23, 12:06 pm, Ross Vandegrift <r...@kallisti.us> wrote:
> On Mon, Nov 23, 2009 at 11:59:03AM -0800, Mike Orr wrote:
> > The answer is to use a generic 'id' variable in the route, and to
> > rename the variable in the action.  This is the philosophy behind the
> > default "/{controller}/{action}/{id}" route.
>
> > map.connect("/api/{action}/{id}", controller="api")
>
> I thought about doing something like this, but this will break
> documentation generation: the parameters will all be generated in the
> docs as "id".  While I might be able to get over that, I have to
> publish this API to programmers that will (fairly) get confused when
> they see a funtion that takes an id, but the documentation claims it's
> something like a name.
>
> I guess I may have to live with a bunch of routes.  That's not really
> the end of the world, since I can write things like:
>
>     map.connect(r"/api/{action:get.*byid}/{id:[0-9]+}", controller="api")
>     map.connect(r"/api/{action:get.*byacct}/{acct:[0-9]+}", controller="api")
>     map.connect(r"/api/{action:get.*bystr}/{searchstr}", controller="api")

This is only tangentially related to your question, but if you are
going for a REST-esque API, I would design the URLs more like this:

1) /api/{resource_name}/{id}

2) /api/{resource_name}/{access_type}/{id}
/api/{resource_name}/{access_type}/{name}

The first type of URL would use the default ID type, perhaps the
numeric ID. The second type of URL would be something like /api/pants/
by_name/{name}. In either case, you'd route to an action based on the
HTTP verb. All of the by_* stuff would really just be different ways
to get to your `show` action, which could have args like
`access_type='id', id=None, name=None`. Of course, your docstring
would have to note that only *one* ID parameter can be passed.

Personally, I would do what Mike O. suggested, perhaps changing the ID
name from `id` to something that's a bit less overloaded and that
indicates different types might be passed. I think even `identifier`
might be better, though it's essentially the same thing--I just think
`id` in particular is often (usually?) automatically translated to
"numeric ID" or "primary key".

Anyway, just a thought or two about API design. Feel free to
disregard. :)

[Mike Orr]

On Mon, Nov 23, 2009 at 12:06 PM, Ross Vandegrift <ro...@kallisti.us> wrote:
> On Mon, Nov 23, 2009 at 11:59:03AM -0800, Mike Orr wrote:
>> The answer is to use a generic 'id' variable in the route, and to
>> rename the variable in the action.  This is the philosophy behind the
>> default "/{controller}/{action}/{id}" route.
>>
>> map.connect("/api/{action}/{id}", controller="api")
>
> I thought about doing something like this, but this will break
> documentation generation: the parameters will all be generated in the
> docs as "id".  While I might be able to get over that, I have to
> publish this API to programmers that will (fairly) get confused when
> they see a funtion that takes an id, but the documentation claims it's
> something like a name.

Maybe. The action name implies the nature of the ID. And you have to
validate/convert the ID anyway, so 'id' is not necessarily the same as
'acct' or 'name'. Action methods by definition take routing variables
as-is, so ``.get_byacct(self, id)`` is not necessarily bad. You
should arguably have a separate set of business methods in the model
that take the real identifiers, to separate the business logic from
the HTTP UI (which is what the actions are).

--
Mike Orr <slugg...@gmail.com>

[Mike Burrows (asplake)]

>
> Maybe.  The action name implies the nature of the ID.  And you have to
> validate/convert the ID anyway, so 'id' is not necessarily the same as
> 'acct' or 'name'.  Action methods by definition take routing variables
> as-is, so ``.get_byacct(self, id)`` is not necessarily bad.  You
> should arguably have a separate set of business methods in the model
> that take the real identifiers, to separate the business logic from
> the HTTP UI (which is what the actions are).
>
> --

> Mike Orr <sluggos...@gmail.com>

That's all fine until you want to nest resources, and then you're
forced to rename at least one of the id's (conventionally the parent's
- at least in Rails) anyway. Why not use meaningful names
throughout? As the author of described_routes I'm interested in
machine readable representations of resource structure so I care more
than most about consistency, but what argument is there in favour of
"id" other than it requires no thought on the part of the developer?

Regards,

Mike
m...@asplake.co.uk
http://positiveincline.com
http://twitter.com/asplake

[Mike Orr]

By described_routes I guess you mean this:
http://github.com/asplake/described_routes/tree

It looks like a pretty printer for route definitions. Does it use a
standard Rails route map, or its own route objects? Does it create a
functional route map, or just its output formats? It says "framework
neutral", but then there must be some framework-specific code to
convert it to a native route map, no?

Pylons does not have a routemap pretty printer. That might be worth adding.

Pylons does not handle nested resources very well anyway, so this
wouldn't be the only problem. But I don't think that
having-to-rename-variables applies. The only problem with combining
routes from multiple sources is if there's a collision in the route
paths or route names, or if a route is over-general and swallows URLs
intended for a later route. But if you're combining nested resources,
the nested one would normally go under a unique URL prefix. If you
tried to overlay two sets of routes at the same URL position, you
might run into collisions, but I doubt many people do that.

As for unique route names, that's just something you have to have in
order to generate them by name. ``map.extend()`` has an argument to
insert a set of routes under a URL prefix. Perhaps it needs a name
prefix argument too.

The 'id' convention is based on the traditional generic route
"/{controller}/{action}/{id}" and ``map.resource()``. I thought both
of these were borrowed from Rails.

I normally do use specific variable names. But that's impossible when
you want to have a generic route that serves a variety of different
purposes. You can't have two routes whose paths differ only by a
variable name, because the variable name does not appear in the
incoming URL, so there's no way for Routes to determine which route is
wanted (and it would select the first matching route anyway).

--
Mike Orr <slugg...@gmail.com>

[Wyatt Baldwin]

On Nov 24, 9:44 am, Mike Orr <sluggos...@gmail.com> wrote:
>
> [...]

>
> I normally do use specific variable names.  But that's impossible when
> you want to have a generic route that serves a variety of different
> purposes.  You can't have two routes whose paths differ only by a
> variable name, because the variable name does not appear in the
> incoming URL, so there's no way for Routes to determine which route is
> wanted (and it would select the first matching route anyway).

That's a good point, which I missed above. Given the following routes,
the second one would never match (unless, of course, a regexp was
added to {id} and/or {name}).

[Mike Burrows (asplake)]

Apologies if this comes out as a (near) dupe - I was sure I had
replied to this!

On Nov 24, 5:44 pm, Mike Orr <sluggos...@gmail.com> wrote:

> By described_routes I guess you mean this:http://github.com/asplake/described_routes/tree

Yes, that's it

> It looks like a pretty printer for route definitions.  Does it use a
> standard Rails route map, or its own route objects?  Does it create a
> functional route map, or just its output formats?  It says "framework
> neutral", but then there must be some framework-specific code to
> convert it to a native route map, no?

Pretty printer, json/yaml/xml metadata generator and (via link headers
and a small Rack middleware component) the provider of a simple
resource discovery protocol. Unfortunately some of the later bits are
better described in my blog than they are in the package itself, so
apologies for that!

> Pylons does not have a routemap pretty printer. That might be worth adding.

I intend to, but it's the usual question of time. A Python port of
the underlying datastructure and some of the format conversions exist
already so it shouldn't be a huge piece of work.

> Pylons does not handle nested resources very well anyway, so this
> wouldn't be the only problem.  But I don't think that
> having-to-rename-variables applies.  The only problem with combining
> routes from multiple sources is if there's a collision in the route
> paths or route names, or if a route is over-general and swallows URLs
> intended for a later route.  But if you're combining nested resources,
> the nested one would normally go under a unique URL prefix.  If you
> tried to overlay two sets of routes at the same URL position, you
> might run into collisions, but I doubt many people do that.

In Rails, collisions are an issue already as each HTTP method
generates a route for each URL (in reality it's a route per action,
not per URL) so I'm used to them already.

One thing that does make me nervous about a Pylons implementation is
the multiplicity of options (no pun intended) around HTTP method
discrimination, namely route requirements, @restrict decorators and
(IIRC - not used them yet) @dispatch decorators. While we're on the
subject, it seems wrong to me that Routes will generate a 404 for a
correct URL but incorrect method. It's easy to guess why it does
this, but it still seems wrong!

> As for unique route names, that's just something you have to have in
> order to generate them by name.  ``map.extend()`` has an argument to
> insert a set of routes under a URL prefix.  Perhaps it needs a name
> prefix argument too.

You may be on to something here. In my work project we have taken to
namespacing route names, but I'm not 100% sure that we've got it right
yet.

> The 'id' convention is based on the traditional generic route
> "/{controller}/{action}/{id}" and ``map.resource()``.  I thought both
> of these were borrowed from Rails.

That's my belief too. I tend to remove that generic route unless I
have a very good reason to keep it though.

> I normally do use specific variable names.  But that's impossible when
> you want to have a generic route that serves a variety of different
> purposes.  You can't have two routes whose paths differ only by a
> variable name, because the variable name does not appear in the
> incoming URL, so there's no way for Routes to determine which route is
> wanted (and it would select the first matching route anyway).

Indeed (route requirements aside). Yes, in the generic case "id" is
as good a name as any other.

[Mike Orr]

On Tue, Nov 24, 2009 at 10:47 AM, Mike Burrows (asplake)
<m...@asplake.co.uk> wrote:
>
> Yes it uses standard Rails routes, and no it doesn't address output
> formats (nor will it).

I'm not sure what this means. XML, JSON, YAML, and text are all
output formats, which it does.

Yes there's a small module to map the Rails
> routes to described_routes' internal model and another to add Rake
> tasks (analagous to adding "paste described_routes" or something).

What would a "paster described_routes" do? Add a bunch of routes and
their various controllers?

I think I would rather see a case where the original route definitions
were in a generic format (and described_routes may be as good a format
as any), and a ``map.import_()`` method that imports them into the
route map. That way the routes are defined in only one place, and the
application is automatically updated (or broken) whenever routes
change. "paster described_routes" could create the controllers, but if
it inserts into the route map you'd have problems when you change the
routes later: because how would you keep routing.py and the original
route definitions in sync?

>> Pylons does not have a routemap pretty printer. That might be worth adding.
>

> I do plan to do this, just a question of time - maybe my next plane
> trip!  A Python version of the underlying data structure exists
> already so it shouldn't be a huge amount of work.

That would be great. The data structures in Mapper and Route are
rather opaque and underdocumented though. We're considering a more
transparent structure for a future version of Routes.

> On a related subject, I'm a little concerned that Pylons has the
> choice of constraining methods via Routes (potentially resulting in
> 404s which seems wrong to me) or via the @restrict decorator.  IIRC
> there's the third choice of a @dispatch decorator but I haven't tried
> that (maybe I should - I'm currently using different URLs for my POST
> actions).

Yes, there are three ways to do it. I don't know when each one was
added to Pylons or why, but I suspect it's because people wanted
TurboGears-style decorators. I'am not sure whether restricting the
methods via Routes or the decorators is better, although I lean toward
Routes to keep it all in one place. The security-obsessed would use
both. : )

--
Mike Orr <slugg...@gmail.com>

[Mike Burrows (asplake)]

On Nov 24, 7:39 pm, Mike Orr <sluggos...@gmail.com> wrote:
> On Tue, Nov 24, 2009 at 10:47 AM, Mike Burrows (asplake)

> > Yes it uses standard Rails routes, and no it doesn't address output
> > formats (nor will it).
>
> I'm not sure what this means.  XML, JSON, YAML, and text are all
> output formats, which it does.

Ah - I just meant that I have no intention to model the formats
produced by the application.

> Yes there's a small module to map the Rails
>
> > routes to described_routes' internal model and another to add Rake
> > tasks (analagous to adding "paste described_routes" or something).
>
> What would a "paster described_routes" do?  Add a bunch of routes and
> their various controllers?

Probably just produce a human-readable report, like "rake routes" does
for vanilla Rails and "rake described_routes" does but better ;-)

> I think I would rather see a case where the original route definitions
> were in a generic format (and described_routes may be as good a format
> as any), and a ``map.import_()`` method that imports them into the
> route map.  That way the routes are defined in only one place, and the
> application is automatically updated (or broken) whenever routes
> change. "paster described_routes" could create the controllers, but if
> it inserts into the route map you'd have problems when you change the
> routes later: because how would you keep routing.py and the original
> route definitions in sync?

See above. Described_routes works with the existing routing so
there's still only one representation to maintain, and I wouldn't wish
to limit frameworks to what described_routes can express so I don't
push it as an input format.

> >> Pylons does not have a routemap pretty printer. That might be worth adding.
>
> > I do plan to do this, just a question of time - maybe my next plane
> > trip!  A Python version of the underlying data structure exists
> > already so it shouldn't be a huge amount of work.
>
> That would be great.  The data structures in Mapper and Route are
> rather opaque and underdocumented though. We're considering a more
> transparent structure for a future version of Routes.

From a brief encounter in the paster shell it's an easy enough
structure to explore. Rail's documentation here isn't great either.

> > On a related subject, I'm a little concerned that Pylons has the
> > choice of constraining methods via Routes (potentially resulting in
> > 404s which seems wrong to me) or via the @restrict decorator.  IIRC
> > there's the third choice of a @dispatch decorator but I haven't tried
> > that (maybe I should - I'm currently using different URLs for my POST
> > actions).
>
> Yes, there are three ways to do it.  I don't know when each one was
> added to Pylons or why, but I suspect it's because people wanted
> TurboGears-style decorators.  I'am not sure whether restricting the
> methods via Routes or the decorators is better, although I lean toward
> Routes to keep it all in one place.  The security-obsessed would use
> both. : )

You ok with the 404s then?

> --
> Mike Orr <sluggos...@gmail.com>

Regards,
Mike

[Wyatt Baldwin]

On Nov 24, 11:39 am, Mike Orr <sluggos...@gmail.com> wrote:
>
> [...]
>

> That would be great.  The data structures in Mapper and Route are
> rather opaque and underdocumented though. We're considering a more
> transparent structure for a future version of Routes.

Sign me up for that, at least as an interested party. I'm responsible
for the current `parent_resource` business, and I've always felt that
it was somewhat of a hack, and it only works for a limited set of use
cases.

I've had some ideas about the resource side of things, like creating a
Resource class with a parent attribute, etc (though I'm not saying
that's necessarily the best approach). I wish that I could take more
up front initiative on this, but I can at least offer some assistance
once things get rolling.

[Mike Orr]

On Tue, Nov 24, 2009 at 1:10 PM, Mike Burrows (asplake)
<m...@asplake.co.uk> wrote:
>
>
> On Nov 24, 7:39 pm, Mike Orr <sluggos...@gmail.com> wrote:
>> On Tue, Nov 24, 2009 at 10:47 AM, Mike Burrows (asplake)

>> > On a related subject, I'm a little concerned that Pylons has the
>> > choice of constraining methods via Routes (potentially resulting in
>> > 404s which seems wrong to me) or via the @restrict decorator. IIRC
>> > there's the third choice of a @dispatch decorator but I haven't tried
>> > that (maybe I should - I'm currently using different URLs for my POST
>> > actions).
>>
>> Yes, there are three ways to do it. I don't know when each one was
>> added to Pylons or why, but I suspect it's because people wanted
>> TurboGears-style decorators. I'am not sure whether restricting the
>> methods via Routes or the decorators is better, although I lean toward
>> Routes to keep it all in one place. The security-obsessed would use
>> both. : )
>
> You ok with the 404s then?

It seems like we should just fix Routes then.

http://bitbucket.org/bbangert/routes/issue/18/match-failure-based-on-http-method-should-return-405

>> > Yes it uses standard Rails routes, and no it doesn't address output
>> > formats (nor will it).
>>
>> I'm not sure what this means.  XML, JSON, YAML, and text are all
>> output formats, which it does.
>
> Ah - I just meant that I have no intention to model the formats
> produced by the application.

Ah, of course not. It can't keep up with every framework's routing convention.

So what do people use their JSON, XML, YAML routemaps for?
Just for display?

I also added a couple tickets for a routemap pretty printer and
"paster describe_routes".

http://bitbucket.org/bbangert/routes/issue/16/routemap-pretty-printer

http://pylonshq.com/project/pylonshq/ticket/661




--
Mike Orr <slugg...@gmail.com>

[Mike Orr]

The code looks like it's still on knowledgetap.

https://www.knowledgetap.com/hg/routes2-dev

This is the basic data structure, but it hasn't been touched for 2
years, and in the meantime some of its features have been implemented
in Routes differently.

I plan to make a ``map.resource2()`` method that implements
view/add/modify/delete only with GET/POST, and using the same URL and
method for the form and action. This would not be friendly to
non-human user agents that expect REST, but not that many applications
will ever be used by non-human agents. I would also have
add/modify/delete flags to tell whether these operations should be
implemented. Some resources are never added or never deleted over the
web.

--
Mike Orr <slugg...@gmail.com>

[Mike Burrows (asplake)]

On Nov 24, 11:09 pm, Mike Orr <sluggos...@gmail.com> wrote:

> It seems like we should just fix Routes then.

Cool. I'd have raised a ticket myself but I'm new round here ;-)

> So what do people use their JSON, XML, YAML routemaps for?
> Just for display?

I wrote a client too ("path-to") which uses the metadata to navigate
the server's model with expressions like app.users["mike"].posts
["hello-world"].comments or app.blog_comments["mike", "hello-world]
which I needed to script a real-world application (a Java-based
trading app). I'm aware of a couple of other projects using it, one
to pull together a distributed app, another which consumes the json
format in the browser.

The path-to client thing has been done before I know, but previous
attempts (to my knowledge) have tended to rely on routing conventions
being followed strictly by the server and therefore (to my mind)
rather fragile. I would be making more noise about both
described_routes and path-to were it not for the frustrating wait for
a significant rewrite of the URI Template specification.

> I also added a couple tickets for a routemap pretty printer and
> "paster describe_routes".

Cool

> --
> Mike Orr <sluggos...@gmail.com>

Regards,
Mike