[ANN] rack-router - routing for the rest of us (feedback requested)

[Carl Lerche]

Hey guys,

I've been working on extracting Merb's router into a rack middleware
library as well as adding a bunch of features to try to get "mountable
apps" working in Merb 1.1. I've been working with Josh Peek on routing
in general so that we can get the best solution done for Rails 3.
Quite a lot of the code and concepts are drawn from his rack-mount
project.

This is a proof of concept that I am submitting to the mailing list
for feedback. All the code is at:

http://github.com/carllerche/rack-router

Abstract
========

Conceptually, rack-router allows you to create a two way map between
HTTP requests and Rack applications. It is built as a piece of
middleware that takes in a set of routes. When a request comes in, the
router will compare that request against the set of routes until it
finds one that matches. It then calls the associated rack app. It can
also generate URL's that you can use to link to other mountable apps.

It also goes quite a bit further and attempts to make reusing rack
applications completely painless (what we are tentatively calling
"mountable apps").

Disclaimer
==========

rack-router is not for most rack middleware and small applications. It
is for any rack based application that needs to generate URIs or allow
other applications to generate URIs that point to it

The Problem
===========

There really are quite a number of problems that we are trying to
solve here.

* Be able to take any Rack application, and have it behave the same no
matter which HTTP space it lives in.

For example, I build a blog rack application. I would like to take
this blog and use it in one rails app and have it live at /blog and
then I would like to add it to a Sinatra application and have it live
at /news. Obviously, the blog application has a set of routes that it
needs to work. I want the routes to work the same no matter which path
prefix it lives in. Conversely, I need to be able to generate links in
the blog application that will be correct no matter where the blog
lives.

* Be able to create dependencies between Rack applications.

Continuing with the blog example, the actual blog gem could expose
multiple rack applications that can be mounted at different spots. For
example, my blog application exposes two rack applications. BlogSite
and BlogAdmin. The first is the actual public facing blog and the
second is the admin section. My end application requirements are that
the admin section is at http://admin.example.org and that the main
site is at http://example.org. The BlogAdmin has to be able to say "I
need the actual blog to exist, but I don't know where it will live".
Somehow, when the BlogAdmin is generating URLs that point to BlogSite,
they have to take into account where BlogSite ends up being mounted.

Another example of this is cloudkit. I've been talking a lot with Jon
Crosby about this router library. Currently in cloudkit, he has an
oauth middleware that cloudkit depends on that he has to vendor and
cannot release as a separate gem because he assumes that all the oauth
routes exist at /oauth. Ideally, cloudkit would be able to say, "I
depend on Rack::OAuth", and then just call a method to generate an
OAuth route and it will just work.

* Be able to specify interfaces that can be implemented differently by
different libraries

Most applications depends on an authentication strategy that
includes :login, :logout, and :signup, but there could be many
different ways to implement this (aka, it isn't necessarily a single
library that is the end all solution). I would like there to be a way
to define an "authentication interface" that requires there to exist a
certain set of routes (say, :login, :logout, :signup) so that, if I
build a blog rack app, I can just say "generate the login route here"
and it just works. This would also be a good thing for OAuth, OpenID,
etc... apps.

* NO MATTER WHAT, the mounted app should not have to worry about the
fact that is mounted or not

* The solution should be Rack specific, and not depend on any rack
compatible framework

Proposed Solution
=================

The rack-router library solves these problems. Besides basic routing
that supports all of Rails' and Merb's routing features (example 1),
it allows you to mount any rack application and isolate it in a URI
(or arbitrary value of the request) space. After a route matches and
before calling the associated Rack application, the router will update
PATH_INFO and SCRIPT_NAME to reflect the mounted location. Any
captures specified in the route will be extracted to a hash that is
available at env['rack_router.params'] (example 2). It will also keep
track of where each router is mounted and provide a url(route_name,
options = {}) method that will generate a URL that matches the route
(example 3).

+ Example 1: Basic Routing

use Rack::Router do |route|
route.map "/users", :to => Users
route.map "/posts", :to => Posts, :name => :posts
end

my_app.url(:posts) # => /posts

Here, my_app is an instance of the Rack::Router and it provides a
url method that generates the routes.

-----

+ Example 2: Accessing captures in the rack app

-----

+ Example 3: Mounting routers

# Blog application
blog = Rack::Router do |route|
route.map "/posts", :to => Posts, :name => :posts
end

# Main application
use Rack::Router do |route|
route.map "/users", :to => Users
route.map "/blog", :to => blog, :name => :blog
end

blog.url(:posts) # => /blog/posts

# It also provides access to child routers
my_app.blog.url(:posts) # => /blog/posts

The point here is that both request recognition and URL generation
are abstracted so that the rack application does not need to worry
about where it lives.

-----

Besides this, the Rack::Router supports the ability to lazily provide
access to rack application dependencies so that URLs can be generated
to them correctly (example 4)


+ Example 4: Application dependencies

module Blog
class Admin
include Rack::Router::Routable
def initialize
prepare :dependencies => { Site => :blog } do |route|
route.map "/posts", :to => Posts, :name => :posts
end
end
end

class Site
include Rack::Router::Routable
def initialize
prepare do |route|
route.map "/posts", :to => Posts, :name => :posts
end
end
end
end

admin = Blog::Admin.new
blog = Blog::Site.new

use Rack::Router do |route|
route.map "/admin", :to => admin
route.map "/blog", :to => blog
end

admin.blog.url(:posts) # => /blog/posts
my_app.admin.url(:posts) # => /admin/posts

So, the Blog::Admin app requires Blog::Site, once it is mounted, the
router has access to it so the application can generate correct URLs.

Other Features
==============

The actual DSL for defining the route is completely pluggable. The DSL
I used above is a pretty simple and low level DSL that I implemented
mostly to write specs. A router that was built using Rails' DSL can
work with one using Merb's. Sinatra would even be able to use this
quite easily.

Rack::Router

What now?
=========

This is still just a proof of concept. I coded it to make it as
readable as possible and did not worry about speed at all. I would
like to get as much feedback as possible before worrying about speed.
Josh Peek has quite a good number of optimizations that can easily
apply to the code base.

So, tell me what you think, does solve problems, would you use this?

Currently, rack-router only generates paths, but I am planning on
adding support for generating full URIs soon as well as getting
information on the HTTP method required for the route.

Outstanding Concerns
====================

Currently, the ability to generate routes doesn't seem as abstracted
as it could be. There is no concept of generating routes currently in
the rack spec, so it's been implemented somewhat arbitrarily. Josh had
an idea of some kind of interface that any rack application could
implement to have it compatible with generating routes given an
arbitrary mount point.

Also, the API for generating mounted app routes (aka, my_app.blog.url
(:posts)) feels slightly strange. My goal is to keep the routes fully
namespaced for each application. I don't want to namespace it with a
symbol name prefix (aka, :blog_posts).

As of now, the problem of "Being able to specify routing interfaces"
is not really handled too elegantly. Ideas for this particular problem
are appreciated.

I also have a bunch of notes in the README. I'm still unsure as to the
best implementation for all this, so please pipe up.

[Scytrin dai Kinthra]

I like the aim and examples of uses. My only worry is just how big and
complex this lib would get.
I've been wiring various routing methods of my own, and while I'm not
a huge fan of large libs if the runtime speed of this was lightweight
it'd be awesome.

--
stadik.net

[Yehuda Katz]

Re: lightweight: At the moment, the codebase is not really optimized, but the intention is to be able to apply the same optimizations as Merb uses, as well as some additional ones Josh came up with, to make it speedy and small. That way, the codebase itself would still be fully operational and comprehensible without the optimizations, but the additional optimizations would make it usable for virtually all cases. For very small route-sets, this router should be as fast as simple URLMap or manual routing once the optimizatios are applied.

For large route sets, it should be faster than the existing routers (Rails/Merb/Sinatra).

-- Yehuda

--
Yehuda Katz
Developer | Engine Yard
(ph) 718.877.1325

[Carl Lerche]

As Yehuda said, the current code base is only to illustrate concepts
and clean code. Also, this library is only for people who need
routing. If you don't need routing, there is no reason to use it. I
have spent a lot of time benchmarking and optimizing the merb router.
Josh has been doing much speed work as well. I am quite confident that
in a very short time, rack-router will be faster than anything anybody
else will be able to come up with in their own rack app to handle
routing. So, in short, if you don't need the feature set, don't use
it. If you do need the feature set, rack-router will be very fast.

Carl Lerche
Engine Yard

> > the admin section is athttp://admin.example.organd that the main
> > site is athttp://example.org. The BlogAdmin has to be able to say "I

[Carl Byström]

First of all, great work! Love the way Rack is heading :)

I have a, perhaps naive, question regarding app/controller mapping.
When doing routing to a Rails 2.3+ controller (which is a Rack app),
how are routes resolved to different controller actions?
I understand that this is Rack-land and such framework specific things
should be handled downstream. But does this imply you need another
framework specific router in addition to rack-router?

> the admin section is athttp://admin.example.organd that the main
> site is athttp://example.org. The BlogAdmin has to be able to say "I

[Yehuda Katz]

We've been talking about making the action_name be the PATH_INFO once Rails gets it. Rails controllers would need a "router", but the router would be limited to send(env["PATH_INFO"]). Sinatra, on the other hand, would want to do something meaningful with the remaining PATH_INFO, and would use rack-router again. I believe Carl posted an example to illustrate this.

-- Yehuda

2009/4/11 Carl Byström <cgby...@gmail.com>

[Carl Byström]

Ok, cool. I guess it boils down to that in the controller's "router".
But if everything is rack specific in rack-router, how/where does the
notion of Rails come in?
If passing the action through env["PATH_INFO"], the router must do
this env preparation before passing on the request.
Is the plan to offer extension points to the rack-router to allow
framework specific routing (in this Rails controller routing/
mapping) ?

Sorry, must have missed Carl's example about that. Could you/someone
give me a pointer?

__
carl

On Apr 11, 9:31 pm, Yehuda Katz <wyc...@gmail.com> wrote:
> We've been talking about making the action_name be the PATH_INFO once Rails
> gets it. Rails controllers would need a "router", but the router would be
> limited to send(env["PATH_INFO"]). Sinatra, on the other hand, would want to
> do something meaningful with the remaining PATH_INFO, and would use
> rack-router again. I believe Carl posted an example to illustrate this.
>
> -- Yehuda
>

> 2009/4/11 Carl Byström <cgbyst...@gmail.com>

> > > the admin section is athttp://admin.example.organdthat the main

> ...
>
> read more »

[Joshua Peek]

On Apr 11, 2009, at 2:31 PM, Yehuda Katz wrote:

> We've been talking about making the action_name be the PATH_INFO
> once Rails gets it. Rails controllers would need a "router", but the
> router would be limited to send(env["PATH_INFO"]). Sinatra, on the
> other hand, would want to do something meaningful with the remaining
> PATH_INFO, and would use rack-router again. I believe Carl posted an
> example to illustrate this.

Rails just pulls the action key from the extracted routing params.
This is what 2.3 already does, it works fine, lets not try to engineer
a new system here. Many of these conventions have already been set out
and proven in Rails.

@controller.send(env["rack.routing_args"][:action]).

*Note this is Rails specific*. There doesn't need to be any sort of
general Rack convention for this.

--
Joshua Peek

[Carl Byström]

On Sat, Apr 11, 2009 at 10:39 PM, Joshua Peek <jo...@joshpeek.com> wrote:

 @controller.send(env["rack.routing_args"][:action]).

Ok, but if you want to map URL directly to an action. How would such a declaration look?
Like this?
route.map "/signup", :to => UserRegistration, :action => "register"

I'm a bit confused when it comes to the strict separation  rack-router <-> framework router.
Perhaps rack-router isn't supposed to solve this problems? I'm just curious and jumping into the discussion here. So bare with me if I'm being too naive :)

__
carl

[Joshua Peek]

2009/4/11 Carl Byström <cgby...@gmail.com>:

> Ok, but if you want to map URL directly to an action. How would such a
> declaration look?
> Like this?
> route.map "/signup", :to => UserRegistration, :action => "register"
> I'm a bit confused when it comes to the strict separation  rack-router <->
> framework router.
> Perhaps rack-router isn't supposed to solve this problems? I'm just curious
> and jumping into the discussion here. So bare with me if I'm being too naive
> :)

The API mapper could look however, you want. But lets take the
existing Rails api for an example.

map.connect "/signup", :controller => "user", :action => "register"

Controller actions are outside the scope of Rack. The only thing the
actual router does, is match "/signup" urls, add the defaults to the
env { :controller => "user", :action => "register" }, and call some
app. In this case Rails would tell the router the app is the
UserController when you are defining your routes. UserController.call
would receive the request and internally route to the correct action.

The only thing I think we should agree on is where to stuff the
routing params. This would be a convention/recommendation, not
required. In WSGI, they have already settled on "routing_args"
(http://www.wsgi.org/wsgi/Specifications/routing_args). So it would be
env["rack.routing_args"] for us. This has already be added to Rails
2.3 and shipped. Its pretty simple and the actual name isn't to
important because its mainly for frameworks.

--
Joshua Peek

[Carl Byström]

On Sat, Apr 11, 2009 at 11:42 PM, Joshua Peek <jo...@joshpeek.com> wrote:

The only thing I think we should agree on is where to stuff the
routing params. This would be a convention/recommendation, not
required. In WSGI, they have already settled on "routing_args"
(http://www.wsgi.org/wsgi/Specifications/routing_args). So it would be
env["rack.routing_args"] for us. This has already be added to Rails
2.3 and shipped. Its pretty simple and the actual name isn't to
important because its mainly for frameworks.

Yep, I agree. There's no reason to change this.
You answered my question, passing all meta data unchanged in a hash makes sense.

[Carl Lerche]

Re: Controller action dispatching. Using PATH_INFO to indicate to the
controller which action to dispatch is simply following the rack spec.
Quoting from the actual spec: "The remainder of the request URL‘s
"path", designating the virtual "location" of the request‘s target
within the application."... for the controller, the virtual "location"
of the request's target within the application IS the action.

Also, I strongly disagree with using rack.routing_args. First, the
router is not going to become part of rack core, so it should not use
the rack namespace (this is according to the SPEC "The prefix rack. is
reserved for use with the Rack core distribution and must not be used
otherwise."). Rails should fix this discrepancy. As for routing_args,
I'm not to fond of the name as it seems to me that the captures
extracted from the route are not involved in the actual routing, but
I'm pretty much open to go with whatever.

Maybe just do env['router.args']

Thoughts?
Carl Lerche
Engine Yard


On Apr 11, 2:42 pm, Joshua Peek <j...@joshpeek.com> wrote:
> 2009/4/11 Carl Byström <cgbyst...@gmail.com>:

[Diogo Terror]

As for feedback, I think this is absolutely awesome. I also believe
the power of this thing is unprecedent in any web open source
community. Django has long been famous for its reusable apps, but
they're django's alone. This middleware is about to be supported by
all major Ruby web frameworks that can now share applications. This is
the closest from the 'UNIX pipes' ideal it can get.

Kudos to all of you.

[Carl Lerche]

Alright, I did a bunch of performance work. The recognition
improvements for large route sets are basically all Josh Peek's (gotta
admit, they were pretty awesome). I also improved route generation to
basically reduce to a single string interpolation. As of now, it's
about 2x faster than Merb's route generation (which is already quite
fast). More recognition performance could squeezed out, but it's fast
enough for now.

Carl Lerche
Software Engineer
Engine Yard