Best practices for splitting tornado application into sub-modules

[Russ Ryba]

Greetings everyone,

Does anyone have any examples or best practices for breaking a "site"
into a collection of application from different directories similar to
django urlpatterns?

I'm thinking it would be something like django
settings.INSTAPPED_APPS and url_patterns.


EXAMPLE:

Assume /tornado_site/blog/blog.py sets up the handlers as follows.

application.handlers = [
(r"/", BlogHomeHandler),
(r"/post", NewPostHandler),
(r"/add_comment", CommentHandler)
]

Then inside /tornado_site/start_tornado.py there is something like...

from blog import blog
...
site_app = tornado.web.application( [
(r"/", SiteHomeHandler),
(r"/blog/", blog.Application)
] )
server = tornado.httpserver.HTTPServer(site_app)



And the result is the same as following tornado url mappings...

site_app = tornado.web.application( [
(r"/", SiteHomeHandler),
(r"/blog/", blog.BlogHomeHandler),
(r"/blog/post", blog.NewPostHandler),
(r"/add_comment", CommentHandler)
] )
server = tornado.httpserver.HTTPServer(site_app)

I've done some testing and I know what I proposed doesn't exist yet.

I think it should be implemented by adding a "add_application"
function to tornado.web.Application that would combine the url mapping
and handlers from another class that extracts the url mappings and
adds a prefix. Application.add_handlers would be modified something
like this...

def add_handlers(self, host_pattern, host_handlers):
"""Appends the given handlers to our handler list."""
if not host_pattern.endswith("$"):
host_pattern += "$"
handlers = []
self.handlers.append((re.compile(host_pattern), handlers))

for handler_tuple in host_handlers:
assert len(handler_tuple) in (2, 3)
pattern = handler_tuple[0]
handler = handler_tuple[1]
if len(handler_tuple) == 3:
kwargs = handler_tuple[2]
else:
kwargs = {}

# Add ability to import whole applications
if type(handler) == 'Application':
prefix_pattern = pattern
self.add_application( prefix_pattern, handler )

if not pattern.endswith("$"):
pattern += "$"
handlers.append((re.compile(pattern), handler, kwargs))

Then add_application would have the task of examining the application
and combining the url mapping handlers and with the appropriate
prefix. It would appear to be very simple with the only complication
being that the Application.__init__ function changes the text based
url mappings and stores the result of re.compile instead.

Has anyone got anything like this working yet?

Does anyone with more experience hacking the source have any
alternatives they could suggest which might achieve the same goal?

Thank you,

Russ Ryba

[Andrew Gwozdziewycz]

I think that a good solution might be even simpler. (this is untested)

Each sub application creates an entry point, say urls.py, which
exports one list variable called handlers:

i.e.

tornado_site/blog/urls.py

from .blog import HomeHandler
<snip>
handlers = [
('/blog/', HomeHandler),
<snip>
]

Then, in tornado_site/start_tornado.py

apps = ['blog', ....]

all_handlers = []
for app in apps:
# import the app, and attempt to find handlers
handlers = find_handlers_in_app(app)
if handlers:
all_handlers.extend(handlers)

app = tornado.web.Application(all_handlers)

Obviously, the all handlers business can be abstracted out of
start_tornado.py, but you get the idea.

The major point is that I don't think there's any reason to create sub
applications within Application--what does that really buy you?

Best,

Andrew

--
http://www.apgwoz.com

[Kuze]

I asked about this a few weeks ago as well but looks like a roll your
own.

Check out my previous post for an example, granted not tornado
specific but should give you some ideas.
http://groups.google.com/group/python-tornado/browse_thread/thread/4c7791d82309c61f/d21d87dfd97d2f0b

Another way you could do this:
1) Create a __init__.py in a specific dir i.e. "/app/modules" which
also houses all your sub applications in their own directory i.e. "/
app/modules/blog".
2) Place your urls.py file in the blogs dir with a patterns list with
all your mapping for blog.
3) In the __init__.py comb over "/app/modules/*" for any urls.py and
do dynamic import module call to grab each patterns list, merge them
into one for use in your root application.handlers i.e:
import app.modules
handlers += app.modules.merged_patterns_list

[Thomas Rampelberg]

This is basically what I'm doing for an app right now. While I'm not
doing "sub-modules", I like having the routes next to the modules that
define them. Obviously, there are trade offs either way, but I like
not having to change something in two different locations and like to
think that my package/module structure is broken up by functionality
well enough that I can go and find that troublesome route pretty
easily.

~Thomas

[Russ Ryba]

On Oct 14, 5:11 pm, Andrew Gwozdziewycz <apg...@gmail.com> wrote:

Hi Andrew,

I've been working on this all day and more or less ended up with the
same idea. I have something like this working. It avoids the problem
of applications storing re.compile output. It also helps out with
"magic" that happens with /static being included that multiple times
when trying to extract the handlers from instantiated applications
instances.

This is also what django uses with their url_patterns - they've had a
lot of time to work it out. I took the django concept and extended it
a bit to specify not just the prefix and module name, but also the
name of the url_handler_list to use. My original goal was to allow
smaller applications to run as stand alone apps for development but it
grew into this more flexible solution. Now I've got a file named
"urls.py" and it has several lists of handlers. Then in the primary
application I define which list name to use.

I really like this solution because I can just define the name of the
url map to use and switch an individual application from debug to
production mode by changing the url handlers list in the primary
file. My solution looks like this...

Here is my latest naming convention:
/tornado_site/module/urls.py

# this is urls.py
production_urls = [
(r"/", ProductionHomeHandler ),
(r"/add", ProductionAddHandler ),
(r"/del", ProductionDeleteHandler ),
]
debug_urls = [
(r"/", DebugHomeHandler ),
(r"/add", DebugAddHandler ),
(r"/del", DebugDeleteHandler ),
(r"/debug", DebugHandler )
]

Then back in /tornado_site/start_tornado_web.py
# Define sub_module url mappings
app_urls = [
(r"/prefixA", 'module_name', 'handler_list_name' ),
(r"/prefixB", 'module_name', 'handler_list_name' ),
(r"/prefixC", 'module_name', 'handler_list_name' ),
(r"/blog", 'blog_demo', 'deubg_urls' ),
]
...

I really like this solution, it gives the following benefits
- not trying to extract urls from re.compile output of instantiated
objects
- there is only one application object at a time
- switching application urls and handlers by simply importing a
different url_list
- can automatically disable debug mode when it detects it's not
running on development servers
- no need to modify tornado.web.py at all

I don't think this is anything novel. It seems the most obvious
solution after only a few hours of trying different ideas.

Regards,
Russ

[Andrew Gwozdziewycz]

I do like the idea of 'handler_list_name' in this situation.

> I really like this solution, it gives the following benefits
> - not trying to extract urls from re.compile output of instantiated
> objects
> - there is only one application object at a time
> - switching application urls and handlers by simply importing a
> different url_list
> - can automatically disable debug mode when it detects it's not
> running on development servers
> - no need to modify tornado.web.py at all
>

I'm not sure I follow this, how would you not have to modify web.py to
accept the 3rd argument? The 3rd argument for URL handlers is normally a
dictionary to use as keyword arguments in the Handler's constructor. Can
you post the code for this? I'd like to look at it, and others may as well.

> I don't think this is anything novel. It seems the most obvious
> solution after only a few hours of trying different ideas.
>
>

I like simple. Actually, I really like the way tornado handles this
right now, but I do foresee issues down the road when I develop other
applications, so finding a solution--whether it be simple or more
complicated--will be good.

Andrew

[Russ Ryba]

I have a few meetings but I'll post what I have later today or
tomorrow.

[Russ Ryba]

> .......

>
> I'm not sure I follow this, how would you not have to modify web.py to
> accept the 3rd argument? The 3rd argument for URL handlers is normally a
> dictionary to use as keyword arguments in the Handler's constructor. Can
> you post the code for this? I'd like to look at it, and others may as well.
>

> .........
> Andrew

The key concept is that a new piece which I'm working on runes before
the server is launched. That piece reads the three part tuple array,
collects the lists of handlers from the sub modules, combines them
into the normal two part tuple array and then feeds that to the
application via add_handlers.

It's a layer on top of tornado - not a part of it.

Code will show you. It's not ready yet.

Russ

[Russ Ryba]

OK. Sorry I don't have GIT or anything setup. I've got a busy
weekend but trying to put out the code I promised. It works, it's in
production. It makes my life easier. If there something I've failed
to get across let me know and I'll try to add it here.

Two tips before you begin
1. Make sure you have "template_path" defined in your application
settings or the system will look for templates in the 'template'
subdirectory of the module where the handler exists.
2. The prefix is added using simple string concatenation. Prefix /
auth/ and url "/login" would result in /auth//login.
Assumed you're smart enough to build your lists right rather than
detecting double slashes


# start_tornado_site.py
import tornado.web
import tornado.ioloop
import tornado.web

class MainHandler(tornado.web.RequestHandler):
def get(self):
self.render('index.html')

default_urls = [
(r"/", MainHandler),
]
module_url_lists = [
(r'/auth', 'auth', 'auth_urls'),
]


def build_module_urls( module_url_lists ):
""" Russ Ryba - 2009-10-16
Given a list of tuples that contain (url prefix, module name, url
list name)
Return a list of tuples that contain (url_prefix, requesthandler)
suitable for passing to the tornado.web.application class
"""
new_handler_list = []
for module_url_def in module_url_lists:
(prefix, module_name, list_name) = module_url_def
url_module = __import__( '%s.urls' % module_name, globals(),
locals(), [list_name], -1)
url_list = url_module.__dict__[list_name]
for handler_def in url_list:
new_handler_def = (prefix + handler_def[0], handler_def
[1])
new_handler_list.append( new_handler_def )
del( url_module )
return new_handler_list


module_urls = build_module_urls( module_url_lists )
app_urls = module_urls + default_urls

application = tornado.web.Application(app_urls, **settings)
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(8877)
tornado.ioloop.IOLoop.instance().start()

------------------------------------------
# auth/urls.py
from handlers import LoginHandler
from handlers import LogoutHandler
from handlers import HomePageRedirectHandler

auth_urls = [
(r"/login", LoginHandler),
(r"/logout", LogoutHandler),
(r"/", HomePageRedirectHandler),
]


The end result is the same as if you started your application with the
following url list:
import auth
app_urls = [
(r"/", MainHandler),
(r"/auth/", auth.HomePageRedirectHandler ),
(r"/auth/", auth.LoginHandler ),
(r"/auth/", auth.LogoutHandler ),
]