(possibly weird) Questions for web2py

[Alex Popescu]

Hi all,

I've spent some time reading the available documentation on web2py
(plus some more). I'll start by saying that I like a lot of things I'm
seeing. Still, there are a couple of weird questions that I'd like to
ask:

1. Routing

According to the documentation URLs are solved according to the
following rule:

protocol://netloc/application/controller/function[(/arg)*][?vars]

My question is: how difficult would be to eliminate the function part
and basically route the request to a method defined in the controller
corresponding to the HTTP request method name (i.e. GET -> get, POST -
> post, PUT -> put, etc.)?

As far as I can tell, for a WSGI deployment the URL parsing happens in
gluon.main.wsgibase, but I am wondering if there are other places I
should look for.

Note: I am looking to build a very RESTful app and I'd like to have
this clear convention in my app.

2. Models and Data Access

This part is a bit more complex to describe, but I hope you'll bare
with me for a couple of paragraphs.

Currently the web2py models are very data centric (please keep in mind
that I'm not saying that this is good or bad):

db.define_table('name', SQLField(...)

What I'd actually like would be to hide this part behind an object
centric approach. I am aware of the fact that basically the approaches
are quite different (ORM vs ActiveRecord), but I'm wondering if some
metaclass magic would be able to allow me to preserve the current
implementation while offering an object oriented perspective. A very
basic example would be the data access layer implementation available
in Google App Engine:

Model.gql('query')....

Simply put the meta-magic will just have to hide the db functions
behind some static Model methods.

Question: do you think that this is possible? how difficult would this
be?

So far these are the only 2 reasons that I haven't signed yet for
web2py. I also believe that 1st idea would be helpful to other web2py
users looking for RESTful apps, while the 2nd one can be seen just as
'syntactic sugar' (there may be other small benefit like intellisense
support in IDEs, readability, refactoring, etc.)

Looking forward to your answer(s),

./alex
--
.w( the_mindstorm )p.
Alexandru Popescu

[Alex Popescu]

[mdipierro]

On Mar 20, 5:16 pm, Alex Popescu <the.mindstorm.mailingl...@gmail.com>
wrote:

> Hi all,
>
> I've spent some time reading the available documentation on web2py
> (plus some more). I'll start by saying that I like a lot of things I'm
> seeing. Still, there are a couple of weird questions that I'd like to
> ask:
>
> 1. Routing
>
> According to the documentation URLs are solved according to the
> following rule:
>
> protocol://netloc/application/controller/function[(/arg)*][?vars]
>
> My question is: how difficult would be to eliminate the function part
> and basically route the request to a method defined in the controller
> corresponding to the HTTP request method name (i.e. GET -> get, POST -
> post, PUT -> put, etc.)?

You cannot acoomplish soley with routes but you can have a catch all
action that checks

if request.post_vars: call_post_action()

> As far as I can tell, for a WSGI deployment the URL parsing happens in
> gluon.main.wsgibase, but I am wondering if there are other places I
> should look for.
>
> Note: I am looking to build a very RESTful app and I'd like to have
> this clear convention in my app.

> 2. Models and Data Access
>
> This part is a bit more complex to describe, but I hope you'll bare
> with me for a couple of paragraphs.
>
> Currently the web2py models are very data centric (please keep in mind
> that I'm not saying that this is good or bad):
>
> db.define_table('name', SQLField(...)
>
> What I'd actually like would be to hide this part behind an object
> centric approach. I am aware of the fact that basically the approaches
> are quite different (ORM vs ActiveRecord), but I'm wondering if some
> metaclass magic would be able to allow me to preserve the current
> implementation while offering an object oriented perspective. A very
> basic example would be the data access layer implementation available
> in Google App Engine:
>
> Model.gql('query')....
>
> Simply put the meta-magic will just have to hide the db functions
> behind some static Model methods.
>
> Question: do you think that this is possible? how difficult would this
> be?

This has been done in order to allow web2py to import Django models:
You can find the code here:
http://mdp.cti.depaul.edu/AlterEgo/default/show/189

[Alex Popescu]

On Mar 21, 1:27 am, mdipierro <mdipie...@cs.depaul.edu> wrote:
> On Mar 20, 5:16 pm, Alex Popescu <the.mindstorm.mailingl...@gmail.com>
> wrote:
>
>
> > 1. Routing
>
> > According to the documentation URLs are solved according to the
> > following rule:
>
> > protocol://netloc/application/controller/function[(/arg)*][?vars]
>
> > My question is: how difficult would be to eliminate the function part
> > and basically route the request to a method defined in the controller
> > corresponding to the HTTP request method name (i.e. GET -> get, POST -
> > post, PUT -> put, etc.)?
>
> You cannot acoomplish soley with routes but you can have a catch all
> action that checks
>
> if request.post_vars: call_post_action()
>

I'm not sure I'm following your answer. While I haven't spent too much
time looking at the code my impression is that it should be as simple
as making some minor changes to the way the Request object is
initialized in gluon.main.wsgibase.

Well, there might be something else needed too, but I haven't found it
yet: generating the URLs.

This looks like an excellent start. I think I can take it from there.

While waiting for your comments on my previous questions I've figured
2 more:

3/ Templating

a/ Is template inheritance supported? (I couldn't find any reference,
but it might be caused by being tired)

b/ Is it possible to invoke Python functions from the template that
are enhancing the template dictionary?

Imagine a homepage that has a couple of 'widget' like fragments. These
are not really part of the controller, so I'd like to just say in the
template: invoke a function and place add the returned value to the
dict used for rendering the template.

c/ Is it possible to include in a template an HTML fragment generated
as a result of a function invocation?

It is a similar concept to the above one, the difference being that
the 'widget' template is completely outside the current page (while in
the above scenario the 'widget' template was part of the current
template)

4/ Request/Response cycle interceptors

Basically this is a way to plug in cross-cutting functionality that
will be automatically triggered at specific moment of the request/
response cycle:

- before_request
- before_controller
- after_controller
- before_view
- after_view
- after_request

I think the Django synonym would be middleware classes.

Many thanks in advance,

[Yarko Tymciurak]

On Fri, Mar 20, 2009 at 6:48 PM, Alex Popescu <the.mindstor...@gmail.com> wrote:

On Mar 21, 1:27 am, mdipierro <mdipie...@cs.depaul.edu> wrote:
> On Mar 20, 5:16 pm, Alex Popescu <the.mindstorm.mailingl...@gmail.com>
> wrote:
>
>

> > 1. Routing
>
> > According to the documentation URLs are solved according to the
> > following rule:
>
> > protocol://netloc/application/controller/function[(/arg)*][?vars]
>
> > My question is: how difficult would be to eliminate the function part
> > and basically route the request to a method defined in the controller
> > corresponding to the HTTP request method name (i.e. GET -> get, POST -
> > post, PUT -> put, etc.)?
>
> You cannot acoomplish soley with routes but you can have a catch all
> action that checks
>
> if request.post_vars: call_post_action()
>

I'm not sure I'm following your answer. While I haven't spent too much
time looking at the code my impression is that it should be as simple
as making some minor changes to the way the Request object is
initialized in gluon.main.wsgibase.

That is, going to something like  protocol://netloc/application/default/index

(the default/index is default for an application - let's leave routes out of this for the moment)

    def index():

        if request.post_vars:

               action=request.post_vars

                if action == 'a':  a()

Maybe an example use case would help describe what you're looking for...

for example, see views/generic:

   {{extend 'layout.html'}}

b/ Is it possible to invoke Python functions from the template that
are enhancing the template dictionary?

   {{import my_template_helpers}}

   {{ for i in list_to_show:}}

          {{ output=my_template_helpers.someaction(i) }}

          {{ =T(output)}}

           {{pass}} 

Imagine a homepage that has a couple of 'widget' like fragments. These
are not really part of the controller, so I'd like to just say in the
template: invoke a function and place add the returned value to the
dict used for rendering the template.

c/ Is it possible to include in a template an HTML fragment generated
as a result of a function invocation?

    {{ =var }}

    {{ =function_call( x ) }}

    {{ =XML( some_otherwise_escaped_value ) }}

    {{ import random

       i = random.randint(1,6)

     }}

     <h{{ =i }}> Look!  A random header!;

       H{{ =i }}

       {{ if i%2: }} is odd {{ else: }} is even {{ pass }}

      </h{{ =i }}>

      

It is a similar concept to the above one, the difference being that
the 'widget' template is completely outside the current page (while in
the above scenario the 'widget' template was part of the current
template)

Oh!  I think you meant this for above:

   {{ def foo():   return a+b}}

   {{ pass }}

   {{ =foo() }}

 

4/ Request/Response cycle interceptors

Basically this is a way to plug in cross-cutting functionality that
will be automatically triggered at specific moment of the request/
response cycle:

I'm not sure what we're talking about here; wsgiserver -> gluon.main.wsgibase which sets up the context, so I think you mean this (I could be wrong - I try not to think about this too much):

- before_request

    your model files are read / run first; I think you mean there... ;  files are executed in alphabetical order, then the requested controller function is initiated, and a like named view.

- before_controller

- after_controller
- before_view
- after_view
- after_request

I think the Django synonym would be middleware classes.

I'm not sure - I think you want to look at "New Tools" and "T2" from here for starters:

http://www.web2py.com/examples/default/docs

Hope some of this has been helpful...

Regards,

Yarko 

[mdipierro]

> 3/ Templating
>
> a/ Is template inheritance supported? (I couldn't find any reference,
> but it might be caused by being tired)

Template inheritance as done by Django is ill-defined and I can
explain more.
You can do
{{extend 'another_template'}}
You can
{{include 'another_template'}}
You can create functions to define blocks before {{extend ...}}. This
is explained in the manual.

> b/ Is it possible to invoke Python functions from the template that
> are enhancing the template dictionary?

Yes if:
1) you define the function in models or
2) you define the funciton in controller and pass it to the view or
3) you import the function from the view
4) you create the function in the view {{def f(a)}}<h1>Helloe {{=a}}</
h1>{{return}}

> Imagine a homepage that has a couple of 'widget' like fragments. These
> are not really part of the controller, so I'd like to just say in the
> template: invoke a function and place add the returned value to the
> dict used for rendering the template.
>
> c/ Is it possible to include in a template an HTML fragment generated
> as a result of a function invocation?

Yes {{=XML(fragment)}}

> It is a similar concept to the above one, the difference being that
> the 'widget' template is completely outside the current page (while in
> the above scenario the 'widget' template was part of the current
> template)
>
> 4/ Request/Response cycle interceptors
>
> Basically this is a way to plug in cross-cutting functionality that
> will be automatically triggered at specific moment of the request/
> response cycle:
>
> - before_request
> - before_controller
> - after_controller
> - before_view
> - after_view
> - after_request

You can do most of this this. In controller place

def mycaller(f):
before_controller()
d=f()
after_controller()
if isinstance(d,dict): d=response.render(d)
after_view()
response._caller=mycaller

You can still do before request and after request using wsgi
middleware.

I apologize for my sketchy responses but I am on vacation and I have
limited connectivity. I suggest you ask your questions again in April.

Massimo

[Alex Popescu]

Firstly thanks for your answers. In order to clarify some of the
remaining things:

1. Routes clarification

HTTP 1.1 defines [1] 7 methods: OPTIONS, GET, HEAD, POST, PUT, DELETE,
TRACE (and reserves the CONNECT). The web2py routing is using to
determine the function name in the controller a part of the URL
according to the rule:

/application/controller/[FUNCTION]/...

What I need to create a RESTful app is that FUNCTION is automatically
deduced from the HTTP Request method and not from the controller
function name. So, my controller will be:

class SomeController(object):
def get(self):
# all GET request to /application/somecontroller/ are getting
here

def post(self):
# all POST requests to /application/somecontroller/ are getting
here

2. Models and Data Access

As I have already said this is a probably a good enough starting point
http://mdp.cti.depaul.edu/AlterEgo/default/show/189 and I will most
probably be able to take it from there.

3. Templates

I'll re-read the documentation, but so far it looks like I should be
able to accomplish all 3 scenarios.

I will not comment on "Django templates are 'ill-defined'" as I have
no idea. As far as I can tell the 2 models that are actually working
are:
- Sitemesh: decoration + basic composition (http://
www.opensymphony.com/sitemesh/
- Tiles: composition

4. Cross-cutting request/response cycle interceptors

I think Yarko's answer is more in the direction I was looking, but
most probably I'll have to dig a bit into the code and come up with a
much more detailed question.

Massimo's first part of the answer is not a solution that it couples
the code with the interceptors and the main goal is exactly the
opposite: to separate the cross-cutting orthogonal concerns.

In conclusion, the only remaining question is if and how is it
possible to create a routing mechanism based on RESTful principles
that uses the HTTP request method information.

Many thanks,

./alex
--
.w( the_mindstorm )p.
Alexandru Popescu

[1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html
[2] http://www.opensymphony.com/sitemesh/
[3] http://struts.apache.org/1.x/struts-tiles/

[Yarko Tymciurak]

On Sat, Mar 21, 2009 at 9:51 AM, Alex Popescu <the.mindstor...@gmail.com> wrote:

Firstly thanks for your answers. In order to clarify some of the
remaining things:

1. Routes clarification

HTTP 1.1 defines [1] 7 methods: OPTIONS, GET, HEAD, POST, PUT, DELETE,
TRACE (and reserves the CONNECT). The web2py routing is using to
determine the function name in the controller a part of the URL
according to the rule:

/application/controller/[FUNCTION]/...

What I need to create a RESTful app is that FUNCTION is automatically
deduced from the HTTP Request method and not from the controller
function name. So, my controller will be:

class SomeController(object):
 def get(self):
    # all GET request to /application/somecontroller/ are getting
here

 def post(self):
    # all POST requests to /application/somecontroller/ are getting
here

In web2py,   requests to application/controller/method  are getting to method;  You can can determine which  method in request.reguest_method and direct the call the desired function;  a quick look at http://www.web2py.com/examples/simple_examples/hello6

will give you an idea of what request and response variables you have to work with.

 

2. Models and Data Access

As I have already said this is a probably a good enough starting point
http://mdp.cti.depaul.edu/AlterEgo/default/show/189  and I will most
probably be able to take it from there.

3. Templates

I'll re-read the documentation, but so far it looks like I should be
able to accomplish all 3 scenarios.

I will not comment on "Django templates are 'ill-defined'" as I have
no idea. As far as I can tell the 2 models that are actually working
are:
- Sitemesh: decoration + basic composition (http://
www.opensymphony.com/sitemesh/
- Tiles: composition

4. Cross-cutting request/response cycle interceptors

I think Yarko's answer is more in the direction I was looking, but
most probably I'll have to dig a bit into the code and come up with a
much more detailed question.

Massimo's first part of the answer is not a solution that it couples
the code with the interceptors and the main goal is exactly the
opposite: to separate the cross-cutting orthogonal concerns.

In conclusion, the only remaining question is if and how is it
possible to create a routing mechanism based on RESTful principles
that uses the HTTP request method information.

Thanks for your thoughtful notes, Alex.  Looking forward to hearing more from you.

Regards,

Yarko

[mdipierro]

http://..../a/b/c/...

request.application is a
request.controller is b
request.function is c
request.args are .... split('/')

On Mar 21, 1:37 pm, Yarko Tymciurak <yark...@gmail.com> wrote:
> On Sat, Mar 21, 2009 at 9:51 AM, Alex Popescu <
>
>
>
> the.mindstorm.mailingl...@gmail.com> wrote:
>
> > Firstly thanks for your answers. In order to clarify some of the
> > remaining things:
>
> > 1. Routes clarification
>
> > HTTP 1.1 defines [1] 7 methods: OPTIONS, GET, HEAD, POST, PUT, DELETE,
> > TRACE (and reserves the CONNECT). The web2py routing is using to
> > determine the function name in the controller a part of the URL
> > according to the rule:
>
> > /application/controller/[FUNCTION]/...
>
> > What I need to create a RESTful app is that FUNCTION is automatically
> > deduced from the HTTP Request method and not from the controller
> > function name. So, my controller will be:
>
> > class SomeController(object):
> >  def get(self):
> >     # all GET request to /application/somecontroller/ are getting
> > here
>
> >  def post(self):
> >     # all POST requests to /application/somecontroller/ are getting
> > here
>
> In web2py,   requests to application/controller/method  are getting to
> method;  You can can determine which  method in request.reguest_method and

> direct the call the desired function;  a quick look athttp://www.web2py.com/examples/simple_examples/hello6

[Alex Popescu]

Yarko, Massimo,

I do understand how web2py routing works. But I do NOT like it as it
is action based while I am looking for clean REST.

As I've pointed out in my first email the URL parsing and 'routing'
happens in 'gluon.main.wsgibase' (at least for a mod_wsgi deployment)
and I am pretty sure I know how to make it work as I want, but my
question is if that is the only modification I need to do or are there
other places where I should look.

Just to clarify the difference between action based URL and REST URL:

1. for action based URLs the HTTP Request method has no impact on
routing, i.e. a GET, POST, PUT can result in the same function
invocation (making a GET or POST request to /myapp/controller/
dosomething will always invoke dosomething)

2. for a REST app, GET, POST, PUT is THE action identifier and so
there is no need of the <function_name> in the URL (basically all is
needed for REST urls is the RESOURCE name)

Here is a very basic example: imagine a book management app that
allows you to Add a book, List existing books and modify the details
of a book.

1. create a new book


REST: POST /myapp/books/

Action based: /myapp/books/create
(Note: you can use even a GET request to create the book)

2. list existing books:

REST: GET /myapp/books/

Action based: /myapp/books/list

3. display an existing book:

REST: GET /myapp/books/<book_identifier>

Action based: /myapp/books/edit/<book_identifier>

4. update existing book:

REST: PUT /myapp/books/<book_identifier>

Action based: /myapp/books/update/<book_identifier>

As you can see there are major differences between the two approaches.
I will not discuss here the pros and cons, but only point out that I
need a way to make web2py RESTy.

Moving away from the REST vs action based URLs, a more generic way to
formulate my question would: is there a way to change the whole
routing mechanism?
My app requires various types of URLs that I don't think I will be
able to route using the current mechanisms.

Examples:

1.
/api/v1.0/<rest_of_url>
/api/v2.0/<rest_of_url>

2.
/resource/identifier.html -> HTML representation of the resource
identified by 'identifier'
/resource/identifier.json -> JSON representation of the resource
identified by 'identifier'
/resource/identifier.xml -> XML representation of the resource
identified by 'identifier'

... and quite a few others so frankly speaking I am looking for a more
descriptive but explicit (non-conventional) way to perform the
routing.

Many thanks in advance,

./alex
--
.w( the_mindstorm )p.
Alexandru Popescu

Disclaimer: We are currently starting to prototype a real app and I
have a pretty clear idea of what I need to accomplish. I have no
intentions to start flamewars or debates and I am only looking for
specific features that I know will be needed.

[mdipierro]

The only relevant files

gluon/main.py (wsgibase)
gluon/rewrite.py

Are you planning to modify the routes_in syntax to allow REST? If so I
am interested in it but let's agree on a syntax first.

Massimo

On Mar 21, 4:13 pm, Alex Popescu <the.mindstorm.mailingl...@gmail.com>
wrote:

[Alex Popescu]

On Mar 22, 4:57 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> The only relevant files
>
> gluon/main.py (wsgibase)
> gluon/rewrite.py
>

I think that gluon/html.py URL will also need some changes for the
case when it is used with a request parameter.

> Are you planning to modify the routes_in syntax to allow REST? If so I
> am interested in it but let's agree on a syntax first.
>

I must confess that I'd prefer to reuse some existing solution and not
try to reinvent a solution as things are not as simple as they may
look at first sight.

I know Google AppEngine has a a basic routing mechanism in place, but
it is far for being complete. Another option I am aware of is Routes
[1] which looks better (I haven't used it extensively though and I
wasn't really able to find the documentation for the latest release
1.10.3 [2]).

If we want to go the routes.py path, what I think it might be needed
is:

1. add a configuration option saying if a non-native web2py routing
should be used
2. allow definition of complex (regex-based) rewrite rules
(f.e.:
r'^/some/url/(controller)/here(.html)$' -> '/myapp/\1?format=\2' #
'controller' is the controller
r'^/other/url/(controller)/(id).(html)$' -> '/myapp/\1/\2?format=\3'
# 'controller' is controller, 'id' will be available through args,
etc
)

Anyways, I don't find the above is really readable, but I don't really
have any other idea on how to preserve as much of the current
execution flow.

There are some more things that will probably need some work.

1. Case insensitive URLs and CamelCase Python class naming convention

Usually I'd prefer to have case insensitive URLs (it is rather stupid
to show serve a resource /a/b.html while returning a 404 for /A/
b.html). On the other hand, Python class names are using the camel
case notation, so when registering the controllers the names will have
to be lowered (same applies to the controller part in the URL). Indeed
this can be added as a configuration option.

2. I am not aware of a way to not have to flatten out my project
structure so that conversions from url_path_fragment to controller
name actually works.

With a decently sized webapp, you may start using different packages
to group the controllers together. Right now, I do think that all
controller names are flatten out (by removing any package/module info
-- this is just a guess for now as I haven't really had the time to
dig through that part of the code).

This is not a major problem though.

Looking forward to hearing your comments,

./alex
--
.w( the_mindstorm )p.
Alexandru Popescu

[1] http://routes.groovie.org
[2] http://pypi.python.org/pypi/Routes

[mdipierro]

This is a complex issue and I am on vacation with limited
connectivity. Can we postpone this discussion to April?

Massimo

On Mar 22, 4:04 pm, Alex Popescu <the.mindstorm.mailingl...@gmail.com>

[Alex Popescu]

On Mar 22, 11:17 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> This is a complex issue and I am on vacation with limited
> connectivity. Can we postpone this discussion to April?
>

Ok. I'll continue with the other framework on my list and get back to
the topic in April.

cheers,

[mdipierro]

Sorry. I did not mean to be dismissive and by April I literally mean
April 1st not later. I will be back in chicago in 3 days and then I am
at PyCon for 3 days.

Massimo

On Mar 22, 10:10 pm, Alex Popescu