Use the source, Luca
Jonathan Lundell
KNOW YOUR INGREDIENTS
Introduction
One of the less visible features of web2py is that it’s a fully integrated framework with a compact and readable source that’s part of your installation. When you have a question that isn’t answered in the web2py book, and isn’t directly addressed by one of these recipes, the answer can generally be found in the source.
Each chapter in this book concludes with a Know Your Ingredients section, that serves as a guide to the portions of the source that make up the subject matter of the chapter. These sections are not detailed descriptions of the source, but are rather high-level roadmaps to assist your own navigation.
Installation and Deployment
Every incoming request to web2py is sent from the web server to gluon.main.wsgibase, which, as the name suggests, supports the WSGI application API. The means by which the request gets from the server to wsgibase is a function of the kind of deployment being used. Different paths require different startup or handler files by which web2py is run.
We can divide the deployment methods into three categories:
- CGI deployment requires web2py to be run as a new process for each request. The CGI handler is cgihandler, which uses the wsgiref module’s CGIHandler to pass the request to wsgibase. This is a pattern that you’ll see repeated in many of the handlers: an API such as CGI is translated to a WSGI call and sent to wsgibase. The Google App Engine is CGI-like, in that web2py runs once for each request. GAE is configured via app.yaml to run gaehandler, which in turn uses standard Google libraries to configure the environment before invoking CGIHandler.py, or else running wsgiref directly if logging is enabled. Notice the configuration parameters near the beginning of gaehandler, and the environment variable SERVER_SOFTWARE a little farther on.
- web2py includes Rocket, a built-in pure-Python WSGI-compatible server that can directly handle web server duties, or act as the back-end server for a proxy such as Apache’s mod_proxy. When using Rocket, web2py is started with the web2py.py script, which simply calls gluon.widget.start to handle command-line options and optionally present the operator with a minimal Tk GUI. Once running, Rocket receives HTTP requests and passes them to Rocket. See gluon.main.HttpServer for the invocation of Rocket.
- Finally, web2py can run continuously and field requests from an external web server though the WSGI API (wsgihandler), the FastCGI API (fcgihandler) or mod_python (modpythonhandler). The WSGI handler is the simplest, because web2py is a native WSGI application. modpythonhandler wraps a mod_python interface around the call to wsgibase with help from mod_python.apache, while fcgihandler uses gluon.contrib.gateways.fcgi, a contributed module, to convert from FastCGI to WSGI, eventually calling wsgibase.
But wait, there’s one more. A recent addition to web2py, the anyserver script, inspired by Bottle’s similar functionality, supports a long list of web servers—just about any server that supports the WSGI API. The list is too long to repeat here, but if you have a favorite server you’d prefer to use, be sure to browse anyserver to see if it’s on the list.
Installation & Startup Scripts
A few installation scripts can be found in the the scripts directory; look for scripts named setup-*, such as setup-web2py-ubuntu.sh.
Likewise, a few startup scripts designed for integration into a hosts boot-time initialization method are found in the scripts directory: web2py.archlinux.sh, web2py.fedora.sh, and web2py.ubuntu.sh.
Dispatching a Request
The web2py request dispatcher, gluon.main.wsgibase, is where each incoming HTTP request ends up once it makes its way through whichever handler is being used. Here’s we’ll briefly describe the top-level flow of a request.
main.wsgibase
- Initialize request-scope globals (request, response, etc)
- Rewrite URL (gluon.rewrite)
- Stream static files via streamer.stream_file_or_304_or_206. Notice that the ultimate handler of a request ends up raising an HTTP exception, which is handled farther down in main.wsgibase.
- Perform per-request housekeeping: parsing the query string, handling cookies, initializing the session and the response headers.
- main.serve_controller: dispatch the request to the application
- compileapp.build_environment: build the environment in which the models, controller and view will run. (This is where the request-scope globals such as URL, T, request, response are supplied.)
- Set the default view, based on the request’s controller, function, extension.
- Run the application’s models: compileapp.run_models_in
- Run the selected controller: compileapp.run_controller_in
- If the controller returns a dict, run the selected controller: compileapp.run_view_in. Notice that both the controller and view see the environment as modified by the models, but the controller does not affect the view’s environment.
- Raise an HTTP exception (typically 200 OK) to return control to main.wsgibase.
- except HTTP: send response to the server
- Static files are served directly.
- Commit database. You’ll notice later that if the request ended in an error, the database is rolled back.
- Save the session on disk.
- Create cookie headers
- finally: run cron if softcron, and respond via http.HTTP.to
- Finish creating the response headers.
- Send status and headers back via WSGI responder callback.
- Return body of response to WSGI caller.
pbreit
Kenneth Lundström
Kenneth
I've been working a little on a proposal for some new cookbook sections, one per chapter, that provide a roadmap to the web2py source that's relevant to the chapter.
My motivation, mentioned below in the Introduction section, is that the full-stack nature of web2py is one of its important features, and can be used to advantage by developers. But the source is intimidating for newcomers, and even for experienced web2py developer's, it's not alway obvious where one should be looking.
US cookbooks often have a chapter or two that talk about ingredients and techniques rather than recipes per se; the standard cookbook Joy of Cooking�calls this chapter "Know Your Ingredients", and I've borrowed the name. My tentative decision is to distribute this material across the existing chapters, as the last section in each chapter. However, and argument could be made for making it a separate chapter. I'm ambivalent on the question.
If you guys are agreeable, I'll proceed with the idea. Below are an introduction and the Use the Source section for Chapter 1.
KNOW YOUR INGREDIENTS
Introduction
One of the less visible features of web2py is that it�s a fully integrated framework with a compact and readable source that�s part of your installation. When you have a question that isn�t answered in the web2py book, and isn�t directly addressed by one of these recipes, the answer can generally be found in the source.
Each chapter in this book concludes with a Know Your Ingredients section, that serves as a guide to the portions of the source that make up the subject matter of the chapter. These sections are not detailed descriptions of the source, but are rather high-level roadmaps to assist your own navigation.
Installation and Deployment
Every incoming request to web2py is sent from the web server to gluon.main.wsgibase, which, as the name suggests, supports the WSGI application API. The means by which the request gets from the server to wsgibase is a function of the kind of deployment being used. Different paths require different startup or handler files by which web2py is run.
We can divide the deployment methods into three categories:
- CGI deployment requires web2py to be run as a new process for each request. The CGI handler is cgihandler, which uses the wsgiref module�s CGIHandler to pass the request to wsgibase. This is a pattern that you�ll see repeated in many of the handlers: an API such as CGI is translated to a WSGI call and sent to wsgibase. The Google App Engine is CGI-like, in that web2py runs once for each request. GAE is configured via app.yaml to run gaehandler, which in turn uses standard Google libraries to configure the environment before invoking CGIHandler.py, or else running wsgiref directly if logging is enabled. Notice the configuration parameters near the beginning of gaehandler, and the environment variable SERVER_SOFTWARE a little farther on.
- web2py includes Rocket, a built-in pure-Python WSGI-compatible server that can directly handle web server duties, or act as the back-end server for a proxy such as Apache�s mod_proxy. When using Rocket, web2py is started with the web2py.py script, which simply calls gluon.widget.start to handle command-line options and optionally present the operator with a minimal Tk GUI. Once running, Rocket receives HTTP requests and passes them to Rocket. See gluon.main.HttpServer for the invocation of Rocket.
- Finally, web2py can run continuously and field requests from an external web server though the WSGI API (wsgihandler), the FastCGI API (fcgihandler) or mod_python (modpythonhandler). The WSGI handler is the simplest, because web2py is a native WSGI application. modpythonhandler wraps a mod_python interface around the call to wsgibase with help from mod_python.apache, while fcgihandler uses gluon.contrib.gateways.fcgi, a contributed module, to convert from FastCGI to WSGI, eventually calling wsgibase.
But wait, there�s one more. A recent addition to web2py, the anyserver script, inspired by Bottle�s similar functionality, supports a long list of web servers�just about any server that supports the WSGI API. The list is too long to repeat here, but if you have a favorite server you�d prefer to use, be sure to browse anyserver to see if it�s on the list.
Installation & Startup Scripts
A few installation scripts can be found in the the scripts directory; look for scripts named setup-*, such as setup-web2py-ubuntu.sh.
Likewise, a few startup scripts designed for integration into a hosts boot-time initialization method are found in the scripts directory: web2py.archlinux.sh, web2py.fedora.sh, and web2py.ubuntu.sh.
Dispatching a Request
The web2py request dispatcher, gluon.main.wsgibase, is where each incoming HTTP request ends up once it makes its way through whichever handler is being used. Here�s we�ll briefly describe the top-level flow of a request.
main.wsgibase�
- Initialize request-scope globals (request, response, etc)
- Rewrite URL (gluon.rewrite)
- Stream static files via streamer.stream_file_or_304_or_206. Notice that the ultimate handler of a request ends up raising an HTTP exception, which is handled farther down in main.wsgibase.
- Perform per-request housekeeping: parsing the query string, handling cookies, initializing the session and the response headers.
- main.serve_controller: dispatch the request to the application
- compileapp.build_environment: build the environment in which the models, controller and view will run. (This is where the request-scope globals such as URL, T, request, response are supplied.)
- Set the default view, based on the request�s controller, function, extension.
- Run the application�s models: compileapp.run_models_in
- Run the selected controller: compileapp.run_controller_in
- If the controller returns a dict, run the selected controller: compileapp.run_view_in. Notice that both the controller and view see the environment as modified by the models, but the controller does not affect the view�s environment.
- Raise an HTTP exception (typically 200 OK) to return control to main.wsgibase.
- except HTTP: send response to the server
- Static files are served directly.
- Commit database. You�ll notice later that if the request ended in an error, the database is rolled back.
Jonathan Lundell
> Probably any additional documentation is a "good thing". I personally like more practical information. My favorite framework documentation is probably CodeIgniter.
Using the source is definitely a backup option, when the books and other docs don't work.
But using the source is particularly useful for web2py, for both a negative and a positive reason.
The negative reason is that there's quite a bit of web2py that simply isn't fully documented anywhere: the source (or asking on the list) is sometimes the only place to find what you need to know.
The positive reason is that web2py is especially friendly to source inspection, because the source is compact and unified. It's a great resource.
DenesL
Well done!.
One doubt, doesn't this sound strange:
"Rocket receives HTTP requests and passes them to Rocket"?.
Jonathan Lundell
That way it'd be easier to keep the source roadmap up to date (and add new material), and the roadmap itself wouldn't get tied up in the book's copyright.
Jonathan Lundell
>
> One doubt, doesn't this sound strange:
> "Rocket receives HTTP requests and passes them to Rocket"?.
Ok, well, details...
Offray Vladimir Luna Cárdenas
Hash: SHA1
Hi,
El 18/01/11 16:12, Jonathan Lundell escribi�:
> I could also be persuaded that the cookbook should have a briefer section on source browsing--just pointers to where to look--and the more detailed roadmap should be a more dynamic online document.
>
> That way it'd be easier to keep the source roadmap up to date (and add new material), and the roadmap itself wouldn't get tied up in the book's copyright.
>
>
I like most this option. As a newbie, I find your contribution to
documentation really valuable and a good reading once I get more
understanding about the basics.
Cheers,
Offray
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJNNhOTAAoJEGiex2828ICw0AYH/3MxvfFXmRrWARsn8J92hLn4
hXYQa37z5//JN9102IAcXY47ELiwDdxvGQhxOmfD/ZdiogAdq0kDpBjLcrqerUfS
8QzeMX6vRzz84alp3EUv7t60WQ/ejRhaQTzxrLM6exxPtBSPmwtRRYts5uIfwcPa
Kk/lAIIfNBXECV5cTb0jK5Vs8Ve8YnEgDpM/K0bAgPW3ekQGfZYd8VCpCIy4T6gP
+j/lhna+7lzkKMeqy4YOcbUJu7cFPNYuUJsMibt+gcGX08EPrfQUz/EtGxzr4VUo
9iAJFq/zkwy3WJhi3WwLMbHMGnFaCUvWpZwYiDwjcSA+hCFflyGA7lGAQuTvAHQ=
=JrPI
-----END PGP SIGNATURE-----
Anthony
cjrh
> I've been working a little on a proposal for some new cookbook sections, one per chapter, that provide a roadmap to the web2py source that's relevant to the chapter.
1) Too little
2) Too much
"Too much" can be bad for at least two reasons: it is a lot of work to
maintain (need many people), and; it becomes difficult to maintain a
cohesive document design in a collaborative environment. Ironically,
even though the web2py project has been accused of "too little
documentation", in truth we already suffer from the symptoms of "too
much". There is gold buried in the web2pyslices.com site, things that
should already be in the book, but are not because the sites are
separate, and it requires significant effort to merge information into
something useful. The solution is not to create more, but to first
improve what we have.
However, having said that, I do think you make a valuable point about
improving the knowledge transfer of the web2py source code itself.
That job can only be properly performed by someone such as yourself,
who knows the code well. There are not *that* many of you, but the
numbers could grow if you write such a document.
I guess the point I am making is that we must be careful to try to
make the greatest impact with the least amount of work.
There are three forms of documentation that emerge (I am including a
lot of extra detail for other readers of this post that may pass by):
a) The book, often called the "Developer's Guide" in other development
tools; this is for the person new to web2py that wants to learn how to
get stuff done in the fastest possible way. It won't carry the full
API, and nor should it, because of (b)
("The book", here: web2py.com/book)
b) The programming API, often called the "Programmer's Guide" in other
development tools, which would presumably be made up of docstrings
harvested directly from the source code
(Epydoc, here: http://www.web2py.com/examples/static/epydoc/index.html).
c) Tutorials, Cookbooks and Howtos
(web2pyslices, here: http://web2pyslices.com/main/default/index)
I really like your suggestion about a guide to the web2py source code,
because it seems that dramatic leverage is available through that
route to get a lot more hands on deck for the development work. I am
less keen on calling that a cookbook. Calling it a tour guide would
be better, if I understand what you're aiming for, and you could break
the hegemony that the cookbook metaphor holds over the programming
space. Enough cooking already! Let's do sights and sounds!
I don't know what the quality of the API documentation is like ([b] in
my list above), because I spend most of my time on (a), and frankly,
the book has been enough for me in the vast majority of situations
(dir() for everything I need more information about!). The web2py
book is a great landmark in the project (not "ingredient"!).
I think I am in favour of your suggestion of putting references to
source units inside the book itself. I would probably suggest that
the format of that inclusion be little more than a list of applicable
source files per section, rather than the level of additional detail
you demonstrate. To add further discussion would create more
maintenance work for the documentation, and would detract, I feel,
from the purpose (and quality, assuming no duplication) of the API
documentation. Epydoc provides a space for writing module-level
comments, and my gut feel is that the higher-level description of the
source code itself should appear here. Perhaps a link from the book
chapter to the Epydoc page for that module? For instance, for the
DAL chapter, we can link immediately to the module-level Epydoc API
documentation right at the start of chapter:
http://www.web2py.com/examples/static/epydoc/toc-web2py.gluon.dal-module.html
And then you and the other core developers could attack the in-source,
docstring documentation, and in particular the module-level docstrings
with expanded explanation and discussion. If you go to the link above,
you'll see the module-level text that is there already; it's a good
start, but room for the very improvement you have demonstrated in your
post.
My ZAR0.02 :)
Anthony
On Tuesday, January 18, 2011 7:21:05 PM UTC-5, cjrh wrote:
I think I am in favour of your suggestion of putting references to
source units inside the book itself. I would probably suggest that
the format of that inclusion be little more than a list of applicable
source files per section, rather than the level of additional detail
you demonstrate. To add further discussion would create more
maintenance work for the documentation, and would detract, I feel,
from the purpose (and quality, assuming no duplication) of the API
documentation. Epydoc provides a space for writing module-level
comments, and my gut feel is that the higher-level description of the
source code itself should appear here. Perhaps a link from the book
chapter to the Epydoc page for that module?
pbreit
cjrh
> These are great thoughts. One comment below...
>
> On Tuesday, January 18, 2011 7:21:05 PM UTC-5, cjrh wrote:
> > documentation. Epydoc provides a space for writing module-level
> > comments, and my gut feel is that the higher-level description of the
> > source code itself should appear here. Perhaps a link from the book
> > chapter to the Epydoc page for that module?
>
> docstrings. For example, Jonathan's "Installation and Deployment" section
> refers to more than one module, so that kind of thing needs to be handled at
> a higher level than just a module docstring.
that I was referring to (curse you Frames): first go here:
http://www.web2py.com/examples/static/epydoc/index.html
And then click on web2py.gluon.dal on the left. You will see module-
level documentation on the right, which is already prose-like and
descriptive. This is what I was referring to as a suggested location
for the "tour guide to the source code".
> He also explains installation
> and startup scripts, which are probably not covered in the API documentation
> at all.
mikech
Anthony
There was a malfunction on my link to the module-level documentation
that I was referring to (curse you Frames): first go here:
http://www.web2py.com/examples/static/epydoc/index.html
And then click on web2py.gluon.dal on the left. You will see module-
level documentation on the right, which is already prose-like and
descriptive. This is what I was referring to as a suggested location
for the "tour guide to the source code".
