mod_wsgi and web2py
carlo
interesting framework. Coming from Karrigell that I used quite
extensively, I recently had some posts in the Karrigell Google group
tackling with the implementation of the framework behind Apache in a
production environment. Some reference links are:
http://groups.google.it/group/karrigell/browse_thread/thread/29552821a97fce8
and the Graham Dumpleton mod_wsgi project:
http://code.google.com/p/modwsgi/
http://code.google.com/p/modwsgi/wiki/IntegrationWithKarrigell
One of the problem arising with Karrigell, when coping with using it
behind Apache, is its embedded webserver is not thread wise and that,
in Graham Dumpleton words, "problems stem from Karrigell not accessing
resources by absolute pathnames, and instead relying on being able to
change the current working directory for each request."
I would like to have your perspective on how web2py confronts this
kind of issues, thank you
carlo
Massimo Di Pierro
web2py wsgibase is thread safe. The only functionality that is not is
web based testing.
The reason is that the doctest module is not thread safe but only
conflicts with itself.
web2py has the option of accessing folders by absolute paths (that is
the default on windowsce).
By default is does access all files as relative paths from the same
web2py working folder, where web2py is installed. web2py never
changes folder.
If you browse the source you may find some chdir(). They are not
actually supposed to change folder except in the case the user code
has a bug and the user code changed the folder. In this case web2py
chdir back to where it is supposed to be and logs the problem.
Massimo
Graham Dumpleton
On Feb 17, 12:47 pm, Massimo Di Pierro <mdipie...@cs.depaul.edu>
wrote:
>
> web2py wsgibase is thread safe. The only functionality that is not is
> web based testing.
> The reason is that the doctest module is not thread safe but only
> conflicts with itself.
>
> web2py has the option of accessing folders by absolute paths (that is
> the default on windowsce).
>
> By default is does access all files as relative paths from the same
> web2py working folder, where web2py is installed. web2py never
> changes folder.
Does web2py internally use os.path.join() to add the relative paths to
a base directory defined by some configuration or dynamic part of
request before the path is passed off to any system functions for
accessing files, or does it pass relative paths direct to the system
functions? If it does the latter it isn't thread safe when run under
Apache where other web applications may be hosted in the same process,
even if web2py does try to restore the working directory to what it
thinks it should be.
> If you browse the source you may find some chdir(). They are not
> actually supposed to change folder except in the case the user code
> has a bug and the user code changed the folder. In this case web2py
> chdir back to where it is supposed to be and logs the problem.
isn't thread safe. A web framework should never need to forcibly
change the working directory at all. If you are trying to protect
users who are using relative paths themselves, it is a mistake as the
user shouldn't be using relative paths either and so you shouldn't be
doing stuff like that to try and ensure it works for them as it will
only give them a false sense of security that their code is portable
to different hosting mechanisms.
Graham
carlo
Web2py is very interesting framework as it has some uncommon features
in the python framework world and I like most of the architectural
choices made by Massimo in his project. Its wsgi compliance made me
hopeful about the possibility to scale up web2py to a more enterprise
oriented environment and I am now looking forward to read a reply from
Massimo.
I worked with some different python frameworks by now and noticed
that, apart from different personal tastes, they all work
satisfactorily when intranet applications matter; when they lack is
about the possibility to easily port the application in a public
hosting service. As many of these problems may be certainly ascribed
to Python itself who never gained much popularity in the hosting
companies, I would like to see at least a full wsgi compliance and no
implementation quirks. TG itself in this respect left me quite
disappointed and some python-specialized hosting forums seem to prove
that problems arise often.
Massimo Di Pierro
web2py is thread safe but to answer your specific questions:
1)
web2py expects to run from web2py/
so that it finds gluon/ applications/ and deposit/ as subfolders.
It never chdir anywhere else. All web2py apps run from there. You
have the option to access all files via relative paths from its
working directory (the default on unix and windows) or via absolute
paths (the default on windows ce since windows ce does not understand
relative paths).
If you run apache with mod_proxy there is no issue.
If you run apache with mod_wsgi you have to install (or symlink)
web2py in the apache running folder.
I agree that if another apache app (not web2py) running in the same
context does a chdir, there may be trouble. In this case you need run
web2py with the windowsce option which requires commenting one line
in main. I will turn this option into a command line option to make
it easier.
2) web2py does not need to do chdir to work propery or, I agree with
you, it would not be thread safe, which it is.
At startup:
working_folder=os.getcwd() # once before creating threads
and it assumes working_folder never changes. For every thread:
try:
.... web2py never chdir ...
except ...
os.chdir(working_folder)
You can comment the latter line and everything works. The latter line
is there in case another app (not web2py's) that shares the same
context does chdir, which as you say is not supposed to happen. The
line is there to try recover and be able to log the issue.
Are you trying to tell me that I can run two mod_wsgi apps (for
example web2py and pylons) and if one of them does chdir, this affect
the other? Hope this is not the case or I will continue use mod_proxy
instead of mod_wsgi. Anyway, web2py will not do this and if another
app does, web2py will try to recover via the above mechanism.
Hope this answers your questions.
Massimo
Graham Dumpleton
wrote:
>
> web2py is thread safe but to answer your specific questions:
more to thread safety than protecting concurrent access to global
Python variables. Specifically, there is no way to control the current
working directory or ensure it is set to a specific value for any
length of time when threads are being used. Thus, any application
which is dependent in any way on the current working directory being a
specific value is flawed in as much as it can't be used with any
certainty that it will work in a multithreaded system.
I'd highly encourage you to change the design of web2py to not be
dependent on the current working directory being a specific value and
not even make it an option as it is the wrong thing to be doing.
> 1)
> web2py expects to run from web2py/
> so that it finds gluon/ applications/ and deposit/ as subfolders.
>
> It never chdir anywhere else. All web2py apps run from there. You
> have the option to access all files via relative paths from its
> working directory (the default on unix and windows) or via absolute
> paths (the default on windows ce since windows ce does not understand
> relative paths).
>
> If you run apache with mod_proxy there is no issue.
>
You can't expect users to be adding a symlink into the root directory
of the operating system.
The other flaw in requiring this is that it prevents someone running
two different web2py applications on the same system.
> I agree that if another apache app (not web2py) running in the same
> context does a chdir, there may be trouble. In this case you need run
> web2py with the windowsce option which requires commenting one line
> in main. I will turn this option into a command line option to make
> it easier.
directory the default and not even allow the other relative path
option. Using relative paths has too much potential for problems.
> 2) web2py does not need to do chdir to work propery or, I agree with
> you, it would not be thread safe, which it is.
>
> At startup:
>
> working_folder=os.getcwd() # once before creating threads
and so you cannot use os.getcwd(). The only way of doing it would be
for a user to explicitly define the location, or to base it some how
on the __file__ attribute of the code file it is contained in.
> and it assumes working_folder never changes. For every thread:
>
> try:
> .... web2py never chdir ...
> except ...
> os.chdir(working_folder)
>
> You can comment the latter line and everything works. The latter line
> is there in case another app (not web2py's) that shares the same
> context does chdir, which as you say is not supposed to happen. The
> line is there to try recover and be able to log the issue.
> Are you trying to tell me that I can run two mod_wsgi apps (for
> example web2py and pylons) and if one of them does chdir, this affect
> the other?
applications within the same Python interpreter mounted at different
URLs. Good WSGI frameworks allow this. Those which were bolted on top
of WSGI as an after thought don't always allow it. For example, Django
and TurboGears don't currently allow it.
A true WSGI friendly framework should even allow multiple distinct
applications based on it to be run. If web2py doesn't allow this
because of dependencies on the current working directory or
dependencies on specific Python global data variables for
configuration, then you are limiting what the user can do with it.
> Hope this is not the case or I will continue use mod_proxy
> instead of mod_wsgi.
hosting solution?
> Anyway, web2py will not do this and if another
> app does, web2py will try to recover via the above mechanism.
protect itself. This in itself could screw up other applications.
> Hope this answers your questions.
mod_wsgi might need to be created which specifically warns users that,
like with Karrigell, its design means it isn't safe to use in a
multithreaded process in conjunction with other web applications. :-)
Graham
Massimo Di Pierro
web2py is absolutely thread safe if used with mod_proxy.
In this case you can also run multiple copies of web2py from
different folders using different ports.
web2py may not be thread safe with mod_wsgi because another app (not
web2py) can do chdir(). In fact I cannot prevent other apps running
in the same context from screwing up web2py. In fact it is more than
chdir(), they share the same memory segments! I would not want web2py
apps to run in this way. It is very unsafe.
Anyway, I hear you. In particular I think there is a value in being
able to specify a working folder as a command line argument and use
absolute paths everywhere. I will add this feature in the next
version. I will release a new version with this change next week.
Massimo
Graham Dumpleton
wrote:
>
> web2py is absolutely thread safe if used with mod_proxy.
environment.
> In this case you can also run multiple copies of web2py from
> different folders using different ports.
>
> web2py may not be thread safe with mod_wsgi because another app (not
> web2py) can do chdir().
persistent processes and which uses multithreading. Basically,
anything except CGI-WSGI bridges.
For example, when using Pylons/Paste or CherryPy WSGI server one can
compose distinct WSGI applications together within the same process.
It is WSGI itself which permits this and nothing specific to do with
mod_wsgi which is only an implementation of the WSGI specification for
Apache web server. Higher level frameworks such as Django and
TurboGears are also going the direction of allowing some degree of
composition of WSGI application components at sub URLs within a large
application.
> In fact I cannot prevent other apps running
> in the same context from screwing up web2py. In fact it is more than
> chdir(), they share the same memory segments! I would not want web2py
> apps to run in this way. It is very unsafe.
processes they can run, and so separation through using processes
isn't always an option.
Graham
Graham Dumpleton
wrote:
> > in the same context from screwing up web2py. In fact it is more than
> > chdir(), they share the same memory segments! I would not want web2py
> > apps to run in this way. It is very unsafe.
>
> In web hosting environments, users often have constraints on how many
> processes they can run, and so separation through using processes
> isn't always an option.
are separation through use of multiple sub interpreters within the
same process and also process separation.
Thus you can run multiple WSGI applications in the same process by
delegating them to run in different sub interpreters and they cannot
readily interfere with each others Python code or data.
If mod_wsgi starts to be adopted for commodity web hosting, the likely
usage scenario would be that users only get one long lived process,
but can run multiple applications (if they conflict when run in same
interpreter) by delegating them to run in different sub interpreters.
If using web2py however, they would only be able to run one instance
of it and would need to make sure that nothing else is dependent on
the current working directory. Thus they would be limited in what they
could do.
Graham
Massimo Di Pierro
I will change what you say because you have one good point: I want to
able to specify the working folder when running with mod_wsgi and I
want to be able to run them from different folders.
The rest of your argument is almost semantic in nature. In fact if
two programs run in the same address space there is no limit to how
bad one can affect the other and there is no protection. This is not
limited to chdir. They all share environment variables, stdout,
stderr, stdin, memory etc. Forget web2py for a second. I can right
now write a pylons application that overwrite the ram memory of any
other pylons application running under mod_wsgi on the same system. I
teach how to do this stuff in my network programming and host
security classes.
If you run multiple programs in the same address space there is no
protection! The only protection is trust that the different
components have some good manners.
Back to web2py. It is designed so that in never chdir, it never
redirects stdout(*), stderr, or stdin. It never changes the values of
environment variables. It does not call python functions that are are
known to be thread unsafe(*) (*=except for doctest and it locks the
admin session when doing so to prevent concurrency). It also assumes
that other applications running in the same address space have the
same good manners. If they don't or if you are not sure, don't run
them with web2py.
What you said about apache running form "/" is not true. This is
against basic security rules since it opens the doors to the entire
filesystem from potential break-ins. Apache apps should run jailed
using chroot. You should chroot to the location where web2py is
installed when running web2py with Apache.
Reference: http://www.faqs.org/docs/securing/chap29sec254.html
Other frameworks may have problems with chroot.
Using mod_proxy of mod_rewrite provide the most separation because
web2py apps will run in its own address space. That is webfaction
runs most python web applications. That is how I suggest running
web2py, I yet have to see a performance penalty.
Massimo
Graham Dumpleton
> We can keep this going forever if you want....
> I will change what you say because you have one good point: I want to
> able to specify the working folder when running with mod_wsgi and I
> want to be able to run them from different folders.
sharing the processes with PHP, other Apache modules etc, and thus it
is a shared address space and environment, you obviously wouldn't want
to do that. Thus using embedded mode not an option in that case.
If running application in mod_wsgi daemon mode, by all means set the
working directory if you want. You can do this from the WSGI script
file when it is first loaded, or preferably use the 'home' option to
WSGIDaemonProcess, just make sure that web2py application is the only
thing running in that particular daemon process group. Delegate any
other WSGI applications, including other web2py instances to another
daemon process group, with potentially a different working directory.
For example:
WSGIDaemonProcess web2py-1 home=/some/path/myapp1
WSGIDaemonProcess web2py-2 home=/some/path/myapp2
WSGIScriptAlias /myapp1 /some/path/myapp1/apache/apache.wsgi
WSGIScriptAlias /myapp2 /some/path/myapp2/apache/apache.wsgi
<Directory /some/path/myapp1/apache>
WSGIProcessGroup web2py-1
</Directory>
<Directory /some/path/myapp2/apache>
WSGIProcessGroup web2py-2
</Directory>
This sets up two daemon process groups with myapp1 and myapp2 running
in each, ie. separate, and with different working directory.
> The rest of your argument is almost semantic in nature. In fact if
> two programs run in the same address space there is no limit to how
> bad one can affect the other and there is no protection.
create multiple sub interpreters in the same process and thus isolate
them. To cause problems for the application in the other sub
interpreters you would have to go to the extent of creating a C
extension module, or use ctypes in some way, to deliberately do things
to the code/data of the other other sub interpreters. In other words
it would have to be deliberately malicious and isn't something that
would inadvertently be done by your typical malfunctioning Python
application.
> This is not
> limited to chdir. They all share environment variables, stdout,
> stderr, stdin,
and stdin. You can replace these handles but bar closing them there
isn't much one can do to interfere with that of another sub
interpreter. In mod_wsgi all three are actually replaced with special
objects which either disable use of them (stdout/stdin), or maps it to
Apache error log (stderr). Thus, if something screws with them, it
only mucks things up for itself as they are no longer linked to an
underlying shared resource such as a file descriptor.
Each sub interpreter also has its own copy of os.environ, although
Python does have an issue with potential leakage of environment
variables when os.environ is changed as it pushes changes to C layer
by also calling putenv(). This means that a sub interpreter created
subsequent to such a change would pick that value up rather than
seeing the original environment variables from when process started.
> memory etc. Forget web2py for a second. I can right
> now write a pylons application that overwrite the ram memory of any
> other pylons application running under mod_wsgi on the same system. I
> teach how to do this stuff in my network programming and host
> security classes.
interpreter. If running in distinct sub interpreters, as above, it
can't be done unless code is deliberately malicious and uses ctypes or
a custom C extension module to inflict damage on other sub
interpreters.
> If you run multiple programs in the same address space there is no
> protection! The only protection is trust that the different
> components have some good manners.
wouldn't use embedded mode of mod_wsgi, similarly why you should never
use mod_python. The whole point of the distinct daemon mode feature of
mod_wsgi is to farm off applications to their own processes. Thus a
process could be given to each user. If a user then wants to be
malicious and screw up their own applications running in separate sub
interpreters of the same process then that is their own problem.
> What you said about apache running form "/" is not true.
assuming that?
> This is
> against basic security rules since it opens the doors to the entire
> filesystem from potential break-ins. Apache apps should run jailed
> using chroot. You should chroot to the location where web2py is
> installed when running web2py with Apache.
> Reference:http://www.faqs.org/docs/securing/chap29sec254.html
> Other frameworks may have problems with chroot.
Apache do not place themselves into any sort of chroot and so they
typically will run with the current working directory as '/'. This
depends on the platform though. On MacOS X it is often set to be '/
Library/WebServer'.
In other words, you would for typical Apache distributions have to do
this sort of isolation yourself as it doesn't come out of the box.
> Using mod_proxy of mod_rewrite provide the most separation because
> web2py apps will run in its own address space. That is webfaction
> runs most python web applications. That is how I suggest running
> web2py, I yet have to see a performance penalty.
like mechanisms.
Graham
Massimo Di Pierro
I spend my last 2 hours doing what you suggested:
http://mdp.cti.depaul.edu/examples/static/web2py_src_v1.23b.zip
which requires some more testing. Not much changed in web2py
libraries but there were many changes in the admin interface to
access the proper paths.
Now I feel I wasted a lot of time since you are telling me that if I
run mod_wsgi in deamon mode I can set my own application folder and
that's all I wanted to do.
I am not a wsgi expert and what you said in this last email is very
valuable.
Perhaps I do not need version 1.23b anymore...
Massimo
Graham Dumpleton
> I think we are now on the same page. ;-)
>
> I spend my last 2 hours doing what you suggested:http://mdp.cti.depaul.edu/examples/static/web2py_src_v1.23b.zip
> which requires some more testing. Not much changed in web2py
> libraries but there were many changes in the admin interface to
> access the proper paths.
>
> Now I feel I wasted a lot of time since you are telling me that if I
> run mod_wsgi in deamon mode I can set my own application folder and
> that's all I wanted to do.
works or changes you have made, but if it was simple to do I can't see
it as being wasted. :-)
On a related issue over the setting of the home directory for the
application, if it can't be done already, you might look at the home
directory being able to be specified through the WSGI environment
passed to the application. Whether you can do this really depends on
whether you store that directory in a global variable or whether it is
always sourced from the dynamic per request configuration.
Being able to control certain aspects of an application through values
supplied in the WSGI environment can be quite powerful. One of the
best examples of that is with Trac, where its project directory can be
specified in that way. This allows with Apache at least, one to do
some quite dynamic stuff where the configuration is determined on the
fly based on URL used to access the application. For some examples of
this in action see latter examples in:
http://code.google.com/p/modwsgi/wiki/IntegrationWithTrac
At the worst, it allows the project directory to be specified in the
Apache configuration file using SetEnv.
Graham
carlo
the fact Massimo got to the ground of his framework technicalities.
Hoping you can have some more spare time between lessons as I do :-)
and after reading
http://www.modpython.org/pipermail/mod_python/2006-December/022841.html
I would like to ask you:
- what prevents mod_proxy/mod_rewrite from being widely used by
hosting companies to offer Python frameworks?
- should we look forward to mod_wsgi as the right key to have Python
hosting wide spreading? Why this solution should be better?
Graham Dumpleton
> just to say your discussion is very valuable and I appreciated a lot
> the fact Massimo got to the ground of his framework technicalities.
>
> Hoping you can have some more spare time between lessons as I do :-)
> I would like to ask you:
>
> - what prevents mod_proxy/mod_rewrite from being widely used by
> hosting companies to offer Python frameworks?
especially in truly low cost commodity web hosting they aren't going
to want to allow the user to be the one dictating what processes are
being run and how if they do allow them. Instead the web hosting
company is going to want to very strictly control how processes get
started and have automated mechanisms in place to kill them easily
when they are idle and not have some user system restarting them
automatically. If they can't control things well, they can't achieve
anything near the site densities they are used to with PHP which means
their cost overheads are much more due to needing more hardware for
the same number of sites. Anyway, that is a very simplistic
description, there is actually a lot more to it than just that such as
issues around load balancing across multiple hosts which makes user
controlled systems not practical.
In practice, with Python they will never achieve the same site
densities as PHP anyway, this is because Python isn't purpose built
for this sort of environment and framework writers for Python aren't
helping by building systems that are so memory hungry. Thus, if they
are getting enough business from PHP sites why would they want to also
support Python at the really low end where it would only serve to trim
their margins.
> - should we look forward to mod_wsgi as the right key to have Python
> hosting wide spreading?
transient daemon processes. Even with that, they may not want to use
it because it can't replace FASTCGI solutions for other programming
languages and as a result those which use FASTCGI may be content to
stick with FASTCGI for Python hosting even though so many people seem
to have trouble with it. In other words they may not want to run
mod_wsgi along size FASTCGI.
> Why this solution should be better?
answer for Python at least. Rather than me try and go into it, you may
want to read some of my blog entries at:
http://blog.dscpl.com.au
Graham
Massimo Di Pierro
would you write an howto mod_wsgi deamon mode on AlterEgo?
Massimo
Graham Dumpleton
http://code.google.com/p/modwsgi/wiki/ConfigurationGuidelines
in conjunction with the rest of that document of course. Also see
documentation for related directives in:
http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives
Graham