Time to deprecate mod_python?
Graham Dumpleton
installation. This is done at some cost though because of mod_python
being broken in various ways. Specifically, the following issues exist
when both mod_wsgi and mod_python loaded at same time, because of
mod_python being broken.
1. When mod_python initialises Python and itself, it doesn't use
Python simplified GIL APIs and creates separate thread state objects
instead. Because of this, mod_python will fail assertion checks
because of incorrect use of Python C APIs, if Python is ever built
with --with-pydebug option to configure. The mod_python package also
doesn't release the GIL after it is initialised in Apache parent
process, with it only being released in context of Apache worker
processes. The mod_wsgi package in order to be able to coexist, has
had to follow this flawed model and so mod_wsgi also doesn't use
Python simplified GIL APIs correctly for certain things in main
interpreter during initialisation. Thus it also fails when
--with-pydebug used to configure Python. This flawed behaviour can
also cause problems with some third party C extension modules.
2. Because mod_python is responsible for initialisating main
interpreter, the mod_wsgi package has to defer to mod_python any
subsequent destruction of the main interpreter. As it is, mod_python
doesn't actually destroy main interpreter on process shutdown. This
means that any Python threads running in main interpreter aren't
cleaned up properly, nor any atexit registered callbacks triggered.
That mod_python doesn't properly destroy Python main interpreter in
Apache parent process on Apache restart also causes some problems with
mod_python and mod_wsgi modules not necessarily unloading properly,
thus forcing a full Apache stop if wanting to upgrade mod_wsgi module
to new version. Not cleaning up Python in Apache parent process also
may result in additional memory leakage on top of normal memory leaks
Python has if Python was destroyed properly.
In the early days, being able to coexist with mod_python was seen as
required, else no one would try mod_wsgi. I feel it is about time with
mod_wsgi 3.0 to say enough is enough and call mod_python for what it
is, broken, and rule that mod_wsgi 3.0 cannot be used at same time as
current versions of mod_python. Intent then would be to have mod_wsgi
check if mod_python also loaded and if it is, refuse to initialise and
thus stop Apache from starting until mod_python is removed.
Having done that, can then fix mod_wsgi to use Python C APIs for
simplified GIL state management and avoid the problems it causes. With
mod_python out of the picture, can then also look at delaying
initialisation of Python interpreter to the Apache worker processes
and mod_wsgi daemon process. That is, Python would not be initialised
in Apache parent process. Being able to do this would avoid memory
leaks that Python exhibits when it is destroyed and reinitialised
within the context of same process. See:
http://code.google.com/p/modwsgi/issues/detail?id=99
Delaying initialisation and not doing it in parent is seen as
reasonable as the time taken to initialise main interpreter isn't that
great, namely in order of 10ms for modern Intel chips. Note that
Python shared library would still get linked into Apache parent
process, only initialisation of Python interpreter would be delayed.
There may possibly be some other benefits that could come from
delaying initialisation. Namely, the ability for daemon process groups
to have different settings for PYTHONHOME, PYTHONOPTIMIZE etc.
Overall, the only problem I can see coming from this is people who are
using non content handlers in mod_python to do strange stuff, as they
wouldn't be able to use mod_python to do that anymore. In mod_wsgi
there already is support for access, authn and authz hooks, albeit
with different APIs for usage in line with new Apache authnz
providers. That just leaves people who use header parse handlers and
fixup handlers to do fiddles on Apache request details. For a lot of
these, mod_headers, mod_rewrite or mod_env could probably do the same
thing, but possibly not everything. if there are strong enough
arguments for mod_wsgi providing some sort of equivalent for doing
these things, would entertain it, but one has to realise that people
needing that sort of functionality would be very few.
Anyway, if you have followed along with my ramble, can anyone see any
reason why mod_wsgi shouldn't ditch the goal of happy coexistence with
mod_python? Note that this wouldn't preclude mod_python and mod_wsgi
working together again in the future if mod_python is fixed to do
things correctly. Frankly though, am not sure if mod_python will ever
be fixed completely as at the moment I am the only one doing any sort
of patching of it.
Comments?
Graham
Clodoaldo Pinto Neto
Apart from not being able to run legacy applications, no. What to do?
Use one server for mod_python and another for mod_wsgi? Given today's
prices that would not be all that expensive if i don't count the
doubled server maintenance work time.
Do you plan to break the coexistence in 3.0? 2.x would be maintained
condescending with mod_python? What is the risk, if any, of people not
adopting mod_wsgi because of its incompatibility with mod_python? What
would be the cost of a compatibility mode compile option? I imagine it
would be difficult to maintain two code paths.
Although my concerns I'm all for making Apache/mod_wsgi the ultimate
web python choice.
Regards, Clodoaldo
Graham Dumpleton
> 2008/10/14 Graham Dumpleton <graham.d...@gmail.com>:
>> Anyway, if you have followed along with my ramble, can anyone see any
>> reason why mod_wsgi shouldn't ditch the goal of happy coexistence with
>> mod_python?
>
> Apart from not being able to run legacy applications, no. What to do?
> Use one server for mod_python and another for mod_wsgi? Given today's
> prices that would not be all that expensive if i don't count the
> doubled server maintenance work time.
>
> Do you plan to break the coexistence in 3.0? 2.x would be maintained
> condescending with mod_python?
Would only do this in mod_wsgi 3.0. Version 2.X would still be
maintained and important/easy fixes backported as necessary.
The error message that mod_wsgi 3.0 would produce if mod_python also
loaded at same time could even explicitly state to use older 2.X
versions if need support for mod_python at same time.
Having to use 2.X shouldn't be a problem for people as long as
important fixes made to it, as the WSGI interface doesn't change. All
you loose from using mod_wsgi 3.0 would be new features or
improvements.
> What is the risk, if any, of people not
> adopting mod_wsgi because of its incompatibility with mod_python?
The biggest obstacle at present that I can see in respect of people
using mod_wsgi over mod_python is that the Django folks still push
mod_python as the preferred option. That and a few web pages and
people in IRC channels still referring to mod_wsgi as young, immature
and potentially full of bugs. I ask, how many years and how many major
sites are needed before people will accept it is working fine. :-(
Other major Python web application frameworks openly push WSGI as one
of the options, if not preferred option. Beyond that, any stuff which
is bound to mod_python either has little mindshare so as not to be
worried about, or is a legacy application for which there are most
likely alternatives.
Of course, this may be fine for people starting on new applications,
who can choose, but obviously a problem for all those old in house
applications that use mod_python.
> What would be the cost of a compatibility mode compile option? I imagine it
> would be difficult to maintain two code paths.
If I didn't want to play with the idea of pushing Python
initialisation into child processes, probably not that hard to
maintain support for mod_python. I would though change it so that it
dynamically detected whether mod_python was used and only include
compatibility hacks/wrong behaviour if mod_python being used. Even
with an option to defer Python initialisation to child processes being
optional, albeit disabled when mod_python used, supporting mod_python
still probably isn't that hard, but just would prefer to draw a line
and not have to have the hacks and complicate the code.
The only other option would be to fix mod_python at same time and say
that if you want mod_python at same time, to use mod_python 3.4 (not
yet available at this stage obviously). Even if mod_python code fixed
in subversion repository, not sure could even muster a release as
don't even have enough active developers on mod_python at the moment
so as to get the required votes to approve it. Would need to drag some
of the prior developers on project management committee out of slumber
to do it.
Graham
MilesTogoe
word - probably need a good post on the Django community blog page -
then maybe some patches to their docs.
Also some info to the major hosting services (ie webfaction, linode,
slicehost...) with a press release for them to send to customers and
docs on how users should install or use mod_wsgi.
There will be a surge of questions so best to have faq ready.
MilesTogoe
> word - probably need a good post on the Django community blog page -
> then maybe some patches to their docs.
> Also some info to the major hosting services (ie webfaction, linode,
> slicehost...) with a press release for them to send to customers and
> docs on how users should install or use mod_wsgi.
> There will be a surge of questions so best to have faq ready.
>
Oh and I meant to add that it would be good to have a msg out to all the
user groups - I know the Utah user group has been trying to have a wsgi
talk
Graham Dumpleton
I think you may be overestimating how many people would be affected,
or at least how many would need both modules. Most cases I have seen
where people have both loaded are people who know no better and just
enabled lots of Apache modules for no reason, or moved to mod_wsgi and
didn't bother disabling mod_python at the same time. In both case they
don't generally use both.
As for a FAQ entry, there is already stuff in FAQ and documentation
warning about using mod_python and mod_wsgi together and people seem
not to read that based on number of problems one sees on various IRC
channels deriving from them doing just that.
Graham
Carl Nobile
Would it be possible to detect in code if mod_python has fixed their issues? If so you could do this test in mod_wsgi then continue to load mod_wsgi based on weather or not mod_python became compliant with the newer GIL calls.
-Carl
gert
Mod_python is like smoking a cigarette, you know its bad but just keep
doing it because everybody ells is doing it and never quit what you
would probably never had be doing in the first place so others would
find you a cool guy :-)
If you want to smoke carrots instead of cigarettes you plant carrots
in your garden. Not manipulated there genes in some very complicated
way so they look like cigarettes but are actually taste like
carrots.
So forget about mod_python and focus on http://en.wikipedia.org/wiki/Carrot
:-)
Graham Dumpleton
> Graham,
>
> Would it be possible to detect in code if mod_python has fixed their issues?
> If so you could do this test in mod_wsgi then continue to load mod_wsgi
> based on weather or not mod_python became compliant with the newer GIL
> calls.
As mod_python code is now, no. Since though I would likely be the
person to change mod_python to fix it, I can add a distinguishing
marker.
What I was intending to do was modify code in mod_python that says:
const char *userdata_key = "python_init";
...
apr_pool_userdata_get(&data, userdata_key, s->process->pool);
if (!data) {
apr_pool_userdata_set((const void *)1, userdata_key,
apr_pool_cleanup_null, s->process->pool);
return OK;
}
Such that instead of setting its marker to '(const void*)1', to set it
to a marker value which is the version string, or combined version
number as integer.
Thus, mod_wsgi could look for the marker and if != '(const void*)1'
assume that mod_python has been fixed. If need be it could look
further at the value of the marker if mod_python is changed again
later and mod_wsgi has to adapt differently again.
This will solve the GIL locking compliance issues, but if mod_wsgi
were to go to deferred initialisation of Python in child processes,
using mod_python at same time will obviously disable that and it would
return to initialisation in parent process as mod_python will take
control. Unless I at the same time rewrite mod_python to also do
deferred initialisation and make mod_wsgi take precedence over
mod_python.
So, yes there are ways and have been investigating them. It is just a
pain to have to maintain such fiddles. Unfortunately I probably will
not have a choice and will have to leave with such fiddles.
Graham
Manuzhai
<graham.d...@gmail.com> wrote:
> At the moment mod_wsgi will coexist with mod_python in same Apache
> installation. This is done at some cost though because of mod_python
> being broken in various ways. Specifically, the following issues exist
> when both mod_wsgi and mod_python loaded at same time, because of
> mod_python being broken.
As someone who currently runs mod_python and mod_wsgi in the same
Apache, it would be very helpful if there was a package that I could
use to morph WSGI requests into a mod_python-like request object, so
there's minimal porting.
Cheers,
Dirkjan
Graham Dumpleton
It isn't necessarily that simple, as mod_python provides a lot more
than just a wrapper for Apache request object.
I haven't touched it for a long time as no one was interested, but for
work in progress see:
http://modwsgi.googlecode.com/svn/sandbox/ap_swig_py/
One must use embedded mode, but idea is that you set:
WSGIApacheExtensions On
Then one can access 'apache.request_rec' from WSGI environment and use
that as input handle for SWIG binding to Apache request object.
I don't have an example of machine I am on at the moment, but will dig
one up tomorrow.
There was still a reasonable amount of work still to go on it though.
Graham
> Cheers,
>
> Dirkjan
>
> >
>