Win32 build

[Adal Chiriliuc]

Hello.

I've attached a patch which fixes two compilation errors in Visual
Studio 2005. The first one (in pre C99 variable declarations must be
right at the start of a function) was really hard to track down, the
error message didn't make any sense.

Unfortunately the mod_wsgi I've built crashes under Apache 2.2. It
could be caused by the compilation settings I used, by the fact the
the Apache libs are from a default Visual Studio 6.0 build while the
Apache binaries were from a Visual Studio 2005 build, or maybe
mod_wsgi itself has a problem. I'll try to solve this.

http://adal.chiriliuc.com/temp/mod_wsgi.c.diff

[Graham Dumpleton]

On 08/07/07, Adal Chiriliuc <adal.ch...@gmail.com> wrote:
>
> I've attached a patch which fixes two compilation errors in Visual
> Studio 2005. The first one (in pre C99 variable declarations must be
> right at the start of a function) was really hard to track down, the
> error message didn't make any sense.

Location of variable definitions fixed in revision 341 in source
repository. The other section you note had already been changed in
revision 339 (one after what you used), for other reasons.

> Unfortunately the mod_wsgi I've built crashes under Apache 2.2. It
> could be caused by the compilation settings I used, by the fact the
> the Apache libs are from a default Visual Studio 6.0 build while the
> Apache binaries were from a Visual Studio 2005 build, or maybe
> mod_wsgi itself has a problem. I'll try to solve this.

I'll reply in more detail about Win32 platform issues later as about
to go out for half a day.

Graham

[Graham Dumpleton]

On 08/07/07, Adal Chiriliuc <adal.ch...@gmail.com> wrote:

> Unfortunately the mod_wsgi I've built crashes under Apache 2.2. It
> could be caused by the compilation settings I used, by the fact the
> the Apache libs are from a default Visual Studio 6.0 build while the
> Apache binaries were from a Visual Studio 2005 build, or maybe
> mod_wsgi itself has a problem. I'll try to solve this.

I haven't personally used Windows for many years, so don't have direct
experience of the possible problems on that platform. All I can do at
this point is relay what I know may cause problems based on experience
with seeing problems and solutions offered up for mod_python.

First off, I would suggest not trying to use mod_python and mod_wsgi
at the same time initially. I have absolutely no idea what might
happen if you try them together on Windows. Although, it may be
helpful in just ensuring you can get Python with Apache to work in the
first place by getting mod_python working first on your machine used
its precompiled installers. Once can get mod_python working, can then
take out its LoadModule line and see how we go with mod_wsgi.

Second, is the crash you see with mod_wsgi as soon as you start
Apache, or only when the first request arrives to Apache, or to a WSGI
application hosted by mod_wsgi. If it is immediately Apache is
started, then advice given in:

http://www.modpython.org/pipermail/mod_python/2006-August/021754.html

may be of use. In particular, when the Apache service is run it may
run as a distinct user and as such, that user must have appropriate
access rights to the various Microsoft C/C++ runtime libraries, plus
mod_wsgi Apache module itself. I unfortunately don't know what the
implications will be if Apache, Python and/or mod_wsgi were compiled
against different versions of the Microsoft C/C++ runtime library.

Third, problems can arise where Python was installed as a yourself and
not as administrator. The problem here is that the Windows registry
entries are installed for you and not globally. This means that when
Apache user tries to initialise Python it can't actually find the
Python modules, or if it can, the access permissions are wrong such
that a different user can't read them. There is a bit of information
about registry problems in:

http://www.modpython.org/pipermail/mod_python/2006-September/021979.html

Starting with that advice, I really need to know go off and construct
a way of building mod_wsgi on Windows. This has been on my todo list
for a while, and the person who usually does the mod_python builds for
Windows has offered to help me come up with some precompiled binaries
for the platform. The approach being taken to build it on Windows will
be to treat it like a Python C extension module. Thus, one is using
the Python distutils infrastructure to trigger the compilations. This
is in effect how mod_python Windows builds are done.

Anyway, that looks like my task for the rest of today now. :-)

Graham

[Graham Dumpleton]

On 08/07/07, Graham Dumpleton <graham.d...@gmail.com> wrote:
> Starting with that advice, I really need to know go off and construct
> a way of building mod_wsgi on Windows. This has been on my todo list
> for a while, and the person who usually does the mod_python builds for
> Windows has offered to help me come up with some precompiled binaries
> for the platform. The approach being taken to build it on Windows will
> be to treat it like a Python C extension module. Thus, one is using
> the Python distutils infrastructure to trigger the compilations. This
> is in effect how mod_python Windows builds are done.
>
> Anyway, that looks like my task for the rest of today now. :-)

Hmmm. I looked at what is required and decided that there is no way
with my knowledge and lack of access to a Windows box, to come up with
something myself that is going to work. Thus, on to plan B.

Plan B entails going to:

http://archive.apache.org/dist/perl/win32-bin/

and downloading the file:

http://archive.apache.org/dist/perl/win32-bin/apxs_win32-0.3.tar.gz

This is a version of apxs which apparently works on Windows. The
instructions for installing it can be seen in:

http://archive.apache.org/dist/perl/win32-bin/apxs_win32-0.3.readme

You will need to have Perl on the machine though as this version of
apxs is implemented in Perl.

Presumably, once you have this installed, you will be able to go something like:

apxs_win32 -I/some/path/to/python/include -c mod_wsgi.c
/some/path/to/python/lib/python25.lib

Not knowing where or how Python installs stuff, not sure what the
include and lib stuff should be set to, so hopefully you know or
someone else does. Not sure if any special -D defines might be
required, eg. -DWIN32.

if you could try this and see if it works would be great, as sounds
like a simpler way of building it.

Graham

[Adal Chiriliuc]

First some context. I'm currently working on a Pylons applications. At
first I planned on deploying it on Linux, but after a little try I
realized that I know very little about this platform and it will take
me probably half a year to become comfortable with it. And I also have
one piece of software used by the web application which runs only on
Windows (3ds max), so I will need a Windows box anyway. Then I did
some research and found out that Apache on Windows XP/2003 is very
stable. So I'll stick with this combination for a while. I also looked
at lighttpd, but it's Windows build seems a little bit experimental,
and I had problems with it.

I did run the Pylons applications on Apache through mod_python and
Python 2.5 on my machine, worked perfectly.

The crash yesterday (on the first request, not on load) was generated
by having a VC80 (Visual Studio 2005) build of mod_wsgi against a VC71
(Visual Studio 2003) build of Python. You can mix Microsoft C Runtime
libraries, but only if you don't pass an object allocated by one of
them to the other (like FILE*). Unfortunately the Python C/API uses
objects like these.

I've replaced the standard Python I had with one built using VC80 like
mod_wsgi. I only replaced python25.dll, and not all the other modules
like _socket.pyd. With this change mod_wsgi was able to run my Pylons
application.

But, it still crashed, something like once every 5-10 requests (rough
guess). It was either on GET or POST request, and on different URLs.
It could still be the mixed compiler issue.

What I need to do right now is to build mod_wsgi using VC71 like
Python 2.4/2.5 is built.

I looked at the mod_python build process. Looks a little complicated
to me since it also has to install the pure Python library. mod_wsgi
for Windows will only need to contain mod_wsgi.so for a particular
Apache/Python combination and maybe some readme files. mod_python also
tries to automatically add a LoadModule directive into Apache.

To build it myself I just created a Visual Studio project and solution
file (mod_python also has a .vcproj file in it's source distribution).
This is not suitable in general since you would need to adjust the
Apache/Python include/lib paths which are set inside them.

And anything requiring installing something (like Perl) isn't very
attractive.

I'll create a Python script in which you can enter the locations of
Apache/Python from your machine and which will automatically generate
the required .vcproj files for each combination based on a template
file and a batch file to compile them. I think this is the easiest
solution. This has the benefit that you can load such a solution in
Visual Studio and compile it/debug it from there (I believe this is
the reason that mod_python has a .vcproj file).

The configuration will be something like:
VISUAL_STUDIO = "C:/Program Files/Microsoft Visual Studio .NET 2003"

PYTHON23 = None
PYTHON24 = "C:/Python24"
PYTHON25 = "C:/Python25"

APACHE20 = "C:/Apache20"
APACHE22 = None

And the script will figure out what combinations to build.
These paths are required since in these folders we have the include/
lib files.

You can then see if this build method is suitable.

There is a small problem with Python 2.3, since it's build using
Visual Studio 6.0 (VC60) and it will require different template file.

[Adal Chiriliuc]

I've implemented a simple build system.

http://adal.chiriliuc.com/temp/win32.zip

The zip should be extracted inside the folder containing mod_wsgi.c.

But first, the SVN version of mod_wsgi doesn't build under Visual
Studio .NET 2003. All the variables named "environ" need to be renamed
to something else ("environment" works) since in stdlib.h "environ" is
a macro which resolves to a function declaration. For some reason this
is not a problem in Visual Studio 2005, even if it's header file looks
very similar.

Steps to build:

1. Install Microsoft Visual Studio .NET 2003
2. Install Python 2.4 and 2.5
3. Install Apache 2.0 and 2.2
4. Run generate.py
5. Run the generated build.bat
6. The new mod_wsgi_py*_apache*_Release folders will each contain the
corresponding mod_wsgi.so binary

If all of the above are installed to the default locations on a 32-bit
Windows, no changes should be required to the paths embedded in
generate.py. Otherwise please adjust.

Python 2.3 is not supported yet (it would take 1-2 hours to add
support for it).

Unfortunately mod_wsgi built for Apache 2.2/Python 2.5 with Visual
Studio 2003 (like Python 2.5 is) still crashes once every 2-10
requests. The crash seems to happen in Adapter_dealloc at the first
line (apr_brigade_destroy(self->bb)). Skipping that line without
executing it still causes a crash, even if "self" looks ok. I will
probably need a debug build of Apache for further investigations.

[Graham Dumpleton]

That apr_brigade_destroy() stuff was a recent change, so possible I
have got something wrong in it.

Try setting the directive:

WSGIOutputBuffering On

This will not stop the call to that function occurring, but will stop
the code which makes use of it from running. This may or may not be
enough to stop whatever the problem is. In the mean time I'll check
through the code again to make sure I am using it correctly.

Thanks.

Graham

[Graham Dumpleton]

Also try latest version of mod_wsgi from subversion repository. I have
disabled the use of buckets for response content on Win32 for now so
you can see whether more conventional way of returning a response is
working or not. The updated code also renames environ to vars.

Graham

[Adal Chiriliuc]

Setting WSGIOutputBuffering to On didn't solve the problem.

Then I updated mod_wsgi.c to the latest subversion version, removed
the WSGIOutputBuffering On directive and it worked.

I tried about 1000 requests, and it didn't crash.

I've also tested these combinations, and all worked:
- Apache 2.0, Python 2.4 (*)
- Apache 2.2, Python 2.4 (*)
- Apache 2.0, Python 2.5
- Apache 2.2, Python 2.5
- Apache 2.2 built with Visual Studio 2005 (http://
www.apachelounge.com), Python 2.5

When using Python 2.4 my Pylons application failed to load. The log
said that Python failed to import "paste.deploy". So I tried using the
simple hello world WSGI application from
http://docs.pythonweb.org/display/pylonscookbook/mod_wsgi+and+workingenv.py

[Graham Dumpleton]

On 11/07/07, Adal Chiriliuc <adal.ch...@gmail.com> wrote:

> Unfortunately mod_wsgi built for Apache 2.2/Python 2.5 with Visual
> Studio 2003 (like Python 2.5 is) still crashes once every 2-10
> requests. The crash seems to happen in Adapter_dealloc at the first
> line (apr_brigade_destroy(self->bb)). Skipping that line without
> executing it still causes a crash, even if "self" looks ok. I will
> probably need a debug build of Apache for further investigations.

There was an issue in the code with how apr_brigade_destroy() was
called from the destructor of the Adapter object. However, I can't see
how it would have been able to be triggered unless the WSGI
application you were testing with was caching the 'write()' method
returned from 'start_response()' beyond the life of the request in
some way.

In other words, if 'write()' was cached then the Adapter object
instance would still be valid until the cached reference to 'write()'
had been replaced. Only at that point would the destructor be called
and apr_brigade_destroy() called. Problem is that at that point, the
request pool would already have been destroyed and the brigade
automatically destroyed. Thus, would be calling apr_brigade_destroy()
on invalid memory.

I have now fixed the code for this issue and committed it back to the
repository. Use of buckets is enabled again for Win32. To disable
again, change:

#if AP_SERVER_MAJORVERSION_NUMBER >= 2
#define MOD_WSGI_WITH_BUCKETS 1
#endif

to:

#ifndef WIN32
#if AP_SERVER_MAJORVERSION_NUMBER >= 2
#define MOD_WSGI_WITH_BUCKETS 1
#endif
#endif

Ensure you completely stop Apache before installing new module, just
to be sure it picks up updated module.

This issue would have affected UNIX systems as well if application was
similarly doing something to cause Adapter object to live beyond end
of request.

Graham

[Adal Chiriliuc]

First I tried a simple Hello World WSGI application with bucket
brigades enabled and without applying the latest fix. It didn't
crashed. So either it didn't cache the write() method like you said,
or it didn't do some other thing that Pylons does.

I than updated to the latest subversion version and the problem was
fixed. The Pylons application didn't crash in 3000 requests.

It shouldn't be to hard to implement a WSGI middleware app to check if
write() lives beyond a request. I'll try doing this.

On Jul 12, 7:15 am, "Graham Dumpleton" <graham.dumple...@gmail.com>
wrote:

[Graham Dumpleton]

With latest version, do ensure that you don't still have
WSGIOutputBuffering set to On as that would also avoid the problem
entirely as the bucket brigade is now only created and used if
buffering is Off (the default).

Graham

[Adal Chiriliuc]

Yes, it works without the WSGIOutputBuffering directive in the
configuration file. I removed it immediately after testing with it.

On Jul 12, 1:49 pm, "Graham Dumpleton" <graham.dumple...@gmail.com>
wrote:

> With latest version, do ensure that you don't still have
> WSGIOutputBuffering set to On as that would also avoid the problem
> entirely as the bucket brigade is now only created and used if
> buffering is Off (the default).
>
> Graham
>

[Adal Chiriliuc]

I've tested the caching of Python objects with this simple WSGI
middleware:
http://adal.chiriliuc.com/temp/caching.py

The results are surprising. First, someone (Apache because of keep
alive?) can keep up to around ten start_response instances alive on my
machine (Win32, Apache 2.2.4, Python 2.5). The start_response objects
seem to be destroyed only when a new request arrives, never in
between.

Some requests (0 with mod_wsgi and 1,2 with mod_python) can outlive
even the next 200 requests.