Multiple PyISAPIe Instances (for Django)

602 views
Skip to first unread message

Randy Syring

unread,
May 10, 2008, 4:52:02 PM5/10/08
to pyis...@googlegroups.com
Hi again,

Ok, my goals are as follows, with the goals at the top being the more important/most efficient ones (at least IMO):
  1. Run multiple instances of Django on the same IIS server with only one instance of PyISAPIe.dll.  This would allow a system admin to setup PyISAPIe once and let non-admin users configure their python setup by using a specially named file in the root of the virtual directory or website.
    • Problem: os.environ['DJANGO_SETTINGS_MODULE'] is required to get Django running correctly.  However, this value is set at the application level.  Therefore, even if that value is set multiple times by Http.Isapi (the PyISAPIe script), Django only cares about the first instance.  It is therefore currently impossible, as best I understand, to run multiple instances of Django under the same PyISAPIe.dll instance.
  2. Run multiple instances of Django on the same IIS server with multiple instances of PyISAPIe.dll on IIS 5.1 or IIS 6 (without multiple application pools).  This still requires admin intervention for every website installed, but would allow development testing on Windows XP with virtual directories and would require less resources on IIS.
    1. Problem 1: when configuring IIS 5.1 in the same setup as noted below using virtual directories, the second application to be initiated doesn't work.
    2. Problem 2: when configuring IIS 6.0 in the same setup as noted below using websites that share an application pool, the system becomes unstable
  3. Run multiple instances of Django on the same IIS server (completed).  It requires a website or virtual directory for each django instances.
    • Each website or virtual directory will need to have its own PyISAPIe.dll and HTTP folder
    • my directory structure is as follows
      • c:\websites\django1
        • pyisapi
          • PyISAPIe.dll
          • HTTP (directory with HTTP files)
        • python
          • djtutorial1  (django project directory)
        • wwwroot
      • c:\websites\django2
        • pyisapi
          • PyISAPIe.dll
          • HTTP (directory with HTTP files)
        • python
          • djtutorial2  (django project directory)
        • wwwroot
    • my websites are as follows
      • django1.example.com
        • home directory is c:\websites\django1
        • Has its own application pool named django1 (this is required, if you try to run multiples instances of PyISAPIe in the same application pool the system becomes unstable and stops working)
      • django2.example.com
        • home directory is c:\websites\django2
        • Has its own application pool named django2
Ok, so the third goal has been accomplished as far as I can tell.  I was initially concerned that having multiple application pools might consume more resources than is necessary, bit it probably isn't an issue based on this comment:
Each application pool will consume resources, so i suppose that is the
disadvantage. You should however compare this to the advantages of
application pools.
10 application pools on a server is however not in any way overkill. One
application pool per website is not strange at all, instead it is a quite
good configuration (of course this depends on the website(s) and its web
applications).
You have too many application pools when the server is running short on
resources because the application pools consume too much resources.
--
Regards,
Kristofer Gafvert (IIS MVP)

found here: http://www.eggheadcafe.com/forumarchives/inetserveriis/Nov2005/post24143506.asp
However, I would still like to see the other two goals accomplished.  Goal #1 probably doesn't have anything to do with PyISAPIe as far as I can tell.  That is probably a Django issue which I have asked about here:

http://groups.google.com/group/django-developers/browse_thread/thread/dfd69014daa36672

The second goal, however, may be a PyISAPIe goal.  Would it possible to run multiple instances if PyISAPIe in the same application pool?  This would use less resources on IIS 6 and enable IIS 5.1 to have multiple Django applications using virtual directories.

Thoughts, comments, questions welcome.

Thanks.

-- 
--------------------------------------
Randy Syring
RCS Computers & Web Solutions
502-644-4776
http://www.rcs-comp.com

"Whether, then, you eat or drink or 
whatever you do, do all to the glory
of God." 1 Cor 10:31

Phillip

unread,
May 13, 2008, 5:45:01 PM5/13/08
to PyISAPIe
Randy,

Thanks for the detailed list; this really puts into perspective what
the important considerations are for getting PyISAPIe to be more
flexible with Django.

For now, I think I'll try to figure out why having multiple instances
of PyISAPIe in the same application pool causes trouble. Are the
worker processes just crashing on you when you request a page?

- Phillip

On May 10, 1:52 pm, Randy Syring <ra...@rcs-comp.com> wrote:
> Hi again,
>
> Ok, my goals are as follows, with the goals at the top being the more
> important/most efficient ones (at least IMO):
>
> 1. Run multiple instances of Django on the same IIS server with only
> one instance of PyISAPIe.dll. This would allow a system admin to
> setup PyISAPIe once and let non-admin users configure their python
> setup by using a specially named file in the root of the virtual
> directory or website.
> * Problem: os.environ['DJANGO_SETTINGS_MODULE'] is required to
> get Django running correctly. However, this value is set at
> the application level. Therefore, even if that value is set
> multiple times by Http.Isapi (the PyISAPIe script), Django
> only cares about the first instance. It is therefore
> currently impossible, as best I understand, to run multiple
> instances of Django under the same PyISAPIe.dll instance.
> 2. Run multiple instances of Django on the same IIS server with
> multiple instances of PyISAPIe.dll on IIS 5.1 or IIS 6 (without
> multiple application pools). This still requires admin
> intervention for every website installed, but would allow
> development testing on Windows XP with virtual directories and
> would require less resources on IIS.
> 1. Problem 1: when configuring IIS 5.1 in the same setup as
> noted below using virtual directories, the second
> application to be initiated doesn't work.
> 2. Problem 2: when configuring IIS 6.0 in the same setup as
> noted below using websites that share an application pool,
> the system becomes unstable
> 3. Run multiple instances of Django on the same IIS server
> (completed). It requires a website or virtual directory for each
> django instances.
> * Each website or virtual directory will need to have its own
> PyISAPIe.dll and HTTP folder
> * my directory structure is as follows
> o c:\websites\django1
> + pyisapi
> # PyISAPIe.dll
> # HTTP (directory with HTTP files)
> + python
> # djtutorial1 (django project directory)
> + wwwroot
> o c:\websites\django2
> + pyisapi
> # PyISAPIe.dll
> # HTTP (directory with HTTP files)
> + python
> # djtutorial2 (django project directory)
> + wwwroot
> * my websites are as follows
> o django1.example.com
> + home directory is c:\websites\django1
> + Has its own application pool named django1 (this
> is required, if you try to run multiples
> instances of PyISAPIe in the same application
> pool the system becomes unstable and stops working)
> o django2.example.com
> + home directory is c:\websites\django2
> + Has its own application pool named django2
>
> Ok, so the third goal has been accomplished as far as I can tell. I was
> initially concerned that having multiple application pools might consume
> more resources than is necessary, bit it probably isn't an issue based
> on this comment:
>
> Each application pool will consume resources, so i suppose that is the
> disadvantage. You should however compare this to the advantages of
> application pools.
> 10 application pools on a server is however not in any way overkill.
> One
> application pool per website is not strange at all, instead it is a
> quite
> good configuration (of course this depends on the website(s) and its
> web
> applications).
> You have too many application pools when the server is running short on
> resources because the application pools consume too much resources.
> --
> Regards,
> Kristofer Gafvert (IIS MVP)
>
> found here:
> http://www.eggheadcafe.com/forumarchives/inetserveriis/Nov2005/post24...
>
> However, I would still like to see the other two goals accomplished.
> Goal #1 probably doesn't have anything to do with PyISAPIe as far as I
> can tell. That is probably a Django issue which I have asked about here:
>
> http://groups.google.com/group/django-developers/browse_thread/thread...
>
> The second goal, however, may be a PyISAPIe goal. Would it possible to
> run multiple instances if PyISAPIe in the same application pool? This
> would use less resources on IIS 6 and enable IIS 5.1 to have multiple
> Django applications using virtual directories.
>
> Thoughts, comments, questions welcome.
>
> Thanks.
>
> --
> --------------------------------------
> Randy Syring
> RCS Computers & Web Solutions
> 502-644-4776http://www.rcs-comp.com

rcs_comp

unread,
May 13, 2008, 10:41:47 PM5/13/08
to PyISAPIe
On May 13, 5:45 pm, Phillip <phillip.sit...@gmail.com> wrote:

> For now, I think I'll try to figure out why having multiple instances
> of PyISAPIe in the same application pool causes trouble. Are the
> worker processes just crashing on you when you request a page?

Phillip,

Using the method described here:

http://groups.google.com/group/pyisapie/web/multiple-instances-of-django-on-iis-with-pyisapie

On Windows XP and IIS 5.1 with two virtual directories the virtual
directory which has it's URL called first displays appropriately.
When the url of the second virtual directory is requested, the request
completes from the browser's perspective, but nothing is returned. I
just get a completely white screen with no errors that I am aware of.

On Windows 2003 server with IIS 6.0 using two websites with the same
application pool I get a response from the first and second website.
However, the response from the second website is slow and eventually
that worker thread crashes.

If you need any more info, please let me know.

Phillip

unread,
May 15, 2008, 2:09:02 AM5/15/08
to PyISAPIe
Ok, I've figured out the problem, which I managed to overlook while
debugging and crashing for some time :)

The issue is with Python. The way PyISAPIe uses Python is meant to be
simple: if you have Python installed, PyISAPIe will find it and use it
as the interpreter. Nowadays this works well because python25.dll, the
Python core, is in the system32 folder. When a worker process starts
up and loads the extension, it sees a dependency for python25 and
loads it. Later on, when you load your *second* copy of the extension,
the dependency for python25 is already satisfied, so the currently-in-
memory copy is used, along with all its global interpreter data.

The reason the worker processes crashed is because the DLL init checks
to see if Python is initialized and loads some important stuff in
there that doesn't exist later for the second instance, since Python
was already loaded.

Alas, my initial advice to use multiple copies of the DLL only applies
to multiple application pools, because that's the only way to
guarantee one Python interpreter per process. There are ways around
this, such as embedding the Python interpreter into the PyISAPIe DLL,
but then it turns into a "special" distribution of Python just for
this app, which I'd like to avoid (although I have successfully done
just that with the python sources).

Sorry I didn't spot that caveat sooner!

- Phillip

On May 13, 7:41 pm, rcs_comp <rsyr...@gmail.com> wrote:
> On May 13, 5:45 pm, Phillip <phillip.sit...@gmail.com> wrote:
>
> > For now, I think I'll try to figure out why having multiple instances
> > of PyISAPIe in the same application pool causes trouble. Are the
> > worker processes just crashing on you when you request a page?
>
> Phillip,
>
> Using the method described here:
>
> http://groups.google.com/group/pyisapie/web/multiple-instances-of-dja...

rcs_comp

unread,
May 15, 2008, 3:12:49 PM5/15/08
to PyISAPIe

> The issue is with Python. The way PyISAPIe uses Python is meant to be
> simple: if you have Python installed, PyISAPIe will find it and use it
> as the interpreter. Nowadays this works well because python25.dll, the
> Python core, is in the system32 folder. When a worker process starts
> up and loads the extension, it sees a dependency for python25 and
> loads it. Later on, when you load your *second* copy of the extension,
> the dependency for python25 is already satisfied, so the currently-in-
> memory copy is used, along with all its global interpreter data.

Just for my own curiosity, is the "global interpreter data" the area
of memory where the django settings are stored (i.e. in the
environment)?

> The reason the worker processes crashed is because the DLL init checks
> to see if Python is initialized and loads some important stuff in
> there that doesn't exist later for the second instance, since Python
> was already loaded.
>
> Alas, my initial advice to use multiple copies of the DLL only applies
> to multiple application pools, because that's the only way to
> guarantee one Python interpreter per process. There are ways around
> this, such as embedding the Python interpreter into the PyISAPIe DLL,
> but then it turns into a "special" distribution of Python just for
> this app, which I'd like to avoid (although I have successfully done
> just that with the python sources).

If you embedded the Python interpreter into PyISAPIe, would that allow
you to run multiple instances of Django (or any other app that puts
values in the memory space shared by multiple processes) in a shared
environment without having to have the system admin configure a new
application pool for each instance? What would that do to
performance?

Obviously I lack the conceptual framework to understand exactly how
Python is interacting with PyISAPIe and IIS. PHP, which is what I
work with mostly, doesn't have a concept of "global interpreter
data." Each request only has access to its own variables regardless
of where they are stored (even in $_GLOBAL) so I have never before run
into an issue like I am having with Django and it's shared settings.

Obviously, I think this would best be solved at the Django level by
moving the settings into a non-global context. But I have put up two
tickets on the developers mailing list and have yet to get a single
response. I thought maybe it was because it was an IIS issue and they
are mostly Linux and Apache users. However, I came across a post by
someone that is having similar issues with Apache and FastCGI where
they have to create seperate application pools for each instance.

Thanks Phillip!

rcs_comp

unread,
May 25, 2008, 2:55:54 PM5/25/08
to PyISAPIe
Just a small update. I found out that IIS 5.1 can use "High
(Isolated)" for the Application Protection setting of a virtual
directory. This will allow multiple instances of Django on IIS 5.1
for testing purposes. More info can be found here:

http://groups.google.com/group/pyisapie/web/multiple-instances-of-django-on-iis-with-pyisapie

Phillip,

Can you comment any further on the possibility of modifying PyISAPIe
to accommodate multiple instances of django with just one global
PyISAPIe instance? Maybe you could so something like allow a WSGI
application to indicate to PyISAPIe that it needs its own instance of
the Python interpreter?

Thanks.

Phillip

unread,
May 29, 2008, 3:38:20 PM5/29/08
to PyISAPIe
Hi Randy,

Sorry for the delay, things have been a little hectic lately.

So, the Python global data problem might be solved by embedding the
interpreter into the DLL in such a way that we can run several
instances in the same process or app pool, having each with its own
Http package. I might even be able to use some tricks to do that
without comi

The global data I was talking about includes the global interpreter
lock (http://docs.python.org/api/threads.html) and the "main"
interpreter instance. The Python C API includes support for multiple
interpreters that provide mostly-isolated environments (if only it was
that simple), which I make use of, but right now I use one specific
interpreter for loading and initial request processing - this is where
two instances of PyISAPIe in the same process will clash. Each
interpreter stores things like the module search path. So I might be
able to fix this by creating interpreters with unique names.

About the GIL, I don't consider it too much of a problem right now
because other request threads run while data is sent to the client.
There is the more complicated issue of Django buffering output within
Python, which doesn't let other request threads run, but I'll save
that for later discussion.

Now, on to getting multiple instances of Django to work by compiling
the Python interpreter/DLL into PyISAPIe - it just won't work. We'll
get the ability to have multiple Python interpreters/DLL instances
coexist, but they will still use os.environ to read Django settings.
Under the hood, it's reading process-level data, so there's nothing we
can do that would allow multiple Django instances in the same process.

Hopefully that's all clear- let me know if it makes sense.

- Phillip

rcs_comp

unread,
May 30, 2008, 12:00:14 PM5/30/08
to PyISAPIe


On May 29, 3:38 pm, Phillip <phillip.sit...@gmail.com> wrote:
> Hi Randy,
>
> Sorry for the delay, things have been a little hectic lately.

No problem at all. I am just glad to have someone interested in
working through this stuff with me. I really appreciate your efforts
and feedback.

> Now, on to getting multiple instances of Django to work by compiling
> the Python interpreter/DLL into PyISAPIe - it just won't work. We'll
> get the ability to have multiple Python interpreters/DLL instances
> coexist, but they will still use os.environ to read Django settings.
> Under the hood, it's reading process-level data, so there's nothing we
> can do that would allow multiple Django instances in the same process.

So, if I understand you correctly, you are saying you can get multiple
interpreter instances to co-exist, but all of them would be running in
the same process? If that is the case, I agree that won't help with
getting multiple instances of Django running. As noted in the other
thread:

http://groups.google.com/group/pyisapie/browse_thread/thread/a01dd5d3ae5ede86

Its not just the DJANGO_SETTINGS_MODULE that is the problem. It would
appear that there are a lot of places that Django assumes one-instance-
per-process and so there is data and settings stored in the process
level modules. It looks like Django is limited to one-instance-per-
process framework for now until someone more familiar with the
framework and Python can make some modifications to the core.

> Hopefully that's all clear- let me know if it makes sense.

Most of it was clear as mud :) But in the end, I think its wise to
keep PyISAPIe as-is, using IIS application isolation as necessary,
until changes to Django can be made.

Thanks again.

Reply all
Reply to author
Forward
0 new messages