How to embed django into a custom web server

[Jim]

Hi,

I'm experimenting with building a web server with embedded python and
I'd like to setup and include Django. The server works by loading a
main python script into memory at startup and then script registers a
callback routine to handle incoming messages. The script looks
something like this:

import MyWebServer

# Initialization code

def my_callback(connection_name, request, size):
MyWebServer.sendReply(connection_name, 'hello world', 11)

MyWebServer.set_callback(my_callback)


First of all, does this seem like a reasonable approach? Second, what
do I need to do to setup the Django environment using this
architecture?

Thanks in advance,
Jim

[Graham Dumpleton]

Make your web server support WSGI API standard and then you can just
use Django WSGI interface. The WSGI specification is documented at:

http://www.python.org/dev/peps/pep-0333/

Graham

[Jim]

Hi Graham,

Thanks for your response. I've taken a quick look over the WSGI spec
and it looks good. I have a couple questions though and I apologize
if they've already been answered in the forum or in the Django
documentation ....

1) Based on your suggestion to use WSGI, I'm assuming Django already
implements WSGI, right?
2) If I implement WSGI, what is the best way to test compliancy? Is
there a testing framework? If not, do you think that using a
compliant framework like Django would be sufficient?
3) My web server has an asynchronous, persistent socket
architecture. Do you see any problems between this architecture and
WSGI?

Thanks!


On Dec 12, 6:08 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:

[Graham Dumpleton]

Jim wrote:
> Hi Graham,
>
> Thanks for your response. I've taken a quick look over the WSGI spec
> and it looks good. I have a couple questions though and I apologize
> if they've already been answered in the forum or in the Django
> documentation ....
>
> 1) Based on your suggestion to use WSGI, I'm assuming Django already
> implements WSGI, right?

Yes. See documentation.

http://code.djangoproject.com/wiki/ServerArrangements

There are links to various places which describe setting it up to work
with WSGI capable hosting mechanisms.

> 2) If I implement WSGI, what is the best way to test compliancy? Is
> there a testing framework? If not, do you think that using a
> compliant framework like Django would be sufficient?

For a WSGI validator, see:

http://docs.python.org/library/wsgiref.html#module-wsgiref.validate

Trying it on a large framework is always a good test, but start with
some simple hello world applications first.

Also look at implementation of:

http://docs.python.org/library/wsgiref.html#module-wsgiref.simple_server

There is also the Paste WSGI server and CherryPy WSGI server you can
look at.

> 3) My web server has an asynchronous, persistent socket
> architecture. Do you see any problems between this architecture and
> WSGI?

Yes. WSGI and Python web applications in general are not suitable for
use on an asynchronous web server. This is because when a request is
being handled the server can't be doing anything else. Thus, if you
expect concurrency within a single process, you would need to use
threads. Other option is to run multiple processes to handle requests.

Why are you developing your own web server in the first place when
there are already so many options available?

Graham

[Jim]

Thanks for the good info. Let me explain why I would want to develop
my own web server... I've got a real time, asynchronous server that
uses a proprietary binary protocol to monitor remote systems using a
push model, and I'd like to expose the incoming data over HTTP using a
web app, also using a push model. The existing server is multi-
threaded and already uses embedded python so I thought I'd look into
implementing a limited web server and run the web server in the same
process as the real-time server. The server is a windows based server
that is built with WIN32 IO completion ports.

So I'm concerned about the multi-threading issues you've brought up
here and I'm worried that I don't fully understand all of the pitfalls
involved in integrating Django using WSGI. If I use standard
synchronization methods such as mutexes, critical sections and
reentrant code, are there any fundamental reasons why I can't make
this work?

Also, I *am* looking at using an available web server to accomplish
this task, but I wanted to look into this first.


On Dec 13, 3:40 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>

[Graham Dumpleton]

On Dec 14, 5:29 pm, Jim <jameswhetst...@comcast.net> wrote:
> Thanks for the good info.  Let me explain why I would want to develop
> my own web server... I've got a real time, asynchronous server that
> uses a proprietary binary protocol to monitor remote systems using a
> push model, and I'd like to expose the incoming data over HTTP using a
> web app, also using a push model.  The existing server is multi-
> threaded and already uses embedded python so I thought I'd look into
> implementing a limited web server and run the web server in the same
> process as the real-time server.  The server is a windows based server
> that is built with WIN32 IO completion ports.
>
> So I'm concerned about the multi-threading issues you've brought up
> here and I'm worried that I don't fully understand all of the pitfalls
> involved in integrating Django using WSGI.  If I use standard
> synchronization methods such as mutexes, critical sections and
> reentrant code, are there any fundamental reasons why I can't make
> this work?

If using Django 1.0, as long as your code uses thread synchronisation
mechanism as necessary, then should be okay. The only example I have
seen illustrating ongoing threading concerns with Django 1.0 relates
to a use case lying outside of the normal in respect of templates.
That is, if you use Django in the standard documented ways then
guidance seems to be that it should have no problems.

Graham