Building Django web application

[mguthrie]

I've been looking into Django for building something that is more web
application than it is website. I understand that Django has been
developed in a sort of CMS mindset but to date I haven't found any
reason why it couldn't create non-content centric web apps as well.

My project requirements are as follows:

1.) I need to be able to host this project for multiple clients. No
customization, just everybody using the same thing. Therefore ideally
they should all share the same codebase.
2.) Each client should have their own user table/authentication since
I want client A to be able to have a user named john.doe and client B
to as well. I would not mind if they shared the same table but they
needed to provide a client id so the login can differentiate.
3.) Media should be separate per client. Client A media should not be
mixed with Client B or vice versa.
4.) Database either needs to be single db per client or one large db
with multiple prefixed tables per client. So each client would get
their own users table (or shared table with client id field), tables
for data, etc.

I've researched the options available and here's what I've found
(correct me if I'm wrong):
1.) I can host each client separately by providing a different
<Location> for each and specifying a different settings file. This
should allow me to specify separate DB's, media locations, etc but I'm
concerned about the overhead of hosting them. I've read that for each
instance of Django requires another python interpreter. If that is
the case wouldn't I run out of RAM quickly hosting several clients?
Is there a way I can do this using only one instance of Django? If I
use Django with multiple <Location>'s per client would that be one
Django instance or multiple? Is mod_python not the way to go for this
project?
2.) I can use django.contrib.sites but every client shares the
authentication. Is there a way I can specify a client id to
distinguish Client A's john.doe from Client B's? If I do this can I
specify where media for either would go? How about they all share the
same DB but have different table prefixes?

I know it's a lot but I wanted to be as specific as I could so I don't
waste someone's time. Is Django probably the wrong framework for this
project? Should this be a Pylons/Turbogears thing or what?

All ideas/critiques/reworkings will be accepted. Thanks in advance.

-MG

[Colin Bean]

Django is quite capable of doing everything you described. You've
almost answered your own question because hosting separate django
instances under different <Location> directives (or virtual hosts)
seems like the ideal solution. I wouldn't worry too much about the
overhead of multiple python interpreters -- if python is using shared
libraries, they will use a little extra ram, but I doubt they will
exhaust your system resources before something else does. I don't
have any numbers to back this, but I've run several django instances
on a single host with no problem, and it seems to be fairly common
practice.

Colin

[mguthrie]

I had read in more than one place that a django instance can eat up to
10mb - 30mb of memory. I don't know whether that is fact or fiction
and I'm also confused by the term instance. Is an instance defined as
multiple different Django projects or a single Django project used
multiple times? If it is true that one Django project used across
multiple <Location>'s eats up that much memory I wouldn't able able to
host very many clients on a server.

I'm trying to devise my infrastructure ahead of time so I'm not stuck
with a finished application that can't be hosted for many clients.

Thanks for replying.

-MG

On Oct 10, 2:58 pm, "Colin Bean" <ccb...@gmail.com> wrote:

[Graham Dumpleton]

On Oct 11, 9:23 am, mguthrie <mitchguth...@gmail.com> wrote:
> I had read in more than one place that a django instance can eat up to
> 10mb - 30mb of memory.  I don't know whether that is fact or fiction
> and I'm also confused by the term instance.  Is an instance defined as
> multiple different Django projects or a single Django project used
> multiple times?  If it is true that one Django project used across
> multiple <Location>'s eats up that much memory I wouldn't able able to
> host very many clients on a server.
>
> I'm trying to devise my infrastructure ahead of time so I'm not stuck
> with a finished application that can't be hosted for many clients.

The comment by Colin about use of multiple sub interpreters not
requiring much more memory is true, but misleading. This is because as
you suspect the main issue is how much Django itself takes and not the
overhead of mod_python or Python sub interpreters.

The problem with using mod_python is that Django instances run in sub
interpreters of Apache child worker processes, thus, how much overall
memory used is in part dictated by which Apache MPM you are using and
how many processes Apache has been configured to use. To clarify,
Apache on UNIX is a multiprocess web server. Thus there isn't just one
instance of Django per site, but number of Apache child worker
processes multiplied by the number of sites.

If you are are using Apache prefork MPM, it is not uncommon to use a
configuration that sees minimum 10-20 processes and maximum of 100.
Thus, if you had 5 sites each running at minimal 20MB, worst case
where Apache had to create maximum number of processes to handle load,
would be 5*20*100 MB. A lot of Django sites run with lot more memory
than 20MB.

Using Apache worker MPM, where each Apache child worker process is
multithreaded can be better, as in that case configuration would
normally see maximum of 5-10 processes. So, maximum would be 5*20*10
MB, which is a lot less.

To use Apache worker MPM though, you would need to use Django 1.0 as
older versions had some question marks over their readiness for
multithreaded use. Your application itself would also need to
multithread safe.

Now, for another option, you may want to use mod_wsgi instead. Whether
you use Apache prefork or worker MPM, with mod_wsgi you would create a
daemon process group for each site and delegate specific WSGI
applications for each site to run in their own daemon group processes.
Using daemon mode gives various benefits. These are they each site
runs in its own processes and no risk of each interfering with each
other. Each site can run as a distinct user. Individual sites can be
restarted without restarting whole of Apache. And the number of
processes used by each site can be be better controlled, with it being
independent of MPM used and how many Apache child worker processes
there are.

In summary, if you need to be mindful of memory usage, you would be
better of using mod_wsgi and its daemon mode. This is because it gives
you separate control over number of processes in use for each site and
number of processes fixed and isn't variable based on load and Apache
making a decision to create more worker processes.

Note that mod_wsgi and daemon mode doesn't preclude you still running
a site within embedded mode, ie., in Apache child worker processes, at
the same time if you feel that the bit of extra speed coming from
embedded mode is important to you for a specific site.

Graham

> > > Django instance or multiple?  Ismod_pythonnot the way to go for this

[mguthrie]

Graham,
Thanks for the detailed response. I have yet to get too much into
the internals of Apache in regards to Python applications. My
background is in PHP which is a whole different beast with it's own
unique way of being tweaked. I'll have to look into the mod_wsgi
setup you mentioned. I've heard a lot of things regarding wsgi with
other Python projects.
Discussing all of this has made me re-think my approach and whether
Django will be a good fit for this specific project. It may be better
for me to simply use a different python framework and let the client
separation be handled via the application and not separated out by
Apache or any other server (lighttpd, nginx, etc).
Like I noted my background is in PHP development so understanding
the Python way of doing things is new territory for me. Thanks for
the feedback.

-MG

On Oct 10, 6:29 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:

[mguthrie]

The following link seems to support Graham's conclusion:

http://www.technobabble.dk/2008/aug/25/django-mod-wsgi-perfect-match/

-MG

[Graham Dumpleton]

On Oct 11, 12:53 pm, mguthrie <mitchguth...@gmail.com> wrote:
>   Discussing all of this has made me re-think my approach and whether
> Django will be a good fit for this specific project.  It may be better
> for me to simply use a different python framework and let the client
> separation be handled via the application and not separated out by
> Apache or any other server (lighttpd, nginx, etc).

Most Python web application frameworks don't really support any
concept of internal separation for distinct applications or sites and
many also don't support multiple instances of the same framework in
one process albeit when in use for distinct applications. As such, you
more often than not don't have much choice but to rely on the hosting
mechanism to somehow handle the separation.

I don't know too much about it, but Zope and it successors in kind may
be the only ones which make an attempt to provide some sort of
separation within the one instance. For Zope this involves use of what
it calls the virtual host monster. Even then, to use it is still needs
to be matched to a hosting environment which can enable its use.

Graham

[Steve Holden]

mguthrie wrote:
> The following link seems to support Graham's conclusion:
>
> http://www.technobabble.dk/2008/aug/25/django-mod-wsgi-perfect-match/
>
>

Since Graham was a major contributor to mod_python and is the author of
mod_wsgi it would be surprising if he'd given you anything other than
the scoop on using Django with Apache.

regards
Steve

[Join.T...@gmail.com]

On Oct 10, 8:53 pm, mguthrie <mitchguth...@gmail.com> wrote:
> Graham,
>    Thanks for the detailed response.  I have yet to get too much into
> the internals of Apache in regards to Python applications.  My
> background is in PHP which is a whole different beast with it's own
> unique way of being tweaked.  I'll have to look into the mod_wsgi
> setup you mentioned.  I've heard a lot of things regarding wsgi with
> other Python projects.
>   Discussing all of this has made me re-think my approach and whether
> Django will be a good fit for this specific project.  It may be better
> for me to simply use a different python framework and let the client
> separation be handled via the application and not separated out by
> Apache or any other server (lighttpd, nginx, etc).
>   Like I noted my background is in PHP development so understanding
> the Python way of doing things is new territory for me.  Thanks for
> the feedback.

The way I see it, any good design will end with you setting up
separate instances of your code per client. You COULD have a "client
id" field in every single model that will be distinguished between
clients, but this will be difficult. For one, you're going to need to
create a new User model (Totally doable, actually, but a hassle), and
trying to maintain all those different clients in the same tables is
just begging for mix-ups and errors.

As said above, if you use apache, I would try using the worker MPM and
mod_wsgi, and just make sure that all your code is thread safe.

[Mitch Guthrie]

I've decided to push forward with the project using Django with separate client instances.  It will keep my development overhead low and allow me to better focus on the application. 

I appreciate all the great feedback.  This has definitely clarified a lot for me.

-MG

[Daniel]

Would it not be possible to create a base model and model manager that
specifies a client, and override the default django.db.models.Model
and django.db.models.Manager classes? Something like this:

class Client(models.Model):
client_name = models.CharField(max_length=64)
# etc...

class ClientManager(models.Manager):
def get_query_set(self):
client = <current client - retrieved from middleware global?>
return super(ClientManager,
self).get_query_set().filter(client=client)

class ClientModel(models.Model):
"""
An abstract base class model that provides a link to client table
"""
client = Client()

class Meta:
abstract = True

django.db.models.Model = ClientModel
django.db.models.Manager = ClientManager

I haven't tested this... this would need to be imported early, before
any of the apps are loaded, and in get_query_set one would need a way
to determine the client that corresponds with the current url - I
think some middleware that inspects the request should be able to
cache that away.

Anyone know if this is possible, or am I way off base?

Dan

[Daniel]

and of course, a save() on the model would be necessary:

class ClientModel(models.Model):
"""
An abstract base class model that provides a link to client table
"""
client = Client()

def save(self):
client = <current client>
super(ClientModel, self).save()

class Meta:
abstract = True