What The Enterprise wants from Django
Jacob Kaplan-Moss
I had a conversation this morning with one of my clients. In the
interests of being a good corporate citizen I'll refrain from
mentioning who other than (a) they're big (Fortune 1000), (b) you've
heard of them, and (c) they're using Django.
Before our chat, they'd invited any engineers internally to bitch
about Django, and they shared some of the pain points they'd come
across. I took some notes; the really good news is that it's a short
list, and we already know about most of it.
Before I share the list, I should mention that I'm not arguing we have
to immediately devote resources to these problems, nor even that we
have to solve them. I'm sharing this just as food for thought, and as
a perspective from a class of developers who mostly don't participate
in our process at all.
So:
The main issue they have is multiple database support. Music to my
ears (and especially Alex's :) since it looks like 1.2 will solve most
(all?) of their issues. My main takeaway from this point was that the
best way to please companies like this one will be to ship 1.2 on
time. Which will happen. So we rock.
The next big one was the "startup signal" issue -- they've got lots of
code that needs to run at startup time, and there's no great mechanism
to do that currently. The core devs have talked about this one a *lot*
over the years, and it's obviously a hard one -- for one, there's no
clear definition of what "startup" means -- but a common theme seems
to be that bigger, more complex applications need some way to cleanly
run one-time, expensive startup tasks.
Next, we discussed the difficulty of maintaining large installations
with multiple sites, a plethora of apps, and unspeakable possible
combinations thereof. Django's settings mechanism is simple and easy
to use, but once you have hundreds of sites it starts getting really
tricky to keep things in sync across hundreds of settings files. We
talked at some length about different possibilities for dynamic
settings infrastructure; this would especially come in handy for folks
wanting to build application servers/services like App Engine or the
nascent Toppcloud.
<digress>If you've not yet checked out Toppcloud, do so now. I'll
wait.</digress>
Finally, we ruminated over the difficulties in building rich internet
applications. Sure, writing HTML/CSS/JS/Python/SQL by hand works fine,
but we doesn't really have a good answer for the people who want
something IDE or GUI-ish. Meanwhile, Adobe and Microsoft are putting
all sorts of marketing dollars into Flex/Silverlight, and although
HTML5 can do some amazing things, the lack of tooling is a big danger.
(I've written at more length about this in the past:
http://jacobian.org/writing/snakes-on-the-web/#s-rich-web-applications).
Of course, there may not be much us backend folks can do about this
problem -- I'm certainly not enough of a GUI application guy to be
able to even start to think about this problem -- but the lack of an
end-to-end solution will nonetheless put some off from writing web
applications with open source tools.
So there we are. This is very much a brain dump, and I don't really
expect any concrete action to result from it. However, I found some
really interesting stuff there, and I thought I'd share.
Jacob
David Cramer
into the settings issue, but would love to see what people can come up
with for solutions (we dont have a good one). Glad to see multi db is
finally shipping, and excited to see what can be done for startup
procs.
mrts
> The first three have been huges ones with us. We're just now running
> into the settings issue, but would love to see what people can come up
> with for solutions (we dont have a good one).
The override_settings() idea that Eric Florenzano describes at
http://djangodose.com/articles/2009/09/handling-development-staging-and-production-enviro/
with some improvements and fixes as described in the following
http://djangodose.com/articles/2009/09/handling-development-staging-and-production-enviro/#comment-16302200
have made my life considerably easier.
Hope the following helps others as well.
# 1.
# settings is a directory, containing host-specific overrides
# and the generic override logic
$ ls settings/
__init__.py override.py host1.py host2.py
# 2.
# __init__.py has the same contents that settings.py
# usually has, except it automatically imports host-specific
# settings in the end
$ tail -2 settings/__init__.py
override.override_settings('projectname', platform.node().split('.')
[0],
sys.modules[__name__])
# 3.
# The generic override logic is as follows
$ cat settings/override.py
from django.utils.importlib import import_module
import warnings
def override_settings(package, name, settings_module):
rel_path = 'settings.' + name
try:
override_module = import_module(rel_path, package=package)
except ImportError:
warnings.warn("Failed to import settings from '%s.%s'" %
(package, rel_path))
else:
for key in dir(override_module):
if key.isupper() and not key.startswith('__'):
setattr(settings_module, key, getattr(override_module,
key))
aviahlaor
place to call the startup work, starting threads, setting logging
level etc. It worked for me, but it's probably much simpler than a
complex corporate deployment.
Vinay Sajip
> The next big one was the "startup signal" issue -- they've got lots of
> code that needs to run at startup time, and there's no great mechanism
> to do that currently. The core devs have talked about this one a *lot*
> over the years, and it's obviously a hard one -- for one, there's no
> clear definition of what "startup" means -- but a common theme seems
> to be that bigger, more complex applications need some way to cleanly
> run one-time, expensive startup tasks.
I've got a mechanism which I described in these posts:
http://groups.google.com/group/django-developers/msg/2b94fa2017b07e67
and
http://groups.google.com/group/django-developers/msg/7f14fc0d9ee92b43
The thread refers to logging but the approach works for any start up
code;
I've provided three points where code can be hooked, which I've called
"bootstrap" (e.g. to initialize logging), "pre-model" (before app
loading, say to
install class_prepared listeners) and "post-model" (after app loading,
to do
any work which needs apps to be initialized and models to be
available).
I would welcome some feedback ... Launchpad branch is here:
https://code.launchpad.net/~vinay-sajip/django/logging
An example settings file which uses the functionality in the branch is
in the
second of the two posts mentioned above.
Regards,
Vinay Sajip
sago
as a Django consultant, Django is a preferred tool) and the issues
your client mentions, Jacob, are very similar to those I deal with.
Some other notes from the concerns of multinational clients:
I've had one very long and complex issue with a major client over
legacy databases with Composite Primary Keys (and other composite keys
more generally), an issue which has also come up in other contexts.
One of my smaller clients switched to a strange bastardization of
Django and SQLAlchemy because this was an issue. I understand the
history of #373 and why it is hard - just for disclosure.
A lack of any mechanism for developers to understand what third party
apps are reliable, what to avoid, known problems, and so on. This has
been a particular complaint since the Django ethos is (rightly) that
the core is the core, and other functionality should be packaged into
third party apps. App authors do a great job of being really generous
with their work, but it is very hard for stuff outside Django's core
to get any adoption. The processes involved in adopting a technology
can be complex and involve lots of stakeholders, legal and technical
due diligence. It is hard enough ticking those boxes to get Django
used, it simply isn't worth doing the same for all the other bits.
There's a lot of wheel reinventing going on, in my experience. A Django
+ project might be useful to give the rock-star third-party apps a
fighting chance at being used. Pinax is somewhat close, but has a very
different purpose and small, agile, company ethos.
Startup and Settings are the big killers though, head and shoulders
above the previous issues. I've nothing much to add to your comments
other than to say that some of the Django deployments I know of are
highly heterogeneous, with various servers running different SOA
elements, some Django some not. The configuration issues are
formidable and involve all kinds of dependencies beyond Django (or
even Python). I struggle to see a way that Django could solve those
issues, but allowing more flexibility and avoiding tying configuration
to an executable module (i.e. settings.py) would be useful. Like the
previous posters, I've also had to roll various custom solutions to
this problem. Is suspect at least some of the solution will always be
custom, though it might be worth making such solutions easier to
create.
It would also be useful for everyone to scream more loudly that monkey-
patching ANYTHING is a very bad idea. Its getting better now (as some
of the major things it was solving are solved properly), but it was a
woefully overused technique for a while that has meant a lot of
deployment pain! That's less about Django and more about evangelizing
bad development practice under the guise of Python-Fu.
Ian.
Luke Plant
> Startup and Settings are the big killers though, head and shoulders
> above the previous issues. I've nothing much to add to your
> comments other than to say that some of the Django deployments I
> know of are highly heterogeneous, with various servers running
> different SOA elements, some Django some not. The configuration
> issues are formidable and involve all kinds of dependencies beyond
> Django (or even Python). I struggle to see a way that Django could
> solve those issues, but allowing more flexibility and avoiding
> tying configuration to an executable module (i.e. settings.py)
> would be useful.
I don't understand how avoiding the settings.py mechanism will produce
*more* flexibility. With an executable module storing settings, you
can implement whatever mechanism you want for settings:
## settings.py ##
import sys
from acme import network_config
network_config.load_django_settings_into_module(sys.modules[__name__])
## end.
Granted, code that does the above might need to be written, but the
point of settings.py is that you *can* write it, without forcing
anyone to use that system.
Luke
--
"Procrastination: Hard work often pays off after time, but laziness
always pays off now." (despair.com)
Luke Plant || http://lukeplant.me.uk/
Jacob Kaplan-Moss
> I don't understand how avoiding the settings.py mechanism will produce
> *more* flexibility.
The problem -- at least as I see it -- is that of a intertwingulment
of "application" settings with "ops" settings.
As organizations get larger, roles stratify more. Thus you end up with
a pretty big gulf between the ops team and the application developers.
In these environments, application developers probably shouldn't be
messing with things like the `DATABASE_*` settings -- in some
organizations its even a breach of policy for the developers to *know*
the database password!
OTOH, ops types don't know what the heck to do with `INSTALLED_APPS`...
And then there's middleware: some middleware (caching, etags) might
need to be controlled by ops; some by apps.
Obviously you can *do* this with Django, and I'd argue it's not
particularly hard. Still, it's something that bigger orgs have to
figure out themselves, and that can be a barrier to adoption.
The enterprisey world solves this with standards like WAR and servers
like Glassfish: if I'm an application developer, all I need to do to
deploy is somehow produce a WAR. And if I'm a ops person, I just take
that WAR and slap it into Glassfish. It's rarely that easy in
practice, of course, but the execs making the tech decisions don't
know that.
Jacob
mtnpaul
> apps are reliable, what to avoid, known problems, and so on. This has
> been a particular complaint since the Django ethos is (rightly) that
> the core is the core, and other functionality should be packaged into
> third party apps. App authors do a great job of being really generous
> with their work, but it is very hard for stuff outside Django's core
> to get any adoption. The processes involved in adopting a technology
> can be complex and involve lots of stakeholders, legal and technical
> due diligence. It is hard enough ticking those boxes to get Django
> used, it simply isn't worth doing the same for all the other bits.
> There's a lot of wheel reinventing going on, in my experience. A Django
> + project might be useful to give the rock-star third-party apps a
> fighting chance at being used. Pinax is somewhat close, but has a very
> different purpose and small, agile, company ethos.
While not perfect I find how the http://plugins.jquery.com/ site
works. It gives me more confidence in picking a plugin to use for a
particular project to see how other users have rated an app. The demos
and examples of the plugins is also very nice. Something similar for
Django apps would be great.
Andy McKay
On 2010-01-20, at 10:54 AM, Jacob Kaplan-Moss wrote:
> On Wed, Jan 20, 2010 at 12:31 PM, Luke Plant <L.Pla...@cantab.net>
> wrote:
>> I don't understand how avoiding the settings.py mechanism will
>> produce
>> *more* flexibility.
>
> The problem -- at least as I see it -- is that of a intertwingulment
> of "application" settings with "ops" settings.
+1
--
Andy McKay, @clearwind
http://clearwind.ca/djangoski
sago
> *more* flexibility.
Bad choice of words on my part, sorry. Of course code is always more
flexible than static data.
The two key bits of required 'flexibility' I didn't have out of the
box (and were difficult to hack and get working) are:
1. App level settings that are mutually exclusive (one app needs
settings different from another - this happens a lot).
2. Setting definitions that aren't tied in timing and scope terms to
Python's import process (I've seen this interact badly with start-up
code, for example).
And it would be cool to have a level of indirection so that settings
could be guaranteed to be dynamic (some are, some aren't, those that
are often need monkey-patch style hacks to change). And tools for
debugging where settings for a failed request are defined. And a
security layer around some settings. And fields and fields of other
ponies.
Seriously though, I don't think the settings.py approach is broken. It
makes sense, makes getting from nothing to runnable code a breeze, is
logical for small deployments, and has bags of flexibility for most
use-cases. I'm happy adjusting it in my projects. I've never deployed
or seen deployed a system that used settings.py in anything like its
default version, but the replacements aren't standard enough to be an
alternative default; every client has different technical, management
and internal politics constraints.
My comments were meant in the spirit of Jacob's. Just FYIs for a
particular niche of users.
I should also say that most of the developers I work with do love
working with Django. So their criticisms have to be taken against a
background of strong positivity.
Mathieu Leduc-Hamel
You write the name of settings of choice in your buildout:
[django]
recipe = djangorecipe
version = 1.1
settings = development
eggs = ${eggs:eggs}
wsgi = true
project = project
And after that you can have by example another production's
configuration file like "prod.cfg" overloading your django part and
using another settings name.
In your project dir, you'll have a file named development.py and
another one named prod.py. In these files you can override any
configuration you want.
For deployement, zc.buildout is working pretty fine with django:
http://pypi.python.org/pypi/djangorecipe
> --
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>
>
>
>
Honza Král
> On Wed, Jan 20, 2010 at 12:31 PM, Luke Plant <L.Pla...@cantab.net> wrote:
>> I don't understand how avoiding the settings.py mechanism will produce
>> *more* flexibility.
>
> The problem -- at least as I see it -- is that of a intertwingulment
> of "application" settings with "ops" settings.
>
> As organizations get larger, roles stratify more. Thus you end up with
> a pretty big gulf between the ops team and the application developers.
> In these environments, application developers probably shouldn't be
> messing with things like the `DATABASE_*` settings -- in some
> organizations its even a breach of policy for the developers to *know*
> the database password!
>
> OTOH, ops types don't know what the heck to do with `INSTALLED_APPS`...
>
> And then there's middleware: some middleware (caching, etags) might
> need to be controlled by ops; some by apps.
>
> Obviously you can *do* this with Django, and I'd argue it's not
> particularly hard. Still, it's something that bigger orgs have to
That's a problem that we have. We solve it very simply by splitting
settings.py into a package and dividing these options. We also provide
some machine wide configuration in /etc/ that won't get overwritten
during update of the project - that's where ops guys store their
passwords and everything.
the mechanism is shortly described here:
http://ella.github.com/tutorial/4-deployment.html#settings-py
project template:
http://github.com/ella/django-base-project/tree/master/djangobaseproject/
I think this can be solved by documenting different approaches (like
http://code.djangoproject.com/wiki/SplitSettings ) or maybe even
picking one and recommending that.
Another problem that we have is error reporting. If your project blows
up and starts generating hundreds of error messages per second, you
really don't want then on your mailserver or mail client. It is also
solvable with django's current tool set, but everybody must also
figure it out on their own.
We are currently working on some simple mechanism to handle this
(offload the exceptions into a queue, group the same errors and store
it in some DB or just log it), we will share the code as soon as it's
finished.
Andy McKay
On 2010-01-20, at 5:59 PM, Honza Král wrote:
> Another problem that we have is error reporting. If your project blows
> up and starts generating hundreds of error messages per second, you
> really don't want then on your mailserver or mail client. It is also
> solvable with django's current tool set, but everybody must also
> figure it out on their own.
There's quite a few other solutions for that. Eg: writing to Apache (if you use it logs, writing to sys log etc) I think the new logging framework addition will help here. Or there are services like Arecibo: http://areciboapp.com/listener/docs/django/ (other services are available) to do that for you.
Russell Keith-Magee
> On Wed, Jan 20, 2010 at 12:31 PM, Luke Plant <L.Pla...@cantab.net> wrote:
>> I don't understand how avoiding the settings.py mechanism will produce
>> *more* flexibility.
>
> The problem -- at least as I see it -- is that of a intertwingulment
> of "application" settings with "ops" settings.
I'm in full agreement that this is a problem.
However, isn't the solution just a matter of adequately documenting
the options that are already available? We could certainly improve the
documentation for complex deployments - e.g., pointing out that:
* settings.py can import base_settings.py
* DJANGO_SETTINGS_MODULE lets you call your settings.py whatever you want
* documenting which settings are operations, and which are application
I'm not convinced that we have anything particularly technical to address here.
> And then there's middleware: some middleware (caching, etags) might
> need to be controlled by ops; some by apps.
>
> Obviously you can *do* this with Django, and I'd argue it's not
> particularly hard. Still, it's something that bigger orgs have to
> figure out themselves, and that can be a barrier to adoption.
It strikes me that this is just one example of a whole class of "best
practice" stuff that we should document, but don't at present - like
project layout, automated deployment, continuous integration,
configuration management, etc.
We've historically avoided having documentation on how to use Django
with external tools (except when we have to, as in the case of
mod_wsgi). A big part of the solution to the problem you've described
is to establish a vetted best practice, and documenting that practice
on our website as a community resource.
The rest of the solution falls to the toolchain outside of Django
itself. It still needs to be solved, but it isn't a Django problem
specifically. One option here might be to take the Apache model, and
use the DSF to incubate tools that have benefits to Django. However,
that's part of a much bigger discussion.
Yours,
Russ Magee %-)
gareth rushgrove
> On Wed, Jan 20, 2010 at 12:31 PM, Luke Plant <L.Pla...@cantab.net> wrote:
>> I don't understand how avoiding the settings.py mechanism will produce
>> *more* flexibility.
>
> The problem -- at least as I see it -- is that of a intertwingulment
> of "application" settings with "ops" settings.
>
> As organizations get larger, roles stratify more. Thus you end up with
> a pretty big gulf between the ops team and the application developers.
> In these environments, application developers probably shouldn't be
> messing with things like the `DATABASE_*` settings -- in some
> organizations its even a breach of policy for the developers to *know*
> the database password!
In the past I've used code like the following in settings files:
# we're going to allow overriding of settings via a yml file.
# This makes live nicer for anyone managing the box
# and means settings can be overloaded without redeploying the site
SETTINGS_OVERIDE = "/etc/$(project).yml"
try:
file = open(SETTINGS_OVERIDE)
for key, value in yaml.load(file).items():
globals()[key]=value
except:
# we don't always have the file around or need the setting
# defined so best to be quite if things go wrong
pass
You can also make this more limited, by allowing only the overriding
certain settings. You can then use tools like puppet on the ops side
to push config files in to /etc (or wherever suites you system).
Example here http://morethanseven.net/2009/02/15/let-you-sys-admin-override-your-django-settings/
This pattern is also used in my project templates paster stuff:
http://github.com/garethr/django-project-templates and
http://pypi.python.org/pypi/django-project-templates/
>
> OTOH, ops types don't know what the heck to do with `INSTALLED_APPS`...
>
> And then there's middleware: some middleware (caching, etags) might
> need to be controlled by ops; some by apps.
>
> Obviously you can *do* this with Django, and I'd argue it's not
> particularly hard. Still, it's something that bigger orgs have to
> figure out themselves, and that can be a barrier to adoption.
>
> The enterprisey world solves this with standards like WAR and servers
> like Glassfish: if I'm an application developer, all I need to do to
> deploy is somehow produce a WAR. And if I'm a ops person, I just take
> that WAR and slap it into Glassfish. It's rarely that easy in
> practice, of course, but the execs making the tech decisions don't
> know that.
>
> Jacob
>
> --
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>
>
>
>
--
Gareth Rushgrove
Web Geek
gareth rushgrove
>
> It strikes me that this is just one example of a whole class of "best
> practice" stuff that we should document, but don't at present - like
> project layout, automated deployment, continuous integration,
> configuration management, etc.
>
> We've historically avoided having documentation on how to use Django
> with external tools (except when we have to, as in the case of
> mod_wsgi). A big part of the solution to the problem you've described
> is to establish a vetted best practice, and documenting that practice
> on our website as a community resource.
I'd certainly be interested in contributing to this effort. I have
lots of this information scattered about on my blog, internal company
notes and presentations from DJUGL and the likes. I've avoided trying
to get it on the wiki for the reasons you describe.
What is the best way of going about documenting this so it can be
vetted by the community and enshrined as somewhat official best
practice - without the work in progress being used before it's been
stamped in someway?
>
> The rest of the solution falls to the toolchain outside of Django
> itself. It still needs to be solved, but it isn't a Django problem
> specifically. One option here might be to take the Apache model, and
> use the DSF to incubate tools that have benefits to Django. However,
> that's part of a much bigger discussion.
>
> Yours,
> Russ Magee %-)
>
Luke Plant
> So there we are. This is very much a brain dump, and I don't really
> expect any concrete action to result from it. However, I found some
> really interesting stuff there, and I thought I'd share.
Thanks for your clarification elsewhere in this thread.
One question I have (for Jacob and anyone else with experience):
I would have guessed that a big issue with Django from an enterprise
perspective is its use of 'singletons'. Though we don't use that
terminology, we have dozens of them in the code - every other setting
we have implies the existence of a singleton, since the setting can
have only one value. For example, EMAIL_BACKEND means that we have
one email component singleton which is used everywhere. If you wanted
one set of views to use a different email backend, you are out of
luck.
The 'admin' app was the first major component to allow multiple
instances, but there are various other apps which might need this
change in theory.
How much is this an issue in practice?
I imagine that for most sub-systems, most people actually want
singletons, but for the cases you don't, it is a big pain. Personally
I think that, while there has been an unhealthy proliferation of
settings which we need to trim, Django has hit the sweet spot between
easy set up and configuration on the one hand, and flexibility on the
other. The alternative - turning everything into components with
dependency injection everywhere - could be horrific for our sweet
spot.
One solution to this problem would be to have multiple Django
instances hosting different parts of your URL space, allowing them to
have multiple settings files. This has obvious limitations, but from
the perspective of a large project, perhaps it's not too bad - I
imagine that large sites are already used to carving up the URL space
to be served by completely different technologies in some cases, at
least for hosts running LAMP or similar.
Regards,
Peter Herndon
On Jan 21, 2010, at 9:58 AM, Luke Plant wrote:
> On Tuesday 19 January 2010 21:26:17 Jacob Kaplan-Moss wrote:
>
>> So there we are. This is very much a brain dump, and I don't really
>> expect any concrete action to result from it. However, I found some
>> really interesting stuff there, and I thought I'd share.
>
> Thanks for your clarification elsewhere in this thread.
>
> One question I have (for Jacob and anyone else with experience):
>
> I would have guessed that a big issue with Django from an enterprise
> perspective is its use of 'singletons'. Though we don't use that
> terminology, we have dozens of them in the code - every other setting
> we have implies the existence of a singleton, since the setting can
> have only one value. For example, EMAIL_BACKEND means that we have
> one email component singleton which is used everywhere. If you wanted
> one set of views to use a different email backend, you are out of
> luck.
>
> The 'admin' app was the first major component to allow multiple
> instances, but there are various other apps which might need this
> change in theory.
>
> How much is this an issue in practice?
Speaking as a web developer using Django in a *very* enterprisey corporation, I will say this singleton issue has been a problem for me. As a self-inflicted example, I wrote an LDAP backend that looks up the necessary settings from settings.py. I need to rewrite it to store the settings in the db, because I need to be able to query multiple LDAP sources. I run into similar issues with lots of different apps. So yes, singletons are an issue for me.
The difference between ops settings vs. app settings has also been a problem. The lack of established, vetted best practices (for deployment, layout, everything Russ mentioned) also has bitten me regularly. In part because I wind up spending a lot of time doing the research to figure out what I need to do, in part because the state of the art moves forward at a rapid pace. But, to put it in perspective, I'm the only person who even *can* deploy a Django app at my place of work. Which is not great, but I'm also a Linux sys-admin, so it is at least in my job description. But none of my fellow SAs have any clue what to do to get stuff working on one of our corporate RHEL 5 boxen, and no inclination to learn.
Anyway, I'm glad to see this discussion taking place. Django indeed has a sweet spot, and it isn't really for enterprise installations -- though it is *very* good in that sweet spot. There are places where the early design decisions to favor convention over configuration impose a lack of flexibility in favor of simplicity. A definite trade-off. It makes it harder to use in places where flexibility and integration are required.
Here's a big question, though: given that satisfying the enterprise crowd, of which I am an occasional part, will require additional "stuff" that may bring along with it additional complexity, making it potentially harder to get started for newbies, is that a direction that the project should head? Is it worth satisfying the enterprise crowd to increase adoption, at the cost of turning into the kind of configuration hell that J2EE servers exemplify? Or, more positively, is it possible to add the required enterprisey bits while still maintaining a certain level of simplicity? Note that Django is already headed towards greater complexity (multi-db, email backends), due to requirements from the current community.
My two cents, appropriately adjusted for the current economic situation.
---Peter Herndon
sago
> perspective is its use of 'singletons'.
>
hundreds of apps, it is often the case that it is absolutely essential
to have settings on a per-app basis. Unfortunately those settings very
often clash.
> It strikes me that this is just one example of a whole class of "best
> practice" stuff that we should document, but don't at present - like
> project layout, automated deployment, continuous integration,
> configuration management, etc.
It would help smaller companies I think, folks who need to deploy a
couple of SOA nodes or a site with ten or twenty apps across a handful
of servers. When it comes to huge deployments, it is so diverse. Best
practice for Google isn't best practice for wisconsinhairstylists.com
Case studies are helpful if folks can share them. But its a political
minefield (or unhelpful pontification, depending on the care you take)
to get into the business of canonizing deployment structures!
Jacob Kaplan-Moss
> I would have guessed that a big issue with Django from an enterprise
> perspective is its use of 'singletons'.
> How much is this an issue in practice?
Other than databases, it hasn't been much of an issue with most of the
BigCos I've been involved with. That's because...
> One solution to this problem would be to have multiple Django
> instances hosting different parts of your URL space, allowing them to
> have multiple settings files.
Bingo.
All of the larger Django installations I've worked with haven't been
just one project -- they've been dozens (hundreds in one case) of
little bits, often cobbled together by different teams and sometimes
different companies (in-house vs. contractors vs. off-the-shelf
products).
Django's URL/view decoupling looks *really* nice in these cases --
Django apps, properly written, don't care *where* in the URL space
they live -- so this means that Django fits in well in these
heterogenous environments. And I've yet to encounter a larger company
that's anything but wildly heterogeneous.
Peter's question below about enterprisy complexity and the dangers of
becoming J2EE are good ones; I'm glad they're part of the discussion.
It's difficult to know where to draw the line between simplicity and
flexibility.
Right now we have exactly 90 documented settings, some of which point
at classes or interfaces that themselves have more complexity. To me
this already sounds way too high, but it's nothing compared to the
configuration options available in other environments -- PHP has about
600 php.ini settings! -- so there's obviously a matter of taste here.
It's also important to keep in mind that part of the reason Django's
had (some) success in corporate environments is *because* of the
simplicity, not despite it. I've heard, over and over again, that
teams that have switched to Django have done so at least in part
because of the simplicity and ease-of-use.
So we should be careful about setting up a false dichotomy where us
hippie open source freaks only value simplicity and build toys and
those square suit types who love FactoryManagerFactories because CIO
magazine tells them to. The real world, as always, is a lot more
complex.
Jacob
Jacob Kaplan-Moss
> Case studies are helpful if folks can share them.
Just a quick note that the DSF is working on gathering case studies;
getting folks who have permission to share is indeed a tricky issue.
Not much is set up yet, but I, at least, would be *thrilled* to help
companies using Django put together case studies that could be shared
publicly. Until we've got something more formal in place, consider it
a standing offer -- drop me a line if you'd like to share a success
story.
Jacob
chris....@gmail.com
environments - even non Fortune 1000 environments. One of the things
we've played with in Satchmo is a database backed solution that allows
you to define reasonable defaults - even override-able in the settings
- but allow people to edit through the admin interface.
Bruce has been working to split it out from Satchmo here -
http://bitbucket.org/bkroeze/django-livesettings/overview/
I'm not saying this is the end all be all solution but it's something
we've been using for a bit so it should be useful in informing the
direction the community wants to take. I also have some ideas for how
it should be improved or tweaked but in my opinion it's a nice start
and it would be good to see what people think about using it in other
apps. It's also consistent with the Django philosophy of building it
outside of core and seeing if it reaches critical mass for the
community.
-Chris
Antoni Aloy
My previous experience was from Java J2EE world and as Jacob said we
adopted Django because it really shines solving some complexity
problems associated to J2EE world as MVC patter, templates, etc., but
I agree that a common way to configure external applications would be
nice.
I'll also have been searching a good RIA to connect with Django. One
of the frameworks I like best is Qooxdoo http://qooxdoo.org/, perhaps
the best javascript opensource RIA frameworks I have seen. In order to
connect with Qooxdoo just Json-RCP or xml-rcp is needed. Perhaps we'll
need an special support for this in Django to mark a view as rcp as
some external projects do as RCP4Django or other. I'd like to see
Qooxdoo as the default Django RIA (not the default javascript
framework) as Qooxdoo people uses also Python and the framework fits
quite well in what I think is an enterprise RIA framework.
I know that this could be against the Django policy of not to bless a
javascript framework, but in my opinion if we want to shine at RIA you
have to choose.
We have created also some HTTP-XML based web services in Django in a
fraction of time we needed to create them in Java. For that we are
using Django without ORM, but we have found a great help to allow
documenting the web service and create the test pages.
Anyway, I'm glad to see Fortune companies are interested in Django :)
--
Antoni Aloy López
Blog: http://trespams.com
Site: http://apsl.net
Andy McKay
> I agree that managing settings gets to be a bit difficult in many
> environments - even non Fortune 1000 environments. One of the things
> we've played with in Satchmo is a database backed solution that allows
> you to define reasonable defaults - even override-able in the settings
> - but allow people to edit through the admin interface.
I've seen that route before and I have to say that I would be strongly against that. The only saving grace could be multi-db which could allow you to have a specific settings database that's separate from the core.
Having data and settings intermingled in the same area just causes too many problems. Further, it detracts from the ability to run the project without the database around. In the scenario of a larger environment with more delineation of roles, it may not be possible.
PauloS
> I've had one very long and complex issue with a major client over
> legacy databases with Composite Primary Keys (and other composite keys
> more generally), an issue which has also come up in other contexts.
> One of my smaller clients switched to a strange bastardization of
> Django and SQLAlchemy because this was an issue. I understand the
> history of #373 and why it is hard - just for disclosure.
Composite primary keys is a major issue over here and we also ended up
with a bastard of SQLAlchemy and Django.
--
Paulo
stuff4ash
On 21 Jan., 17:45, Andy McKay <a...@clearwind.ca> wrote:
> On 2010-01-21, at 9:15 AM, chris.moff...@gmail.com wrote:
>
> > I agree that managing settings gets to be a bit difficult in many
> > environments - even non Fortune 1000 environments.
Its also an issue for us that write and use portable apps. Luckily a
way for apps to provide defaults could easily be implemented in a
admin.autodisover sort of way.
Somebody already implemented something similar to what i wanted to
write today: http://code.djangoproject.com/ticket/1282
For the rest of the "probelms", as someone who is constantly
integrating heterogenous systems with django, I think the solution
might be "outside" of django.
Some problems are sometimes dealt easier in python space. Just as a
deliverance similar/friendly solution might be a better solution than
making a django admin skin "engine", a more generic configuration app
might be better for advanced uses.
Luc Saffre
> Finally, we ruminated over the difficulties in building rich internet
> applications. Sure, writing HTML/CSS/JS/Python/SQL by hand works fine,
> but we doesn't really have a good answer for the people who want
> something IDE or GUI-ish. Meanwhile, Adobe and Microsoft are putting
> all sorts of marketing dollars into Flex/Silverlight, and although
> HTML5 can do some amazing things, the lack of tooling is a big danger.
> (I've written at more length about this in the past:
> http://jacobian.org/writing/snakes-on-the-web/#s-rich-web-applications).
It's a bit early to speak about Lino, but Jacob's post incited me to
write a kind of preview for Django developers:
http://code.google.com/p/lino/wiki/LinoAndDjango
Encouraging words are welcome now, but wait a few months before you
seriously investigate this project.
Luc