I want a pony: Django Cheeseshop

1,016 views
Skip to first unread message

mrts

unread,
Sep 10, 2008, 10:31:05 AM9/10/08
to Django developers
This is for Django 2.0.

Rationale:

a) useful (and less-useful) Django apps are proliferating on Google
Code, djangoplugables.com and Django resources wiki page,
b) apps need refactoring (#3591,
http://code.djangoproject.com/wiki/VersionOneFeatures#INSTALLED_APPSrefactoring
)
c) reusable apps are immensely useful and may be one of Djangos great
selling points once they are easy to deploy and some order is enforced
upon the current chaos.

Proposal:
=======

Infrastructure
-------------------
* create a central app index à la Cheeseshop
* create an automated system similar to easy_install for installing
apps from
o that central repository
o or a custom repository/URL
to
o either globally to Python packages -- *but under django namespace!
*
o or locally into a concrete project
* provide app dependency handling like setuptools does for
o python package dependencies (identical to setuptools 'depends')
o Django app dependencies (e.g. 'django_app_depends')
* bundle the install tool either as a command for manage.py or a
separate utility in bin
* create the solution so that it can be adopted by other projects by
clearly decoupling Django-specific functionality (i.e. engage more
brainpower and devs)

Apps
-------------------
* create rich app objects that contain at least the fields available
in setuptools setup files (i.e. app metadata),
* extend them with Django-specific functionality to
o lessen magic: explicitly specify files containing models,
templates and admin bits, i.e. obsolete all forms of autodiscovery and
path hacking
o find a way to expose media
o handle all issues outlined in #3591

etc.

I see that spec pages are popping up in wiki, e.g.
http://code.djangoproject.com/wiki/ReplacingGetAbsoluteUrl -- thanks,
Simon! This should eventually land in a spec page as well (unless
there will be a unanimous "No pony for you!" reply).

Regards,
Mart Sõmermaa

James Bennett

unread,
Sep 10, 2008, 11:57:04 AM9/10/08
to django-d...@googlegroups.com
On Wed, Sep 10, 2008 at 9:31 AM, mrts <mr...@mrts.pri.ee> wrote:
> * create a central app index à la Cheeseshop

Doesn't the Cheese Shop already exist?

> * create an automated system similar to easy_install for installing
> apps from
> o that central repository

"easy_install django-registration" works fine for me right now. Why
not encourage people to use standard Python practices for packaging
and distribution?

> o either globally to Python packages -- *but under django namespace!
> *
> o or locally into a concrete project

Does anybody else actually do this? Last I checked, Pylons, TurboGears
and Zope apps didn't install or need to be installed into
framework-specific locations. Django applications are just Python
modules, and that's a *strength* from where I sit.

> * provide app dependency handling like setuptools does for
> o python package dependencies (identical to setuptools 'depends')
> o Django app dependencies (e.g. 'django_app_depends')

Or just, you know, use setuptools.

> * bundle the install tool either as a command for manage.py or a
> separate utility in bin

Or just, you know, install setuptools.

> * create the solution so that it can be adopted by other projects by
> clearly decoupling Django-specific functionality (i.e. engage more
> brainpower and devs)

Or just use the existing packaging and distribution tools Python already has.

Have I made my point clear enough yet?


--
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."

Eric Holscher

unread,
Sep 10, 2008, 12:12:50 PM9/10/08
to django-d...@googlegroups.com
Yea, I totally agree with this.

I wrote a blog post about how to use setuptools and distutils to distribute your django apps. There is no reason to reinvent the wheel here, especially after what Mark talked about at Djangocon (Django being considered seperate from the Django community).

I have been tempted to package other people's apps and put them on Pypi and then hand over control if they wanted it, but that seemed rather heavy handed. Some kind of official suggestion, or just a culture of putting apps up there would be huge. I really think that everyones apps should be up on Pypi.

Eric
--
Eric Holscher
Web Developer at The World Company in Lawrence, Ks
http://www.ericholscher.com
er...@ericholscher.com

Karen Tracey

unread,
Sep 10, 2008, 12:17:32 PM9/10/08
to django-d...@googlegroups.com
On Wed, Sep 10, 2008 at 12:12 PM, Eric Holscher <eric.h...@gmail.com> wrote:
Yea, I totally agree with this.

I wrote a blog post about how to use setuptools and distutils to distribute your django apps. There is no reason to reinvent the wheel here, especially after what Mark talked about at Djangocon (Django being considered seperate from the Django community).

Did you mean 'Python community' there?  (I did not see the talk, so trying to understand...)

Karen

Eric Holscher

unread,
Sep 10, 2008, 12:19:47 PM9/10/08
to django-d...@googlegroups.com
Haha, yea, sorry.

mrts

unread,
Sep 10, 2008, 12:20:03 PM9/10/08
to Django developers
On Sep 10, 6:57 pm, "James Bennett" <ubernost...@gmail.com> wrote:
> Have I made my point clear enough yet?

Your point is clear but is likely to bring less order to the current
chaos. And it doesn't handle project-local installation (arguably this
can be done with virtualenv, but that will just clutter user-specific
"app-space" instead of the global one). Most of the apps are really
Django-specific and don't conceptually belong in global Python
namespace.

If setuptools remains the recommended way of packaging apps (I don't
necessarily oppose this, but I'd like to hear other opinions; what I
don't like is littering CheeseShop with stuff that is unusable in
general Python context), at least a fixed policy is needed:
* designate a global namespace where all apps should live (e.g.
django.apps.mycompany.fooapp or just django.apps.fooapp)
* find a way for avoiding name collisions there (mycompany would deal
with that)
* document all the above and the steps and requirements for creating
and publishing a reusable app.

Joseph Kocherhans

unread,
Sep 10, 2008, 12:20:21 PM9/10/08
to django-d...@googlegroups.com
On Wed, Sep 10, 2008 at 8:57 AM, James Bennett <ubern...@gmail.com> wrote:
>
> On Wed, Sep 10, 2008 at 9:31 AM, mrts <mr...@mrts.pri.ee> wrote:
>> * create a central app index à la Cheeseshop
>
> Doesn't the Cheese Shop already exist?
>
>> * create an automated system similar to easy_install for installing
>> apps from
>> o that central repository
>
> "easy_install django-registration" works fine for me right now. Why
> not encourage people to use standard Python practices for packaging
> and distribution?

So, if TurboGears hasn't rewritten setuptools, I think there's
probably a reason. They aren't idiots, and they're much better at this
packaging and dependency stuff than we are.

>> o either globally to Python packages -- *but under django namespace!
>> *
>> o or locally into a concrete project
>
> Does anybody else actually do this? Last I checked, Pylons, TurboGears
> and Zope apps didn't install or need to be installed into
> framework-specific locations. Django applications are just Python
> modules, and that's a *strength* from where I sit.

100% with James here. I had a discussion about namespace packages [1]
with Mark Ramm at PyCon, and hist short answer was "don't do it, it
isn't worth it". Seeing as he's the maintainer of TurboGears and
probably knows setuptools about as well as anyone, I'd tend to trust
him on that one. I understand the appeal, but the consequences aren't
worth it.

Joseph

[1] http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages

mrts

unread,
Sep 10, 2008, 12:30:18 PM9/10/08
to Django developers
On Sep 10, 7:12 pm, "Eric Holscher" <eric.holsc...@gmail.com> wrote:
> your django apps. There is no reason to reinvent the wheel here, especially
> after what Mark talked about at Djangocon (Django being considered seperate
> from the Python community).

Although I don't know anything about the talk, that's exactly what I
wanted to say: as Django and Python communities are different,
overflooding CheeseShop with Django apps does not serve Python
community at all.

And I never had re-implementing setuptools in mind -- what I meant was
to extend and build upon setuptools and easy_install.

Steve Holden

unread,
Sep 10, 2008, 1:00:18 PM9/10/08
to django-d...@googlegroups.com
I don't see why Django can't be just as much a part of the Python
community as, say, Zope, who frequently distribute code through PyPi. I
don't see the advantage of fragmenting the infrastructure. That's what
it's there for - supporting Python users.

regards
Steve
[PSF chairman]

mrts

unread,
Sep 10, 2008, 1:28:40 PM9/10/08
to Django developers
On Sep 10, 8:00 pm, Steve Holden <holden...@gmail.com> wrote:
> I don't see why Django can't be just as much a part of the Python
> community as, say, Zope, who frequently distribute code through PyPi. I
> don't see the advantage of fragmenting the infrastructure. That's what
> it's there for - supporting Python users.

Right, be it the CheeseShop then if we have the blessing of Steve
Holden.

Integration-wise, Django is more like Zope (apps are coupled to the
framework) than TurboGears (apps are loosely coupled), so namespace
packages may make sense. A namespace should be agreed upon anyhow
(i.e. even if namespace packages will not be used) though. And
packaging guidelines are direly needed.

And what's with Django itself? Now that 1.0 is out it's high time for
`easy_install django` to work (James?).

Dan Fairs

unread,
Sep 10, 2008, 1:41:20 PM9/10/08
to django-d...@googlegroups.com
>>
>> Does anybody else actually do this? Last I checked, Pylons,
>> TurboGears
>> and Zope apps didn't install or need to be installed into
>> framework-specific locations. Django applications are just Python
>> modules, and that's a *strength* from where I sit.
>
> 100% with James here. I had a discussion about namespace packages [1]
> with Mark Ramm at PyCon, and hist short answer was "don't do it, it
> isn't worth it". Seeing as he's the maintainer of TurboGears and
> probably knows setuptools about as well as anyone, I'd tend to trust
> him on that one. I understand the appeal, but the consequences aren't
> worth it.
>
> J

+1 on just packaging stuff as eggs, and being able to easy_install
them. Works great with my buildout infrastructure for managing Django
builds too. I wouldn't worry about putting stuff that's not useful
outside a Django context up; that's exactly what the categories system
is for. There's plenty of Zope stuff there, for example, and I
personally haven't seen any complaints. Maybe I just missed them.

On the namespace package front, I'd be wary; I started trying to use
them, but a client of mine on Windows has experienced problems using
them in Django (in places where some __import__ magic is done). I'll
raise a ticket when I get down to a minimal example.

Cheers,
Dan

--
Dan Fairs <dan....@gmail.com> | http://www.stereoplex.com/

Ludvig Ericson

unread,
Sep 10, 2008, 1:42:29 PM9/10/08
to django-d...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 10, 2008, at 18:30, mrts wrote:

> as Django and Python communities are different, overflooding
> CheeseShop
> with Django apps does not serve Python community at all.

Sounds like an "issue" with PyPI at worst -- actually a potential
feature:
grouping packages, trove doesn't really cut it.

+1 for PyPI usage.

Ludvig "toxik" Ericson
ludvig....@gmail.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAkjIBwUACgkQXnZ94Kd6KaehKwCePlVLpBQ/gKuRKIMSMeeKuwXt
dfYAnROOUsS0gEmGpyxJ5X0OVOAgLHaG
=kRxg
-----END PGP SIGNATURE-----

Brett Hoerner

unread,
Sep 10, 2008, 2:00:08 PM9/10/08
to django-d...@googlegroups.com
On Wed, Sep 10, 2008 at 11:20 AM, mrts <mr...@mrts.pri.ee> wrote:
> And it doesn't handle project-local installation (arguably this
> can be done with virtualenv, but that will just clutter user-specific
> "app-space" instead of the global one).

At some point the Django app you're trying to installed has to go
somewhere. If it doesn't go in global or user's local, where do you
want it to go?

With virtualenv you can (and I would encourage that you) have an env
for each Django project, so any installing you do is only 'cluttering'
that project, and at that point it isn't even 'clutter'.

Brett

Jannis Leidel

unread,
Sep 10, 2008, 3:21:18 PM9/10/08
to django-d...@googlegroups.com

Am 10.09.2008 um 18:20 schrieb mrts:

> If setuptools remains the recommended way of packaging apps (I don't
> necessarily oppose this, but I'd like to hear other opinions; what I
> don't like is littering CheeseShop with stuff that is unusable in
> general Python context), at least a fixed policy is needed:
> * designate a global namespace where all apps should live (e.g.
> django.apps.mycompany.fooapp or just django.apps.fooapp)
> * find a way for avoiding name collisions there (mycompany would deal
> with that)
> * document all the above and the steps and requirements for creating
> and publishing a reusable app.

Although this is a shameless plug, I'd like to point to a small helper
app that is able to find resuable Django apps that are packaged as egg
files. It's independent from the PYTHONPATH and hooks into the
INSTALLED_APPS setting to automatically find every Django app that has
a so called entry point, a pretty nice feature of setuptools to find
components/plugins/pluggables [1]. It also works well with virtualenv.

For more information head over to http://code.google.com/p/django-reusableapps/
and start making eggs.

Cheers,
Jannis


1: http://peak.telecommunity.com/DevCenter/setuptools#dynamic-discovery-of-services-and-plugins

mrts

unread,
Sep 10, 2008, 3:48:24 PM9/10/08
to Django developers
I think we largely have a consensus now (unless someone speaks up) --
CheeseShop and the global (or virtualenv) package space is the way to
go for all apps when packaging rules have been documented.

If someone wants to take this further, a ticket + a documentation
patch that outlines how to package apps would be most welcome.
Namespace issues still need to be resolved though. Looks like there
should be two topics
* "How do I install external packages?" -- outline the two
strategies, global and virtualenv (for self-contained project
environment), how to find packages etc;
* "How should I package and distribute a reusable Django app?" --
what namespace to use, uploading to CheeseShop, registering in
djangoplugables (i.e. there should be a semi-official richer
environment that lists only Django apps, djangoplugables looks good),
dependencies, media etc.

On Sep 10, 9:00 pm, "Brett Hoerner" <bretthoer...@gmail.com> wrote:
> On Wed, Sep 10, 2008 at 11:20 AM, mrts <m...@mrts.pri.ee> wrote:
> > And it doesn't handle project-local installation (arguably this
> > can be done with virtualenv, but that will just clutter user-specific
> > "app-space" instead of the global one).
>
> At some point the Django app you're trying to installed has to go
> somewhere.  If it doesn't go in global or user's local, where do you
> want it to go?

Into project dir of course. Most of the existing apps can be trivially
installed by cd myproject; wget somewhere/app.tgz; tar zxf app.tgz.

> With virtualenv you can (and I would encourage that you) have an env
> for each Django project, so any installing you do is only 'cluttering'
> that project, and at that point it isn't even 'clutter'.

That holds only if you don't live on shared hosting and can afford
direct project <-> user (evironment) mapping.

If you can't afford that, installing stuff either globally or into
virtualenv can create trouble, e.g. project X is written against app Z
version 1, project Y is written against (incompatible) version 2 of
the same app, both X and Y have to run under the same user (shared
hosting) -- the only way to run them is to have a Y-project-local copy
of Z.

This is a minor, uninteresting problem though, so no need to discuss
this further.

Malcolm Tredinnick

unread,
Sep 10, 2008, 9:00:51 PM9/10/08
to django-d...@googlegroups.com

On Wed, 2008-09-10 at 12:48 -0700, mrts wrote:
> I think we largely have a consensus now (unless someone speaks up) --

Huh?!

In my current timezone, the first mail in this thread arrived at 07:31
and you declared "consensus" at 12:48. A total of ten people
participated in the thread in that five hour period, of over 3000
subscribed to the list.

I know you said this was for 2.0, and, if you really believe that's the
case, you have to really ensure that premature conclusions aren't drawn.

Regards,
Malcolm


Marc Fargas

unread,
Sep 10, 2008, 10:15:39 PM9/10/08
to django-d...@googlegroups.com
Hi,

On Wed, Sep 10, 2008 at 4:31 PM, mrts <mr...@mrts.pri.ee> wrote:
> Apps
> -------------------


> * extend them with Django-specific functionality to
> o lessen magic: explicitly specify files containing models,
> templates and admin bits, i.e. obsolete all forms of autodiscovery and
> path hacking
> o find a way to expose media
> o handle all issues outlined in #3591
>

I'd say that having CheeShop the only really important thing (my POV)
would be giving developers a few pointers on how to make their apps
more decoupled, i.e:
* Your app templates should all {% extends %} a base.html which is
the "standard" name of the base template.
* Your should work on the content, content-related, and
content-extra blocks to provide content in your example/live templates
* Your URL's should always use named urls (atleast on those meant
to be used by users).
etc. In brief, that kind of things that make an app "plug&play" or
"jumper-based" :)

Now blame me if such document already exists (in docs/), and such
document doesn't need to wait for 2.0 ;)

Regards,
Marc

--
http://www.marcfargas.com - will be finished someday.

Martin Diers

unread,
Sep 13, 2008, 6:35:34 PM9/13/08
to django-d...@googlegroups.com
On Sep 10, 2008, at 9:31 AM, mrts wrote:

>
> This is for Django 2.0.
>
> Rationale:
>
> a) useful (and less-useful) Django apps are proliferating on Google
> Code, djangoplugables.com and Django resources wiki page,
> b) apps need refactoring (#3591,
> http://code.djangoproject.com/wiki/VersionOneFeatures#INSTALLED_APPSrefactoring
> )
> c) reusable apps are immensely useful and may be one of Djangos great
> selling points once they are easy to deploy and some order is enforced
> upon the current chaos.
>

I do not believe that this has anything to do with Django 2.0 or any
version of Django.

The answer to the current package chaos is not a centralized
repository. I do not like that idea one bit. It's too storm-
trooperish. Who is going to decide whether a package is repository-
worthy?

The answer is community packaging guidelines. Somebody needs to write
or adapt an existing doc on how to package django apps using existing
Python tools (which are excellent already), how to name them, etc.
That doc could then be incorporated into djangoproject.com. The usual
community-driven caveats apply: let it be discussed, debated, etc. If
the major Django contributors begin using it for their own package
submissions, and the guidelines make sense, the community as a whole
will follow suit.

Julien Phalip

unread,
Sep 13, 2008, 7:48:29 PM9/13/08
to Django developers
On Sep 14, 8:35 am, Martin Diers <mar...@diers.us> wrote:
> The answer to the current package chaos is not a centralized  
> repository. I do not like that idea one bit. It's too storm-
> trooperish. Who is going to decide whether a package is repository-
> worthy?

I would take the example of jQuery. They have a single repository at
google code to store pretty much every plugin [1], the good the bad
and the ugly. So, no discrimination, anyone can have their plugin
hosted there. I think the jQuery team have a deal with Google to
handle such big volumes of data, for free.

There is also a centralized interface [2] for all the community
activities related to plugin development: discussion boards, voting,
bug tracking, etc.

I think that is a very sensible and useful approach. Everything is in
one place, and it is much easier to compare plugins and find the best
ones. It both keeps the jQuery community focused and makes users' life
a lot easier.

I would argue that this system was part of jQuery's success, and I
would love to see a similar thing with Django.

> The answer is community packaging guidelines. Somebody needs to write  
> or adapt an existing doc on how to package django apps using existing  
> Python tools (which are excellent already), how to name them, etc.  
> That doc could then be incorporated into djangoproject.com. The usual  
> community-driven caveats apply: let it be discussed, debated, etc. If  
> the major Django contributors begin using it for their own package  
> submissions, and the guidelines make sense, the community as a whole  
> will follow suit.

I agree and I like this idea. While I think packaging an app should
remain the responsibility of each app's developer, some documentation
with best practices (from a Django's perspective) would be welcome.

[1] http://code.google.com/p/jqueryjs/source/browse/#svn/trunk/plugins
[2] http://plugins.jquery.com/

Russell Keith-Magee

unread,
Sep 13, 2008, 8:23:49 PM9/13/08
to django-d...@googlegroups.com
On Sun, Sep 14, 2008 at 6:35 AM, Martin Diers <mar...@diers.us> wrote:
>
> The answer is community packaging guidelines. Somebody needs to write
> or adapt an existing doc on how to package django apps using existing
> Python tools (which are excellent already), how to name them, etc.

Somebody really is a great guy. He does a lot of great work for the
community, doesn't he :-)

If you think this document is required, why not write it? Don't wait
for permission - just write it!

Of course, if you actually sat down to write this document, you would
rapidly discover why it hasn't been written - the questions you need
to answer haven't got canonical answers yet. Are you familiar with the
Django Hotclub? This project started out to answer exactly these
questions, and they still don't have canonical answers. Pinax is an
attempt to discover best practices through empirical means, but they
don't have a complete set of answers.

Yours,
Russ Magee %-)

Matic Žgur

unread,
Sep 14, 2008, 6:46:08 AM9/14/08
to django-d...@googlegroups.com
I think it's a great idea to have a central repository of all apps.
Googlecode became an unofficial host of majority of third party apps,
so there seems to be a tendency for this kind of hosting. In my
opinion there is no need to reinvent hot water due to the fact that
Python already provides such facilities, so it would be the best if we
started using pypi. However, I see some space for improvements. Maybe
a voting system (like Ubuntu has with Add/Remove) to reward the good
apps and to separate them from the bad ones. Maybe we should work with
the upstream Python community and implement such system.

Matic Žgur

Martin Diers

unread,
Sep 14, 2008, 2:04:59 PM9/14/08
to django-d...@googlegroups.com


Look, that's exactly my point. Authorship aside, just throwing up a
platform for submitting packages accomplishes nothing without
packaging guidelines. Besides, we already have that. It's called
Google Code.

Pinax provides a model perhaps for the manner of packaging a project,
but as a whole is more cathedral than bazar. Is it the answer? I don't
know.

I don't know because I am not qualified to write such guidelines. I
have not been using Django long enough to understand all its parts
well enough, nor to propose how others should work. But I sure as hell
don't want to see yet another repository canonized before the footwork
is complete. Thus here's my vote: No.

Steve Holden

unread,
Sep 14, 2008, 6:18:46 PM9/14/08
to django-d...@googlegroups.com
Just to return to the original theme, I've been in touch with Richard
Jones, Pypi's (the Cheeseshop's) maintainer. He points out that there's
already a "Frameworks :: Django" Trove discriminator available, and
suggests that if Django developers have a need to change the code
(though I am not sure why they would) that too would be possible.

The only proviso to that is that the existing web APIs used by distutils
and setuptools need to be retained. If they weren't, of course, it would
kind of nullify the point of using PyPi in the first place.

regards
Steve

Kevin Teague

unread,
Sep 14, 2008, 10:52:32 PM9/14/08
to Django developers
As primarily a Zope/Plone/Grok developer, I thought I'd share some of
my experiences in working with the python packaging, dependencies and
builds aspects.

From my perspective, packaging and deploying Python applications are
one of the areas of Python that could use the most improvement - there
is a tonne of good and interesting work out there already, but the
approachability of this ecosystem can be fairly challenging. I've
attempted to teach new Python developers about project and dependency
management, but there are a lot of things to learn and a lack of good
beginner documentation, so at some point after rattling on about
setuptools, virtualenv, buildout, easy_install, PyPI and eggs they get
glazed over look and say, "symlinks and shell scripts work for me" too
which I answer, "yes, for your requirements, I think you're right."
Even many very talented, experienced developers have made mistaken
assumptions about this ecosystem at some point, there are threads on
the distutils-sig where people get stuck or confused on what should be
fairly straightforward things.

note: any mistakes in the following text is mearly proof that I'm a
"very talented, experienced developer" :)


PyPI
-----

Everyone knows what this is (hopefully!). A small point to note is
that it was originally called The Cheeseshop, there was a long thread
started by people who didn't like the name, and it was renamed to PyPI
and use of the name Cheeseshop is politely discouraged.

While there is only one PyPI, tools that pull packages from here
(easy_install and buildout) can easily pull from additional package
repositories. These can be mirrors, local caches, private repos, or
whatever. Creating your own repo is as easy as dropping a Python
packaged .tar.gz file into a directory and exposing it using Apache's
default directory views feature. There are also cool projects recently
started such as z3c.pypimirror which lets you create a local PyPI
mirror, but rather than mirror all 4+ GB of PyPI, it will only mirror
packages *as you request them*, very cool and probably something that
would be quite neat in a shared hosting environment.

Django really needs to update their PyPI page - it hasn't been updated
since 0.91! And it's not classified under the Frameworks::Django trove
either:

http://pypi.python.org/pypi/Django/

As far as "flooding" PyPI with packages, Zope2+Zope3+Plone has 1,236
packages on PyPI right now. There have been one or two grumbles about
this (it would seem people will grumble about just about anything),
but I'm pretty sure the consensus is if the number of packages becomes
unwieldly, the solution is to improve PyPI itself and not "start more
PyPIs".

The only valid grumbles about heavy PyPI usage are: packages with
really sparse classification and documentation, releasing -dev
packages onto PyPI (don't do this!), and especially *don't remove a
older packages* (really please, please don't do this!).

Plone add-ons were released to the Plone Software Center before PyPI
existed. Tarek Ziade has done awesome work at making it possible to
seamlessly release Plone add-ons to both plone.org and PyPI. This way
a package can have additional Plone-specific metadata, but still have
a PyPI presence without creating a bunch of extra maintenance work for
the add-on developer.


VirtualEnv
----------

http://pypi.python.org/pypi/virtualenv

VirtualEnv is fairly new, but it's become fairly popular fairly
quickly, since it's a very handy tool. It's basically a "Python
symlink utility" for cloning an existing Python installation so that
you can easily install packages into a different location than the
site-packages directory of your Python installation. With the --no-
site-packages switch you can also use a Python install even if you've
already hosed it with a million packages :P

Often people will use one virtualenv per project, installing packages
into here is called your "project-global" space.

VirtualEnv is awesome for "trying out" a collection of packages,
especially higher level packages that depend upon a number of other
packages and you don't want to pull a big gnarly mess into your site-
packages - as anyone who's every had the unfortunate experience of
doing a "sudo easy_install Plone" or "sudo easy_install grok" has
experienced.


Distutils
---------

http://docs.python.org/dist/dist.html

The original Python module installation tool. Typically invoked with
"python setup.py install", which installs packages into Python
installation global location (site-packages). Great for beginners
since they don't have to worry about PYTONPATHs, eggs or any of that
stuff but eventually you can wind up with a huuuuge ball of files in
your site-packages directory.

Distutils is so old that the documentation still reads, "Distributing
Python Modules" and not "Distributing Python Packages". Note that
distutils scope was for installing Python modules and packages - it's
not a very good tool for managing the installation of a complete web
app!

So while this venerable tool is a Python standard, please don't ask it
to do too much. It's very annoying when you run "python setup.py
install" and the package does more than just install itself, but also
tries to act as an "application installer" puts files in unexpected
places on your OS.


Eggs
----

A packaging format for Python, that requires the setuptools package
installed to use.

Eggs != setuptools! Eggs != easy_install!

It's a common misconception that if you are using eggs, you have to
use easy_install. Breaking setuptools into it's a core pkg_resources
library and a separate easy_install tool is something that people will
likely use the Python time machine for at some point :)

The egg format is an extension of the distutils format, changes
between the two formats is documented here:

http://peak.telecommunity.com/DevCenter/setuptools#new-and-changed-setup-keywords

The most interesting field is 'install_requires':

- install_requires: this allows you to specify what packages your
package depends upon. If you've just updated your django-foo package
to use 1.0 and it's incompatible with older Django releases, you could
note this with:

install_requires = ['django >= 1.0']

If the Django devs later create a concrete road map that makes
explicit your package will break in Django 1.1 you can even write:

install_requires = ['django >= 1.0 < 1.1']

This is useful, because if you're using a tool such as buildout, it
will use this information to fetch from PyPI the *newest release that
doesn't break your package*.

A bit of a red-herring field is the 'zip_safe' field. This lets you
specify if a package will still work while zipped. However, all
packages *always* work when unzipped! zipped packages can be pretty
annoying to deal with, and the resources saved by zipping packages is
pretty minimal and not a concern to most people.

Eggs is also not yet an "official Python standard". This isn't
entirely bad, since it's easier right now to lobby to extend this
format somewhat (e.g. install_recommends has been suggested in
addition to install_requires).


Namespace Packages
------------------

"Namespaces are one honking great idea -- let's do more of those!"
-- Tim Peters

Zope 3 started life as a single source tree, that looked something
like this:

zope/
__init__.py
testing/
interface/
app/

As the project grew, some people just wanted to be able to use a few
core libraries and not have to install the whole damn kitchen sink. If
you are distributiong zope.testing and zope.interface as separate
packages so that you have:

/opt
/zope
/testing

/opt2
/zope
/interface

There is no way to unify these two packages with symlinks or
PYTHONPATHs so that you can import from both. Namespaces simply solves
this problem.

Culturally, namespaces are also used to reduce the chance of namespace
collisions. In the Zope world there are:

- zope.* : Zope's original namespace package
- zope.app.* : Zope uses "app" as a nested namespace for packages
which are concerned with application code (e.g. Views n' URLs n'
Models).
- z3c.* : Short for the "Zope 3 Community", and typically these
packages are all managed in the main Zope svn repo.
- megrok.* : Namespace for add-ons written for Grok.
- plone.* : Namespace for code written by the Plone project that can
be used outside Plone.
- plone.app.* : Namespace for code written by the Plone project which
*can't* be used outside Plone.
- collective.* : Very early on in the Plone project, in the dark days
of the internet when we only had SourceForge, it was far too much
hassle to create an entire SourceForge project for a very simple Plone
add-on. So they created an umbrella project called the Collective and
were *very* free with allowing anyone who requested it access to this
svn repo. As a testament to the FOSS community in general, it's
interesting just how rarely open access to a source tree is abused.
- lovely.* : Namespace for packages written by Lovely Systems.
Various other Zope-centric companies will release code that they
contributed under the name of their organization.


easy_install
------------

An application for downloading and installing Python packages.

By default it installs into a Python global location. People new to
this tool often "shoot themselves in the foot" by doing "sudo python
easy_install package_name" and making a right mess of their site-
packages directory. At which point they decide they hate easy_install.

However, easy_install can be used to install to other locations! Or
more commonly, combined with VirtualEnv to install into a project
global location. This is the recommended install process for
TurboGears 2, where they are publishing a PyPI-like index for each
release of TurboGears:

$ easy_install -i http://www.turbogears.org/2.0/downloads/current/index
tg.devtools


zc.buildout
-----------

http://pypi.python.org/pypi/zc.buildout

A tool for for creating applications, especially Python applications.

The phrase "repeatable deployments" gets used a lot with Buildout.
This just means that you can take an application, checkout the project
on a clean system, run ./bin/buildout, and the same configuration and
installation actions will be performed. Think of it as maintaining a
buildout.cfg file for your project instead of an INSTALL.txt file.

This is similar to Ruby's Capistrano tool - although the two tools
really differ quite a bit in what they do. Capistrano puts a large
focus on describing your deployment infrastructure and managing
commands remotely. Buildout is smaller in scope in that it is only
concerned with installing parts of your web application locally.

Buildout is a configuration-driven build tool. You define a
buildout.cfg file that describes the parts that compose your project
or application. Only when the configuration of a part changes does
buildout re-build that part. This is different from a "source code
recompilication" focused build tools such as SCons. If you are working
in a compiled language, buildout is not the tool for you.

One of Buildout's unique innovations is that the recipes that describe
how a part gets installed are themselves Python eggs. These recipes
can be be included as part of your project, or they can be published
on PyPI and shared with other buildout users.

A sample buildout.cfg file might look like this:

[buildout]
parts = oldervirtualpython newervirtualpython

[oldervirtualpython]
recipe = zc.recipe.egg
interpreter = oldpython
eggs = zope.interface == 3.3

[newervirtualpython]
recipe = zc.recipe.egg
interpreter = newpython
eggs = zope.interface

When buildout is run with this configuration, it will create two
scripts (named oldpython and newpython) that act as wrappers around
the Python interpreter. The 'oldpython' wrapper will have
zope.interface version 3.3 installed, while the 'newpython' wrapper
will have the latest version of zope.interface installed (currently
3.4.1). In this way you can address more complex use cases, such as a
web site where you have one section of the site relying on an older
version of Django, and another section of the site using the latest
version.

There are Buildout recipes to install Django such as this one:

http://pypi.python.org/pypi/djangorecipe

Buildout has a bit of a reputation for being "opinionated", although
this isn't entirely true. Buildout is completely agnostic as to what/
how it builds, but there are certain conventions within existing
recipes, such as recipes in the zc.recipe.* being very egg-centric or
having the policy that if the configuration for a part changes, it's
rebuild by blowing away the existing part and re-install from scratch.

Note that Recipes such as zc.recipe.egg will work in a shared hosting
environment. At my organization I like it because it lets less
experienced developers install Python packages into their home
directories without needing to either give them root or teach them how
to compile Python.

It's worth noting that there are few more competitors in the Python-
based project build space. Tools such as Fassembler, Vellum and Paver
all have interesting ideas and could be a reasonable replacement for
Buildout (or Rake or Capistrano if you like that Ruby junk :P).
However, I'm only going to talk about Buildout since it is the only
actively developed Python "repeatable deployment" project to date to
have passed the 1.0 mark.

So why go to the hassle of learning a build tool and maintaining a
build configuration for your projects? Tarek's recent OSCON slides
(http://tarekziade.wordpress.com/2008/07/25/my-oscon-slides-online/)
touch on one point. See slide 6, which describes a Zope-based
application install as "5 hours in 2006: install python extra
packages, get zope, install zope, create an instance, get extra
products, read extra products docs, install extra products
dependencies, install extra products, doesn't work, ahh right, install
python-ldap, checkout products in development, doesn't work, ahh right
wrong python-ldap version ... start to work." then the next slide, "5
minutes in 2008".

There is a second ancillary motivation, and that's to enable the
Django community to "play nicer" (note I said "nicer" I ain't saying
you folks are mean ... :P) in the Python ecosytem. A good way to do
this is to use more non-Django python packages and to contribute back
to them, and to spin-off the non-Django specific parts of a project so
that others can more easily use them. However, in order to do this,
your project's install is going to require an ever growing list of
package dependencies. If you don't have a tool for automatically
managing these dependencies, you will drive yourself batty doing
manual installs and the impetus to "just keep it all together in a
single, monolithic package" will be much greater. Since adopting
Buildout I've even found a couple instances where I'll take some
internally developed code and split it into a seperate egg only
intended for internal re-use, when in the past I would have just
employed cut-n-paste re-use.

Zope 3 started life as a single source tree, and it wasn't broken into
eggs until 2006. As a result of not wanting to break backwards
compatability, even though Zope 3 was split into some 80+ individual
python packages, many packages have unfortunate couplings between each
other. This has the effect that often pulling in one specific Zope 3
package to your project you "pull in the world" and you end up getting
all of Zope 3 :(

http://mail.zope.org/pipermail/zope-dev/2008-September/033085.html

A very cool side-effect of having an egg-heavy project is being able
to visualize the dependency tree. Marius Gedminas used the tl.eggdeps
package to visualize the dependency tree expressed in the Zope 3
'install_requires' fields:

http://mg.pov.lt/blog/2007/09

Another cool side-effect is the ability to easily determine what has
changed between two releases of a framework (yeah, I know, that's what
CHANGELOGs are for). Say I've got a web app using Grok 0.11 and want
to upgrade to Grok 0.13. The list of packages and versions that
compose these two releases are here:

http://grok.zope.org/releaseinfo/grok-0.11.cfg
http://grok.zope.org/releaseinfo/grok-0.13.cfg

I can take these two files and run a diff between them. I also started
(but didn't get very far due to lack of time and motiviation) to
create a web app which would allow you to better visualize the
differences between two lists of eggs, highlighting packages whose
major or medium version numbers had incremented. This would for
example highlight 'django.form' in bright red between the 0.96 and 1.0
release. Even cooler would be for developers to create "playlists" of
interesting sets of packages (such as suite of testing and code
analysis tools), and share them with each other.

Note that Buildout is often "developer centric". This means that it's
not always intended to replace your operating systems existng
packaging system. Your SysAdmins may really like Debian or Red Hat and
insist upon production deployments using as their OS packaging system
- especially if they are security conscious. However, it's often a
PITA to develop using the same environment used in production (and
with Google App Engine it's not even possible to at all), so Buildout
is often used to repeat the install *only for development
environments*. For example my own small web-based LDAP management tool
has a buildout.cfg file that only bothers to configure LDAP on Mac OS
X, since that's what I use for development (http://www.bcgsc.ca/
platform/bioinfo/software/gum). In no way did I suggest to our
sysadmins that they use Buildout to configure our production LDAP
servers, since they'd rightfully just look at me like I was nuts.


grokproject
-----------

http://pypi.python.org/pypi/grokproject

In the Grok framework (yay Grok!) (and also zopeproject for Zope 3),
the framework does not have an installer built into it! Instead the
framework installation experience is as follows:

1. Install grokproject.

2. Run grokproject and answer the prompts to create a sample project.
This will generate a starting buildout.cfg file and run Buildout
against that file.

Or alternatively:

1. Checkout an existing sample project from SVN.

2. Run the "bootstrap.py" script to install Buildout.

3. Run ./bin/buildout to install the sample app, as well as the Grok
framework.

Yes, it's possible to easy_install the Grok framework inside a
VirtualEnv, but since the Grok framework is composed of 107 (zoiks!)
python packages, run easy_install at any given point is can give you a
different set of versions of the packages installed. TurboGears 2 also
works around this by publishing their own "TurboGears PyPI repo" for
every release of TG2.

Since installing that many packages means a lot of network
connections, grokproject now downloads a zip file containing all the
packages (there are also other tools for packaging up all the files
fetched over the net when using buildout so that purely off-line
installs are possible). The beauty of this approach is you can also
have a user-specific config file that states that you'd like to have a
local cache of all eggs, so if you already have a bunch of apps
installed, installing new ones is very quick and requires very little
disk space. For example, my ~/.buildout/default.cfg has:

[buildout]
eggs-directory = /Users/kteague/buildouts/shared/eggs
download-cache = /Users/kteague/buildouts/shared/cache

The important thing to note though, is that the framework doesn't have
an installer. In the olden days, we would run the Zope 2 installer,
and then we'd have a location where we could put our own Python code.
This is totally bass-ackwards. Instead you write own Python code, and
then either in setup.py or buildout.cfg, you state that your code
depends upon the framework.

For example, in one Grok app that I'm working on I have in the
setup.py file:

install_requires=['setuptools',
'grok',
],

The Grok line is a bit of red-herring, since I'm using a supplying a
list of packages and versions to "pin" my app to a specific release of
Grok. But when I wanted to use the Grok add-on 'megrok.form', I
changed the install_requires to read:

install_requires=['setuptools',
'grok',
'megrok.form',
],

Inside the megrok.form 0.25 package is the following install_requires:

install_requires=['setuptools',
'grok',
'zc.resourcelibrary == 1.0.1',
'z3c.widget == 0.1.6',
'zc.datetimewidget == 0.5.2',
'collective.namedfile == 1.1',
'collective.namedblobfile == 0.3',
],

However, I do not have to install megrok.form manually, nor do I have
to track down and manually install the correct versions of
zc.resourcelibrary, z3c.widget, zc.datetimewidget,
collective.namedfile, collective.namedfile. Instead I just add one
line to my project's install_requires, re-run './bin/buildout' to
freshen my project's install, and huzzah! the packages are
automagically installed (and not in a global location!). Then I check
that change into Mercurial, ping the developer I'm collaborating with,
and he pulls my changes to his workstation, runs './bin/buildout' -
and we've seamleslsy shared the exact same install process and have
the same working set of python packages installed :)

If a new developer starts on the project, they clone the Mercurial
project, run "python bootstrap.py; ./bin/buildout" and a few minutes
later they also have a running application that they can begin hacking
on. Beautiful!


Conclusion
----------

We'll I've not even touched on anything Django-specific, and this e-
mail is already head-spinningly long. As Malcom pointed out, reaching
a consensus on some of these issue is going to take more than a few
hours. As you can tell, I'm pretty happy with the switch from
distutils/framwork install process to the eggs/buildout install
process. But there are other tools and approaches to these problems,
so I'd encourage people to try out different things, and let others
know what worked, what didn't work, where you're getting stuck, and
wouldn't worry too much about deciding upon a "standard" just yet. And
most of all, hopefully some people will contribute back in the form of
beginner friendly documentation so that we can get to the point where
Python gets a reputation as being "easy to deploy" :)

Jacob Kaplan-Moss

unread,
Sep 15, 2008, 4:58:42 AM9/15/08
to django-d...@googlegroups.com
On Mon, Sep 15, 2008 at 3:52 AM, Kevin Teague <ke...@bud.ca> wrote:
> [...]

Holy crap, Kevin, this is fantastic. Can you post this somewhere other
than the mailing list so that it'll have a permanent home?
Alternatively, do you mind if I do?

Jacob

Marc Garcia

unread,
Sep 15, 2008, 5:33:38 AM9/15/08
to Django developers
Hey Jacob,

not sure if it's what you want, but Kevin's message already has a
permanent (and individual) url:
http://groups.google.com/group/django-developers/msg/5407cdb400157259

Of course a blog post, or any pther place would be better.

On Sep 15, 1:58 am, "Jacob Kaplan-Moss" <jacob.kaplanm...@gmail.com>
wrote:

Jacob Kaplan-Moss

unread,
Sep 15, 2008, 5:57:30 AM9/15/08
to django-d...@googlegroups.com
On Mon, Sep 15, 2008 at 10:33 AM, Marc Garcia <garci...@gmail.com> wrote:
> not sure if it's what you want, but Kevin's message already has a
> permanent (and individual) url:
> http://groups.google.com/group/django-developers/msg/5407cdb400157259

Yeah, yeah, I know... but there's a very important mental difference
between a link to a mailing list post and a "real" web page; the
latter has a bit more gravitas, and I'd like to make sure that Kevin's
information gets read as much as possible.

Jacob

mrts

unread,
Sep 15, 2008, 6:15:21 AM9/15/08
to Django developers
A few days ago I started a wiki page to consolidate all the info in
this thread. However, someone has cleared the page. Is this
intentional?

The initial (incomplete mindmap-ish) version is here:
http://code.djangoproject.com/wiki/DjangoSpecifications/AppGuidelines?action=diff&version=5

Should I continue adding information there or is someone planning to
write the guidelines in some different medium?

Simon Willison

unread,
Sep 15, 2008, 6:32:40 AM9/15/08
to Django developers
On Sep 15, 9:58 am, "Jacob Kaplan-Moss" <jacob.kaplanm...@gmail.com>
wrote:
> On Mon, Sep 15, 2008 at 3:52 AM, Kevin Teague <ke...@bud.ca> wrote:
> > [...]
>
> Holy crap, Kevin, this is fantastic.

Completely agree. This is the missing manual - in five minutes you've
taught me more about the Python packaging environment than several
years of failed attempts at understanding the setuptools documentation.

Erik Allik

unread,
Sep 15, 2008, 6:35:14 AM9/15/08
to django-d...@googlegroups.com
This question might sound stupid, but couldn't we put this text on the
Django docs page or at least integrate it with the docs? This is
something we might want every newcomer to (at least have the
opportunity to) read.

Erik

mrts

unread,
Sep 15, 2008, 6:40:43 AM9/15/08
to Django developers
Kevin's "manual" needs to be amended with Django-specific concrete
HOWTO-like guidelines.

mrts

unread,
Sep 16, 2008, 3:16:52 AM9/16/08
to Django developers
Let me try to wrap this up:

* there seems to be a general consensus that amending setuptools to
create Django-specific extensions is not required
* a separate repository is not required, apps should be published to
PyPi Famework :: Django category but a central site that tracks the
apps there to further classify, rate and enhance them would be useful
(like djangoplugables.com)
* app naming and namespace issues should be resolved (should all apps
have a django-prefix, the de facto standard? should they live in
django.apps namespace?)
* dependencies should be handled with setuptools facilities
* Django releases should be packaged and published in PyPi, apps
should state the version of Django they depend on
* extensibility (at least templates and override points) and
publishing app media (symlinks?) should be addressed
* generally we should strive for a hassle-free experience so that
`easy_install django-foo` gives you an expected entry point (`from
django.apps import foo`) and works-out-of-the-box feel
* there should be official guidelines for both app authors and users
that document the above, give best-practice advice (virtualenv),
provide HOWTO-like instructions and configuration examples for
packagers; docs/internals/apps.txt perhaps.

James, as you have been visibly active in the reusable apps front (the
"Writing reusable applications" slides are excellent), can you perhaps
lieutenant this?

Simon Willison

unread,
Sep 16, 2008, 3:28:37 AM9/16/08
to Django developers
On Sep 16, 8:16 am, mrts <m...@mrts.pri.ee> wrote:
> * app naming and namespace issues should be resolved (should all apps
> have a django-prefix, the de facto standard? should they live in
> django.apps namespace?)

I don't understand the benefit of having third party apps live inside
the "django" namespace. At the moment, that namespace is reserved for
official django releases, which makes a lot of sense to me. Django
doesn't care where an application lives as long as it is on the Python
path - at GCap, we've been putting all of our apps in "gcap.apps" and
it's worked well for us. Third party Django apps can live anywhere.

Cheers,

Simon

mrts

unread,
Sep 16, 2008, 3:46:24 AM9/16/08
to Django developers
See Kevin Teague's post above, section "Namespace packages" for
general considerations.

Apart from that, having all packages in a known namespace provides a
good entry point for introspection, e.g. you can ask questions like
"What Django applications are installed in my environment?" and do
several interesting things with the answer (creating symlinks for
media, adding apps to settings.py etc).

Erik Allik

unread,
Sep 16, 2008, 4:01:45 AM9/16/08
to django-d...@googlegroups.com
The django.apps.* prefix idea is kinda odd. Mostly I've seen
namespaces that are like org.orgname.* or com.compname.* etc.

Erik

Simon Willison

unread,
Sep 16, 2008, 4:02:00 AM9/16/08
to Django developers
On Sep 16, 8:46 am, mrts <m...@mrts.pri.ee> wrote:
> On Sep 16, 10:28 am, Simon Willison <si...@simonwillison.net> wrote:
> > I don't understand the benefit of having third party apps live inside
> > the "django" namespace. At the moment, that namespace is reserved for
> > official django releases, which makes a lot of sense to me. Django
> > doesn't care where an application lives as long as it is on the Python
> > path - at GCap, we've been putting all of our apps in "gcap.apps" and
> > it's worked well for us. Third party Django apps can live anywhere.
>
> See Kevin Teague's post above, section "Namespace packages" for
> general considerations.

Kevin's post explained what they were, but didn't really touch on why
they were a good idea. My understanding is that zope.* is reserved for
zope core stuff, whereas z3c.* is the "Zope community" namespace which
sounds a bit more like what we'd be doing with django apps.

> Apart from that, having all packages in a known namespace provides a
> good entry point for introspection, e.g. you can ask questions like
> "What Django applications are installed in my environment?" and do
> several interesting things with the answer (creating symlinks for
> media, adding apps to settings.py etc).

That's an interesting ability, but a principle aim of the
INSTALLED_APPS setting is to allow lots of apps to be installed on a
system but only specific apps to be "turned on" for a given Django
powered site. If there are significant disadvantages to namespace
packages for third party apps (they feel pretty icky to me, but I
can't exactly explain why) I don't see introspection as a strong
reason to have them. I'd need some pretty solid convincing at any
rate.

Cheers,

Simon

mrts

unread,
Sep 16, 2008, 4:27:58 AM9/16/08
to Django developers
On Sep 16, 11:02 am, Simon Willison <si...@simonwillison.net> wrote:
> That's an interesting ability, but a principle aim of the
> INSTALLED_APPS setting is to allow lots of apps to be installed on a
> system but only specific apps to be "turned on" for a given Django
> powered site. If there are significant disadvantages to namespace
> packages for third party apps (they feel pretty icky to me, but I
> can't exactly explain why) I don't see introspection as a strong
> reason to have them. I'd need some pretty solid convincing at any
> rate.

Namespace packages and packages living in a namespace are different
things. I'd say the latter is sufficient.

As of now, media handling could be one of the main selling points of
coordinated namespace access (as there are no interesting use cases
for general introspection yet):
* create a single symlink from central app media location to
MEDIA_ROOT
* serve that location with Apache
* add APP_MEDIA setting that behaves like ADMIN_MEDIA to Django
settings
* app `foo` expects it's media to live in APP_MEDIA + 'foo'

As for naming, `django.apps` is an arbitrary name, it can be
`djangoapps` or whatnot. To avoid bikeshedding, core devs could decide
on that.

Of course, company sub-namespaces are entirely welcome, i.e.
`django.apps.mycompany.foo` or `django.apps.com.mycompany.foo` should
be equivalently accepted. I personally don't like too deeply nested
and verbose namespaces and they are less common in Python than they
are in Java, but YMMV.

Kevin Teague

unread,
Sep 16, 2008, 6:51:30 AM9/16/08
to Django developers


On Sep 15, 1:58 am, "Jacob Kaplan-Moss" <jacob.kaplanm...@gmail.com>
wrote:
Aww, thanks :)

I've made a copy of the mail on my internet web log:

http://bud.ca/blog/pony

Folks are free to copy text into anything documentationish if'n they'd
like - although I'd recommend liberal editing, some of text is a bit
rambly.

Malcolm Tredinnick

unread,
Sep 16, 2008, 7:10:30 AM9/16/08
to django-d...@googlegroups.com

On Tue, 2008-09-16 at 00:16 -0700, mrts wrote:
> Let me try to wrap this up:
>
> * there seems to be a general consensus that amending setuptools to
> create Django-specific extensions is not required
> * a separate repository is not required, apps should be published to
> PyPi Famework :: Django category but a central site that tracks the
> apps there to further classify, rate and enhance them would be useful
> (like djangoplugables.com)
> * app naming and namespace issues should be resolved (should all apps
> have a django-prefix, the de facto standard? should they live in
> django.apps namespace?)

As others have pointed out, this isn't necessary and would be bad
practice. Everybody should feel free to use whatever namespace they like
to avoid clashes and so that we avoid lowering the quality of the Django
brand (as soon as somebody installs a few low-quality "packages" and
they misbehave in any way, it's going to be "Django is a pile of
rubbish" because it's all coming from the Django namespace).

> * dependencies should be handled with setuptools facilities

As an option, only. This can't be a requirement, since setuptools will
often download random stuff from the internet, doesn't support signed
packages as a rule, doesn't integrate well with existing packaging
systems such as rpm and apt (which includes the fact that it downloads
random extra stuff without asking) and doesn't make it easy to validate
the source code at a later date. In other words, it's insecure by design
and therefore inappropriate for large system installs. People might be
happy to use it on their laptops or whatever, but a lot of system
administrators should rightly feel very nervous about using setuptools.

Regards,
Malcolm

mrts

unread,
Sep 16, 2008, 8:06:12 AM9/16/08
to Django developers
Both of these arguments are valid, but will not help to resolve the
problem: smooth experience in distributing and using pluggable
applications and some order in the current chaos.

On Sep 16, 2:10 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
> On Tue, 2008-09-16 at 00:16 -0700, mrts wrote:
> > Let me try to wrap this up:
>
> > * there seems to be a general consensus that amending setuptools to
> > create Django-specific extensions is not required
> > * a separate repository is not required, apps should be published to
> > PyPi Famework :: Django category but a central site that tracks the
> > apps there to further classify, rate and enhance them would be useful
> > (like djangoplugables.com)
> > * app naming and namespace issues should be resolved (should all apps
> > have a django-prefix, the de facto standard? should they live in
> > django.apps namespace?)
>
> As others have pointed out, this isn't necessary and would be bad
> practice. Everybody should feel free to use whatever namespace they like
> to avoid clashes and so that we avoid lowering the quality of the Django
> brand (as soon as somebody installs a few low-quality "packages" and
> they misbehave in any way, it's going to be "Django is a pile of
> rubbish" because it's all coming from the Django namespace).

Lets not use the `django` namespace then -- `djangoapps`, `djc` (as
django community, mimicking z3c), whatnot. Do you think the
introspection and media arguments are irrelevant? These would provide
practical gain apart from the general feel of coherence I'm trying to
advocate.

As for coherence: standardized toplevel namespace would help to
discern at a glance which bits of a project are Django-related and
which are not, just by looking at the imports in a .py. Everybody
should feel free to use whatever namespace or namespace hierarchy they
like *under* that namespace.

As for the brand, the existing practice for naming packaged Django
apps is to use the 'django-' prefix anyway. Should that be discouraged
(which doesn't sound good to me)?

> > * dependencies should be handled with setuptools facilities
>
> As an option, only. This can't be a requirement, since setuptools will
> often download random stuff from the internet, doesn't support signed
> packages as a rule, doesn't integrate well with existing packaging
> systems such as rpm and apt (which includes the fact that it downloads
> random extra stuff without asking) and doesn't make it easy to validate
> the source code at a later date. In other words, it's insecure by design
> and therefore inappropriate for large system installs. People might be
> happy to use it on their laptops or whatever, but a lot of system
> administrators should rightly feel very nervous about using setuptools.

apt and rpm don't mix with project-specific packages and versioning
(virtualenv/buildout). It is just not feasible (or even possible if
something is built against django-0.96 and something else against
django-1.0) to dump everything into the global package space if you
have diverse projects hosted on a single server.

setuptools is a necessary evil for the agile developer who frequently
tracks updates for the bits he relies upon. Hopefully it will be
improved (the clamor around it is ever-ongoing), but unless we want to
do it ourselves, we have to accept the state of things.

As for security: a centralized approach provides better protection,
e.g. sha256sums could be published on the app tracker as well as
security alerts. Otherwise, everybody has to review the code
themselves. As mentioned above, other large projects already have a
semi-controlled extension environment, so it is not a new problem.

Tarek

unread,
Sep 16, 2008, 11:03:36 AM9/16/08
to Django developers
Great mail !

This has to be published somewhere like Jacob is suggesting.

I'd like to react on the PyPI part:


On 15 sep, 04:52, Kevin Teague <ke...@bud.ca> wrote:
>
> PyPI
> -----
>
> Everyone knows what this is (hopefully!). A small point to note is
> that it was originally called The Cheeseshop, there was a long thread
> started by people who didn't like the name, and it was renamed to PyPI
> and use of the name Cheeseshop is politely discouraged.
>
> While there is only one PyPI, tools that pull packages from here
> (easy_install and buildout) can easily pull from additional package
> repositories. These can be mirrors, local caches, private repos, or
> whatever. Creating your own repo is as easy as dropping a Python
> packaged .tar.gz file into a directory and exposing it using Apache's
> default directory views feature. There are also cool projects recently
> started such as z3c.pypimirror which lets you create a local PyPI
> mirror, but rather than mirror all 4+ GB of PyPI, it will only mirror
> packages *as you request them*, very cool and probably something that
> would be quite neat in a shared hosting environment.

As far as I know (and as I said to Andreas Jung when they created this
package during a sprint lately),
z3c.pypimirror is a *full* mirror. It doesn't allow you to get only
some packages. It has a name filter,
so you can get for instance all package that starts with "foo." but
this is a non sense because this
package might have a "bar" package in its dependency, it won't grab
it.

So basically it is a rsync, and rsyncing 5 gigas is taking a bit of
time (I think they are taking care of this aspect now, I am not sure
though)

We developed before z3c.pypimirror a package called iw.eggproxy that
is a proxy that will simply get all the package
tools like easy_install or buildout are asking for. So if you work in
a zope environment it will get only the package you work on.

>
> Django really needs to update their PyPI page - it hasn't been updated
> since 0.91! And it's not classified under the Frameworks::Django trove
> either:
>
> http://pypi.python.org/pypi/Django/
>
> As far as "flooding" PyPI with packages, Zope2+Zope3+Plone has 1,236
> packages on PyPI right now. There have been one or two grumbles about
> this (it would seem people will grumble about just about anything),
> but I'm pretty sure the consensus is if the number of packages becomes
> unwieldly, the solution is to improve PyPI itself and not "start more
> PyPIs".
>
> The only valid grumbles about heavy PyPI usage are: packages with
> really sparse classification and documentation, releasing -dev
> packages onto PyPI (don't do this!), and especially *don't remove a
> older packages* (really please, please don't do this!).
>
> Plone add-ons were released to the Plone Software Center before PyPI
> existed. Tarek Ziade has done awesome work at making it possible to
> seamlessly release Plone add-ons to both plone.org and PyPI. This way
> a package can have additional Plone-specific metadata, but still have
> a PyPI presence without creating a bunch of extra maintenance work for
> the add-on developer.

I tried to use PyPi software at first but it was hard to reuse.

I have created this PloneSoftwareCenter extension in order to be able

- to make plone.org accept eggs through distutils calls, like you said
(otherwise the products section of plone.org is doomed, facing pypi's
one)
- to run my own PyPI for private packages

But there's a very important point about this work: I don't want PyPI
to be a single point of failure. PyPI server is pretty robust,
but I don't want my developers to depend on it while they are working.

It happened in the past that we couldn't work because of that, because
zc.buildout grabs a myriad of eggs at pypi to build Plone,
and because we are building and upgrading buildouts all day long. (We
have mirrors and caches everywhere now ;) )

Our strategy for the future on this will be to be able to use
distutils/setuptools and zc.buildout with several packages indexes
(pypi, plone.org or any other), and
to push packages on several places and sometimes with extra metadata
like you said (plone.org will have a rating system)

I have changed Python 2.6 for that (http://tarekziade.wordpress.com/
2008/05/12/the-new-pypirc-format-in-python-distutils/) so you
can deal with several pypi-like servers and I have a patch in
setuptools that is pending, so easy_install can play with several
servers as well/

So basically, a developer will be able to push any package to any
server, and a tool like zc.buildout or easy_install
will be able to get a package from a list of pypi-enabled servers.
(and if one of them is down , the package might be in to another one)

So if you decide to deploy Django like zope, at pypi, I would
encourage you to set up your own community pypi-like server, like we
are going to do on plone.org.

The work done on python, setuptools, zc.buildout and Plone.org to make
this possible in Plone
will be presented in a tutorial at the plone conf in DC in two weeks.
I think it can be interesting for the django community to take a look
at it.

Cheers

Tarek

mrts

unread,
Sep 16, 2008, 11:20:33 AM9/16/08
to Django developers
On Sep 16, 2:10 pm, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
> On Tue, 2008-09-16 at 00:16 -0700, mrts wrote:
> > Let me try to wrap this up:
>
> > * there seems to be a general consensus that amending setuptools to
> > create Django-specific extensions is not required
> > * a separate repository is not required, apps should be published to
> > PyPi Famework :: Django category but a central site that tracks the
> > apps there to further classify, rate and enhance them would be useful
> > (like djangoplugables.com)
> > * app naming and namespace issues should be resolved (should all apps
> > have a django-prefix, the de facto standard? should they live in
> > django.apps namespace?)
>
> As others have pointed out, this isn't necessary and would be bad
> practice. Everybody should feel free to use whatever namespace they like
> to avoid clashes and so that we avoid lowering the quality of the Django
> brand (as soon as somebody installs a few low-quality "packages" and
> they misbehave in any way, it's going to be "Django is a pile of
> rubbish" because it's all coming from the Django namespace).

Another issue I have with this: we grab generic toplevel names like
"tagging", "registration" etc for apps that are coupled to Django.
Semantic names that clearly state the purpose of a package help to
make code more readable are of course good. But what if TurboGears
people want to name their tagging app "tagging" as well (instead of
Tasty as it's named now)? Ideally, I'd like to see code like this:

from djc import tagging # or `from django.apps import tagging`

def foo():
tagging.do_bar() # OK, at a glance I can see that this line
relates to tagging

instead of

import tasty # `import tasty as tagging` *can* be done, but oh well...

def foo():
tasty.do_bar() # What is tasty?! *Waste time for looking it up*
Aha, this line is for tagging


Also, there may be differences in toplevel generic `frobnicate`
(usable in Python in general) and `django.apps.frobnicate` (coupled to
Django and packaged as django-frobnicate) that adapts the former to
Django.

Malcolm Tredinnick

unread,
Sep 16, 2008, 7:19:32 PM9/16/08
to django-d...@googlegroups.com

Don't tell that to my laptop and some systems I admin which have
completely parallel installation of Python 2.4 and everything else I
need to do Plone work, all managed by rpms, then. They don't like being
told that what they're doing is apparently impossible. :-)

Yes, it doesn't work trivially with virtualenv, but so what? It's not
part of the use-case of virtualenv and buildout which are targeted at an
different audience and use-case. Sandboxed development environments have
been in use for years and they work very well when you're developing.
They're also not how you deploy things to dozens of machines at once in
a very large installation. At that point you use the sandboxed
environment to build packages that work with the system they're being
installed on and which install in parallel. Not every sandbox
environment is designed to do that and that is fine.

> to dump everything into the global package space if you
> have diverse projects hosted on a single server.
>
> setuptools is a necessary evil for the agile developer who frequently
> tracks updates for the bits he relies upon. Hopefully it will be
> improved (the clamor around it is ever-ongoing), but unless we want to
> do it ourselves, we have to accept the state of things.

Wow. You just said we should entirely ditch security because it isn't
convenient. That's exactly the attitude that has made Microsoft the
laughing stock of the professional IT community for the last 15 years.

If my original mail had said "nobody should ever use setup tools" or
anything like that, you reply might be valid. Go back and read it. I
said "optional", not "forbidden". There are multiple audiences here. If
Django ever had a hard dependency on setuptools for using third-party
packages it would make it very difficult for system administrators to
use Django because of the large level of security uncertainty and extra
overhead that comes with using it. So, again, it's fine for people who
want to to use setuptools and virtualenv and buildout and eggs and
rinky-dinky-tool-of-choice. People use things knowing that there are
always trade-offs and compromises. It's bad once policy starts being
enforced on something that is simply unusable in certain environments.

Regards,
Malcolm


Steve Holden

unread,
Sep 16, 2008, 8:06:39 PM9/16/08
to django-d...@googlegroups.com
Malcolm Tredinnick wrote:
> On Tue, 2008-09-16 at 05:06 -0700, mrts wrote:
>
> [...]

>> setuptools is a necessary evil for the agile developer who frequently
>> tracks updates for the bits he relies upon. Hopefully it will be
>> improved (the clamor around it is ever-ongoing), but unless we want to
>> do it ourselves, we have to accept the state of things.
>>
>
> Wow. You just said we should entirely ditch security because it isn't
> convenient. That's exactly the attitude that has made Microsoft the
> laughing stock of the professional IT community for the last 15 years.
>
>
Which, sadly, hasn't affected either their profits or their market
dominance.

> If my original mail had said "nobody should ever use setup tools" or
> anything like that, you reply might be valid. Go back and read it. I
> said "optional", not "forbidden". There are multiple audiences here. If
> Django ever had a hard dependency on setuptools for using third-party
> packages it would make it very difficult for system administrators to
> use Django because of the large level of security uncertainty and extra
> overhead that comes with using it. So, again, it's fine for people who
> want to to use setuptools and virtualenv and buildout and eggs and
> rinky-dinky-tool-of-choice. People use things knowing that there are
> always trade-offs and compromises. It's bad once policy starts being
> enforced on something that is simply unusable in certain environments.
>
Yes, anyone looking for "the one true way" is doomed to disappointment.
You only have to review recent correspondence on python-dev to realize
that no one person is aware of all the requirements of the various
platforms for which modern applications must be packaged.

I don't much like the pejorative implication of matching
"rinky-dinky-tool-of-choice" with the other listed alternative, however.
Personally I think virtualenv is a good solution to a tricky problem.

The thing that bugs me most of all is Python developers who assume that
the answer to all Python distribution problems is the assumption that a
single Python installation can support all required applications. In
this day and age it's quite practical to look at distributing the
required interpreter as a component of your application or framework.
There is no reason why your Python program and mine need have any
components in common, and increasingly less reason to enforce this
requirement to save disk space.

Having said which, Django and other frameworks don't have that luxury:
if I'm running Django on 2.4 then every app my Django installation
supports also has to run on 2.4.

regards
Steve

Kevin Teague

unread,
Sep 16, 2008, 8:51:59 PM9/16/08
to Django developers
> > apt and rpm don't mix with project-specific packages and versioning
> > (virtualenv/buildout). It is just not feasible (or even possible if
> > something is built against django-0.96 and something else against
> > django-1.0)
>
> Don't tell that to my laptop and some systems I admin which have
> completely parallel installation of Python 2.4 and everything else I
> need to do Plone work, all managed by rpms, then. They don't like being
> told that what they're doing is apparently impossible. :-)

Yes, it's perfectly possible to use Buildout and rpms in conjunction.
Zope Corp, who created Buildout, even uses Buildout for development
and production deployments using rpms:

http://mail.python.org/pipermail/distutils-sig/2007-August/008158.html

> setuptools is a necessary evil for the agile developer who frequently
> tracks updates for the bits he relies upon. Hopefully it will be
> improved (the clamor around it is ever-ongoing), but unless we want to
> do it ourselves, we have to accept the state of things.

Only a small (and arguably non-evil) slice of the setuptools package
is a necessary evil. It would be nice if the namespace support and the
"include more metadata about our Python packages" parts of setuptools
were separate from the rest of setuptools, so that the distinction
between using easy_install and using eggs was a lot clearer. Eggs are
just a packaging format, and one that can work complementary to
packaging formats such as rpms and dpkgs.

Buildout only uses a small slice of functionality from setuptools.
It's egg-installation code lives outside setuptools and Buildout
entirely in the zc.recipe.egg package (http://pypi.python.org/pypi/
zc.recipe.egg). If someone doesn't like the way zc.recipe.egg handles
the install of egg-based packages, it's perfectly possible create a
recipe which does things the way you desire.

mengel

unread,
Sep 17, 2008, 3:49:38 PM9/17/08
to Django developers


On Sep 16, 2:16 am, mrts <m...@mrts.pri.ee> wrote:
> * generally we should strive for a hassle-free experience so that
> `easy_install django-foo` gives you an expected entry point (`from
> django.apps import foo`) and works-out-of-the-box feel

Just a slightly twisted thought, what if a site's manage.py had a
'python manage.py easy_install foo' that would install the foo
application in the current site directory, and automagically add a
sample urls.py entry for it?

Yuri Baburov

unread,
Sep 17, 2008, 5:53:30 PM9/17/08
to django-d...@googlegroups.com

wow, that'd be pretty much enough!
maybe even without urls.py modification (this can cause troubles)

--
Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
MSN: bu...@live.com

HenrikV

unread,
Sep 18, 2008, 9:32:07 AM9/18/08
to Django developers
> Namespace packages and packages living in a namespace are different
> things. I'd say the latter is sufficient.

There are two different name spacing related issues. One is PyPI
organisation, and one is the local one.
Assuming that deployment environments are only set up with the
applications needed for the deployed projects.
Local environment is really as installed apps for on or more installed
projects.

PyPI name spacing can be solved with categories/tags/egg-prefixes. On
the other hand I am not convinced that
a formal name spacing of application modules is needed.

> Of course, company sub-namespaces are entirely welcome, i.e.
> `django.apps.mycompany.foo` or `django.apps.com.mycompany.foo` should
> be equivalently accepted. I personally don't like too deeply nested
> and verbose namespaces and they are less common in Python than they
> are in Java, but YMMV.

Flat is better than nested.

Django does a great job sticking to this, lets not diverge from that.

It is annoying enough as it is to have have 'django_foo' applications.
Lets not turn that into 'django.apps.mybiz.foo'.

Rajeev J Sebastian

unread,
Sep 18, 2008, 2:44:32 PM9/18/08
to django-d...@googlegroups.com
On Tue, Sep 16, 2008 at 1:57 PM, mrts <mr...@mrts.pri.ee> wrote:
> As for naming, `django.apps` is an arbitrary name, it can be
> `djangoapps` or whatnot. To avoid bikeshedding, core devs could decide
> on that.
>
> Of course, company sub-namespaces are entirely welcome, i.e.
> `django.apps.mycompany.foo` or `django.apps.com.mycompany.foo` should
> be equivalently accepted. I personally don't like too deeply nested
> and verbose namespaces and they are less common in Python than they
> are in Java, but YMMV.

I am -1 on this, because all my reusable apps are under "dynamo.apps"
namespace :)

Really, there is no need for this. Introspecting applications is never
going to happen in django itself (atleast from what I understand of
the devs).

Eggs have an "entry point" concept, which could be used for this
purpose [1]. If apps are going to be reusable, might as well
distribute them as eggs.

Or you could just have people drop packages into a specific directory
in your project. (Pinax for e.g., has apps/)

Regards
Rajeev J Sebastian

[1] http://peak.telecommunity.com/DevCenter/setuptools#dynamic-discovery-of-services-and-plugins

Reply all
Reply to author
Forward
0 new messages