Convince us to continue using setuptools
Adrian Holovaty
requires a working Internet connection. That's because we're using
ez_setup/setuptools
(http://peak.telecommunity.com/DevCenter/setuptools), which downloads
the setup program automatically if you don't have it installed.
The thing is, we don't actually *use* any of the advanced
functionality in setuptools. Frankly, the only reason we're using it
is because setuptools makes it easier to specify which files we want
to include in the distribution (via some globbing syntax that's more
advanced than the standard Python distutils). This is because we
figured it'd be too much of a pain to create a MANIFEST file and keep
it updated. Yes, this is a *horrible* reason to use setuptools rather
than distutils.
So, convince us to continue using setuptools. What incentive do we
have to keep using it? I'm not sure the convenience of easily being
able to specify a manifest outweighs the horrid stain of requiring an
Internet connection just to install our software. Are there any other
ways we can take advantage of it, perhaps?
I really want to be convinced to continue using setuptools, but I'm
drawing a blank... Insights are appreciated!
Adrian
--
Adrian Holovaty
holovaty.com | djangoproject.com
Jay Parlar
need for it in Django.
I say dump it.
Jay P.
Michael Radziej
> I really want to be convinced to continue using setuptools, but I'm
> drawing a blank... Insights are appreciated!
How about bundling setuptools so that you don't need to download
it later?
Michael
du...@habmalnefrage.de
I wondered what essential function of setuptools is used and I think that only find_packages() is neccessary.
Try the following patch for setup.py and a MANIFEST.in
If this works for you, than I won't suggest setuptools bundled with Django.
I tested the following usecases:
python setup.py sdist
python setup.py bdist
python setup.py build
python setup.py bdist_rpm
Regards,
Dirk
--
Echte DSL-Flatrate dauerhaft für 0,- Euro*. Nur noch kurze Zeit!
"Feel free" mit GMX DSL: http://www.gmx.net/de/go/dsl
jpel...@gmail.com
> have to keep using it? I'm not sure the convenience of easily being
> able to specify a manifest outweighs the horrid stain of requiring an
> Internet connection just to install our software. Are there any other
> ways we can take advantage of it, perhaps?
Here are a couple that I've been thinking about, both using setuptools
entry points.
Easy: make database backends pluggable. There's no good reason for all
db backends in the world to go in core, nor for users to be unable to
use custom backends without hacking core. This could be solved with a
tiny change in django.db: if the backend isn't in core, look under the
'django.db.backends' entry point and load whatever you find there under
the requested name.
Harder: make app loading use entry points. models.loading is ... not
the least hacky part of django. :) The app/model structure constraints
are quite difficult to work around. Entry points could make app loading
much easier and more flexible. Just iterate the django.applications
entry point and call everything you find there an app. No more need to
dictate package layouts ... plus with apps distributed as eggs, you
have a solution to the absolute/relative package loading problems
caused by the practice of copying apps from one project to another.
In summary: entry points *rule*. My experience with using them to add
plugin support to my test runner (nose) left me with a great respect
for their power to make a framework much more open and hacker-friendly,
at minimal cost. Dependency loading and versioning and such are great,
but to me entry points are *the* reason to use setuptools.
JP
Eugene Lazutkin
>
> So, convince us to continue using setuptools. What incentive do we
> have to keep using it? I'm not sure the convenience of easily being
> able to specify a manifest outweighs the horrid stain of requiring an
> Internet connection just to install our software. Are there any other
> ways we can take advantage of it, perhaps?
>
> I really want to be convinced to continue using setuptools, but I'm
> drawing a blank... Insights are appreciated!
Users can do custom builds with ease. For example: I was able to build a
NSIS-based Windows installer for Django 0.95 as promised before (sent to
Jacob) in a matter of minutes. It would be difficult to do, if you
didn't use setuptools packaging.
Thanks,
Eugene
Jay Parlar
> Here are a couple that I've been thinking about, both using setuptools
> entry points.
Hmm... I forgot about that. Jason's right, entry points do rule. At
least from what I've seen of them in nose.py :)
If there is an effort to move to entry points, then yeah, keep
setuptools. Otherwise though...
Jay P.
Eugene Lazutkin
>
> So, convince us to continue using setuptools. What incentive do we
Which reminds me that you used setuptools 0.6c1, which is a developer's
version, if I am not mistaken. setuptools tries to update itself from
the current version on my computer and cannot find 0.6c1 version. The
latest available version is 0.6a10. I had to edit DEFAULT_VERSION in
ez_setup.py to make it work. Probably it should be corrected in the distro.
Thanks,
Eugene
Deryck Hodge
> >
> > So, convince us to continue using setuptools. What incentive do we
The easy packaging is a big plus, and I think, too, that "python
setup.py install" is becoming somewhat of the defacto python
installation standard. If it's just the network connection that's the
major issue, don't do ez_setup. Just require users have setuptools
installed. It's so common, I think, they'll have to install
setuptools eventually when dealing with Python.
Cheers,
deryck
--
Deryck Hodge http://www.devurandom.org/
Web Developer, Naples News http://www.naplesnews.com/
Samba Team http://www.samba.org/
Jay Parlar
>
> > Adrian Holovaty wrote:
> > >
> > > So, convince us to continue using setuptools. What incentive do we
>
> The easy packaging is a big plus, and I think, too, that "python
> setup.py install" is becoming somewhat of the defacto python
> installation standard. If it's just the network connection that's the
> major issue, don't do ez_setup. Just require users have setuptools
> installed. It's so common, I think, they'll have to install
> setuptools eventually when dealing with Python.
"python setup.py install" has been the defacto standard for years and
years, it's a distutils feature.
Jay P.
Malcolm Tredinnick
Why does this require setuptools, though? I mean, the Python XML
utilities used entry points for extra loaders (e.g. 4Suite) for years
before setuptools came along.
I'm also a bit confused about your application loading case. What huge
constraints are we putting on app loading at the moment? Relative
imports work -- in the sense that you don't need project name in the
import path -- so copying apps between projects should be reasonably
simple. I would have thought that most of the time somebody is using an
application with Django, there is no need to hack the source at all. The
app has to be on the Python path and importable. Now clearly this is as
obvious to you as it is to me, so what else are you wanting to do here
that we are currently making hard?
Whenever I'm evaluating additions to the code, one of my questions is
always "can this be done outside of the source?" Usually the answer is
yes (template loaders, apps, template tags, filters, ...). Database
backend additions are one that can't be, as you point out. We can fix
that without needing setuptools, it seems.
Malcolm
Deryck Hodge
> "python setup.py install" has been the defacto standard for years and
> years, it's a distutils feature.
>
You're right. Not really sure what I was thinking there. Must be the
stupidity of a too hurriedly written email. :-)
(Stupidity for confusing distutils and setuptools. I really didn't
realize setup.py install had been used for years and years. I've only
been writing Python a couple years myself. :-)
jpel...@gmail.com
> utilities used entry points for extra loaders (e.g. 4Suite) for years
> before setuptools came along.
"Before" is the key word there, I think. Now that setuptools is a
defacto standard, I think it would be an odd decision to rewrite pieces
it. What's the benefit to doing that, when (as someone suggested
upthread) you could just bundle setuptools itself, if ez_setup is out?
> I'm also a bit confused about your application loading case. What huge
> constraints are we putting on app loading at the moment?
Apps must be packages which (if they have models) must have models in a
file called models.py. You can't even use the ORM without following
that convention. Why should that be, except for the loading logic in
models.loading? Why can't a class be an app? An instance? Why can't
models be anywhere? Tight coupling leads to brittle systems; apps and
models are, in my opinion, too tightly coupled to each other and to
package layout. Allowing loading of apps through entry points is one
way to loosen the ties.
> Now clearly this is as
> obvious to you as it is to me, so what else are you wanting to do here
> that we are currently making hard?
What first triggered my thoughts was something seemingly really simple:
I was trying to write a test that defined two models related by a
many-many relationship. This is currently possible only through hideous
hacks, because get_model() only loads models that are in apps, and apps
can only be packages and their models can only be in app.models.
Defining and installing a model ought to be enough to make it usable.
That's not true now, for all models, and the failure cases are very
surprising, even when you've been digging around in the model/db code
quite a bit. This is easily fixable without setuptools (just let
get_models() return all models when called without an app arg instead
of iterating the app list and recursing), but it made me think about
app loading and packaging and lead to my conclusions above.
> Database
> backend additions are one that can't be, as you point out. We can fix
> that without needing setuptools, it seems.
Sure, but why? Setuptools is here, the only detriment to its use is
that installing django from source requires either an internet
connection or that setuptools is installed already. Why reinvent the
wheel? Wouldn't that time be better spent on creating new features? For
me this is a classic "build or buy" decision that lands strongly on
"buy".
JP
Malcolm Tredinnick
> > Why does this require setuptools, though? I mean, the Python XML
> > utilities used entry points for extra loaders (e.g. 4Suite) for years
> > before setuptools came along.
>
> "Before" is the key word there, I think. Now that setuptools is a
> defacto standard, I think it would be an odd decision to rewrite pieces
> it. What's the benefit to doing that, when (as someone suggested
> upthread) you could just bundle setuptools itself, if ez_setup is out?
See, this is where I think you are wanting to do more than just using
extra loading hooks. There's no big rewrite involved. Have a think back
to how XML stuff was imported in, say, Python 1.5: it was like one line
of code to do the extra search (inside Python core libraries, not
something the user had to worry about).
> > I'm also a bit confused about your application loading case. What huge
> > constraints are we putting on app loading at the moment?
>
> Apps must be packages which (if they have models) must have models in a
> file called models.py. You can't even use the ORM without following
> that convention. Why should that be, except for the loading logic in
> models.loading? Why can't a class be an app? An instance? Why can't
> models be anywhere? Tight coupling leads to brittle systems; apps and
> models are, in my opinion, too tightly coupled to each other and to
> package layout. Allowing loading of apps through entry points is one
> way to loosen the ties.
This is wandering very far from "should we use setuptools". You are
talking about a large redesign of the model system towards infinite
flexibility. That's probably a topic for another thread. It's not really
tied to setuptools usage beyond you think it would make it easier to
leverage those changes in the future if setuptools were available (or am
I misunderstanding?).
[... snip...]
> > Database
> > backend additions are one that can't be, as you point out. We can fix
> > that without needing setuptools, it seems.
>
> Sure, but why? Setuptools is here, the only detriment to its use is
> that installing django from source requires either an internet
> connection or that setuptools is installed already. Why reinvent the
> wheel? Wouldn't that time be better spent on creating new features? For
> me this is a classic "build or buy" decision that lands strongly on
> "buy".
Maybe I'm not being clear: how is setuptools even involved in the case
of looking for extra database backends? Either they are available
underneath a search from some module path (which could be a standard
location of a configuration option so that we can have backends out of
tree) or they aren't. What does setuptools add to the picture? (Serious
question. I think I must be missing something fundamental, since I have
plenty of hooks in applications I've written that don't use setuptools,
so I'm wondering what else it's bringing along.)
Cheers,
Malcolm
jpel...@gmail.com
> talking about a large redesign of the model system towards infinite
> flexibility. That's probably a topic for another thread. It's not really
> tied to setuptools usage beyond you think it would make it easier to
> leverage those changes in the future if setuptools were available (or am
> I misunderstanding?).
That's true. I was just trying to explain my thinking. Let's drop that
example use case, it doesn't really add anything -- one good use for
entry points is enough. :)
> Maybe I'm not being clear: how is setuptools even involved in the case
> of looking for extra database backends?
Backends would be registered by packages as setuptools entrypoints.
Django would iterate the entries in the 'django.db.backends' entry
point to load backends outside of core. This change, requiring only a
few lines, would make it possible for any package to provide a django
db backend. No need for custom loading hooks or enforcing package
naming conventions or anything.
> What does setuptools add to the picture?
It adds the fact that it already exists and works. Entry points provide
a widely-used and well understood way of adding extensibility to an
application. Again, to me, it's a simple build or buy. Why spend time
building something that won't be any better than what you can get right
now, for the low-low price of using ez_setup, something django already
does? After all, we're talking about *dumping* setuptools, not adopting
it. This is a reason not to: it provides this very useful entry points
feature that can be used to support things like pluggable db backends
with very little effort.
JP
Michael Twomey
>
> It has bothered me for a while that Django's installation (setup.py)
> requires a working Internet connection. That's because we're using
> ez_setup/setuptools
> (http://peak.telecommunity.com/DevCenter/setuptools), which downloads
> the setup program automatically if you don't have it installed.
>
You can specify a local url in the use_setuptools, e.g.
import ez_setup
import os
ez_setup.use_setuptools(download_base = os.path.join("file://",
os.getcwd()) + "/")
(not the most robust, but you get the idea)
This will look for the egg in the same directory as the setup.py is
run from, usually the django source. This way you can bundle the egg.
The nice thing here is that it shouldn't intefere with anyone using
setuptools already.
> The thing is, we don't actually *use* any of the advanced
> functionality in setuptools. Frankly, the only reason we're using it
> is because setuptools makes it easier to specify which files we want
> to include in the distribution (via some globbing syntax that's more
> advanced than the standard Python distutils). This is because we
> figured it'd be too much of a pain to create a MANIFEST file and keep
> it updated. Yes, this is a *horrible* reason to use setuptools rather
> than distutils.
>
If you have a lot of shared data to include this is actually a very
good reason to use setuptools :)
Projects like matplotlib got mired in so many installation issues
trying to get all their shared data installed correctly on so many
configurations that they switched to setuptools (well, that was one of
the main reasons I think).
> So, convince us to continue using setuptools. What incentive do we
> have to keep using it? I'm not sure the convenience of easily being
> able to specify a manifest outweighs the horrid stain of requiring an
> Internet connection just to install our software. Are there any other
> ways we can take advantage of it, perhaps?
>
Well, if no feature other than the manifest management is used then
there isn't much incentive to keep using it. However, folks like JP
mentioned some of the more advanced features which could be used in
future.
The main one which springs to mind is optional dependencies. If
database eggs are available then you could go "easy_install
Django[sqlite]" to pull in the sqlite egg for example. This
potentially can make installation a lot easier, provided eggs for each
of the supported databases are available of course :)
Entry points are potentially useful if you want to provide optional
components or pluggable parts. I think they are more useful to an app
using django rather than django itself.
Another useful part of setuptools is the pkg_resources part which lets
you locate and open shared data from inside the egg (which could be a
zip file). This is a more robust replacement for __file__.
> I really want to be convinced to continue using setuptools, but I'm
> drawing a blank... Insights are appreciated!
>
I think this is a tricky question to answer. Projects like turbogears
have been built from the ground up using eggs, so the dependency logic
in setuptools is vital to them. Django has been built to have as few
deps as possible, so this aspect is not particularly useful.
I think keeping setuptools keeps more options open in future. However
if the default distutils is used instead then I would ask that django
be kept egg compatible, i.e. don't do anything fruity in the setup.py
which breaks easy_install. It's quite annoying to write your own
setup.py so you can depend on other packages :)
cheers,
Michael
Istvan Albert
automatic setuptools installation has caused me some grief as it
installs an obsolete version ... I forgot the exact specifics but it
took me quite a while to track down that installing django was causing
all kinds of strange problems with other python modules.
I think it is fair to say that having the situation where installing
django will also covertly install setuptools is greatly undesirable.
As many have pointed out, using a MANIFEST.in provides a very simple
means to use file globbing for files and directories. If that was the
main reason for using setuptools you'd have a straightforward solution
right away.
i.
ps. Nowadays I don't ever install it, I just simply add the django
folder to my python path and have not had any problems. I would not
treat django as a python module anymore, installing into the python
site libraries makes keeping track of different releases very difficult.