replace distutils with setuptools
Vinzent Steinberg
1. Depenencies. Sympy depends on pyglet and mpmath and they are thus
included in a separate thirdparty directory. This has some drawbacks:
Other programs than sympy can't use them and it ignores existing
installations of these modules. Furthermore it's involved in issues
like the following one.
2. Upgrading. The current, distutils-based setup.py doesn't remove old
files before installing, which can break sympy completely:
http://code.google.com/p/sympy/issues/detail?id=864
I think at least the first problem can be solved when using setuptools/
easy_install instead of distutils, as one of the main advantages of
setuptools is dependencies-handling. The second problem can probably
be solved too (there seems to be support for upgrading/uninstalling),
but I'm not sure.
I just tried easy_install for ipython:
"
# easy_install ipython
Searching for ipython
Reading http://pypi.python.org/simple/ipython/
Reading http://ipython.scipy.org
Reading http://ipython.scipy.org/dist
Best match: ipython 0.8.4
Downloading http://ipython.scipy.org/dist/ipython-0.8.4-py2.5.egg
Processing ipython-0.8.4-py2.5.egg
creating /usr/local/lib/python2.5/site-packages/ipython-0.8.4-
py2.5.egg
Extracting ipython-0.8.4-py2.5.egg to /usr/local/lib/python2.5/site-
packages
Adding ipython 0.8.4 to easy-install.pth file
Installing ipython script to /usr/local/bin
Installing pycolor script to /usr/local/bin
Installed /usr/local/lib/python2.5/site-packages/ipython-0.8.4-
py2.5.egg
Processing dependencies for ipython
Finished processing dependencies for ipython
"
IMHO it's awesome. Let's switch to this way of distribution of sympy
and its third party modules. (Additionally there could be of course a
tar ball with all dependencies included.) What do you think?
Links:
http://pypi.python.org/pypi/setuptools/
http://peak.telecommunity.com/DevCenter/setuptools
http://peak.telecommunity.com/DevCenter/EasyInstall
It was going to be included in Python 2.5 but the author refused:
http://mail.python.org/pipermail/python-dev/2006-April/064145.html
Vinzent
Vinzent Steinberg
# easy_install --upgrade sympy
Searching for sympy
Reading http://pypi.python.org/simple/sympy/
Reading http://code.google.com/p/sympy
Best match: sympy 0.5.15
Downloading http://sympy.googlecode.com/files/sympy-0.5.15.tar.gz
Processing sympy-0.5.15.tar.gz
Running sympy-0.5.15/setup.py -q bdist_egg --dist-dir /tmp/
easy_install--T05uk/s
ympy-0.5.15/egg-dist-tmp-CC1Wrm
package init file 'sympy/concrete/tests/__init__.py' not found (or not
a regular
file)
package init file 'sympy/core/tests/__init__.py' not found (or not a
regular fil
e)
[...]
package init file 'sympy/utilities/tests/__init__.py' not found (or
not a regula
r file)
Creating missing __init__.py for sympy.simplify.tests
Creating missing __init__.py for sympy.concrete.tests
Creating missing __init__.py for sympy.numerics.tests
Creating missing __init__.py for sympy.test_external
Creating missing __init__.py for sympy.ntheory.tests
Creating missing __init__.py for sympy.core.tests
Creating missing __init__.py for sympy.physics.tests
Creating missing __init__.py for sympy.functions.elementary.tests
Creating missing __init__.py for sympy.functions.special.tests
Creating missing __init__.py for sympy.functions.combinatorial.tests
Creating missing __init__.py for sympy.parsing.tests
Creating missing __init__.py for sympy.polynomials.tests
Creating missing __init__.py for sympy.polys.tests
Creating missing __init__.py for sympy.integrals.tests
Creating missing __init__.py for sympy.statistics.tests
Creating missing __init__.py for sympy.matrices.tests
Creating missing __init__.py for sympy.geometry.tests
Creating missing __init__.py for sympy.plotting.tests
Creating missing __init__.py for sympy.utilities.tests
Creating missing __init__.py for sympy.series.tests
Creating missing __init__.py for sympy.solvers.tests
Creating missing __init__.py for sympy.printing.pretty.tests
Creating missing __init__.py for sympy.printing.tests
Creating missing __init__.py for sympy.specfun.tests
zip_safe flag not set; analyzing archive contents...
sympy.__init__: module references __file__
sympy.utilities.pkgdata: module references __file__
sympy.utilities.tests.test_code_quality: module references __file__
sympy.utilities.tests.test_tests_names: module references __file__
Adding sympy 0.5.15 to easy-install.pth file
Installing isympy script to /usr/local/bin
Installed /usr/local/lib/python2.5/site-packages/sympy-0.5.15-
py2.5.egg
Processing dependencies for sympy
Finished processing dependencies for sympy
We could let it additionally install pyglet and mpmath as
dependencies.
Vinzent
Brian Jorgensen
It's been a while, but I'm the one who committed the original sin of including pyglet. In general, I agree with you. The intent back then was to allow users to plot out-of-the-box, without any external dependencies. I've since come to believe that our users are generally python programmers who can handle the installation of dependencies.
On the other hand, I want to avoid the situation that used to happen with PyOpenGL on windows: you had to install specific, non-standard versions of numeric, PIL, etc (see http://www.visionegg.org/install-windows-details.html). Things to think about:
* How would this work for people using the windows installer? Can it handle dependencies somehow, or should pyglet be a separate download?
* Plotting hasn't been updated to use the latest versions of pyglet (we depend on < 1.0, I think). If someone uses a more recent pyglet, how do we know it's going to work? We might end up getting a lot of bug reports relating to version mismatch.
Regrettably, I haven't had enough time to contribute regularly, but I've been toying here and there with different ways of doing plotting. I've been thinking it might be a good idea to make a separate package sympy-plotting, sympy-extras, or similiar. We could offer two windows installers, one with plotting and one without, though it would create quite a bit of extra work for each release.
So, is sympy an everything-but-the-kitchen-sink, batteries-included library, or a svelte library with lots of optional add-ons?
Brian
Ondrej Certik
Thanks for starting this discussion and sharing the ideas. My own thoughts:
* As to mpmath, it contains essential things that should imho be part
of sympy, also it's just 8 files, no nested directories, so we just
plain copy them to sympy and that's it. That's imho the best solution
here and this is how it is done now. So I think mpmath is a non-issue.
* pyglet: I think we could distribute sympy-pure.tar.gz (without
pyglet) and sympy.tar.gz (with pyglet). Unfortunately this means more
work for the release manager, as now he needs to test two tarballs,
but if it's worthy, let's do it. The only real problem here is the
sys.path hack, but that is imho only needed in python2.4, so this
should soon become non-issue anyway, as I think in python2.5 it coud
be fixed by relative imports and we don't need any sys.path things
anymore. An argument could be the size of pyglet --- is there a
problem with it? For me the size is ok.
> So, is sympy an everything-but-the-kitchen-sink, batteries-included library,
> or a svelte library with lots of optional add-ons?
Both. It should be easy enough to get the job done. Imho it needs to
be judged on case by case basis, so for mpmath and pyglet, see my
thoughts above.
Ondrej
Vinzent Steinberg
automatically. The user wouldn't notice any difference. I'd keep the
full tar ball. We could offer an alternative with setuptols. IMHO not
being able to use mpmath outside of sympy (or installing mpmath twice)
just sucks. Especially if there's a large sympy overhead when using
mpmath from sympy.
A good example: TurboGears uses easy_install excessively:
http://docs.turbogears.org/1.0/Install
In my opinion, this kind of modularity is great. There could be for
example an old pyglet version for plotting. A problem are offline
installations. But for this the tarball still exists.
Might be possible that some Windows users prefer graphical installers,
but I prefer easy_install over such things (even when using Windows).
It's possible to make an installer though (I think setuptools is able
to do so).
Vinzent
Brian Jorgensen
- Download ez_setup.py
- Start menu, run "cmd"
- Type "python ez_setup.py"
- Type "easy_install -U sympy"
- Huh, command not found? Oh yeah, I probably have to add something to my PATH. What, though?
- Based on a cunning google search, it seems I need to add "C:\Python25\Scripts" to my PATH. How do I do that again? Should I add it under "User Variables" or "System Variables"?
So I do think that having an executable installer is important to many users who read about sympy on reddit and just want to try it out. If inelegant, it probably won't be much worse than what we've been doing--at worst, we can just release installers for major versions.
Brian
Vinzent Steinberg
possible to add a Python script you have to double-click on and which
would do all the stuff for you?
On 2 Jun., 18:08, "Brian Jorgensen" <brian.jorgen...@gmail.com> wrote:
> I completely agree with you in principle. And I personally love the command
> line, even on windows. But a majority of windows users are unwilling to do
> anything with it--including many talented programmers. Here's how it goes
> for many people I've known:
>
> 2. Start menu, run "cmd"
> 3. Type "python ez_setup.py"
> 4. Type "easy_install -U sympy"
> 5. Huh, command not found? Oh yeah, I probably have to add something to
> my PATH. What, though?
> 6. Based on a cunning google search, it seems I need to add
> under "User Variables" or "System Variables"?
>
> At every step, we're likely to lose part of the audience, converging to 0
> windows users as N goes to infinity ;-)
>
> So I do think that having an executable installer is important to many users
> who read about sympy on reddit and just want to try it out. If inelegant, it
> probably won't be much worse than what we've been doing--at worst, we can
> just release installers for major versions.
>
> Brian
>
> On Mon, Jun 2, 2008 at 8:24 AM, Vinzent Steinberg <
>
Ondrej Certik
<vinzent....@googlemail.com> wrote:
>
> Yes, command line usage on Windows is a pain. But wouldn't it be
> possible to add a Python script you have to double-click on and which
> would do all the stuff for you?
Sure, I am not against. As always, the only real requirement is that
the thing is easy to maintain, test and produce (e.g. now it's enough
for me to do setup.py bdist_wininst).
OK, so to move things forward, the reason for using a particular
mpmath and a particular pyglet is that those are versions that are
known to work with sympy.
So let's try to figure out how to use pyglet without touching
sys.path. If we succeed, is anyone against leaving both mpmath and
pyglet in sympy?
If we fail, then we'd have to release two tarballs, one with, the
other without, pyglet.
Ondrej
Vinzent Steinberg
> On Tue, Jun 3, 2008 at 8:47 AM, Vinzent Steinberg
>
>
> > Yes, command line usage on Windows is a pain. But wouldn't it be
> > possible to add a Python script you have to double-click on and which
> > would do all the stuff for you?
>
> Sure, I am not against. As always, the only real requirement is that
> the thing is easy to maintain, test and produce (e.g. now it's enough
> for me to do setup.py bdist_wininst).
>
> OK, so to move things forward, the reason for using a particular
> mpmath and a particular pyglet is that those are versions that are
> known to work with sympy.
mpmath. Of course this means that someone needs to fix sympy when
there's something fundamentally changed in a third party package. I
would prefer this solution, but if it's a too big effort let's include
our own versions.
> So let's try to figure out how to use pyglet without touching
> sys.path. If we succeed, is anyone against leaving both mpmath and
> pyglet in sympy?
otherwise it would be too difficult to maintain .
> If we fail, then we'd have to release two tarballs, one with, the
> other without, pyglet.
>
> Ondrej
Vinzent
Ondrej Certik
>> mpmath and a particular pyglet is that those are versions that are
>> known to work with sympy.
>
> IMHO sympy should just work with the latest stable version of pyglet/
> mpmath. Of course this means that someone needs to fix sympy when
> there's something fundamentally changed in a third party package. I
> would prefer this solution, but if it's a too big effort let's include
> our own versions.
>
>> So let's try to figure out how to use pyglet without touching
>> sys.path. If we succeed, is anyone against leaving both mpmath and
>> pyglet in sympy?
>
> As I said I don't like this solution but it's the way to go if
> otherwise it would be too difficult to maintain .
Why do you not like it? Because of the size of the included files?
There shouldn't be any other side effect.
>
>> If we fail, then we'd have to release two tarballs, one with, the
>> other without, pyglet.
>>
>
> What's so wrong with sys.path hacking?
Because it's leaving side effects for the user, which is very, very
bad. Also it does't work always, for example on the google app engine.
Ondrej
Gael Varoquaux
> IMHO it's awesome. Let's switch to this way of distribution of sympy
> and its third party modules. (Additionally there could be of course a
> tar ball with all dependencies included.) What do you think?
Eggs work by monkey patching sys.patch, which has some "interesting"
consequences. IMHO thisis not acceptable. It has been pointed out that
these consequences can be avoided by patching setuptools. This patch is
not happening, and setuptools development process is not transparent. I
consider setuptools as a neat toy, but not mature and useable for
production. Not everybody agrees with me :).
Gaėl
Ondrej Certik
I do! :)
Ondrej
Vinzent Steinberg
On Jun 9, 8:38 am, Gael Varoquaux <gael.varoqu...@normalesup.org>
wrote:
I don't know much about setuptools as I did not try it yet. My
judgement was completely superficial. :)
Thank you for sharing your experiences, this is exactly what I wanted
to know.
Could you be please more concrete and tell what "interesting"
consequences are to be expected?
Do you think distutils is better than setuptools? I want it to work
like every good Windows installer. It should remove old files and not
break installations.
Guido for example apparently does not agree with you. :) (http://
mail.python.org/pipermail/python-dev/2006-April/063964.html)
Vinzent
Gael Varoquaux
> Could you be please more concrete and tell what "interesting"
> consequences are to be expected?
For instance you can no longer override the import of a module by
installation a newer version of it... unless you use setuptools. Users
also get confused as to what is imported, and this results in strange bug
reports.
> Do you think distutils is better than setuptools?
It is orthogonal. These two tools attempt to solve two different
problems: installation and packaging.
> I want it to work like every good Windows installer. It should remove
> old files and not break installations.
I don't know if it does this for windows. It certainly does not do this
for Linux.
> Guido for example apparently does not agree with you. :) (http://
> mail.python.org/pipermail/python-dev/2006-April/063964.html)
That's old news. There has been quite a flame at Pycon this year (2008)
about setuptools. I do not have the impression Guido is terribly
satisfied with the state of the project. The problem is that it is the
only project trying to tackle this challenge.
Gaël
Ondrej Certik
I am not against setuptools per se, only I want sympy to work with the
traditional tools, which currently is distutils. My own experience
with projects using setuptools is that it is very fragile, but if
someone is interested in supporting setuptools, let's do it (under the
assumption that the distutils way will still just work).
Ondrej
Sebastian Haase
I was just trying to find out about "the general acceptance" of
setuptools in the python community.
It seems impossible:
python.org does not seem to mention it at all --except
PEP 365 -- Adding the pkg_resources module -- which was rejected
and
PEP 361 -- Python 2.6 and 3.0 Release Schedule --which makes it
sound like it is already in Python 2.5
What is the current state ? Are "eggs" the all-agreed-upon future ?
Thanks,
Sebastian Haase