Unbundle mpmath

[Sergey Kirpichev]

There is a long term issue [1] to make mpmath an external dependency.  It was suggested by
Aaron (see discussion in PR [2]), that we should discuss this on the mailing list.

Perhaps, the main reason to prefer bundled mpmath copy is a
mystical "complex installation".  Not clear enough.  For example, I don't
think that additional steps for package installation from mpmath manual does matter.  For non-root users
there shouldn't be any problems too, at least for Python 2.6+ (per-user site-package directory).

[1] http://code.google.com/p/sympy/issues/detail?id=2482
[2] https://github.com/sympy/sympy/pull/2192

[Stefan Krastanov]

I am also of the opinion that we should unbundle it. Python has an
extremely easy dependency tracking system and libraries can be easily
installed without root privileges (with virtualenv and pip). Moreover,
for the "lazy" scientists there are many pre-bundled python
distributions ready to use. Even if considering the google app engine,
installing mpmath is as difficult/easy as installing sympy.

> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sympy.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Ondřej Čertík]

Hi,

We already agreed in the issue to allow to use systemwide version of
mpmath, preferably by adding a simple switch to setup.py, while using
our version of mpmath if the switch is not given. Patches would be
appreciated. :)

I vote yes to this.

As far as physically removing sympy.mpmath, is the issue its size?
The sympy/ dir on my computer has 15M, sympy/mpmath has 1.9M. So the
size of mpmath can hardly be a reason.

That being said, if projects like binstar (https://binstar.org/),
which was announced *yesterday*, take off and allow easy
installation on all platforms (including Windows...), we can revisit
this. Clearly, the issue of distribution of packages has not been
fixed *today*. As such, I vote no to physically removing sympy.mpmath
today, and I vote for keeping the extra 1.9M in our repository. Note
that it compresses extremely well (0.44M as tgz on my machine). If
there is any argument why this extra little space is a problem, I
would like to hear it. Note that we are in great company, as packages
like VTK also ship their little dependencies (alglib expat
ftgl jpeg libxml2 oggtheora TclTk verdict zlib
Cosmo freerange gl2ps jsoncpp mrmpi png tiff VPIC
exodusII freetype hdf5 libproj4 netcdf sqlite utf8
xdmf2), and allow to use systemwide using cmake options.
VTK has 140M, and their ThirdParty lib has 54M, so that's 38%. In our
case the percentage is 12%, so we are even doing better.

Ondrej

[Sergey B Kirpichev]

On Fri, Jun 28, 2013 at 04:20:29PM -0500, Ondřej Čertík wrote:
> As far as physically removing sympy.mpmath, is the issue its size?
> The sympy/ dir on my computer has 15M, sympy/mpmath has 1.9M. So the
> size of mpmath can hardly be a reason.

Not the size, of course. The reasons to drop this bundled library is
exactly the reasons why people prefer to use external libraries
instead of bundled copies. Independent (security) updates for
library, prevent forking, etc.

> That being said, if projects like binstar (https://binstar.org/),
> which was announced *yesterday*, take off and allow easy
> installation on all platforms (including Windows...), we can revisit
> this. Clearly, the issue of distribution of packages has not been
> fixed *today*.

I don't see any reasons why this new project is an argument. Can
you list some real problems (better for python >= 2.5)?

[Stefan Krastanov]

@Sergey, I think Ondrej's point is that installing dependencies is not
as easy as I have suggested. I guess this is open for debate.

There is probably also a psychological barrier. `mpmath` is the only
hard requirement for running sympy, and having it bundled permits the
project to say that it has no dependencies (to be fair, I was left
with the impression that most of mpmath was developed under the
umbrella of sympy).

[Ondřej Čertík]

One of them is for example that sympy.mpmath has received 13 patches
since the last release of mpmath 0.17 over 2 years ago:

ondrej@eagle:~/repos/sympy/sympy/mpmath(master)$ git shortlog -ns
--since="February 2011" . | wc -l
13

Some of these are important fixes, which are needed for sympy to work
or test correctly. There is a reason
for each of these patches. As far as I know we try to push these
patches to mpmath as well, but
it's quite a lot of work to create a new release, for example right
now we are struggling to get out a new release
of sympy. At the moment we don't seem to have the manpower to also
manage a release of another external project (mpmath). It's extremely
important to manage hard dependencies well. If we are going to depend
on some project, it needs to be well supported project with lots of
developers. At the moment, the best way to manage it is inside sympy.

The projects like binstar (see the link above) allow more people to
create binaries and my hope is that it will become easier to release
and ship binaries for all architectures.


Some of these problems have been greatly explained in the blog post
that I posted into the issue, but let me re-post it for others:

http://vagabond.github.io/2013/06/21/z_packagers-dont-know-best/



There is another big problem, that I hadn't mentioned before. At the
moment, you can use git bisect and go into history of sympy and easily
figure out if something broke. The moment we start depending on
mpmath, and we commit just one fix that breaks sympy (by adding it to
mpmath directly and creating a new release), the git bisect stops
working, and different sympy git hashes will depend on different
versions of mpmath. That will be a huge nightmare.

Another issue, mentioned by Fredrik in the issue is that currently
mpmath is tested with sympy. This way it gets great testing on various
architectures and receives lots of fixes from us, just browse some of
the commits. Since we hard-depend on mpmath, we need it to be in great
shape, and it is, because it is in sympy. Once we leave it aside, it
stops being tested so much.

Ondrej

[Sergey B Kirpichev]

On Sat, Jun 29, 2013 at 01:05:45AM +0200, Stefan Krastanov wrote:
> @Sergey, I think Ondrej's point is that installing dependencies is not
> as easy as I have suggested. I guess this is open for debate.

Yes, I see. The question is how "not as easy" it is.

Let's stick with mpmath's installation guide:
http://mpmath.googlecode.com/svn/trunk/doc/build/setup.html#download-and-installation

[Sergey B Kirpichev]

On Fri, Jun 28, 2013 at 09:38:35PM -0500, Ondřej Čertík wrote:
> >> That being said, if projects like binstar (https://binstar.org/),
> >> which was announced *yesterday*, take off and allow easy
> >> installation on all platforms (including Windows...), we can revisit
> >> this. Clearly, the issue of distribution of packages has not been
> >> fixed *today*.
> >
> > I don't see any reasons why this new project is an argument. Can
> > you list some real problems (better for python >= 2.5)?
>
> One of them is for example that sympy.mpmath has received 13 patches
> since the last release of mpmath 0.17 over 2 years ago:
>
> ondrej@eagle:~/repos/sympy/sympy/mpmath(master)$ git shortlog -ns
> --since="February 2011" . | wc -l
> 13

It's also - not an argument in favor of some mystical
problems with mpmath's installation.

Actually, this just means that we have mpmath fork.

> At the moment we don't seem to have the manpower to also
> manage a release of another external project (mpmath).

Do you think, that mpmath is not supported well?

> There is another big problem, that I hadn't mentioned before. At the
> moment, you can use git bisect and go into history of sympy and easily
> figure out if something broke. The moment we start depending on
> mpmath, and we commit just one fix that breaks sympy (by adding it to
> mpmath directly and creating a new release), the git bisect stops
> working, and different sympy git hashes will depend on different
> versions of mpmath. That will be a huge nightmare.

We already have this situation for others external dependecies.

> Another issue, mentioned by Fredrik in the issue is that currently
> mpmath is tested with sympy.

Are you about sympy/mpmath/tests/*? Why this can't be tested with
travis for a separate project?

[Ondřej Čertík]

On Sat, Jun 29, 2013 at 1:52 AM, Sergey B Kirpichev
<skirp...@gmail.com> wrote:
> On Fri, Jun 28, 2013 at 09:38:35PM -0500, Ondřej Čertík wrote:
>> >> That being said, if projects like binstar (https://binstar.org/),
>> >> which was announced *yesterday*, take off and allow easy
>> >> installation on all platforms (including Windows...), we can revisit
>> >> this. Clearly, the issue of distribution of packages has not been
>> >> fixed *today*.
>> >
>> > I don't see any reasons why this new project is an argument. Can
>> > you list some real problems (better for python >= 2.5)?
>>
>> One of them is for example that sympy.mpmath has received 13 patches
>> since the last release of mpmath 0.17 over 2 years ago:
>>
>> ondrej@eagle:~/repos/sympy/sympy/mpmath(master)$ git shortlog -ns
>> --since="February 2011" . | wc -l
>> 13
>
> It's also - not an argument in favor of some mystical
> problems with mpmath's installation.

Well I think it is --- these patches are not in the latest release of mpmath.
Projects like binstar will hopefully provide easy means for people to contribute
releases and binaries for all platforms.

Currently the easiest way for us has been to maintain our own version of mpmath,
making sure the codebase is up-to-date and contributing our patches
back to mpmath.

>
> Actually, this just means that we have mpmath fork.

That's right.

>
>> At the moment we don't seem to have the manpower to also
>> manage a release of another external project (mpmath).
>
> Do you think, that mpmath is not supported well?

I CCed Fredrik, so that he can join the discussion.

Look, the whole problem of any project is to manage its dependencies well.
This is a deep problem, that one has to very carefully weigh pros and cons.
One should only depend on libraries, that are well supported and widely used.
Mpmath is not. For example on my Ubuntu, here are all the projects that depend
on mpmath:

ondrej@hawk:~$ wajig dependents python-mpmath
s python-mpmath-doc


So it's the mpmath (documentation) itself. Compare it's popularity:

http://qa.debian.org/popcon.php?package=mpmath

versus sympy:

http://qa.debian.org/popcon.php?package=sympy

As such, sympy is the only project that actually depends on mpmath, and also
sympy is vastly (e.g. more than 5x according to the stats) more popular. So
the result is that 80% of all people will have to bother installing
mpmath, which otherwise
they would not use separately. As such, that is not a healthy
dependency. At least not yet.

Now compare this for example with numpy. Let's list all packages that
depend on it:

ondrej@hawk:~$ wajig dependents python-numpy
d python-numpy-dbg
d cain
d castle-combat
d childsplay
d code-aster-run
d connectomeviewer
d dicompyler
d eficas
d epigrass
r expeyes
d fofix
d fonttools
d gausssum
r gnudatalanguage
d grass
s ipython
d labyrinth
d libsyfi1.0-dev
d mayavi2
d memaker
d mmass
d model-builder
d mypaint
d ninix-aya
d novnc
d nulog
d pdb2pqr
d psychopy
d pybik
s pykaraoke
d pymca
d pymca-data
s python-aubio
d python-biopython
d python-biosig
d python-brian
d python-brian-lib
d python-cclib
d python-cfflib
d python-chaco
d python-cmor
d python-cogent
d python-csa
r python-dballe
r python-dicom
d python-dipy
d python-dolfin
d python-enable
d python-epr
d python-fabio
d python-ferari
d python-fiat
d python-freenect
r python-gamera
d python-gdal
d python-getfem++
d python-gnuplot
d python-guiqwt
d python-h5py
r python-instant
r python-joblib
d python-magics++
d python-mathgl
d python-matplotlib
d python-mdp
d python-mlpy
d python-mlpy-lib
s python-mpi4py
d python-mpikmeans
d python-mvpa
d python-mvpa-lib
d python-mvpa2
d python-mvpa2-lib
d python-necpp
r python-networkx
d python-nibabel
d python-nifti
d python-nipy
d python-nipy-lib
d python-nipy-lib-dbg
d python-nitime
s python-nltk
d python-numexpr
s python-opengl
d python-openopt
d python-pandas
d python-pandas-lib
d python-pebl
d python-plplot
d python-plplot-qt
d python-pybiggles
d python-pyentropy
d python-pyepl
r python-pyevolve
d python-pyfits
d python-pygame
d python-pygetdata
d python-pygrace
s python-pykaraoke
d python-pymt
d python-pytango
d python-pytools
d python-pytrilinos
d python-pywt
d python-qwt5-qt4
d python-rpy
d python-rpy2
d python-scikits.statsmodels
d python-scipy
d python-scipy-dbg
d python-scitools
s python-shapely
d python-sidl
d python-sklearn
d python-sklearn-lib
d python-sparse
d python-sphere
r python-spyderlib
d python-stfio
d python-swiginac
d python-syfi
r python-sympy
d python-tables
r python-uncertainties
d python-vigra
d python-viper
d python-visual
d python-whiteboard
d pytrainer
r qiime
r science-mathematics-dev
r science-numericalcomputation
r screenlets-pack-all
r screenlets-pack-basic
d sfc
d shogun-elwms-static
d shogun-python-modular
d shogun-python-static
d singularity
d snowballz
d specto
d stimfit
d tpclient-pywx
d veusz
d veusz-helpers
d vistrails
d vitables
d wsjt
d wxbanker
d wxgeometrie
d yade
d mgltools-gle
d mgltools-molkit
d mgltools-opengltk
d mgltools-pyautodock
d mgltools-sff
d mgltools-symserv
d mgltools-volume
d python-pygpu
d python-pyopencl
d sixpack
d python-numpy-dbg
r gnudatalanguage
d grass
d libsyfi1.0-dev
d mayavi2
d mypaint
d ninix-aya
d novnc
d pdb2pqr
d pybik
d pymca
s python-aubio
d python-biopython
d python-biosig
d python-brian-lib
d python-chaco
d python-cmor
d python-cogent
r python-dballe
d python-dolfin
d python-enable
d python-epr
d python-fabio
d python-freenect
r python-gamera
d python-gdal
d python-getfem++
d python-guiqwt
d python-h5py
d python-magics++
d python-mathgl
d python-matplotlib
d python-mlpy-lib
s python-mpi4py
d python-mpikmeans
d python-mvpa-lib
d python-mvpa2-lib
d python-necpp
d python-nifti
d python-nipy-lib
d python-nipy-lib-dbg
d python-numexpr
d python-pandas-lib
d python-pebl
d python-plplot
d python-plplot-qt
d python-pybiggles
d python-pyepl
d python-pyfits
d python-pygame
d python-pygetdata
s python-pykaraoke
d python-pymt
d python-pytango
d python-pytrilinos
d python-pywt
d python-qwt5-qt4
d python-rpy
d python-rpy2
d python-scipy
d python-scipy-dbg
s python-shapely
d python-sidl
d python-sklearn-lib
d python-sparse
d python-sphere
d python-stfio
d python-swiginac
d python-syfi
d python-tables
d python-vigra
d python-visual
r qiime
d shogun-elwms-static
d shogun-python-modular
d shogun-python-static
d stimfit
d veusz-helpers
d wsjt
d yade
d mgltools-gle
d mgltools-opengltk
d mgltools-sff
d python-pyopencl
r inkscape
s pitivi
s ibid
d python-netcdf
d python-qwt3d-qt4
d python-scientific
d python-scientific-doc
r inkscape
d python-netcdf
d python-qwt3d-qt4
d conservation-code
d python-pycuda
d python-pycuda
r inkscape


And its popularity:

http://qa.debian.org/popcon.php?package=python-numpy

So that's a healthy dependency in the sense that we should not include
sources of numpy in our project, but
rather depend on it if needed.

By ignoring all this and just seeing "mpmath" in sympy sources and
thus requiring that we remove it and hard
depend on it, would not be a wise decision. Again, I am repeating what
the blogpost that I linked above discusses,
I encourage you to read it and think about the arguments.

>
>> There is another big problem, that I hadn't mentioned before. At the
>> moment, you can use git bisect and go into history of sympy and easily
>> figure out if something broke. The moment we start depending on
>> mpmath, and we commit just one fix that breaks sympy (by adding it to
>> mpmath directly and creating a new release), the git bisect stops
>> working, and different sympy git hashes will depend on different
>> versions of mpmath. That will be a huge nightmare.
>
> We already have this situation for others external dependecies.
>
>> Another issue, mentioned by Fredrik in the issue is that currently
>> mpmath is tested with sympy.
>
> Are you about sympy/mpmath/tests/*?

Yes.

> Why this can't be tested with
> travis for a separate project?

It could and it should, but arguably mpmath gets much better testing
when many of sympy users run our test suite.

Again, if we were talking about numpy, then my argument would be quite
ridiculous, in the light of the stats that I discussed above, because
vastly more users install numpy than sympy. But we are talking about
mpmath, and there it stands, because at least 80% of Debian users only
install sympy, not mpmath.


Now for projects like Debian or Fedora, which do allow easy dependency
installation, we should provide an easy switch in our setup.py. That
way they can manage mpmath at one place and any Debian specific
patches and fixes that they contribute will automatically apply to
sympy. But this is specific to Debian/Fedora, as well as homebrew and
other distributions like Anaconda. On other platforms the users would
have to install mpmath from its webpage, so they will have an older
version than it is in sympy. Any security fixes that you mentioned (as
one possible worry), we would simply apply to sympy itself.


I apologize for the longer email, but this issue is something that I
want to make clear, as it is my highest priority that sympy remains
easy to use and install.

Ondrej

[Sergey Kirpichev]

On Sun, Jun 30, 2013 at 11:04:03AM -0600, Ondřej Čertík wrote:
> > Actually, this just means that we have mpmath fork.
>
> That's right.
>

> [...]

>
> One should only depend on libraries, that are well supported and widely
used.
> Mpmath is not. For example on my Ubuntu, here are all the projects that
depend
> on mpmath
>

> [...]

>
> As such, sympy is the only project that actually depends on mpmath

Taking into account all this. Do you think, mpmath should actually be
a separate project?

If it's a pet project for sympy - why it's not just a part of sympy?

If not, I don't think that the small current number of mpmath's users
(not total, but ones using mpmath AND NOT sympy) is an argument that
this project is not well supported and we should encourage forking.

> It could and it should, but arguably mpmath gets much better testing
> when many of sympy users run our test suite.

mpmath could run own test suite, in mpmath/tests/ (perhaps, we need a
patch for this?). There should be no decrease in test coverage at
least.

> Now for projects like Debian or Fedora, which do allow easy dependency
> installation, we should provide an easy switch in our setup.py.

Ok. I think, we got your points. Let's see, if there is other opinions.

> I apologize for the longer email, but this issue is something that I
> want to make clear, as it is my highest priority that sympy remains
> easy to use and install.

Through, you don't provide any examples of difficulties with mpmath's
installation.

[Ondřej Čertík]

On Wed, Jul 3, 2013 at 11:23 AM, Sergey Kirpichev <skirp...@gmail.com> wrote:
> On Sun, Jun 30, 2013 at 11:04:03AM -0600, Ondřej Čertík wrote:
>> > Actually, this just means that we have mpmath fork.
>>
>> That's right.
>>
>> [...]
>>
>> One should only depend on libraries, that are well supported and widely
> used.
>> Mpmath is not. For example on my Ubuntu, here are all the projects that
> depend
>> on mpmath
>>
>> [...]
>>
>> As such, sympy is the only project that actually depends on mpmath
>
> Taking into account all this. Do you think, mpmath should actually be
> a separate project?
>
> If it's a pet project for sympy - why it's not just a part of sympy?

Well it is part of sympy. :) It started as part of SymPy and so far we
have been treating it as part of SymPy. Fredrik eventually split it,
kept it and supported it as a separate project and called
it mpmath. Somebody then created a separate Debian package and so on.
And that's fine,
as mpmath does not depend on the rest of sympy, and if it is useful to
at least some people outside of sympy, then why not make it available
as a separate package too.

>
> If not, I don't think that the small current number of mpmath's users
> (not total, but ones using mpmath AND NOT sympy) is an argument that
> this project is not well supported and we should encourage forking.

Matthew had a great lightning talk at SciPy 2013 about taking common
code and taking
it out of the big package into a separate project. There are lots of
things in SymPy that
can be taken apart. For example with some extra work, sympy.polys
could be shipped as a separate package as well. Many things in
sympy.utilities too, for example memoization.py etc. Also
sympy.parsing could probably be made sympy independent (sympy would
only register the sympify callback).

So we can create all these little or bigger packages, like mpmath,
memoization, polys, decorators, cythonutils, parsing. And make the
user install all those in order to run sympy. I don't think it would
make things simpler for the user. I think it would make things more
complex.

>
>> It could and it should, but arguably mpmath gets much better testing
>> when many of sympy users run our test suite.
>
> mpmath could run own test suite, in mpmath/tests/ (perhaps, we need a
> patch for this?). There should be no decrease in test coverage at
> least.
>
>> Now for projects like Debian or Fedora, which do allow easy dependency
>> installation, we should provide an easy switch in our setup.py.
>
> Ok. I think, we got your points. Let's see, if there is other opinions.

Now when my opinion is clear, can you clearly write your opinion as well? :)

We have agreement on Debian/Fedora as well as that the size of mpmath
is not an issue.
We also have an agreement that for people like you (and others) who
prefer to install it separately
should have that option and it should be easy.

So why do you want to make this extra step and remove it from sympy completely?
Clearly it's not going to make *your* life any easier, since you'll
already be installing mpmath separately.
But you make it more complex for all the users who don't want to
bother with mpmath and just want sympy to work. So I still don't
understand your argument.

>
>> I apologize for the longer email, but this issue is something that I
>> want to make clear, as it is my highest priority that sympy remains
>> easy to use and install.
>
> Through, you don't provide any examples of difficulties with mpmath's
> installation.

Just the fact that you have to install it.

Ondrej

[Ronan Lamy]

I've stayed silent until then, but this discussion is getting annoying. I'm starting to think that the only way to restore sanity is to fork sympy.

2013/7/3 Ondřej Čertík <ondrej...@gmail.com>

On Wed, Jul 3, 2013 at 11:23 AM, Sergey Kirpichev <skirp...@gmail.com> wrote:
> On Sun, Jun 30, 2013 at 11:04:03AM -0600, Ondřej Čertík wrote:
>> > Actually, this just means that we have mpmath fork.
>>
>> That's right.
>>
>> [...]
>>
>> One should only depend on libraries, that are well supported and widely
> used.
>> Mpmath is not. For example on my Ubuntu, here are all the projects that
> depend
>> on mpmath
>>
>> [...]
>>
>> As such, sympy is the only project that actually depends on mpmath
>
> Taking into account all this. Do you think, mpmath should actually be
> a separate project?
>
> If it's a pet project for sympy - why it's not just a part of sympy?

Well it is part of sympy. :)

No, it's not. As you pointed out yourself, sympy.mpmath and mpmath are not the same thing.

 

It started as part of SymPy and so far we
have been treating it as part of SymPy.

No, we have not. Current policy is not to modify anything in sympy.mpmath.
 

Fredrik eventually split it,
kept it and supported it as a separate project and called
it mpmath. Somebody then created a separate Debian package and so on.
And that's fine,
as mpmath does not depend on the rest of sympy, and if it is useful to
at least some people outside of sympy, then why not make it available
as a separate package too.

>
> If not, I don't think that the small current number of mpmath's users
> (not total, but ones using mpmath AND NOT sympy) is an argument that
> this project is not well supported and we should encourage forking.

If mpmath does not have many users, it's in part because sympy robbed it of its users (it's basically impossible to use the real mpmath together with sympy)

>> It could and it should, but arguably mpmath gets much better testing
>> when many of sympy users run our test suite.

No, mpmath doesn't get any testing by sympy users, because what they use is sympy.mpmath, not mpmath.

>
> mpmath could run own test suite, in mpmath/tests/ (perhaps, we need a
> patch for this?).  There should be no decrease in test coverage at
> least.
>
>> Now for projects like Debian or Fedora, which do allow easy dependency
>> installation, we should provide an easy switch in our setup.py.
>
> Ok. I think, we got your points. Let's see, if there is other opinions.

Now when my opinion is clear, can you clearly write your opinion as well? :)

We have agreement on Debian/Fedora as well as that the size of mpmath
is not an issue.

Size is irrelevant. Code duplication is an issue in itself.
 

We also have an agreement that for people like you (and others) who
prefer to install it separately
should have that option and it should be easy.

Making it an option is not a solution. It might help packagers but it'll make the code for installation even more complicated, and force us to test both versions.

 

So why do you want to make this extra step and remove it from sympy completely?
Clearly it's not going to make *your* life any easier, since you'll
already be installing mpmath separately.

It would make *my* life easier, because the code for testing and installing would be seriously simplified.

 

But you make it more complex for all the users who don't want to
bother with mpmath and just want sympy to work. So I still don't
understand your argument.

It's "pip install sympy" in one case and "pip install sympy" in the other. You can call it more complex if you like, but it's not strictly more complex.

[Sergey B Kirpichev]

On Wed, Jul 03, 2013 at 09:52:20PM +0200, Ronan Lamy wrote:
> I've stayed silent until then, but this discussion is getting annoying.

Sorry :(

On Wed, Jul 03, 2013 at 12:17:42PM -0600, Ondřej Čertík wrote:
> Now when my opinion is clear, can you clearly write your opinion as well? :)

I thought, it was clear from beginning. I prefer to unbundle mpmath.
Reasons: nothing special. If it's a stable external project - why we
should fork it?

> > Through, you don't provide any examples of difficulties with mpmath's
> > installation.
>
> Just the fact that you have to install it.

What's the difference? As already Ronan said, why "pip install sympy"
can't handle both cases?

Only for people, using git clone - things won't work out of the box.

On Wed, Jul 03, 2013 at 09:52:20PM +0200, Ronan Lamy wrote:
> Making it an option is not a solution.

It's a minimal variant. Just in the case we want
to stay with bundled copy.

[Ondřej Čertík]

Hi Ronan,

On Wed, Jul 3, 2013 at 1:52 PM, Ronan Lamy <ronan...@gmail.com> wrote:
> I've stayed silent until then, but this discussion is getting annoying.

First of all, I am very sorry if we or I annoyed you. That was not my intention,
all I want is to rationally discuss pros and cons of unbundling mpmath.
For this I would like to thank you Sergey, that we kept the discussion based
on rational arguments and mutual respect.

> I'm starting to think that the only way to restore sanity is to fork sympy.

Well I would be very sad if we can't resolve things, especially since we agreed
on everything except the last point of actually removing mpmath completely.

> 2013/7/3 Ondřej Čertík <ondrej...@gmail.com>
>>
>> On Wed, Jul 3, 2013 at 11:23 AM, Sergey Kirpichev <skirp...@gmail.com>
>> wrote:
>> > On Sun, Jun 30, 2013 at 11:04:03AM -0600, Ondřej Čertík wrote:
>> >> > Actually, this just means that we have mpmath fork.
>> >>
>> >> That's right.
>> >>
>> >> [...]
>> >>
>> >> One should only depend on libraries, that are well supported and widely
>> > used.
>> >> Mpmath is not. For example on my Ubuntu, here are all the projects that
>> > depend
>> >> on mpmath
>> >>
>> >> [...]
>> >>
>> >> As such, sympy is the only project that actually depends on mpmath
>> >
>> > Taking into account all this. Do you think, mpmath should actually be
>> > a separate project?
>> >
>> > If it's a pet project for sympy - why it's not just a part of sympy?
>>
>> Well it is part of sympy. :)
>
>
> No, it's not. As you pointed out yourself, sympy.mpmath and mpmath are not
> the same thing.
>
>>
>> It started as part of SymPy and so far we
>> have been treating it as part of SymPy.
>
>
> No, we have not. Current policy is not to modify anything in sympy.mpmath.

I guess the correct way to put it would be: Ondrej did but Ronan didn't. ;)

>> Fredrik eventually split it,
>> kept it and supported it as a separate project and called
>> it mpmath. Somebody then created a separate Debian package and so on.
>> And that's fine,
>> as mpmath does not depend on the rest of sympy, and if it is useful to
>> at least some people outside of sympy, then why not make it available
>> as a separate package too.
>>
>> >
>> > If not, I don't think that the small current number of mpmath's users
>> > (not total, but ones using mpmath AND NOT sympy) is an argument that
>> > this project is not well supported and we should encourage forking.
>
>
> If mpmath does not have many users, it's in part because sympy robbed it of
> its users (it's basically impossible to use the real mpmath together with
> sympy)

We all already agreed that we should fix this, so that sympy can be easily
used with externally installed mpmath.

>>
>>
>> >> It could and it should, but arguably mpmath gets much better testing
>> >> when many of sympy users run our test suite.
>
>
> No, mpmath doesn't get any testing by sympy users, because what they use is
> sympy.mpmath, not mpmath.

Which, as you pointed above, the policy is not to modify sympy.mpmath,
or a least
send the patches upstream, as we did. So mpmath does get testing by sympy users.

>>
>> >
>> > mpmath could run own test suite, in mpmath/tests/ (perhaps, we need a
>> > patch for this?). There should be no decrease in test coverage at
>> > least.
>> >
>> >> Now for projects like Debian or Fedora, which do allow easy dependency
>> >> installation, we should provide an easy switch in our setup.py.
>> >
>> > Ok. I think, we got your points. Let's see, if there is other opinions.
>>
>> Now when my opinion is clear, can you clearly write your opinion as well?
>> :)
>>
>> We have agreement on Debian/Fedora as well as that the size of mpmath
>> is not an issue.
>
>
> Size is irrelevant. Code duplication is an issue in itself.
>
>>
>> We also have an agreement that for people like you (and others) who
>> prefer to install it separately
>> should have that option and it should be easy.
>
>
> Making it an option is not a solution. It might help packagers but it'll
> make the code for installation even more complicated, and force us to test
> both versions.

I haven't thought about this before, but you are right. If we start
depending on mpmath as an external project, then we have to test all
versions that we decide to support. So that's a major pain and that is
a great argument why we should include a version of mpmath in sympy
and making sure that this particular version works great with sympy on
all platforms and Python versions.

>
>>
>> So why do you want to make this extra step and remove it from sympy
>> completely?
>> Clearly it's not going to make *your* life any easier, since you'll
>> already be installing mpmath separately.
>
>
> It would make *my* life easier, because the code for testing and installing
> would be seriously simplified.

Are we talking about

python setup.py install --no-mpmath

versus

python setup.py install

or about something else?

>
>>
>> But you make it more complex for all the users who don't want to
>> bother with mpmath and just want sympy to work. So I still don't
>> understand your argument.
>
>
> It's "pip install sympy" in one case and "pip install sympy" in the other.
> You can call it more complex if you like, but it's not strictly more
> complex.

That's if you use virtualenv. Many times I install from tar.gz or git,
and many times offline.
In all these cases, it will be more complex. Other users install .exe
on Windows.
And so on.


Sergey, I think I already stated my opinion to all your questions from
your last email, so just
briefly:

>> Now when my opinion is clear, can you clearly write your opinion as well? :)
>

> I thought, it was clear from beginning. I prefer to unbundle mpmath.
> Reasons: nothing special. If it's a stable external project - why we
> should fork it?
>

I stated all the reasons why we should include it in sympy in my
previous emails. If your only reason against is "nothing special", in
my opinion that's not good enough reason to make life more complex to
a lot of our users.

Ronan did state however a valid reason:

* code duplication

and I think that's your main reason as well, right? That is a valid reason, yes.



Anyway, I think the conclusion is clear -- make it so that sympy can
use external mpmath, then we'll go from there.

Ondrej

[Sergey B Kirpichev]

On Wed, Jul 03, 2013 at 04:33:50PM -0600, Ondřej Čertík wrote:
> I haven't thought about this before, but you are right. If we start
> depending on mpmath as an external project, then we have to test all
> versions that we decide to support. So that's a major pain and that is
> a great argument why we should include a version of mpmath in sympy
> and making sure that this particular version works great with sympy on
> all platforms and Python versions.

Why we can't just depend on this particular version of mpmath? If it
has a stable api and if it's well supported - what's a difference?
It's hardly a reason to include a bundled copy...

> I stated all the reasons why we should include it in sympy in my
> previous emails. If your only reason against is "nothing special", in
> my opinion that's not good enough reason to make life more complex to
> a lot of our users.

I think the major reasons for unbundling of mpmath are the reasons why
few programs ship their own libraries. Nothing special.

Your reasons are applicable to any external dependency of sympy. The
only difference: mpmath is not optional.

> That's if you use virtualenv. Many times I install from tar.gz or git,
> and many times offline.

It seems to be not very common setups. Especially, an offline setup
with git...

> Other users install .exe on Windows.

Why we can't include mpmath here (but from external source)?

> Anyway, I think the conclusion is clear -- make it so that sympy can
> use external mpmath, then we'll go from there.

I don't see why we can't decide now whether we want include a copy of
mpmath or not, but reiterate again...

[Ronan Lamy]

2013/7/4 Ondřej Čertík <ondrej...@gmail.com>

Hi Ronan,

On Wed, Jul 3, 2013 at 1:52 PM, Ronan Lamy <ronan...@gmail.com> wrote:
> I've stayed silent until then, but this discussion is getting annoying.

First of all, I am very sorry if we or I annoyed you. That was not my intention,
all I want is to rationally discuss pros and cons of unbundling mpmath.

That's the thing. I don't think you're being rational on this issue. You seem to claim the benefits both of forking and of not forking mpmath. With the current situation, we get neither.
 

For this I would like to thank you Sergey, that we kept the discussion based
on rational arguments and mutual respect.

> I'm starting to think that the only way to restore sanity is to fork sympy.

Well I would be very sad if we can't resolve things, especially since we agreed
on everything except the last point of actually removing mpmath completely.

Well, I think there's room for a light-weight, more modular version of sympy. But don't worry, I don't have the time to actually do it.

I doubt that all the patches have been sent upstream. In any case, patches to mpmath have not been applied to sympy.mpmath, so what is tested is not mpmath. (and even if sympy.mpmath were a perfect copy of some version of mpmath, "sympy.mpmath is not mpmath" would still be True)

So mpmath does get testing by sympy users.

 

>>

>> We also have an agreement that for people like you (and others) who
>> prefer to install it separately
>> should have that option and it should be easy.
>
>
> Making it an option is not a solution. It might help packagers but it'll
> make the code for installation even more complicated, and force us to test
> both versions.

I haven't thought about this before, but you are right. If we start
depending on mpmath as an external project, then we have to test all
versions that we decide to support. So that's a major pain and that is
a great argument why we should include a version of mpmath in sympy
and making sure that this particular version works great with sympy on
all platforms and Python versions.

We already have that problem with all our "optional" dependencies.

>
>>
>> So why do you want to make this extra step and remove it from sympy
>> completely?
>> Clearly it's not going to make *your* life any easier, since you'll
>> already be installing mpmath separately.
>
>
> It would make *my* life easier, because the code for testing and installing
> would be seriously simplified.

Are we talking about

python setup.py install --no-mpmath

versus

python setup.py install

or about something else?

I'm talking about the code inside setup.py and everything that it calls.

>
>>
>> But you make it more complex for all the users who don't want to
>> bother with mpmath and just want sympy to work. So I still don't
>> understand your argument.
>
>
> It's "pip install sympy" in one case and "pip install sympy" in the other.
> You can call it more complex if you like, but it's not strictly more
> complex.

That's if you use virtualenv. Many times I install from tar.gz or git,
and many times offline.

You can use pip without virtualenv. You can also install from git or from a local .tar.gz.
 

In all these cases, it will be more complex. Other users install .exe
on Windows.
And so on.

 

Anyway, I think the conclusion is clear -- make it so that sympy can

use external mpmath, then we'll go from there. 

Well, if we do that, we'll have to support all relevant versions of mpmath, so we don't gain anything compared to just depending on it. Also, this will certainly require fixing the interaction between sympy.mpmath and mpmath, which means modifying sympy.mpmath.

It seems that you actually want sympy.mpmath to be a fork of mpmath, and a fully integrated part of sympy, so why don't you just say so?

[Ondřej Čertík]

On Wed, Jul 3, 2013 at 6:03 PM, Ronan Lamy <ronan...@gmail.com> wrote:
> 2013/7/4 Ondřej Čertík <ondrej...@gmail.com>
>>
>> Hi Ronan,
>>
>> On Wed, Jul 3, 2013 at 1:52 PM, Ronan Lamy <ronan...@gmail.com> wrote:
>> > I've stayed silent until then, but this discussion is getting annoying.
>>
>> First of all, I am very sorry if we or I annoyed you. That was not my
>> intention,
>> all I want is to rationally discuss pros and cons of unbundling mpmath.
>
>
> That's the thing. I don't think you're being rational on this issue. You
> seem to claim the benefits both of forking and of not forking mpmath. With
> the current situation, we get neither.

That's why we discuss it so that we can clarify things out. That's the only way
I know of how to figure out what's best for sympy.

>
>>
>> For this I would like to thank you Sergey, that we kept the discussion
>> based
>> on rational arguments and mutual respect.
>>
>> > I'm starting to think that the only way to restore sanity is to fork
>> > sympy.
>>
>> Well I would be very sad if we can't resolve things, especially since we
>> agreed
>> on everything except the last point of actually removing mpmath
>> completely.
>
>
> Well, I think there's room for a light-weight, more modular version of
> sympy. But don't worry, I don't have the time to actually do it.

There is also room for faster sympy. I want sympy to become faster and
more modular.

I think sympy.mpmath is the exact copy of the latest release of mpmath.
Plus a few of our patches. If Fredrik releases a new version, we
simply update it,
again exactly. That's how we've been doing it in the past.

>
>> So mpmath does get testing by sympy users.
>
>
>>
>> >>
>>
>> >> We also have an agreement that for people like you (and others) who
>> >> prefer to install it separately
>> >> should have that option and it should be easy.
>> >
>> >
>> > Making it an option is not a solution. It might help packagers but it'll
>> > make the code for installation even more complicated, and force us to
>> > test
>> > both versions.
>>
>> I haven't thought about this before, but you are right. If we start
>> depending on mpmath as an external project, then we have to test all
>> versions that we decide to support. So that's a major pain and that is
>> a great argument why we should include a version of mpmath in sympy
>> and making sure that this particular version works great with sympy on
>> all platforms and Python versions.
>
>
> We already have that problem with all our "optional" dependencies.

That's right.

>>
>>
>> >
>> >>
>> >> So why do you want to make this extra step and remove it from sympy
>> >> completely?
>> >> Clearly it's not going to make *your* life any easier, since you'll
>> >> already be installing mpmath separately.
>> >
>> >
>> > It would make *my* life easier, because the code for testing and
>> > installing
>> > would be seriously simplified.
>>
>> Are we talking about
>>
>> python setup.py install --no-mpmath
>>
>> versus
>>
>> python setup.py install
>>
>> or about something else?
>
>
> I'm talking about the code inside setup.py and everything that it calls.

Ok. I think this can be resolved so that it is not messy or complex.

>>
>>
>> >
>> >>
>> >> But you make it more complex for all the users who don't want to
>> >> bother with mpmath and just want sympy to work. So I still don't
>> >> understand your argument.
>> >
>> >
>> > It's "pip install sympy" in one case and "pip install sympy" in the
>> > other.
>> > You can call it more complex if you like, but it's not strictly more
>> > complex.
>>
>> That's if you use virtualenv. Many times I install from tar.gz or git,
>> and many times offline.
>
>
> You can use pip without virtualenv. You can also install from git or from a
> local .tar.gz.
>
>>
>> In all these cases, it will be more complex. Other users install .exe
>> on Windows.
>> And so on.
>
>
>>
>> Anyway, I think the conclusion is clear -- make it so that sympy can
>>
>> use external mpmath, then we'll go from there.
>
>
> Well, if we do that, we'll have to support all relevant versions of mpmath,
> so we don't gain anything compared to just depending on it. Also, this will
> certainly require fixing the interaction between sympy.mpmath and mpmath,
> which means modifying sympy.mpmath.
>
> It seems that you actually want sympy.mpmath to be a fork of mpmath, and a
> fully integrated part of sympy, so why don't you just say so?

I want sympy to be self contained and very robustly tested and work out of the
box in the basic configuration, yes. I do not want to fork mpmath.

My definition of a "fork" is a project, that becomes incompatible with
the original project.
That is definitely not the case and should not be the case for mpmath.
sympy.mpmath is like "git fork", it is equal to the latest release + a
few of our patches
that make sure that things work together. More importantly, sympy is
adapted to work
with whatever mpmath we have.

Once we drop Python 2.5, it should be possible to use relative
imports. If this is
of utter importance to you guys, I'll try to spend some time to resolve it.
I do think that it would be a big improvement to add a switch to setup.py,
to use systemwide mpmath. If nothing, about 80% of the above discussion
will not be relevant, because the whole discussion would be about making
"python setup.py --no-mpmath" the default, and about "rm -rf sympy.mpmath",
both of which are technically very easy to implement.

Also, we can run a google docs poll among our users to see what they prefer.

Ondrej

[Joachim Durchholz]

Am 03.07.2013 20:17, schrieb Ondřej Čertík:
> So we can create all these little or bigger packages, like mpmath,
> memoization, polys, decorators, cythonutils, parsing. And make the
> user install all those in order to run sympy. I don't think it would
> make things simpler for the user. I think it would make things more
> complex.

Erm... no?
If a user needs one package, the other packages can be pulled as needed
via dependencies.
Plus a meta-package that just contains dependencies on every module in
SymPy.

Sholdn't make the users' lives more complex, except that they have more
options (which can be a complication in itself, but if smaller packages
are desired ... *shrug*).

It would make development harder because dependencies between packages
would have to be spelled out, and the correctness of the dependency
information would need to be unit tested. (I have no idea to do that
kind of dependency check in Python, so maybe it's not reasonably
feasible at all.)

[Stefan Krastanov]

I am pretty much ok with any of the _non-complex_ suggestions
mentioned in the thread (for an appropriate definition of "complex"),
but I guess the following datapoint could be useful whenever you
decide to continue the discussion:

- on more than one occasion I have wanted to install only the
polynomials module without the rest of sympy

- the same goes for the core+polynomials without anything else

The reasons were usually revolving around building a better mental
model of the library, searching for unnecessary or even stupid
dependencies, etc.

It would also be very useful for experimentation. Currently sympy does
_not_ have a simple core. At least in the mental model of more casual
user like myself.

[Freddie Witherden]

As someone who has an application which uses *both* SymPy *and* mpmath I
am absolutely against removing the embedded copy. There are several
reasons for this.

When we first started developing our application the plan was to allow
for both system and embedded mpmath installations. The result was code
along the lines of:

try:
import mpmath as mp
except ImportError:
from sympy import mpmath as mp

which was something of a nuisance. Worse, if the user had opted to
install their own version of mpmath but SymPy was configured to use its
own version things were found to go seriously awry. Eventually we gave
up and just switched to using the embedded copy within SymPy. Removing
mpmath would reopen this can of worms for very little benefit.

Now, if mpmath was a flourishing project I would be open to the idea of
splitting the two apart. However, as of June 2013, it isn't. There
have been no commits to the official subversion repository in quite some
time (Nov 2011) with the version that ships with SymPy, for all intents
and purposes, being the de facto revision. Case in point: Google
mpmath, the second result for me is:

http://docs.sympy.org/dev/modules/mpmath/index.html

which is from the SymPy project.

I would much rather see mpmath brought back under the umbrella of SymPy
at least until a new maintainer can be found. At the very least I do
not believe that removing the embedded copy of mpmath should even be on
the table until all of the patches (or functional equivalents thereof)
have been merged upstream and a new mpmath release has been made.

Polemically yours, Freddie.

[Aaron Meurer]

On Tue, Jul 9, 2013 at 11:56 AM, Freddie Witherden <fre...@witherden.org> wrote:

On 28/06/13 14:42, Sergey Kirpichev wrote:
> There is a long term issue [1] to make mpmath an external dependency.
> It was suggested by
> Aaron (see discussion in PR [2]), that we should discuss this on the
> mailing list.
>
> Perhaps, the main reason to prefer bundled mpmath copy is a
> mystical "complex installation".  Not clear enough.  For example, I don't
> think that additional steps for package installation from mpmath manual
> does matter.  For non-root users
> there shouldn't be any problems too, at least for Python 2.6+ (per-user
> site-package directory).
>
> [1] http://code.google.com/p/sympy/issues/detail?id=2482
> [2] https://github.com/sympy/sympy/pull/2192

As someone who has an application which uses *both* SymPy *and* mpmath I
am absolutely against removing the embedded copy.  There are several
reasons for this.

Thank you for your opinion. I think it's important to get the views of the people who are actually using sympy and mpmath.

 

When we first started developing our application the plan was to allow
for both system and embedded mpmath installations.  The result was code
along the lines of:

try:
    import mpmath as mp
except ImportError:
    from sympy import mpmath as mp

which was something of a nuisance.  Worse, if the user had opted to
install their own version of mpmath but SymPy was configured to use its
own version things were found to go seriously awry.  Eventually we gave
up and just switched to using the embedded copy within SymPy.  Removing
mpmath would reopen this can of worms for very little benefit.

Now, if mpmath was a flourishing project I would be open to the idea of
splitting the two apart.  However, as of June 2013, it isn't.  There
have been no commits to the official subversion repository in quite some
time (Nov 2011) with the version that ships with SymPy, for all intents
and purposes, being the de facto revision.  Case in point: Google
mpmath, the second result for me is:

The source of mpmath is now at https://github.com/fredrik-johansson/mpmath, but I think your point is still valid. There haven't been any major changes other than bugfixes since 2011 (all the changes since 2011 fit on one GitHub page https://github.com/fredrik-johansson/mpmath/commits/master), and there have been a total of three pull requests in the two years that the GitHub repo has existed.

Aaron Meurer

[Ondřej Čertík]

I agree with you that removing mpmath brings more cons than benefits overall.

However, I just want to understand the root of the particular problem
you mentioned.
If sympy removes mpmath, then sympy cannot be configured to use its own mpmath.
In other words, you always do just "import mpmath as mp". What things
go seriously
awry? Is it some incompatibility between mpmath and sympy?

Or did you mean that if sympy ships its own mpmath (as it does now),
and the user
installs systemwide mpmath and you start mixing the two? In other
words, that would
be an argument for removing mpmath from sympy.

>
> Now, if mpmath was a flourishing project I would be open to the idea of
> splitting the two apart. However, as of June 2013, it isn't. There
> have been no commits to the official subversion repository in quite some
> time (Nov 2011) with the version that ships with SymPy, for all intents
> and purposes, being the de facto revision. Case in point: Google
> mpmath, the second result for me is:
>
> http://docs.sympy.org/dev/modules/mpmath/index.html
>
> which is from the SymPy project.
>
> I would much rather see mpmath brought back under the umbrella of SymPy
> at least until a new maintainer can be found. At the very least I do
> not believe that removing the embedded copy of mpmath should even be on
> the table until all of the patches (or functional equivalents thereof)
> have been merged upstream and a new mpmath release has been made.

Yep, I agree. I've been trying to make this point over and over in this thread.
Thanks for sharing your opinion.

Ondrej

[Freddie Witherden]

On 09/07/13 18:55, Ondřej Čertík wrote:
> I agree with you that removing mpmath brings more cons than benefits
> overall.
>
> However, I just want to understand the root of the particular
> problem you mentioned. If sympy removes mpmath, then sympy cannot be
> configured to use its own mpmath. In other words, you always do just
> "import mpmath as mp". What things go seriously awry? Is it some
> incompatibility between mpmath and sympy?
>
> Or did you mean that if sympy ships its own mpmath (as it does now),
> and the user installs systemwide mpmath and you start mixing the two?
> In other words, that would be an argument for removing mpmath from
> sympy.

If sympy removes mpmath then we would indeed switch to "import mpmath as
mp". The issues arise at the moment when people accidentally mix a
system-wide mpmath install with the one in sympy. Hence why we only
support using the copy embedded in sympy.

Unbundling mpmath would mean that we would have to start -- at least in
the interim -- supporting the current version of sympy (which will use
the system wide mpmath) and legacy versions (via their embedded copy).
At this point I would need to check when I import mpmath that I am
pulling in the same version being used by sympy and no accidentally
using the system version when I should be using the embedded version.

Polemically yours, Freddie.

[Stefan Krastanov]

The issue is that `mpmath.whatever` and `sympy.mpmath.whatever` are considered by the interpreter to be different classes even if the same versions of mpmath are used, hence methods like `__add__` raise NotImplementedError. I have heard this issue to be raised occasionally before.

[Ondřej Čertík]

Right. But again, playing the devil's advocate, this would be an
argument for removing mpmath from sympy completely.

There is also another option (unfortunately not perfect either, see
below), if we choose to depend on mpmath --- we can use git
submodules. The sympy master hash then contains the exact hash of all
submodules. If we update mpmath in sympy, all we need to do is to pull
in more patches for mpmath from its own repo, and then we need to
create a new commit in sympy. Then when you checkout any previous
commit in sympy, and do "git submodule update" (unfortunately you have
to do this by hand), it will checkout the *exact* mpmath that was
known to work with that particular commit of sympy. So we would still
only support and test exactly one exact version (commit) of mpmath for
each commit of sympy. In other words, almost no change to what we do
now. We would need to do something like:

rm -rf sympy/mpmath
git submodule add https://github.com/sympy/mpmath sympy/mpmath

where the https://github.com/sympy/mpmath repository would
unfortunately only contain the "mpmath" directory of the mpmath repo
(this point is not very satisfactory though). So for this last point I
am now happy with this solution either, but at least it's good to know
that git submodules fix the problem of what exact version of mpmath we
depend on as time goes.

Ondrej

[Ondřej Čertík]

Actually, one can use a symlink to fix this issue. I created this PR:

https://github.com/sympy/sympy/pull/2267

if anyone wants to play with it (see the PR for details).
Note that I am -0.5 and Aaron is -1 on this PR,
but it is nevertheless good to know about this approach.

Ondrej

[Sergey Kirpichev]

Hello,

Perhaps, it's time to revive this old discussion, as sympy devs seems
to be more positive for allowing external dependencies:
https://github.com/sympy/sympy/issues/7339

Thus, it's a ping for mailing list about
https://github.com/sympy/sympy/pull/2192
(as suggested by Ondřej)

[Ondřej Čertík]

On Sat, Apr 26, 2014 at 10:52 AM, Sergey Kirpichev <skirp...@gmail.com> wrote:
> Hello,
>
> Perhaps, it's time to revive this old discussion, as sympy devs seems
> to be more positive for allowing external dependencies:
> https://github.com/sympy/sympy/issues/7339
>
> Thus, it's a ping for mailing list about
> https://github.com/sympy/sympy/pull/2192
> (as suggested by Ondřej)

I am fine with depending on mpmath, if this is what the community prefers.

It seems that setuptools would allow us to specify which exact
versions we are compatible with. But it adds another dependency to
sympy.

Ondrej

>
>
> On Friday, June 28, 2013 5:42:07 PM UTC+4, Sergey Kirpichev wrote:
>>
>> There is a long term issue [1] to make mpmath an external dependency. It
>> was suggested by
>> Aaron (see discussion in PR [2]), that we should discuss this on the
>> mailing list.
>>
>> Perhaps, the main reason to prefer bundled mpmath copy is a
>> mystical "complex installation". Not clear enough. For example, I don't
>> think that additional steps for package installation from mpmath manual
>> does matter. For non-root users
>> there shouldn't be any problems too, at least for Python 2.6+ (per-user
>> site-package directory).
>>
>> [1] http://code.google.com/p/sympy/issues/detail?id=2482
>> [2] https://github.com/sympy/sympy/pull/2192
>>

> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sympy.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sympy/6b54d310-2754-4a4c-acb9-15d77447b6e3%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Sergey B Kirpichev]

On Sat, Apr 26, 2014 at 09:49:57PM -0600, Ondřej Čertík wrote:
> It seems that setuptools would allow us to specify which exact
> versions we are compatible with. But it adds another dependency to
> sympy.

But we ship a bootstrap module, which allow user to automatically
download and install this dependency.

[Sergey Kirpichev]

FYI: now #2192 was merged.

Please be aware about new installation instructions for SymPy:
https://github.com/sympy/sympy#installation
https://github.com/sympy/sympy/wiki/Development-workflow#install-mpmath

[Joachim Durchholz]

Am 02.01.2015 um 19:25 schrieb Sergey Kirpichev:
> FYI: now #2192 was merged.
>
> Please be aware about new installation instructions for SymPy:
> https://github.com/sympy/sympy#installation
> https://github.com/sympy/sympy/wiki/Development-workflow#install-mpmath

Requiring manual installation of dependencies is going to give
propective users a bad first impression.
This is going to get worse as more dependencies are being added (pep8.py
is already being discussed).

I assume that this can be fixed by using pip in the installation
instructions, and noting the proper dependencies.
The pip docs list some scenarios where the wrong version of a dependency
might be installed, so we probably need to add a check that we indeed
have the correct version.

[Sergey B Kirpichev]

On Fri, Jan 02, 2015 at 10:57:34PM +0100, Joachim Durchholz wrote:
> Requiring manual installation of dependencies is going to give
> propective users a bad first impression.

If you are using setuptools (i.e. with pip), everything will
just works. Just use "pip install sympy", as now.

> I assume that this can be fixed by using pip in the installation
> instructions, and noting the proper dependencies.

I think, we can add instructions for using pip in the next release.

> The pip docs list some scenarios where the wrong version of a
> dependency might be installed

Could you be more precise?

[Joachim Durchholz]

Am 02.01.2015 um 23:15 schrieb Sergey B Kirpichev:
> On Fri, Jan 02, 2015 at 10:57:34PM +0100, Joachim Durchholz wrote:
>> Requiring manual installation of dependencies is going to give
>> propective users a bad first impression.
>
> If you are using setuptools (i.e. with pip), everything will
> just works. Just use "pip install sympy", as now.

Does that install the dependencies?
Because I didn't find the requirements.txt file in the SymPy install. (I
might be too unfamiliar with the packaging system to check everything well.)

>> I assume that this can be fixed by using pip in the installation
>> instructions, and noting the proper dependencies.
>
> I think, we can add instructions for using pip in the next release.

Hopefully that menans "upcoming release"... I'd hate it if a SymPy
release with that kind of install went out.

>> The pip docs list some scenarios where the wrong version of a
>> dependency might be installed
>
> Could you be more precise?

See https://pip.pypa.io/en/latest/user_guide.html, search for "pip
doesn’t have true dependency resolution".

[Sergey B Kirpichev]

On Sat, Jan 03, 2015 at 12:08:04AM +0100, Joachim Durchholz wrote:
> Does that install the dependencies?

Yes. Simple test:
$ pip install 'git+https://github.com/sympy/sympy.git'
Downloading/unpacking git+https://github.com/sympy/sympy.git
Cloning https://github.com/sympy/sympy.git to /tmp/pip-iU8YtD-build
Running setup.py egg_info for package from git+https://github.com/sympy/sympy.git

no previously-included directories found matching 'doc/_build'
warning: no files found matching 'TODO'
Downloading/unpacking mpmath>=0.19 (from sympy==0.7.6-git)
Downloading mpmath-0.19.tar.gz (498Kb): 498Kb downloaded
Running setup.py egg_info for package mpmath

Installing collected packages: mpmath, sympy
Running setup.py install for mpmath

Running setup.py install for sympy
changing mode of build/scripts-2.7/isympy from 640 to 755

no previously-included directories found matching 'doc/_build'
warning: no files found matching 'TODO'
changing mode of /home/sk/.virtualenvs/test-sympy/bin/isympy to 755
Successfully installed mpmath sympy
Cleaning up...

> Because I didn't find the requirements.txt file in the SymPy
> install. (I might be too unfamiliar with the packaging system to
> check everything well.)

Yes, you definitely should be more familar with this.

> >>I assume that this can be fixed by using pip in the installation
> >>instructions, and noting the proper dependencies.
> >
> >I think, we can add instructions for using pip in the next release.
>
> Hopefully that menans "upcoming release"... I'd hate it if a SymPy
> release with that kind of install went out.

In fact, we can add pip instructions right now. Nothing will
be changed.

> >>The pip docs list some scenarios where the wrong version of a
> >>dependency might be installed
> >
> >Could you be more precise?
>
> See https://pip.pypa.io/en/latest/user_guide.html, search for "pip
> doesn’t have true dependency resolution".

Oh, this. There are workarounds. Also, I don't know
any other packages, that depend on mpmath.

[Francois Bissey]

I am guessing here that the concern is that the latest version available
will be installed. So you have to be sure that mpmath never introduce
changes that are breaks compatibility with your current release (and if it
does you are welcome to rush out a release of sympy that is compatible).

Also you have to make sure that all your mpmath patches are included upstream
not like in the latest sympy release. Although in all fairness the current sympy
works with an unbundled mpmath at 0.19.

Disclaimer: I did the first unbundling of mpmath in sympy for Gentoo (based
on work done on Mandriva before me) and I am also responsible for the current
unbundling in Gentoo.

François

[Sergey B Kirpichev]

On Fri, Jan 02, 2015 at 11:51:35PM +0000, Francois Bissey wrote:
> I am guessing here that the concern is that the latest version available
> will be installed. So you have to be sure that mpmath never introduce
> changes that are breaks compatibility with your current release

Lets hope that mpmath will not break backward compatibility.

> Also you have to make sure that all your mpmath patches are included upstream

Yes, I sure.

[Joachim Durchholz]

Am 03.01.2015 um 01:16 schrieb Sergey B Kirpichev:
> On Fri, Jan 02, 2015 at 11:51:35PM +0000, Francois Bissey wrote:
>> I am guessing here that the concern is that the latest version available
>> will be installed. So you have to be sure that mpmath never introduce
>> changes that are breaks compatibility with your current release
>
> Lets hope that mpmath will not break backward compatibility.

I wouldn't want to rely on hope.
Let's stick to the one version we test for during a release. Anything
else is going to slow down bug fixing.

>> Also you have to make sure that all your mpmath patches are included upstream
>
> Yes, I sure.

Í do not think that that is a serious problem in the first place.
If it comes to the worst and the project is hijacked by somebody who
makes incompatible changes, we can still fork.

[Sergey B Kirpichev]

On Sat, Jan 03, 2015 at 12:27:41PM +0100, Joachim Durchholz wrote:
> >Lets hope that mpmath will not break backward compatibility.
>
> I wouldn't want to rely on hope.

In fact, this is a sane assumption about library development.

> Let's stick to the one version we test for during a release.

Why? This is unnecessary and only add difficulties for
packagers.

[Joachim Durchholz]

Am 03.01.2015 um 13:02 schrieb Sergey B Kirpichev:
> On Sat, Jan 03, 2015 at 12:27:41PM +0100, Joachim Durchholz wrote:
>>> Lets hope that mpmath will not break backward compatibility.
>>
>> I wouldn't want to rely on hope.
>
> In fact, this is a sane assumption about library development.

I strongly disagree about this being a sane assumption.

It may be sane for libraries that already have a varied set of
consumers, which is proof that they have committed to a stable API. Even
there, I have seen libraries make horrifying decisions.

>> Let's stick to the one version we test for during a release.
>
> Why? This is unnecessary

Why? For the reasons I gave.
These reasons are unrefuted, you just snip them and claim the opposite -
what do you suppose that I do, bow to your superior intellect?
Sarcasm aside, you're not going to convince me by "because I say so".

> and only add difficulties for packagers.

Well, that's a problem for the packagers to solve.

I'd say they should be helped if they present concrete problems and
cooperate for a solution. For example, if they want to allow additional
versions, and we find the testing overhead exceeds what we can do on
Travis, it would be appropriate if we could use their test farm. Or
maybe they can convince the Travis guys to give us faster testing.

If the packagers can't help us, then maybe their problems aren't that
relevant to the distribution that they're working for, and they'd need
to solve the problems themselves.
I'm sympathetic with their situation, I'm not going to let their
priorities override ours, which has reliability as its primary purpose.

If this is going to be a problem, then maybe unbundling mpmath was the
wrong decision.

[Ondřej Čertík]

On Sat, Jan 3, 2015 at 1:27 PM, Joachim Durchholz <j...@durchholz.org> wrote:
> Am 03.01.2015 um 13:02 schrieb Sergey B Kirpichev:
>>
>> On Sat, Jan 03, 2015 at 12:27:41PM +0100, Joachim Durchholz wrote:
>>>>
>>>> Lets hope that mpmath will not break backward compatibility.
>>>
>>>
>>> I wouldn't want to rely on hope.
>>
>>
>> In fact, this is a sane assumption about library development.
>
>
> I strongly disagree about this being a sane assumption.
>
> It may be sane for libraries that already have a varied set of consumers,
> which is proof that they have committed to a stable API. Even there, I have
> seen libraries make horrifying decisions.

Fredrik said that he will try to keep the API stable, so it will work out fine.

>
>>> Let's stick to the one version we test for during a release.
>>
>>
>> Why? This is unnecessary
>
>
> Why? For the reasons I gave.
> These reasons are unrefuted, you just snip them and claim the opposite -
> what do you suppose that I do, bow to your superior intellect?
> Sarcasm aside, you're not going to convince me by "because I say so".
>
>> and only add difficulties for packagers.
>
> Well, that's a problem for the packagers to solve.
>
> I'd say they should be helped if they present concrete problems and
> cooperate for a solution. For example, if they want to allow additional
> versions, and we find the testing overhead exceeds what we can do on Travis,
> it would be appropriate if we could use their test farm. Or maybe they can
> convince the Travis guys to give us faster testing.
>
> If the packagers can't help us, then maybe their problems aren't that
> relevant to the distribution that they're working for, and they'd need to
> solve the problems themselves.
> I'm sympathetic with their situation, I'm not going to let their priorities
> override ours, which has reliability as its primary purpose.
>
> If this is going to be a problem, then maybe unbundling mpmath was the wrong
> decision.

I think we can solve technical problems like testing on Travis.

I agree with you that pip is a horrible way to install and maintain
software, but fortunately there are now pretty good package source
managers like Conda or Hashdist, so you don't need to use it. Neither
of these two existed when we first wanted to decouple mpmath, but
today there solutions exist and while not perfect, they are improving
and are very usable.

Ondrej

[Joachim Durchholz]

Am 03.01.2015 um 22:59 schrieb Ondřej Čertík:
> On Sat, Jan 3, 2015 at 1:27 PM, Joachim Durchholz <j...@durchholz.org> wrote:
>> Am 03.01.2015 um 13:02 schrieb Sergey B Kirpichev:
>>>
>>> On Sat, Jan 03, 2015 at 12:27:41PM +0100, Joachim Durchholz wrote:
>>>>>
>>>>> Lets hope that mpmath will not break backward compatibility.
>>>>
>>>> I wouldn't want to rely on hope.
>>>
>>> In fact, this is a sane assumption about library development.
>>
>> I strongly disagree about this being a sane assumption.
>>
>> It may be sane for libraries that already have a varied set of consumers,
>> which is proof that they have committed to a stable API. Even there, I have
>> seen libraries make horrifying decisions.
>
> Fredrik said that he will try to keep the API stable, so it will work out fine.

I still see risks, though there's a limit to which discussing failure
scenarios makes sense. After a while, the probability of overlooking a
failure scenario becomes higher than the probability of each individual
failure scenario.

However, even if the risk is small, I wouldn't want to promote using
SymPy together with an mpmath version that it wasn't tested against.

> I think we can solve technical problems like testing on Travis.

That's going to be a combinatorial explosion.

> I agree with you that pip is a horrible way to install and maintain
> software,

What are the problems with that?
Right now I'm aware of that versioning issue, which I think can be
worked around though I'm not 100% sure yet.

> but fortunately there are now pretty good package source
> managers like Conda or Hashdist,

How are they better than pip?
I'm not opposed to any of these actually, I just don't know what the
differences are.

I'm generally much in favor of unbundling, I'm just extremely cautious
about avoiding reliability or bugfixing problems. What I wouldn't want
is if SymPy got cluttered with code that used different workarounds
depending on what version of a dependency is installed - such things are
reliability and productivity killers (been there, done that, didn't like
it).

[Ondřej Čertík]

pip wasn't even able to uninstall packages, see e.g. the second link
in Google for "pip uninstall":
http://stackoverflow.com/questions/6625597/installing-uninstalling-my-module-with-pip

But I think they fixed it now:
https://pip.pypa.io/en/latest/reference/pip_uninstall.html

For my particular use case, I need to be able to install non-python
packages, multiple versions of the same package, use various platforms
(Linux, Mac, Windows, clusters), handle package dependencies, work
without root access, etc. That's my use case, there are other as well.

In the past, various packages managers satisfied some of my use cases,
but not others, e.g. Debian works great, except that it requires a
root access and only works on Linux in fact, it is Linux. Also it is
heavy weight, if you want to quickly add and change packages, i.e.
virtual environments. Virtualenv has virtual environments but it is
tied to Python. pip can install (and now uninstall) packages, but only
Python packages. And so on. Only Conda and Hashdist fix all the
problems. Neither of them are perfect, but are improving. There are
also more package managers popping up all the time lately, but these
two seem to have the largest community.

>
> I'm generally much in favor of unbundling, I'm just extremely cautious about
> avoiding reliability or bugfixing problems. What I wouldn't want is if SymPy
> got cluttered with code that used different workarounds depending on what
> version of a dependency is installed - such things are reliability and
> productivity killers (been there, done that, didn't like it).

It's definitely a concern, and as you know, I used to be firmly
opposed to it. But I think with more projects now that handle source
package management on multiple platforms, things can be managed for
almost all use cases (and things are improving). And we just need to
use good judgement to make sure things work reasonably well with SymPy
and dependencies. If you'll see something that is broken, definitely
bring it up.

Ondrej

[Joachim Durchholz]

>>> but fortunately there are now pretty good package source
>>> managers like Conda or Hashdist,
>>
>> How are they better than pip?
>> I'm not opposed to any of these actually, I just don't know what the
>> differences are.
>
> pip wasn't even able to uninstall packages, see e.g. the second link
> in Google for "pip uninstall":
> http://stackoverflow.com/questions/6625597/installing-uninstalling-my-module-with-pip
>
> But I think they fixed it now:
> https://pip.pypa.io/en/latest/reference/pip_uninstall.html

Ah, I just read the latest docs and wasn't aware that uninstalling is a
new feature.
OTOH I think that pip is a solution for user-local installs, which means
if it doesn't work anymore you throw away the directory.

> For my particular use case, I need to be able to install non-python
> packages, multiple versions of the same package, use various platforms
> (Linux, Mac, Windows, clusters), handle package dependencies, work
> without root access, etc. That's my use case, there are other as well.

That's system-wide installs.
Is there a use case where SymPy needs to be a system-wide install?
You also mentioned that quickly changing installs is important to you -
does that really apply to system-wide installs? (Just curious, I'm not
seeing that kind of use case myself, but then I don't see all use cases.)

> Only Conda and Hashdist fix all the
> problems. Neither of them are perfect, but are improving. There are
> also more package managers popping up all the time lately, but these
> two seem to have the largest community.

Hm. New tools. I'm not sure that we're going to be happy with that.
(Of course, the same concern applies to newly added pip features such as
uninstallation.)

> It's definitely a concern, and as you know, I used to be firmly
> opposed to it. But I think with more projects now that handle source
> package management on multiple platforms, things can be managed for
> almost all use cases (and things are improving). And we just need to
> use good judgement to make sure things work reasonably well with SymPy
> and dependencies. If you'll see something that is broken, definitely
> bring it up.

Will do :-)

[Matthew Brett]

On Sun, Jan 4, 2015 at 12:07 PM, Joachim Durchholz <j...@durchholz.org> wrote:
>>>> but fortunately there are now pretty good package source
>>>> managers like Conda or Hashdist,
>>>
>>>
>>> How are they better than pip?
>>> I'm not opposed to any of these actually, I just don't know what the
>>> differences are.
>>
>>
>> pip wasn't even able to uninstall packages, see e.g. the second link
>> in Google for "pip uninstall":
>>
>> http://stackoverflow.com/questions/6625597/installing-uninstalling-my-module-with-pip
>>
>> But I think they fixed it now:
>> https://pip.pypa.io/en/latest/reference/pip_uninstall.html
>
>
> Ah, I just read the latest docs and wasn't aware that uninstalling is a new
> feature.
> OTOH I think that pip is a solution for user-local installs, which means if
> it doesn't work anymore you throw away the directory.

As far as I can see pip has had uninstall since version 0.6, about 5
years ago. The stackoverflow post is someone trying to uninstall
using the wrong command line.

>> For my particular use case, I need to be able to install non-python
>> packages, multiple versions of the same package, use various platforms
>> (Linux, Mac, Windows, clusters), handle package dependencies, work
>> without root access, etc. That's my use case, there are other as well.
>
>
> That's system-wide installs.
> Is there a use case where SymPy needs to be a system-wide install?
> You also mentioned that quickly changing installs is important to you - does
> that really apply to system-wide installs? (Just curious, I'm not seeing
> that kind of use case myself, but then I don't see all use cases.)

I use pip all the time for virtualenvs, and, at least on OSX, it's
very effective. We OSX'ers bundle the needed libraries in the wheels,
and so far that has been a practical solution to installing difficult
packages like scikit-image [1] and matplotlib [2]

>> Only Conda and Hashdist fix all the
>> problems. Neither of them are perfect, but are improving. There are
>> also more package managers popping up all the time lately, but these
>> two seem to have the largest community.
>
>
> Hm. New tools. I'm not sure that we're going to be happy with that.
> (Of course, the same concern applies to newly added pip features such as
> uninstallation.)

I suspect the new tool that gets really widespread adoption will first
need to persuade the Python Packaging Authority [3].

Cheers,

Matthew

[1] https://pypi.python.org/pypi/matplotlib
[2] https://pypi.python.org/pypi/scikit-image
[3] https://www.pypa.io/en/latest/

[Joachim Durchholz]

Am 04.01.2015 um 14:51 schrieb Matthew Brett:
> As far as I can see pip has had uninstall since version 0.6, about 5
> years ago. The stackoverflow post is someone trying to uninstall
> using the wrong command line.

LOL. That's what you get from just skimming a problem report.

So, pip is in the clear about uninstalling.
Are there other problems with it that would affect us?

> I suspect the new tool that gets really widespread adoption will first
> need to persuade the Python Packaging Authority [3].

I'd be very happy if that's truly the case, because their policy sounds
very much like what we need (very careful not to break backwards
compatibility etc.)
Can we verify that the PPA is really authoritative? At least in those
ways that count, i.e. if what they decide is quickly and widely adopted,
that's good enough for me, authoritative or not.

[Matthew Brett]

On Sun, Jan 4, 2015 at 5:43 PM, Joachim Durchholz <j...@durchholz.org> wrote:
> Am 04.01.2015 um 14:51 schrieb Matthew Brett:
>>
>> As far as I can see pip has had uninstall since version 0.6, about 5
>> years ago. The stackoverflow post is someone trying to uninstall
>> using the wrong command line.
>
>
> LOL. That's what you get from just skimming a problem report.
>
> So, pip is in the clear about uninstalling.
> Are there other problems with it that would affect us?

For sympy / mpmath, no, I don't think so. I'm happy to be corrected
if someone can think of something.

>> I suspect the new tool that gets really widespread adoption will first
>> need to persuade the Python Packaging Authority [3].
>
>
> I'd be very happy if that's truly the case, because their policy sounds very
> much like what we need (very careful not to break backwards compatibility
> etc.)
> Can we verify that the PPA is really authoritative? At least in those ways
> that count, i.e. if what they decide is quickly and widely adopted, that's
> good enough for me, authoritative or not.

I think PPA is authoritative on general Python packaging. I think that
conda and so on are trying to establish themselves as de-facto
standards in scientific Python.

The route that most projects I know have taken is to build things that
work with pip (like wheels for matplotlib etc) and let Continuum and
the conda team develop conda packages.

Cheers,

Matthew

[Aaron Meurer]

On Sun, Jan 4, 2015 at 11:01 AM, Matthew Brett <matthe...@gmail.com> wrote:

On Sun, Jan 4, 2015 at 5:43 PM, Joachim Durchholz <j...@durchholz.org> wrote:
> Am 04.01.2015 um 14:51 schrieb Matthew Brett:
>>
>> As far as I can see pip has had uninstall since version 0.6, about 5
>> years ago.  The stackoverflow post is someone trying to uninstall
>> using the wrong command line.
>
>
> LOL. That's what you get from just skimming a problem report.
>
> So, pip is in the clear about uninstalling.
> Are there other problems with it that would affect us?

For sympy / mpmath, no, I don't think so.  I'm happy to be corrected
if someone can think of something.

I think you're right. The proglems I know of with pip are

- packages that have bad metadata in setup.py (something we can easily avoid)

- combinations of packages that have some version restrictions. I doubt people will want anything but the latest mpmath. The only issue might be if mpmath breaks compatibility in some subtle way that breaks an old version of SymPy, and someone needs to use an old version of SymPy. In that case, they will need to know what version of mpmath to install in addition to SymPy. Conda does help here because in addition to having a true dependency solver, it lets you add dependency restrictions to old versions of packages retroactively. But honestly, with mpmath, it's a non-issue, and I doubt it will come up in any serious way.

- packages that require compilation. This is pip's worst sore, because it compiles packages from source, unless there are wheels, and compiling from source on a user's machine is destined to fail. But SymPy is and will remain pure Python. This is more an issue with CSymPy, and again, you can work around it by building wheels. 

(by the way, in case you didn't know, I work on conda for my day job, so I'm very biased in that direction)

Aaron Meurer

>> I suspect the new tool that gets really widespread adoption will first
>> need to persuade the Python Packaging Authority [3].
>
>
> I'd be very happy if that's truly the case, because their policy sounds very
> much like what we need (very careful not to break backwards compatibility
> etc.)
> Can we verify that the PPA is really authoritative? At least in those ways
> that count, i.e. if what they decide is quickly and widely adopted, that's
> good enough for me, authoritative or not.

I think PPA is authoritative on general Python packaging. I think that
conda and so on are trying to establish themselves as de-facto
standards in scientific Python.

The route that most projects I know have taken is to build things that
work with pip (like wheels for matplotlib etc) and let Continuum and
the conda team develop conda packages.

Cheers,

Matthew

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAH6Pt5p8b46L0ehP%2B_SGjnGem%3DAPR5qN11EeNLqygunWuP9EYA%40mail.gmail.com.

[Ondřej Čertík]

On Sun, Jan 4, 2015 at 1:07 PM, Joachim Durchholz <j...@durchholz.org> wrote:
>>>> but fortunately there are now pretty good package source
>>>> managers like Conda or Hashdist,
>>>
>>>
>>> How are they better than pip?
>>> I'm not opposed to any of these actually, I just don't know what the
>>> differences are.
>>
>>
>> pip wasn't even able to uninstall packages, see e.g. the second link
>> in Google for "pip uninstall":
>>
>> http://stackoverflow.com/questions/6625597/installing-uninstalling-my-module-with-pip
>>
>> But I think they fixed it now:
>> https://pip.pypa.io/en/latest/reference/pip_uninstall.html
>
>
> Ah, I just read the latest docs and wasn't aware that uninstalling is a new
> feature.
> OTOH I think that pip is a solution for user-local installs, which means if
> it doesn't work anymore you throw away the directory.
>
>> For my particular use case, I need to be able to install non-python
>> packages, multiple versions of the same package, use various platforms
>> (Linux, Mac, Windows, clusters), handle package dependencies, work
>> without root access, etc. That's my use case, there are other as well.
>
>
> That's system-wide installs.

No, my use-case is users installing into their home directories or
some other place, as I wrote *without* root access. Not system-wide.

Ondrej

> Is there a use case where SymPy needs to be a system-wide install?
> You also mentioned that quickly changing installs is important to you - does
> that really apply to system-wide installs? (Just curious, I'm not seeing
> that kind of use case myself, but then I don't see all use cases.)
>
>> Only Conda and Hashdist fix all the
>> problems. Neither of them are perfect, but are improving. There are
>> also more package managers popping up all the time lately, but these
>> two seem to have the largest community.
>
>
> Hm. New tools. I'm not sure that we're going to be happy with that.
> (Of course, the same concern applies to newly added pip features such as
> uninstallation.)
>
>> It's definitely a concern, and as you know, I used to be firmly
>> opposed to it. But I think with more projects now that handle source
>> package management on multiple platforms, things can be managed for
>> almost all use cases (and things are improving). And we just need to
>> use good judgement to make sure things work reasonably well with SymPy
>> and dependencies. If you'll see something that is broken, definitely
>> bring it up.
>
>
> Will do :-)
>

> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sympy.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/sympy/54A92CF3.3080101%40durchholz.org.

[Ondřej Čertík]

Right. For CSymPy we actually allow "pip install" and test it
regularly on Travis
(https://github.com/sympy/csympy/blob/9e6384713b2254413d15980c366521ac39624e1d/bin/test_travis.sh#L54),
but the way it works is that there is a very thin setup.py
(https://github.com/sympy/csympy/blob/9e6384713b2254413d15980c366521ac39624e1d/setup.py)
that just calls cmake under the hood. I personally don't use pip and
just use cmake for CSymPy. But that's a separate question, unrelated
to SymPy and Mpmath.

>
> (by the way, in case you didn't know, I work on conda for my day job, so I'm
> very biased in that direction)

(I am contributing to Hashdist in my free time and also would like to
finish the Conda's CSymPy package eventually.)

Ondrej

> https://groups.google.com/d/msgid/sympy/CAKgW%3D6JERR9%2BFA_-O1Zmw%2Bh8N4bc95ek4Lh6hFzPdLbdbasgiw%40mail.gmail.com.

[Joachim Durchholz]

Am 04.01.2015 um 20:38 schrieb Aaron Meurer:
> I think you're right. The proglems I know of with pip are
>
> - packages that have bad metadata in setup.py (something we can easily
> avoid)

Or at least fix because it's a bug :-)

> - combinations of packages that have some version restrictions.

That's unavoidable.

In the Java world, people work around this by renaming modules.
I.e. if module A requires library com.fubar.biglib and module B requires
the same but at a different version, integrators include two copies of
com.fubar.biglib, but one version renamed (e.g. com.fubar.A.biglib).
The catch is that all code in A now needs to refer to com.fubar.A.biglib.

Similar things could be done in Python.
The failure mode here is when class names are constructed from strings,
because the renaming pass does not find all occurrences of
com.fubar.biglib in A. But actually that's just the same problem as with
Java, and probably no more often than in Java.

> I doubt
> people will want anything but the latest mpmath. The only issue might be if
> mpmath breaks compatibility in some subtle way that breaks an old version
> of SymPy, and someone needs to use an old version of SymPy. In that case,
> they will need to know what version of mpmath to install in addition to
> SymPy. Conda does help here because in addition to having a true dependency
> solver, it lets you add dependency restrictions to old versions of packages
> retroactively. But honestly, with mpmath, it's a non-issue, and I doubt it
> will come up in any serious way.

Well, even seemingly nonserious problems can cause trouble.

OTOH if somebody needs to keep some old combo of software alive, they're
going to freeze it anyway.

I'm more concerned about people who use SymPy together with some other
software. They might end using a version of mpmath that's incompatible
with either current SymPy or that other software.

> (by the way, in case you didn't know, I work on conda for my day job, so
> I'm very biased in that direction)

Well, from a packaging point of view, all packaging software uses the
same format for describing dependencies. I.e. if everybody uses
requirements.txt, it's easiest to set something up that will work for
all users, regardless of what packaging software they use.