mpmath missing, installation instructions fail

440 views
Skip to first unread message

Joachim Durchholz

unread,
Jan 6, 2015, 3:38:29 PM1/6/15
to sy...@googlegroups.com
tl;dr: Sergey's "unbundle mpmath" patches have created a substantial
entry barrier for GSoC students (and people with similar low Python
experience), and complicated the process of starting a debugging or test
session.
Also, I'm pretty p*ssed because I predicted exactly this, and got some
dismissive non-answers instead of seeing the issues addressed.


Full story:


I have to say that my fears have fully become reality: The process of
installing mpmath, while simple, isn't described in a way that "just
works", and still invites subtle error.

As a developer, I expect to be able to download the git repo and start
working. Possibly after reading a README file.

Okay, let's see how a new dev with little knowledge of Python and no
knowledge of SymPy would fare...

* README.rst is available. Fair enough.

* It says "git clone" - okay, I just did that.

* It sends me to http://docs.sympy.org/dev/install.html - erh, a detour.
Let's put that aside, I haven't fully read the readme and don't even
know yet that anything is missing.

* Okay, "Documentation and Usage", that sounds promising.
Says "Everything is at" and another URL - also instructions for
generating docs locally: cd docs; make html
Oh. I'm getting this:

Makefile:121: warning: overriding commands for target
`_build/cheatsheet/cheatsheet.pdf'
Makefile:116: warning: ignoring old commands for target
`_build/cheatsheet/cheatsheet.pdf'
rm -rf _build/logo
mkdir -p _build/logo
python ./generate_logos.py -d
INFO:root: File saved: ./_build/logo/sympy-notail.svg
INFO:root: File saved: ./_build/logo/sympy-notailtext.svg
INFO:root: File saved: ./_build/logo/sympy-notext.svg
ERROR:root:rsvg-convert: command not found
make: *** [logo] Error 127

OK, that's "just a detour" because I'm really after mpmath, not html,
and I was able to ask Ubuntu what package I needed to get installed.
However, this would add to the problems a new dev would have to
overcome; maybe it's a good idea to list rsvg-convert among the required
installs in the documentation building docs.

The same goes for convert (from imagemagick).

I'm giving up at sphinx-build since it does not have a Debian package.
I suppose I can easily install it via a Python installer, a newbie might
not know that.

So... no local docs, skipping that section. (Also, getting a bit
frustrated.)
(Taking off my newbie hat, I also didn't want to pursue that detour
further. Maybe somebody else has more interest in getting the docs
building instructions back on track.)

* Now the "Usage" part of "Documentation and Usage"
It says I should just start python and run a few import statements.
... aaand I'm getting this:

$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from sympy import Symbol, cos
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "sympy/__init__.py", line 37, in <module>
from .core import *
File "sympy/core/__init__.py", line 8, in <module>
from .expr import Expr, AtomicExpr
File "sympy/core/expr.py", line 7, in <module>
from .evalf import EvalfMixin, pure_complex
File "sympy/core/evalf.py", line 9, in <module>
import mpmath.libmp as libmp
ImportError: No module named mpmath.libmp

So okay, I bet the instructions are elsewhere, but somebody who just
walks down the README.rst is hosed.

As expectation had it, bin/isympy would fail with exactly the same error.

* Oh, *now* there's an "Installation" section!

* python setup.py install
Which gives me this:

running install
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

[Errno 13] Permission denied:
'/usr/local/lib/python2.7/dist-packages/test-easy-install-15567.write-test'

There's a long explanation and a link... but it does not explain what to
do with a "permission denied" situation. Luckily, I know that virtualenv
helps (but it's still a pain because virtualenv is a pain for me,
somehow I'm not finding a good way to set up shortcuts so I'm still
forced to type a whole lot of incantations in each new shell window just
to get into the virtualenv).

Still. I'm not after almost 100 lines of writeup, with several problems
along the way.
Sergey said "it's easy" and "you should do your reading". Well, I did
and it still isn't, it took me more than an hour to try all these things.
Compared to the way before the unbundling, it's a big step backwards. I
could simply download git repo, start bin/isympy or bin/test, and it all
worked out of the box; now I have an odysseus tour through various
procedures and problems (the writeup now nearing the 100-line mark), I
have to know virtualenv, I have to organize my shortcuts, before I can
start hacking... Sergey, this wasn't the finest thing to do,
particularly for a change that never had a clear motive (except making
life easier for packagers - heck no, sympy was standalone and worked
well with that, and if somebody really ran into a library conflict,
then, well, the conflict wouldn't go away by unbundling anyway, so what
was the point???)

Yeah, sorry for the rant.
I'm not saying that unbundling libraries was misguided.
But I'm really questioning the wisdom of unbundling the most critical
library first. We should have done this with some other dependency.
pep8.py comes to mind - it's not needed for running SymPy, and the tests
that it is useful for are entirely optional, so that would have been a
good testbed for trying out packaging and stuff.

Ah well. Dropped eggs and all that.

Fair warning: Given the way Sergey tried to talk me out of predicting
exactly this kind of problem, and being somewhat down the nose about it
as well, I'm having a very, very short fuse about this all.

Regards,
Jo

Aaron Meurer

unread,
Jan 6, 2015, 4:01:19 PM1/6/15
to sy...@googlegroups.com
On Tue, Jan 6, 2015 at 1:38 PM, Joachim Durchholz <j...@durchholz.org> wrote:
tl;dr: Sergey's "unbundle mpmath" patches have created a substantial entry barrier for GSoC students (and people with similar low Python experience), and complicated the process of starting a debugging or test session.
Also, I'm pretty p*ssed because I predicted exactly this, and got some dismissive non-answers instead of seeing the issues addressed.


Full story:


I have to say that my fears have fully become reality: The process of installing mpmath, while simple, isn't described in a way that "just works", and still invites subtle error.

As a developer, I expect to be able to download the git repo and start working. Possibly after reading a README file.

Okay, let's see how a new dev with little knowledge of Python and no knowledge of SymPy would fare...

* README.rst is available. Fair enough.

* It says "git clone" - okay, I just did that.

* It sends me to http://docs.sympy.org/dev/install.html - erh, a detour. Let's put that aside, I haven't fully read the readme and don't even know yet that anything is missing.

Clearly this document needs to be updated.
 

* Okay, "Documentation and Usage", that sounds promising.
Says "Everything is at" and another URL - also instructions for generating docs locally: cd docs; make html
Oh. I'm getting this:

Makefile:121: warning: overriding commands for target `_build/cheatsheet/cheatsheet.pdf'
Makefile:116: warning: ignoring old commands for target `_build/cheatsheet/cheatsheet.pdf'
rm -rf _build/logo
mkdir -p _build/logo
python ./generate_logos.py -d
INFO:root: File saved: ./_build/logo/sympy-notail.svg
INFO:root: File saved: ./_build/logo/sympy-notailtext.svg
INFO:root: File saved: ./_build/logo/sympy-notext.svg
ERROR:root:rsvg-convert: command not found
make: *** [logo] Error 127

OK, that's "just a detour" because I'm really after mpmath, not html, and I was able to ask Ubuntu what package I needed to get installed. However, this would add to the problems a new dev would have to overcome; maybe it's a good idea to list rsvg-convert among the required installs in the documentation building docs. 

The same goes for convert (from imagemagick).

I'm giving up at sphinx-build since it does not have a Debian package.
I suppose I can easily install it via a Python installer, a newbie might not know that.

The docs require a handful of things to build actually. We should document this. sphinx-build is just part of sphinx.
 

So... no local docs, skipping that section. (Also, getting a bit frustrated.)
(Taking off my newbie hat, I also didn't want to pursue that detour further. Maybe somebody else has more interest in getting the docs building instructions back on track.)

* Now the "Usage" part of "Documentation and Usage"
It says I should just start python and run a few import statements.
... aaand I'm getting this:

$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from sympy import Symbol, cos
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "sympy/__init__.py", line 37, in <module>
    from .core import *
  File "sympy/core/__init__.py", line 8, in <module>
    from .expr import Expr, AtomicExpr
  File "sympy/core/expr.py", line 7, in <module>
    from .evalf import EvalfMixin, pure_complex
  File "sympy/core/evalf.py", line 9, in <module>
    import mpmath.libmp as libmp
ImportError: No module named mpmath.libmp

So okay, I bet the instructions are elsewhere, but somebody who just walks down the README.rst is hosed.

As expectation had it, bin/isympy would fail with exactly the same error.

Maybe we should import mpmath at the top of sympy/__init__.py and raise an ImportError with a better message.
 

* Oh, *now* there's an "Installation" section!

* python setup.py install
Which gives me this:

running install
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/test-easy-install-15567.write-test'

There's a long explanation and a link... but it does not explain what to do with a "permission denied" situation. Luckily, I know that virtualenv helps (but it's still a pain because virtualenv is a pain for me, somehow I'm not finding a good way to set up shortcuts so I'm still forced to type a whole lot of incantations in each new shell window just to get into the virtualenv).

Still. I'm not after almost 100 lines of writeup, with several problems along the way.
Sergey said "it's easy" and "you should do your reading". Well, I did and it still isn't, it took me more than an hour to try all these things.
Compared to the way before the unbundling, it's a big step backwards. I could simply download git repo, start bin/isympy or bin/test, and it all worked out of the box; now I have an odysseus tour through various procedures and problems (the writeup now nearing the 100-line mark), I have to know virtualenv, I have to organize my shortcuts, before I can start hacking... Sergey, this wasn't the finest thing to do, particularly for a change that never had a clear motive (except making life easier for packagers - heck no, sympy was standalone and worked well with that, and if somebody really ran into a library conflict, then, well, the conflict wouldn't go away by unbundling anyway, so what was the point???)

Yeah, sorry for the rant.
I'm not saying that unbundling libraries was misguided.
But I'm really questioning the wisdom of unbundling the most critical library first. We should have done this with some other dependency. pep8.py comes to mind - it's not needed for running SymPy, and the tests that it is useful for are entirely optional, so that would have been a good testbed for trying out packaging and stuff.

Ah well. Dropped eggs and all that.

Fair warning: Given the way Sergey tried to talk me out of predicting exactly this kind of problem, and being somewhat down the nose about it as well, I'm having a very, very short fuse about this all.

Regards,
Jo

Most of these issues are general to the way Python handles dependencies and packaging and not specific to SymPy or mpmath.  The ImportError is generally the best error message you get when a package is missing. The permissions issues are due to the way Python gets installed by the system in a place that isn't user-writable. You either have to use sudo or know how to install somewhere that is user-writable.

Even so, I can guarantee that the following will work (once we do a release):

- Installing SymPy via Anaconda or conda will get the mpmath dependency.
- pip installing SymPy will get mpmath
- If you have setuptools installed, python setup.py install will download and install mpmath, if it isn't installed already. 

The toughest situation will be for people who are developing on SymPy (i.e., they just clone the repo, without installing). They will need to know to install mpmath, which requires better documentation and error messages.

Aaron Meurer



--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscribe@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.
To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/54AC47C0.7000405%40durchholz.org.
For more options, visit https://groups.google.com/d/optout.

Joachim Durchholz

unread,
Jan 6, 2015, 6:07:18 PM1/6/15
to sy...@googlegroups.com
Am 06.01.2015 um 22:00 schrieb Aaron Meurer:
> Most of these issues are general to the way Python handles dependencies and
> packaging and not specific to SymPy or mpmath.

Sure.
I was more venting off that the docs weren't updated, and I have yet to
determine how to properly use virtualenv - right now it doesn't meld
well into the workflow.

> The ImportError is
> generally the best error message you get when a package is missing. The
> permissions issues are due to the way Python gets installed by the system
> in a place that isn't user-writable. You either have to use sudo or know
> how to install somewhere that is user-writable.

Exactly. I guess a hint at virtualenv would be appropriate.
The mpmath installer sadly does not say so.

> Even so, I can guarantee that the following will work (once we do a
> release):
>
> - Installing SymPy via Anaconda or conda will get the mpmath dependency.
> - pip installing SymPy will get mpmath
> - If you have setuptools installed, python setup.py install will download
> and install mpmath, if it isn't installed already.
>
> The toughest situation will be for people who are developing on SymPy
> (i.e., they just clone the repo, without installing). They will need to
> know to install mpmath, which requires better documentation and error
> messages.

100% ack.

Aaron Meurer

unread,
Jan 6, 2015, 7:25:19 PM1/6/15
to sy...@googlegroups.com
On Tue, Jan 6, 2015 at 4:07 PM, Joachim Durchholz <j...@durchholz.org> wrote:
Am 06.01.2015 um 22:00 schrieb Aaron Meurer:
Most of these issues are general to the way Python handles dependencies and
packaging and not specific to SymPy or mpmath.

Sure.
I was more venting off that the docs weren't updated, and I have yet to determine how to properly use virtualenv - right now it doesn't meld well into the workflow.

> The ImportError is
generally the best error message you get when a package is missing. The
permissions issues are due to the way Python gets installed by the system
in a place that isn't user-writable. You either have to use sudo or know
how to install somewhere that is user-writable.

Exactly. I guess a hint at virtualenv would be appropriate.
The mpmath installer sadly does not say so.

conda environments are also useful. It's not really mpmath's fault. For Python installations you don't need to worry about this (e.g., if you install Anaconda, it is user-writable by default).



Even so, I can guarantee that the following will work (once we do a
release):

- Installing SymPy via Anaconda or conda will get the mpmath dependency.
- pip installing SymPy will get mpmath
- If you have setuptools installed, python setup.py install will download
and install mpmath, if it isn't installed already.

The toughest situation will be for people who are developing on SymPy
(i.e., they just clone the repo, without installing). They will need to
know to install mpmath, which requires better documentation and error
messages.

100% ack.

It's still way better than with other projects. Try contributing to matplotlib, which has three dependencies (more depending on how you count), and must be compiled and installed (you can't just import matplotlib from the development directory).

Aaron Meurer



--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscribe@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

Joachim Durchholz

unread,
Jan 7, 2015, 3:20:42 AM1/7/15
to sy...@googlegroups.com
Am 07.01.2015 um 01:24 schrieb Aaron Meurer:
> It's still way better than with other projects. Try contributing to
> matplotlib, which has three dependencies (more depending on how you count),
> and must be compiled and installed (you can't just import matplotlib from
> the development directory).

One of the reasons I've been around with SymPy, and not matplotlib :-)

Sergey Kirpichev

unread,
Jan 7, 2015, 8:38:14 AM1/7/15
to sy...@googlegroups.com
Hello,


On Tuesday, January 6, 2015 11:38:29 PM UTC+3, Joachim Durchholz wrote:
I have to say that my fears have fully become reality: The process of
installing mpmath, while simple, isn't described in a way that "just
works", and still invites subtle error.

Lets hope, I can understand this from the following verbose text...
 
As a developer, I expect to be able to download the git repo and start
working. Possibly after reading a README file.

Can you just literally follow instructions in that file?  What's wrong?
I don't see your command history...
 
* It sends me to http://docs.sympy.org/dev/install.html - erh, a detour.

No.  README has all required instructions.
 
I haven't fully read the readme

Then please do read Installation section before posting such things.
 
OK, that's "just a detour" because I'm really after mpmath, not html

We have README in the docs directory.  Perhaps, this stuff should
be elsewhere.

How this related to mpmath at all?
 
However, this would add to the problems a new dev would have to
overcome;

New developers are redirected to
https://github.com/sympy/sympy/wiki/Development-workflow

Everything doc-related is documented here, section "Install other software".
 
maybe it's a good idea to list rsvg-convert among the required
installs in the documentation building docs.

It listed.
 
The same goes for convert (from imagemagick).

Same.

Indeed, we can improve docs documentation, but that's different issue.
 
It says I should just start python and run a few import statements.
... aaand I'm getting this:

$ python   [...]
ImportError: No module named mpmath.libmp

Did you install mpmath as described in the README.rst?

If no, what do you expect?
 
The following error occurred while trying to add or remove files in the
installation directory:

     [Errno 13] Permission denied:
'/usr/local/lib/python2.7/dist-packages/test-easy-install-15567.write-test'

There's a long explanation and a link... but it does not explain what to
do with a "permission denied" situation.

As you don't explain what you do before - it's hard to guess what's was wrong.
Perhaps, first time you did sudo python install ..

Still. I'm not after almost 100 lines of writeup, with several problems
along the way.

IS THIS ALL?

The one problem I see - it's just the specific user, who can't even read
the README.rst.  Let's wait for real bugreport before listing something as a problem.

Exactly one import error - all mpmath's problem you have.  The reason
for one is simple: please do install mpmath first.  That's the first sentence of the
installation instructions in the README.
 
Sergey said "it's easy" and "you should do your reading".

Clearly, you don't.  That's a problem.

for a change that never had a clear motive (except making
life easier for packagers

Please read relevant discussions first.
 
We should have done this with some other dependency.
pep8.py comes to mind

We don't have hard dependences on anything else, including
pep8.  Right now, we don't use pep8 at all.

Joachim Durchholz

unread,
Jan 7, 2015, 10:13:05 AM1/7/15
to sy...@googlegroups.com
Am 07.01.2015 um 14:38 schrieb Sergey Kirpichev:
> Hello,
>
> On Tuesday, January 6, 2015 11:38:29 PM UTC+3, Joachim Durchholz wrote:
>>
>> I have to say that my fears have fully become reality: The process of
>> installing mpmath, while simple, isn't described in a way that "just
>> works", and still invites subtle error.
>
> Lets hope, I can understand this from the following verbose text...

Dismissive right from the start?

>> As a developer, I expect to be able to download the git repo and start
>> working. Possibly after reading a README file.
>
> Can you just literally follow instructions in that file?

No. The file does not even have a linear path.

> What's wrong?

Read the rest...

> I don't see your command history...

I mentioned all commands. Actually I'm not

>> * It sends me to http://docs.sympy.org/dev/install.html - erh, a detour.
>
> No. README has all required instructions.

The text does not say this.

>> I haven't fully read the readme
>
> Then please do read Installation section before posting such things.

You're quoting out of context.
I'm describing the situation at that point when reading the README.
So please do at least try to understand before posting such things...

>> OK, that's "just a detour" because I'm really after mpmath, not html
>
> We have README in the docs directory. Perhaps, this stuff should
> be elsewhere.
>
> How this related to mpmath at all?

Not really. I hit the mpmath problem and started to read the README, at
which point I stumbled upon the doc building instructions.

>> However, this would add to the problems a new dev would have to
>> overcome;
>
> New developers are redirected to
> https://github.com/sympy/sympy/wiki/Development-workflow

Not mentioned in the github readme.
Besides, how many pages do you expect people to read? Or do you really
expect everybody to follow a single read flow?
In my opinion, the README at the root of the git repo should be
self-sufficient in the sense that for everything one might want to do
it, it should either be described there, or have a link to a full
description (which should live inside the repo as well).

> Everything doc-related is documented here, section "Install other software".

Then that reference is missing.

>> maybe it's a good idea to list rsvg-convert among the required
>> installs in the documentation building docs.
>
> It listed.

Not in the README.
I do not think the root README needs to contain all the information.
However, I think it should not pretend to describe a process that
doesn't work.

>> It says I should just start python and run a few import statements.
>> ... aaand I'm getting this:
>>
>> $ python [...]
>>
>> ImportError: No module named mpmath.libmp
>
> Did you install mpmath as described in the README.rst?

That is described *after* the instruction to try try SymPy.

> If no, what do you expect?

Do you expect people to read and memorize the entire text before trying
out things?

>> The following error occurred while trying to add or remove files in the
>> installation directory:
>>
>> [Errno 13] Permission denied:
>> '/usr/local/lib/python2.7/dist-packages/test-easy-install-15567.write-test'
>>
>>
>> There's a long explanation and a link... but it does not explain what to
>> do with a "permission denied" situation.
>
> As you don't explain what you do before - it's hard to guess what's was
> wrong.
> Perhaps, first time you did sudo python install ..

Hey, Python comes preinstalled.
Of course it's the system Python it's trying to install into.

> Still. I'm not after almost 100 lines of writeup, with several problems
>> along the way.
>
> IS THIS ALL?
>
> The one problem I see - it's just the specific user, who can't even read
> the README.rst.

Yeah.
*plonk*

Sergey B Kirpichev

unread,
Jan 7, 2015, 10:57:49 AM1/7/15
to sy...@googlegroups.com
On Wed, Jan 07, 2015 at 04:12:59PM +0100, Joachim Durchholz wrote:
> >No. README has all required instructions.
>
> The text does not say this.

It shouldn't. It just has it.

> Not mentioned in the github readme.

README.rst is in the tree (not github's readme) - for users in
first place, not for developers.

> Besides, how many pages do you expect people to read?

For users, just README.

For developers, also mentioned link. Maybe we should add
that text in the sympy tree too (something like HACKING.rst, like in scipy).

> >Did you install mpmath as described in the README.rst?
>
> That is described *after* the instruction to try try SymPy.

Why this should matter at all? We have all installation instructions
for users in the README in the specific section. How can someone
miss that?

> >If no, what do you expect?
>
> Do you expect people to read and memorize the entire text before
> trying out things?

I expect that people read installation instructions
before blaming that we have some installation problems.

Probably, it only one assumption we can do about our users.
For sure, this can be violated, but it's waste of time
to workarround this.

> >As you don't explain what you do before - it's hard to guess what's was
> >wrong.
> >Perhaps, first time you did sudo python install ..
>
> Hey, Python comes preinstalled.

The problem here is certainly not with Python.

Ondřej Čertík

unread,
Jan 7, 2015, 12:57:55 PM1/7/15
to sympy
We should put step by step instructions for developers. I would be
inclined to just put instructions using Conda, i.e. use Conda to
install mpmath, then one can hack on sympy immediately. That way
people can get started, and if they need a different setup, they can
at least start with Conda and then go from there.

Ondrej

Matthew Brett

unread,
Jan 7, 2015, 1:10:00 PM1/7/15
to sympy
I am happy to contribute the pip / virtualenv instructions too.

Cheers,

Matthew

Joachim Durchholz

unread,
Jan 7, 2015, 1:12:26 PM1/7/15
to sy...@googlegroups.com
Am 07.01.2015 um 18:57 schrieb Ondřej Čertík:
> We should put step by step instructions for developers.

I'm very +1 about that.

Ideally, it would be a single-line command which would set up a
development directory with:
- a Python 2.6 virtualenv (no newer Python so people don't accidentally
code against newer features; upgrade Python version as we drop support;
maybe a script or an instruction how to add virtualenvs for more Python
versions)
- all dependencies installed into the virtualenvs, at the versions SymPy
needs/wants them
- a clone of the SymPy repo, with source tracking properly setup,
possibly even with a commit hook that prevents mis-commits on the mater
branch (I can't count how often that happened to me); ideally, also with
the connection to the upload repo properly set up, too, though that
might require too much user interaction and be too fragile to be workable.

That's just my personal wishlist, feel free to add or remove your own
items :-)

> I would be
> inclined to just put instructions using Conda, i.e. use Conda to
> install mpmath, then one can hack on sympy immediately. That way
> people can get started, and if they need a different setup, they can
> at least start with Conda and then go from there.

I'm not sure what the implications of using Conda are.
+1 if it can easily support my wishlist items ;-) ... well, not just MY
wishlist, of course.
+0 otherwise.

Regards,
Jo
Reply all
Reply to author
Forward
0 new messages