Allow for --user pip installs for sage?

[Nils Bruin]

Presently,

$ sage -pip install --user .....

fails (at least for me, with a vanilla sage install on FC33) with

ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

Perhaps it's worth having this?

Also, I noticed:

$ sage --python3 -c "import sys; print(sys.path)" ['', '/usr/lib64/python39.zip', '/usr/lib64/python3.9', '/usr/lib64/python3.9/lib-dynload', '/usr/local/sage/sage-git/local/lib64/python3.9/site-packages', '/usr/local/sage/sage-git/local/lib/python3.9/site-packages']

note that the first enstries in this list actually refer to the system python, not the sage python! I think that means that any patched package installs that sage uses will be shadowed by the system installations. That's probably not what is meant.

[Matthias Koeppe]

On Monday, September 27, 2021 at 12:46:30 PM UTC-7 Nils Bruin wrote:

I noticed:

$ sage --python3 -c "import sys; print(sys.path)" ['', '/usr/lib64/python39.zip', '/usr/lib64/python3.9', '/usr/lib64/python3.9/lib-dynload', '/usr/local/sage/sage-git/local/lib64/python3.9/site-packages', '/usr/local/sage/sage-git/local/lib/python3.9/site-packages']

note that the first enstries in this list actually refer to the system python, not the sage python! I think that means that any patched package installs that sage uses will be shadowed by the system installations.

No, that's not how it works. Sage uses a venv over the system python. The first items in the list provide access to python's standard library. System-installed packages would be found in a directory like /usr/lib64/python3.9/site-packages. Our venv is configured so that it does not provide access to system packages, which is why you don't see this directory in the path.

 

[Matthias Koeppe]

On Monday, September 27, 2021 at 12:46:30 PM UTC-7 Nils Bruin wrote:

Presently,

$ sage -pip install --user .....

fails (at least for me, with a vanilla sage install on FC33) with

ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

Perhaps it's worth having this?

This is deliberately disabled (look for PYTHONUSERBASE in src/bin/sage-env) because users tend to have random trash installed in the user scheme. Because these packages would take precedence over Sage-provided packages, this would lead to endless bug reports on the mailing lists.

 

[William Stein]

On CoCalc I think we explicitly patch the Sage we provide to include ~/.local:

~$ sage --python3 -c "import sys; print(sys.path)"
['', '/ext/sage/sage-9.3/local/lib/python39.zip',
'/ext/sage/sage-9.3/local/lib/python3.9',
'/ext/sage/sage-9.3/local/lib/python3.9/lib-dynload',
'/home/user/.local/lib/python3.9/site-packages',
'/ext/sage/sage-9.3/local/lib/python3.9/site-packages',
'/ext/sage/sage-9.3/local/lib/python3.9/site-packages/awalipy-0.4-py3.9-linux-x86_64.egg']

Also note that all the other path entries are our sage install (not
the system python, which is crazy, as you point out).

For us, it's very very important that "pip install --user" just works.
Also, for us, usually people are working in a very clean well defined
environment (a cocalc project), which they are using for a very
specific "project" (e.g., teaching or taking a specific class, or
writing a paper), so using ~/.local is very reasonable. I see at
[1] that our page about installing packages "into sage is not
supported", and it should be updated to say that "sage --pip install"
does work. We also make pip default to --user so actually "pip
install" does just work.

I just found a "random" package on pypi:

~$ sage --pip install pytype
Defaulting to user installation because normal site-packages is not writeable
Collecting pytype
...
~$ sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.3, Release Date: 2021-05-09 │
│ Using Python 3.9.2. Type "help()" for help. │
└────────────────────────────────────────────────────────────────────┘
sage: import pytype
sage: # yeah

IMHO, this is pretty good user experience.

Sometimes people do break their sage installs via installing stuff
into ~/.local, but it's also easy to fix by just deleting ~/.local
and starting over, and it's usually very clear this is the problem
based on the traceback.

-- William

[William Stein]

This is a very good point.

However, as I mentioned in my other email, we explicitly enable it back in CoCalc by patching the 

Sage install.  This leads to probably one bug report a month, if that (so yes, it's endless, but very easy to

manage), and I'm not sure those are even Sage related, since the same problem can happen with any

Python install.  We make bug reports very, very easy for users to create (it's 2 clicks), and there is a lot [1] of usage 

of Sage on CoCalc.   The main thing that  makes explicitly enabling this pip functionality a good idea for cocalc

is that cocalc projects are clean Docker containers, and people use them for specific purposes.  Also support

is stupid easy due to it being a web application.   So the problem you mention with random trash goes away

*for us*.

I think the best solution for the official Sage distribution is to provide an *option* for systems admins installing

Sage to all or not allow user pip installs.  Probably the default should be "no", but there should be a documented

supported way of changing the behavior to "yes".  I don't like having to patch sage just because of this, but I do,

and *definitely* will continue to do so, because the reasoning for removing this functionality does not apply

to my use case. 

William

 [1] https://cocalc.com/info/status

 

[Thierry]

Hi,

for what it worth, i am +1 for letting the random user to pip-install
her own packages, either on ~/.local as William suggested, and/or in
~/.sage/local as it used to be the case (this location ensures that
Python packages unrelated to some Sage activity will not add noise).

Ciao,
Thierry

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CACLE5GD2n47_2QHpPNa82NqypuEqC6iV2Djg_0PTsy_%2BdqLwpA%40mail.gmail.com.

[Nils Bruin]

On Monday, 27 September 2021 at 13:06:58 UTC-7 Matthias Koeppe wrote:

On Monday, September 27, 2021 at 12:46:30 PM UTC-7 Nils Bruin wrote:

$ sage -pip install --user .....

This is deliberately disabled (look for PYTHONUSERBASE in src/bin/sage-env) because users tend to have random trash installed in the user scheme. Because these packages would take precedence over Sage-provided packages, this would lead to endless bug reports on the mailing lists.

If we'd mangle the version of the sage-installed python (make it python3.9sage or something like that), it would make it very unlikely we'd stumble into random stuff the user didn't explicitly install in sage's python. Interestingly enough, this would also automatically prevent system libraries from taking precedence over the sage-provided ones. I don't know how doable this mangling is?

Of course, all of this goes out the window if one is running sage using the system python (modulo virtualenv's, which I don't know anything about). But I think it should.

[Matthias Koeppe]

On Monday, September 27, 2021 at 1:24:47 PM UTC-7 wst...@gmail.com wrote:

On Monday, September 27, 2021 at 1:06:58 PM UTC-7 Matthias Koeppe wrote:

On Monday, September 27, 2021 at 12:46:30 PM UTC-7 Nils Bruin wrote:

Presently,

$ sage -pip install --user .....

fails (at least for me, with a vanilla sage install on FC33) with

ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

Perhaps it's worth having this?

This is deliberately disabled (look for PYTHONUSERBASE in src/bin/sage-env) because users tend to have random trash installed in the user scheme. Because these packages would take precedence over Sage-provided packages, this would lead to endless bug reports on the mailing lists.

However, as I mentioned in my other email, we explicitly enable it back in CoCalc by patching the 

Sage install. [...]

I think the best solution for the official Sage distribution is to provide an *option* for systems admins installing

Sage to all or not allow user pip installs.  Probably the default should be "no", but there should be a documented

supported way of changing the behavior to "yes". 

Much better than any of these options is to use user-defined venvs. 

This has been available since Sage 9.2:

https://wiki.sagemath.org/ReleaseTours/sage-9.2#Reusable_wheels_for_the_Python_packages_built_by_the_Sage_distribution

[Nils Bruin]

On Monday, 27 September 2021 at 14:45:44 UTC-7 Matthias Koeppe wrote:

Much better than any of these options is to use user-defined venvs. 

This has been available since Sage 9.2:

https://wiki.sagemath.org/ReleaseTours/sage-9.2#Reusable_wheels_for_the_Python_packages_built_by_the_Sage_distribution

Cool! Whatever "make build" does will be what's relevant for most people (and that includes me). I assume that's also how binary distributions will package it.

[Nils Bruin]

Reiterating:

I think it is worthwhile to have something like "sage --pip install --user ..." work, because people may well have a sage-install on which they don't have write permission, or on which they don't *want* to write. If it needs to be something else, like "sage --pipuser install ..." that's fine too, but I think it needs to be a one-liner one can just use on a sage install, without having to have done funny stuff on original installation: the target audience wouldn't have had the chance to do that and now it's too late. The target audience also has no knowledge of venv or user-venv, so if those tools are required then a good document telling exactly what to execute should be given; otherwise it basically isn't an option we offer.

[Matthias Koeppe]

On Sunday, October 10, 2021 at 1:57:55 PM UTC-7 Nils Bruin wrote:

 a good document telling exactly what to execute should be given

We have tickets for this, including https://trac.sagemath.org/ticket/29784, which need contributions.

 

[Matthias Koeppe]

On Sunday, October 10, 2021 at 1:57:55 PM UTC-7 Nils Bruin wrote:

I think it is worthwhile to have something like "sage --pip install --user ..." work [...] The target audience also has no knowledge of venv [...]

The user scheme (--user) is an outdated mechanism. Python, by design, disables it in virtual environments. https://packaging.python.org/tutorials/installing-packages/#installing-to-the-user-site

This is not specific to Sage. 

It could, however, be a good idea to extend

https://doc.sagemath.org/html/en/installation/source.html#installation-in-a-multiuser-environment to give advice to system administrators who install Sage systemwide. 

They may want to advise their users that they can use a startup script (https://doc.sagemath.org/html/en/reference/repl/startup.html) to add to the search path; and that they can use "sage -pip install --prefix=/SOME/WHERE" to install in their custom install directory.

It will then be the system administrator's problem when the accumulated stuff in a user's custom install directory stops an upgraded installation of Sage from working.