Proposal: install all pip-installable components of Sage via pip, directly

[Dima Pasechnik]

Presently we distribute a number of tarballs that are copies of pip-installable Python packages, which are dependencies of something, and maintain placeholders for them in build/pkg/ (this will be completely true as soon as #14840 is merged).

This is a lot of unnecessary stuff to carry around, and is quite different to what is done in pythonic (as well as ocaml, haskell, Macaulay2, etc) universe.

My proposal is to get rid of these, and install pip-installable dependencies directly. Perhaps maintaining a catalog of such dependencies in some way; perhaps such facilities are provided by pip already, one needs to check this out.

We may keep (some) placeholders in build/pkg, for the purpose of bootstrapping.

One added advantage is  that we can provide a sane way to install packages which cannot be installed from distributed sources, such as ssl support.

In case one wants all the sources needed for a complete Sage install packaged locally,

this is easy to achieve by providing a custom pip repo; such a repo may become some kind of optional component.

[Thierry]

Hi,

i am not sure to understand your proposal, but currently:
- you can install any pip-installable by 'sage -pip install <pkg>'
- pip-installable optional packages can have their source directly
downloaded and installed from pip. For this, just write 'pip' in the
file "${SAGE_ROOT}/build/pkgs/<pkg>/type"
- for standard packages, the source must be provided since the tarball is
supposed to be autonomous.

I do not get what is missing there ?

Ciao,
Thierry

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

[Volker Braun]

I think there are two different tasks:

1) Use pip to install python packages, regardless of where the package is. Maybe we bundle the tarball, maybe its on PyPI, maybe its on github. Pip can install either just fine. IMHO we should always use pip, but pip depends on openssl. We'll save us much hassle if we can just wait until openssl gets around to fixing their license as announced.

2) Download package tarballs from PyPI instead of our own mirror. IMHO thats just a tooling problem, either should be trivial. I have a script that searches for PyPI updates, downloads the latest version, and updates package_version.txt and checksums.ini accordingly. Thats about as much work as updating the version in a requirements.txt file. 

[Simon King]

Hi Thierry,

On 2016-04-16, Thierry <sage-goo...@lma.metelu.net> wrote:
> - for standard packages, the source must be provided since the tarball is
> supposed to be autonomous.

If I understand correctly, that'S exactly what the proposal would
change: Sage sources would contain the information that package "foo"
has to be pip-installed by default, but it does *not* provide foo's
source tar-ball, since that's not needed in order to pip-install it.

I suppose there could be a file in the Sage sources providing a list
of everything that should be pip-installed by default, and then as soon
as Sage has built its own version of Python, it would go through that
list and install stuff.

And that's a kind of modularisation that I'd like.

Best regards,
Simon

[Jeroen Demeyer]

What is the problem that your proposal fixes?

[Jeroen Demeyer]

On 2016-04-16 12:15, Volker Braun wrote:
> IMHO we should always use pip, but pip
> depends on openssl.

Not for local packages, as far as I know.

[Volker Braun]

$ mv local/lib/python/lib-dynload/_ssl.so local/lib/python/lib-dynload/_ssl.so-backup

$ sage -pip -h

Traceback (most recent call last):

  File "/home/vbraun/Code/sage.git/local/bin/pip", line 9, in <module>

    load_entry_point('pip==8.1.1', 'console_scripts', 'pip')()

  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 549, in load_entry_point

  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2542, in load_entry_point

  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2202, in load

  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 2208, in resolve

  File "/home/vbraun/Code/sage.git/local/lib/python2.7/site-packages/pip-8.1.1-py2.7.egg/pip/__init__.py", line 16, in <module>

    from pip.vcs import git, mercurial, subversion, bazaar  # noqa

  File "/home/vbraun/Code/sage.git/local/lib/python2.7/site-packages/pip-8.1.1-py2.7.egg/pip/vcs/subversion.py", line 9, in <module>

    from pip.index import Link

  File "/home/vbraun/Code/sage.git/local/lib/python2.7/site-packages/pip-8.1.1-py2.7.egg/pip/index.py", line 30, in <module>

    from pip.wheel import Wheel, wheel_ext

  File "/home/vbraun/Code/sage.git/local/lib/python2.7/site-packages/pip-8.1.1-py2.7.egg/pip/wheel.py", line 39, in <module>

    from pip._vendor.distlib.scripts import ScriptMaker

  File "/home/vbraun/Code/sage.git/local/lib/python2.7/site-packages/pip-8.1.1-py2.7.egg/pip/_vendor/distlib/scripts.py", line 14, in <module>

    from .compat import sysconfig, detect_encoding, ZipFile

  File "/home/vbraun/Code/sage.git/local/lib/python2.7/site-packages/pip-8.1.1-py2.7.egg/pip/_vendor/distlib/compat.py", line 31, in <module>

    from urllib2 import (Request, urlopen, URLError, HTTPError,

ImportError: cannot import name HTTPSHandler

[Jeroen Demeyer]

Great... that still doesn't work.

[Michael Orlitzky]

On 04/16/2016 08:06 AM, Jeroen Demeyer wrote:
> What is the problem that your proposal fixes?
>

It gets us one step closer to a Sage that will use what's already
installed on the system. Users can upgrade pip-installed packages
themselves (pip --update, or whatever). And since Sage won't be looking
for its own copy of the library, using one installed by the system
package manager should become seamless.

It will also make the Sage tarballs smaller, simplify licensing (we
won't be distributing anything), and make dependency upgrades easier
(just edit the requirements file).

[Jeroen Demeyer]

On 2016-04-16 16:19, Michael Orlitzky wrote:
> It gets us one step closer to a Sage that will use what's already
> installed on the system.

No, it does not do that. It's just a different way of installing packages.

> It will also make the Sage tarballs smaller

I don't think that "small tarballs" is a goal of Sage. On the other
hand, I do think that "tarballs containing all needed sources" is a goal
of Sage.

> make dependency upgrades easier

It's already pretty easy, but that might indeed be a minor advantage.

[Dima Pasechnik]

On Saturday, April 16, 2016 at 4:06:18 PM UTC+1, Jeroen Demeyer wrote:

On 2016-04-16 16:19, Michael Orlitzky wrote:
> It gets us one step closer to a Sage that will use what's already
> installed on the system.

No, it does not do that. It's just a different way of installing packages.

> It will also make the Sage tarballs smaller

I don't think that "small tarballs" is a goal of Sage. On the other
hand, I do think that "tarballs containing all needed sources" is a goal
of Sage.

Any other largish Python-based  project that does these kind of tarballs these days?

Of course it's hard to change this sort of attitude: 

(warning, the image on that link might not be very PC...)

[Dima Pasechnik]

is this a clean experiment? I imagine one would try building pip over an ssl-less python...

[Jeroen Demeyer]

On 2016-04-16 17:29, Dima Pasechnik wrote:
> Any other largish Python-based project that does these kind of tarballs
> these days?

The Sage source tarball is a distribution containing many packages, it's
not a "Python project".

> Of course it's hard to change this sort of attitude

I don't think that there is a problem of attitude, we are most likely
just misunderstanding eachother.

[Dima Pasechnik]

On Saturday, April 16, 2016 at 4:34:30 PM UTC+1, Jeroen Demeyer wrote:

On 2016-04-16 17:29, Dima Pasechnik wrote:
> Any other largish Python-based  project that does these kind of tarballs
> these days?

The Sage source tarball is a distribution containing many packages, it's
not a "Python project". 

whatever. Distributions nowadays do not carry megamegatarballs with them, either

(think e.g. Haskell Platform, or Macaulay2) 

If a Sage package (say, sagenb) needs to do 'pip install Blah' it should just do it.

As opposed to to putting Blah.tar.gz into upstream, creating a placeholder in build/pkgs/Blah,

doing a vote on sage-devel on whether Blah might be included (oh, and if, oops, Blah has a wrong

sort of license, it's a no-starter...), hearing much cringing on how this would bloat Sage even more,

etc etc...

> Of course it's hard to change this sort of attitude

I don't think that there is a problem of attitude, we are most likely
just misunderstanding eachother.

basically, I propose to reduce the over-packaging of dead-trivial things that

can be pulled from the net with zero effort.

You say it's no big deal, and I think you are plain wrong here.

 

[William Stein]

On Sat, Apr 16, 2016 at 8:54 AM, Dima Pasechnik <dim...@gmail.com> wrote:
>
>
> On Saturday, April 16, 2016 at 4:34:30 PM UTC+1, Jeroen Demeyer wrote:
>>
>> On 2016-04-16 17:29, Dima Pasechnik wrote:
>> > Any other largish Python-based project that does these kind of tarballs
>> > these days?
>>
>> The Sage source tarball is a distribution containing many packages, it's
>> not a "Python project".
>
>
> whatever.

(1) Providing a source tarball from which one can build Sage without
internet access is a basic requirement from a company that provides
some funding for Sage development over the years. We don't
contractually have to do it, but I think it is a good to be able to do
so. (2) It is also helpful when we have Sage days workshops at places
with crappy internet (frequent), since at least everybody can build
Sage locally from a single tarball.

- William

> Distributions nowadays do not carry megamegatarballs with them,
> either
> (think e.g. Haskell Platform, or Macaulay2)
> If a Sage package (say, sagenb) needs to do 'pip install Blah' it should
> just do it.
> As opposed to to putting Blah.tar.gz into upstream, creating a placeholder
> in build/pkgs/Blah,
> doing a vote on sage-devel on whether Blah might be included (oh, and if,
> oops, Blah has a wrong
> sort of license, it's a no-starter...), hearing much cringing on how this
> would bloat Sage even more,
> etc etc...
>
>>
>> > Of course it's hard to change this sort of attitude
>>
>> I don't think that there is a problem of attitude, we are most likely
>> just misunderstanding eachother.
>
>
> basically, I propose to reduce the over-packaging of dead-trivial things
> that
> can be pulled from the net with zero effort.
> You say it's no big deal, and I think you are plain wrong here.
>
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

--
William (http://wstein.org)

[Jeroen Demeyer]

Let me also remind everyone that you don't have to use the full source
tarball. If you like to download at build-time from the internet, clone
the git repo (which does not contain tarballs) and build from that.

[Dima Pasechnik]

On Saturday, April 16, 2016 at 5:18:17 PM UTC+1, William wrote:

On Sat, Apr 16, 2016 at 8:54 AM, Dima Pasechnik <dim...@gmail.com> wrote:
>
>
> On Saturday, April 16, 2016 at 4:34:30 PM UTC+1, Jeroen Demeyer wrote:
>>
>> On 2016-04-16 17:29, Dima Pasechnik wrote:
>> > Any other largish Python-based  project that does these kind of tarballs
>> > these days?
>>
>> The Sage source tarball is a distribution containing many packages, it's
>> not a "Python project".
>
>
> whatever.

(1) Providing a source tarball from which one can build Sage without
internet access is a basic requirement from a company that provides
some funding for Sage development over the years.   We don't
contractually have to do it, but I think it is a good to be able to do
so. (2) It is also helpful when we have Sage days workshops at places
with crappy internet (frequent), since at least everybody can build
Sage locally from a single tarball.

one can easily make a local pip repo containing all the needed extra things, if 

required. I mentioned this in my original post.

[Dima Pasechnik]

Don't you see that making copies of perfectly good stuff that one can install using pip

raises few eyebrows?

See, some people are serious about security, and you are forcing them to check every bloody

tarfile for authenticity (and you rename them once in a while, making their job even harder)...

  

[Dima Pasechnik]

On Saturday, April 16, 2016 at 5:18:17 PM UTC+1, William wrote:

On Sat, Apr 16, 2016 at 8:54 AM, Dima Pasechnik <dim...@gmail.com> wrote:
>
>
> On Saturday, April 16, 2016 at 4:34:30 PM UTC+1, Jeroen Demeyer wrote:
>>
>> On 2016-04-16 17:29, Dima Pasechnik wrote:
>> > Any other largish Python-based  project that does these kind of tarballs
>> > these days?
>>
>> The Sage source tarball is a distribution containing many packages, it's
>> not a "Python project".
>
>
> whatever.

(1) Providing a source tarball from which one can build Sage without
internet access is a basic requirement from a company that provides
some funding for Sage development over the years.   We don't
contractually have to do it, but I think it is a good to be able to do
so. (2) It is also helpful when we have Sage days workshops at places
with crappy internet (frequent), since at least everybody can build
Sage locally from a single tarball. 

PS. in my eyes, my proposal would bring Sage closer to "normal open-source development

practices" that you were arguing for recently. No normal project I know creates

megamegatarball of everything; reducing these would be a step in the right direction.

[kcrisman]

 

(1) Providing a source tarball from which one can build Sage without
internet access is a basic requirement from a company that provides
some funding for Sage development over the years.   We don't
contractually have to do it, but I think it is a good to be able to do
so. (2) It is also helpful when we have Sage days workshops at places
with crappy internet (frequent), since at least everybody can build
Sage locally from a single tarball. 

PS. in my eyes, my proposal would bring Sage closer to "normal open-source development

practices" that you were arguing for recently. No normal project I know creates

megamegatarball of everything; reducing these would be a step in the right direction.

Remind me again how this helps with (2)?  But more seriously, is there a way to have the "normal" distribution be pip or whatever but have a scriptable way to create a tarball for everything that is just not the default one people are pointed to?  I assume yes.

[Dima Pasechnik]

here is one I found

http://doc.devpi.net/latest/quickstart-pypimirror.html 

it will create a pypi cache that can be served locally.

Then, I saw a link saying that a complete pypi mirror is "just" 120GB.

Even if this is 5 times too optimistic, it would still fit on a portable 1TB or so disk you can take with you.

Perhaps a single tarball was important to have 10 years ago, nowadays it's really not something so important...

[Isuru Fernando]

Or you can just download the packages to SAGE_ROOT/upstream from pip and create a tarball for everything and use the downloaded packages to install using pip without an internet connection

./sage -pip install --download $SAGE_ROOT/upstream sympy  # Download sympy and dependencies to $SAGE_ROOT/upstream
./sage -pip install --no-index --find-links=$SAGE_ROOT/upstream --upgrade sympy  # Use the downloaded packages

 

Perhaps a single tarball was important to have 10 years ago, nowadays it's really not something so important...

[William Stein]

This fails to solve the problem of a security-paranoid person with an air gapped computer, since such a person would be careful about what data they put on their computer.  They would prefer to just put the sage-related packages, not all pypi.

 

Perhaps a single tarball was important to have 10 years ago, nowadays it's really not something so important...

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

--
Sent from my massive iPhone 6 plus.

[Dima Pasechnik]

On Sunday, April 17, 2016 at 3:43:15 PM UTC+1, William wrote:

On Sunday, April 17, 2016, Dima Pasechnik <dim...@gmail.com> wrote:

On Sunday, April 17, 2016 at 2:14:09 AM UTC+1, kcrisman wrote:

 

(1) Providing a source tarball from which one can build Sage without
internet access is a basic requirement from a company that provides
some funding for Sage development over the years.   We don't
contractually have to do it, but I think it is a good to be able to do
so. (2) It is also helpful when we have Sage days workshops at places
with crappy internet (frequent), since at least everybody can build
Sage locally from a single tarball. 

PS. in my eyes, my proposal would bring Sage closer to "normal open-source development

practices" that you were arguing for recently. No normal project I know creates

megamegatarball of everything; reducing these would be a step in the right direction.

Remind me again how this helps with (2)?  But more seriously, is there a way to have the "normal" distribution be pip or whatever but have a scriptable way to create a tarball for everything that is just not the default one people are pointed to?  I assume yes.

here is one I found

http://doc.devpi.net/latest/quickstart-pypimirror.html 

it will create a pypi cache that can be served locally.

Then, I saw a link saying that a complete pypi mirror is "just" 120GB.

Even if this is 5 times too optimistic, it would still fit on a portable 1TB or so disk you can take with you.

This fails to solve the problem of a security-paranoid person with an air gapped computer, since such a person would be careful about what data they put on their computer.  They would prefer to just put the sage-related packages, not all pypi.

 

sorry, William, I don't see what you mean.

1) We talk about a workshop at a remote location situation, bad internet access etc.

2) Putting data on your computer does not equal to installing crud on your computer, although people wearing tinfoil hats

might disagree. And I never said: "copy the whole of pypi to evreyone's machine".

I said, basically, have a server on a local net with a mirror of pypi (or its part). 

3) if Sage must be approved for use in psychiatric wards, it's a different story all together. :-)

 

Perhaps a single tarball was important to have 10 years ago, nowadays it's really not something so important...

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscribe@googlegroups.com.

To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[William Stein]

Dima:

> sorry, William, I don't see what you mean.

[...]

> 2) Putting data on your computer does not equal to installing crud on your
> computer, although people wearing tinfoil hats
> might disagree.

They do disagree. And they are right.

--
William (http://wstein.org)

[Dima Pasechnik]

On Sunday, April 17, 2016 at 4:15:12 PM UTC+1, William wrote:

Dima:
> sorry, William, I don't see what you mean.
[...]
> 2) Putting data on your computer does not equal to installing crud on your
> computer, although people wearing tinfoil hats
> might disagree.

They do disagree.  And they are right.  

Why would they trust somebody putting hundreds of megabytes of 3rd party sources into Sage? Is it because we mislead them into thinking that we did some kind of source review?

 

--
William (http://wstein.org)

[Dima Pasechnik]

they actually would be less paranoid about having the 3rd party from its original source rather than re-packaged, and this is precisely a part of my proposal. And, by the way, http://doc.devpi.net/latest/quickstart-pypimirror.html

is precisely a way to create a mini-repo containing only things needed by Sage.

Anyhow, it seems that what I propose is not rrrrevolutional enough to merit a normal discussion. :-(

 

--
William (http://wstein.org)

[William Stein]

On Sun, Apr 17, 2016 at 8:27 AM, Dima Pasechnik <dim...@gmail.com> wrote:
>
>
> On Sunday, April 17, 2016 at 4:20:42 PM UTC+1, Dima Pasechnik wrote:
>>
>>
>>
>> On Sunday, April 17, 2016 at 4:15:12 PM UTC+1, William wrote:
>>>
>>> Dima:
>>> > sorry, William, I don't see what you mean.
>>> [...]
>>> > 2) Putting data on your computer does not equal to installing crud on
>>> > your
>>> > computer, although people wearing tinfoil hats
>>> > might disagree.
>>>
>>> They do disagree. And they are right.
>>
>>
>> Why would they trust somebody putting hundreds of megabytes of 3rd party
>> sources into Sage? Is it because we mislead them into thinking that we did
>> some kind of source review?

You are right. However, such a tinfoil hat person would rather
(themselves) inspect the packages that are just needed for Sage,
rather than all of Pypi.

> they actually would be less paranoid about having the 3rd party from its
> original source rather than re-packaged, and this is precisely a part of my
> proposal. And, by the way,
> http://doc.devpi.net/latest/quickstart-pypimirror.html
> is precisely a way to create a mini-repo containing only things needed by
> Sage.

Perfect -- I agree that is the right approach.


--
William (http://wstein.org)

[mmarco]

Would it be possible to ship such a pip repo (with only the packages needed by sage) in the sage source tarball? Or maybe have two such tarballs, one with it (so it would allow to compile Sage offline, and source review before compiling), and another that would just rely on downloading packages from pypi?

[Dima Pasechnik]

On Sunday, April 17, 2016 at 9:51:03 PM UTC+1, mmarco wrote:

Would it be possible to ship such a pip repo (with only the packages needed by sage) in the sage source tarball? Or maybe have two such tarballs, one with it (so it would allow to compile Sage offline, and source review before compiling), and another that would just rely on downloading packages from pypi?

It looks like it would make more sense to provide a cache of pypi packages needed for Sage building.

It might either be a tarball, or just a list of files to download from a pypi mirror.

The less duplication, the better...

[Erik Bray]

On Sat, Apr 16, 2016 at 10:59 AM, Dima Pasechnik <dim...@gmail.com> wrote:
> Presently we distribute a number of tarballs that are copies of
> pip-installable Python packages, which are dependencies of something, and
> maintain placeholders for them in build/pkg/ (this will be completely true
> as soon as #14840 is merged).
> This is a lot of unnecessary stuff to carry around, and is quite different
> to what is done in pythonic (as well as ocaml, haskell, Macaulay2, etc)
> universe.
>
> My proposal is to get rid of these, and install pip-installable dependencies
> directly. Perhaps maintaining a catalog of such dependencies in some way;
> perhaps such facilities are provided by pip already, one needs to check this
> out.
> We may keep (some) placeholders in build/pkg, for the purpose of
> bootstrapping.
> One added advantage is that we can provide a sane way to install packages
> which cannot be installed from distributed sources, such as ssl support.
>
> In case one wants all the sources needed for a complete Sage install
> packaged locally,
> this is easy to achieve by providing a custom pip repo; such a repo may
> become some kind of optional component.

Hi Dima,

I haven't read the rest of this thread yet, but I already started part
of this work back in http://trac.sagemath.org/ticket/20218 which I'm
nearly done testing a patch for.

The above isn't the whole store because it just switches from using
`setup.py install` (which has problems that I described in the ticket)
to using `pip install --no-deps .` for installing most Python
packages.

You're talking about going a step further and I'm inclined to agree,
but I haven't looked at what the others have stated on this issue.
Still, I think something like #20218 would be a good starting point
before proceeding further.

Thanks,
Erik

[Erik Bray]

On Sat, Apr 16, 2016 at 2:13 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> Great... that still doesn't work.

I agree that should be fixed. It's like a two line patch that I'll
submit to pip.

[Dima Pasechnik]

except that it's probably not  that easy, according to

https://github.com/pypa/pip/issues/1165

and references therein.

[Michael Orlitzky]

On 04/16/2016 11:06 AM, Jeroen Demeyer wrote:
> On 2016-04-16 16:19, Michael Orlitzky wrote:
>> It gets us one step closer to a Sage that will use what's already
>> installed on the system.
>
> No, it does not do that. It's just a different way of installing packages.
>

I had `pip install --user` in mind.

Suppose I use pip to install libfoo as part of the sage-8.0
installation. Then I wipe out my sage-8.0 directory and try to install
sage-9.0. Wouldn't it find the libfoo that I already installed?

Those pip-installed libraries should persist, and that means our "does
this library exist?" test should be something like "import foo" and not
"is the libfoo directory exactly where I think it should be". The
"import foo" test would also pick up libfoo if I've installed it with my
system's package manager.

[Volker Braun]

It should be fixed in distlib which is vendored into pip...