Proposal: Policy for standard packages from binary wheels

[Matthias Koeppe]

Currently all "standard" packages of the Sage distribution, by policy (https://doc.sagemath.org/html/en/developer/packaging.html; recommended reading), 

- either can be installed from source ("normal" packages);

- or they are Python packages that can be installed from platform-independent wheels.

(We added the second option in order to treat Javascript assets as prebuilt data, thus avoiding the complexity of Javascript build infrastructure in the Sage distribution.)

In https://github.com/sagemath/sage/pull/38219 (needs review), I propose the following mild policy change:

- as a third option, a "standard" package is allowed to be a Python package that can be installed from platform-dependent (binary) wheels. In this case, a "configure" option must be provided that disables the package.

(Again this new option is motivated by avoiding the complexity of new build infrastructure related to another language; in this case, Rust. See the PR for two example Rust-based example packages; previous discussion: https://groups.google.com/g/sage-devel/c/JiSryImYGe0/m/4B8z6F4AEwAJ)

[Kwankyu Lee]

Does this mean that there will be different sage tarballs for supported platforms?

[Georgi Guninski]

On Sun, Aug 4, 2024 at 9:03 PM Matthias Koeppe <matthia...@gmail.com> wrote:
>

> In https://github.com/sagemath/sage/pull/38219 (needs review), I propose the following mild policy change:
> - as a third option, a "standard" package is allowed to be a Python package that can be installed from platform-dependent (binary) wheels. In this case, a "configure" option must be provided that disables the package.
>
> (Again this new option is motivated by avoiding the complexity of new build infrastructure related to another language; in this case, Rust. See the PR for two example Rust-based example packages; previous discussion: https://groups.google.com/g/sage-devel/c/JiSryImYGe0/m/4B8z6F4AEwAJ)
>

I would recommend running antivirus just before `make install` and
commercial supply chain backdoor scanner. Especially if there is no
way to verify the integrity.

[Nathan Dunfield]

This proposal makes sense to me.  Something like this is necessary for us to upgrade Jupyter since it depends on rdps-py which needs Rust to build, and this approach preserves the other features of current standard packages: the version is pinned (including checksums on the binary wheels) and all dependencies explicitly included as other standard packages.

Nathan

[Dima Pasechnik]

It is amusing that it is put forward as a new proposal. This is my proposal I have been making in  various forms during the past 10 months or so,
mainly in context of standardisation of a couple of packages, such as pytest.
And it is put forward by a very vocal opponent of my proposal.

At the very least, proper credits are due.

Best,
Dima

[Bagas Sanjaya]

On Sat, Aug 03, 2024 at 11:15:13PM -0700, Matthias Koeppe wrote:
> Currently all "standard" packages of the Sage distribution, by policy
> (https://doc.sagemath.org/html/en/developer/packaging.html; recommended
> reading),
> - either can be installed from source ("normal" packages);
> - or they are Python packages that can be installed from
> platform-independent wheels.
>
> (We added the second option in order to treat Javascript assets as prebuilt
> data, thus avoiding the complexity of Javascript build infrastructure in
> the Sage distribution.)
>
> In https://github.com/sagemath/sage/pull/38219 (needs review), I propose
> the following mild policy change:
> - as a third option, a "standard" package is allowed to be a Python package
> that can be installed from platform-dependent (binary) wheels. In this
> case, a "configure" option must be provided that disables the package.

pip packages?

Confused...

--
An old man doll... just what I always wanted! - Clara

[Dima Pasechnik]

Nathan,
this is essentially the proposal I put forward in Feb 2024:
https://groups.google.com/g/sage-devel/c/5kmxaw105lg/m/QGShvy6-AAAJ

You didn't like it then.
https://groups.google.com/g/sage-devel/c/5kmxaw105lg/m/N-eEEQTeAgAJ

Dima

PS. Nobody seems to remember that conversation? Is it forbidden to
mention my proposals on sage-develop nowadays?

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/b1ccf359-c759-46e7-a08a-f1176d91f755n%40googlegroups.com.

[Nathan Dunfield]

Dima,

As I understand it, while both proposals result in binary wheels being pulled off PyPI for certain standard packages, they differ in that:

a) The current proposal requires the version of the package to be pinned (with checksums).

b) The current proposal does not allow a package to pull in any dependencies that are not explicitly included as other Sage packages.

As you know, these are both part of the behavior of the current "wheel packages", which are currently required to be pure Python, with no compiled extensions.

In contrast, our "pip packages", used for so many optional packages, do not require either (a) or (b), and I think most do not.  Your proposal, as I understood it, was to allow standard packages to that do not do either (a) or (b).   My comment you refer to is my reasons for why I think (a) and (b) are important.  Later in that same thread, said I was in favor of Sage using upstream binary wheels from PyPI if we did (a) and (b):

https://groups.google.com/g/sage-devel/c/5kmxaw105lg/m/fu-g4zhaEwAJ

Best wishes,

Nathan

[Dima Pasechnik]

On Tue, Aug 6, 2024 at 9:45 PM Nathan Dunfield <nat...@dunfield.info> wrote:
> As I understand it, while both proposals result in binary wheels being pulled off PyPI for certain standard packages, they differ in that:
>
> a) The current proposal requires the version of the package to be pinned (with checksums).

Does this mean using `--require-hashes` while calling pip, i.e.
https://pip.pypa.io/en/stable/topics/secure-installs/#hash-checking-mode ?
That's fine with me - if there is a ./configure option to skip these
hashes tests.

This allows the added security of hashes without an extra burden to
keep packages on mirrors.

>
> b) The current proposal does not allow a package to pull in any dependencies that are not explicitly included as other Sage packages.

This seems like an unneeded extra bureaucracy. With `--require-hashes`
one has to list all the
dependencies of the package with their versions and hashes. E.g. to
install pytest with --require-hashes
into a Python venv one would have to create a requirements file such as

$ cat reqs.txt
iniconfig == 2.0.0
--hash=sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374
packaging == 24.1
--hash=sha256:5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124
pluggy == 1.5.0
--hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669
pytest == 8.3.2
--hash=sha256:4ba08f9ae7dcf84ded419494d229b48d0903ea6407b030eaec46df5e6a73bba5

and run

$ pip install --require-hashes -r reqs.txt

So no need to create spkgs for each dependency - they are already
listed, with pinned versions and hashes, in reqs.txt

>
> As you know, these are both part of the behavior of the current "wheel packages", which are currently required to be pure Python, with no compiled extensions.
>
> In contrast, our "pip packages", used for so many optional packages, do not require either (a) or (b), and I think most do not. Your proposal, as I understood it, was to allow standard packages to that do not do either (a) or (b). My comment you refer to is my reasons for why I think (a) and (b) are important.

Well, as long as the user can skip these extra tests, it's fine to have them.

Dima

[Bagas Sanjaya]

In Windows, that's quite common for people to have antivirus installed.
But for people on Linux, they usually don't install one.

[Michael Orlitzky]

On Wed, 2024-08-07 at 06:41 +0700, Bagas Sanjaya wrote:
>
> In Windows, that's quite common for people to have antivirus installed.
> But for people on Linux, they usually don't install one.
>

On Linux, people aren't usually dumb enough to download exes from
strangers. Yet here we are.

[Georgi Guninski]

On Wed, Aug 7, 2024 at 4:17 AM Bagas Sanjaya <bagas...@gmail.com> wrote:
>
>
> In Windows, that's quite common for people to have antivirus installed.
> But for people on Linux, they usually don't install one.
>

On GNU/Linux malware is open source :)
> Find attached .tar.gz, do `configure ; make`
or
> git clone chix_for_free

:)

[Georgi Guninski]

On Wed, Aug 7, 2024 at 5:54 AM Michael Orlitzky <mic...@orlitzky.com> wrote:
>

> On Linux, people aren't usually dumb enough to download exes from
> strangers. Yet here we are.
>

Some people didn't realize mine was sarcasm.
As for dumb people running exe's, some of the notable windoze malware
is zero interaction, resulting in pandemia.
Some linux developers are dumb enough to allow committing obfuscated backdoors.

[Marc Culler]

I think it would be possible to use a single spkg for all of the additional binary wheels, where the main content of that spkg is a requirements.txt file listing versions and hashes.  I also think that would be easier to maintain than many spkg directories, one for each binary wheel.  The only reason I know of for using separate directories is to allow a separate documentation file for each wheel.  I don't think that documentation is really needed, as it could be replaced by boilerplate which references pypi.  And, in practice, those individual documentation files would probably be constructed by copying and pasting from pypi anyway.

- Marc

[Nathan Dunfield]

On Thursday, August 8, 2024 at 7:57:12 AM UTC-5 Marc Culler wrote:

I think it would be possible to use a single spkg for all of the additional binary wheels, where the main content of that spkg is a requirements.txt file listing versions and hashes. 

Yes, that is approach Matthias implemented in the accompanying PR (https://github.com/sagemath/sage/pull/38219), though the data is in the "checksums.ini" file.  For example:

https://github.com/sagemath/sage/blob/fdc545367173f836890b9cddbce0d7c032b10d2b/build/pkgs/rpds_py/checksums.ini 

Best,

Nathan

[Marc Culler]

As far as I can tell, that is quite different from what I was suggesting.  It involves a separate package directory for each binary wheel, not one directory for many binary wheels which would all get installed together at the end of the build process in one big pip run.

However, I was naively looking at this from the point of view of someone maintaining a port of Sage to a single platform, which also happens to allow a single binary wheel to work for multiple versions of the OS. Looking at that linked file reveals that the linux situation is infinitely more complex.  The version "pinning" apparently must be done separately for  each release of each distro.  Probably there is no way to tame that maintenance headache.  Possibly Sage could hope to "delegate" the job of deciding which version of each binary wheel is optimal for each release of each distro to the package maintainer(s) for that distro and to copy the data for those checksum.ini files from some known location within each distro.  However, I don't see how that will work for the majority of distros given that they are not close to supporting the current version of Sage.

I don't know if this scheme will end up being simpler than installing Rust in the Sage build directory (which involves very little more than downloading and running rustup.sh).  I hope it will.

- Marc

[Matthias Koeppe]

On Friday, August 9, 2024 at 7:37:34 AM UTC-7 Marc Culler wrote:

I was naively looking at this from the point of view of someone maintaining a port of Sage to a single platform, which also happens to allow a single binary wheel to work for multiple versions of the OS. Looking at that linked file [https://github.com/sagemath/sage/blob/fdc545367173f836890b9cddbce0d7c032b10d2b/build/pkgs/rpds_py/checksums.ini] reveals that the linux situation is infinitely more complex.  The version "pinning" apparently must be done separately for  each release of each distro.

These are not really tied to "Linux distro releases". Instead the Python PA has defined common-denominator platforms called "manylinux" that are basically keyed to the GLIBC version. (That's what "2.17" in "manylinux_2_17" refers to.) In addition, there are common-denominator platforms for a different libc implementation, "musllinux".

Probably there is no way to tame that maintenance headache.  Possibly Sage could hope to "delegate" the job of deciding which version of each binary wheel is optimal for each release of each distro to the package maintainer(s) for that distro and to copy the data for those checksum.ini files from some known location within each distro.  However, I don't see how that will work for the majority of distros given that they are not close to supporting the current version of Sage.

That's the upstream projects' burden, not ours; in fact, the upstream projects delegate this burden to a large extent to the infrastructure that is provided by https://cibuildwheel.pypa.io/en/stable/

I don't know if this scheme will end up being simpler than installing Rust in the Sage build directory (which involves very little more than downloading and running rustup.sh).  I hope it will.

I also don't know for sure, but I'm simply not keen on taking on the work of maintaining anything Rust-related just to know with greater certainty.

It's good to have a clear boundary of the project.

[Marc Culler]

Oh.  I wasn't paying very close attention when I looked at that file.  I am familiar with manylinux wheels, of course.  I guess I was too busy being impressed by the size of the file.

The real driver of the complexity, then, is the number of different versions of python which Sage is obliged to support, and that in turn is driven by which Python versions are provided by the various distros that Sage supports.  Of course the macOS app uses only one version of python, which it embeds, namely the one built with Sage's spkg.

When I build Sage for the macOS app my current process is to first run Sage's standard build (disabling all "system" libraries and enabling as many optional packages as possible) and then, when that is done, I use Sage's pip to install current versions of pillow, jupyterlab and notebook from binary wheels.  I use the following arguments to pip:

  install --no-user --force-reinstall --upgrade-strategy eager

That means that several packages that Sage built from source get clobbered and replaced by packages installed from binary wheels.  But Sage ends up with current (as of the time of my build)  versions of jupyterlab and notebook, and those two packages end up having the dependencies against which they have actually been tested.  (I don't believe that even a project as large as jupyter has the capacity to test against all combinations of dependencies which would meet their specifications, but I do believe that they test pretty thoroughly against the newest allowed version of each dependency.)  I do a sanity check to see that the installed versions of jupyterlab and notebook work at a basic level, using the Sage kernel, but I don't really have any way of testing jupyter or its dependencies more carefully than the jupyter project itself does.  So, for the macOS app, I have basically delegated the choice of wheel versions for the jupyterlab and notebook dependencies to the jupyter project.  I think that few of the dependencies of those two packages are critical to other parts of Sage, with notable exceptions being IPython and its dependencies.

- Marc

PS I am also delegating the choice of version for many image processing libraries to the Pillow project, and trusting them to compile the libraries.

--
You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/vFsNl87zLxo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/03d03a54-609c-444d-b834-7fb0663a8c7fn%40googlegroups.com.

[Matthias Koeppe]

On Sunday, August 11, 2024 at 8:05:42 AM UTC-7 Marc Culler wrote:

The real driver of the complexity, then, is the number of different versions of python which Sage is obliged to support

Sure. My current implementation of the "sage --package create --pypi" for these types of packages does not even bother to filter out any of the unsupported Python versions - so you see a bunch of "cp38" files (https://github.com/sagemath/sage/blob/fdc545367173f836890b9cddbce0d7c032b10d2b/build/pkgs/rpds_py/checksums.ini#L196) ... and also win32 and win_amd64 files ( https://github.com/sagemath/sage/blob/fdc545367173f836890b9cddbce0d7c032b10d2b/build/pkgs/rpds_py/checksums.ini#L316). Easy to refine, but I think it's good enough for now.

When I build Sage for the macOS app my current process [...] Sage ends up with current (as of the time of my build)  versions of jupyterlab and notebook, and those two packages end up having the dependencies against which they have actually been tested.

I do the same in upgrade PRs such as https://github.com/sagemath/sage/pull/38471 (needs review). Just updating every Jupyter-related package to the latest and then running with "pip check" whether any of the declared version bounds for dependencies are violated. 

(Thanks to your code to extract runtime dependencies from PyPI wheels that is now part of "sage --package create --pypi", preparing such upgrades has become much smoother.)

 

(I don't believe that even a project as large as jupyter has the capacity to test against all combinations of dependencies which would meet their specifications, but I do believe that they test pretty thoroughly against the newest allowed version of each dependency.)

I agree.

 

  I do a sanity check to see that the installed versions of jupyterlab and notebook work at a basic level, using the Sage kernel [...]

I think it would be be valuable to prepare a checklist for making such updates and add it to our developer documentation. I think Eric Gourgoulhon has some Jupyter notebooks that are a good test of graphics and typesetting, some of which is Sage-specific.

[Eric Gourgoulhon]

Le lundi 12 août 2024 à 02:45:03 UTC+2, Matthias Koeppe a écrit :

I think it would be be valuable to prepare a checklist for making such updates and add it to our developer documentation. I think Eric Gourgoulhon has some Jupyter notebooks that are a good test of graphics and typesetting, some of which is Sage-specific.

FWIW, the test notebook is available here:

https://github.com/egourgoulhon/SageMathTest/blob/master/Notebooks/test_display.ipynb

or via nbviewer:

https://nbviewer.org/github/egourgoulhon/SageMathTest/blob/master/Notebooks/test_display.ipynb

Eric.

 

[Matthias Koeppe]

On Thursday, August 8, 2024 at 5:57:12 AM UTC-7 Marc Culler wrote:

I think it would be possible to use a single spkg for all of the additional binary wheels, where the main content of that spkg is a requirements.txt file listing versions and hashes.

I did initially look into using requirements.txt files before settling on the implementation in https://github.com/sagemath/sage/pull/38219, but I had to abandon this idea because it turns out that this format is not expressive enough to allow preparing multi-platform requirements with hashes. 

The reason: PEP-508 environment markers (https://packaging.python.org/en/latest/specifications/dependency-specifiers/#environment-markers) are not able to distinguish between ABI variants expressed in wheel tags such as manylinux vs. musllinux.

[Matthias Koeppe]

Thanks, Eric, for the link. I've opened https://github.com/sagemath/sage/pull/38503 for mentioning it in our review checklist.

[Kwankyu Lee]

> Does this [platform-dependent standard package] mean that there will be different sage tarballs for supported platforms?

I want to here an answer for this question.

[Matthias Koeppe]

On Tuesday, August 13, 2024 at 4:36:18 PM UTC-7 Kwankyu Lee wrote:

> Does this [platform-dependent standard package] mean that there will be different sage tarballs for supported platforms?

Yes. There are multiple "tarballs" (actually wheel files) listed in checksums.ini and uploaded/downloaded to/from GitHub Releases and the Sage mirrors  

Together they cover the supported platforms.

See branch of https://github.com/sagemath/sage/pull/38219 for the implementation.

[Matthias Koeppe]

Nathan explained it correctly.

[Kwankyu Lee]

Yes. There are multiple "tarballs" (actually wheel files) listed in checksums.ini and uploaded/downloaded to/from GitHub Releases and the Sage mirrors  

Together they cover the supported platforms.

Thanks. I confess that I misunderstood things about installing sage from source. I thought, for example, sage-10.4.zip contains all upstream source tarballs (files in /upstream) so that it can be used to install sage from it without internet connection. My question was based on the misunderstanding. Unzipping the zip file (first time for me) proved that I am wrong.

 

[Matthias Koeppe]

If you a referring to the "Source code (zip)" file that you find in https://github.com/sagemath/sage/releases/tag/10.4 (https://github.com/sagemath/sage/archive/refs/tags/10.4.zip): That file is automatically generated by GitHub from the commit of the tag. It has no relation to the present discussion.

[Matthias Koeppe]

On the other hand, the "sage-10.4.tar.gz" tarball that is built by the Release Manager and put on the Sage mirrors (as well as the "sage-10.4.tar.gz" tarball that is built using the same procedure and put on GitHub Releases - https://github.com/sagemath/sage/releases/tag/10.4) does contain an upstream/ directory that is populated with the upstream tarballs/wheel files of all standard normal/wheel packages.

With the branch of https://github.com/sagemath/sage/pull/38219, also the new binary wheel files (such as rpds_py-0.18.1-cp310-cp310-macosx_11_0_arm64.whl and rpds_py-0.18.1-cp310-cp310-musllinux_1_2_i686.whl) will appear in this directory.

[Kwankyu Lee]

On Thursday, August 15, 2024 at 4:11:50 AM UTC+9 Matthias Koeppe wrote:

On the other hand, the "sage-10.4.tar.gz" tarball that is built by the Release Manager and put on the Sage mirrors (as well as the "sage-10.4.tar.gz" tarball that is built using the same procedure and put on GitHub Releases - https://github.com/sagemath/sage/releases/tag/10.4) does contain an upstream/ directory that is populated with the upstream tarballs/wheel files of all standard normal/wheel packages.

Do you mean this https://github.com/sagemath/sage/archive/refs/tags/10.4.tar.gz ? This one also does not have "upstream/" directory. 

The installation guide does not tell me where I can download sage-10.4.tar.gz that contains upstream/ directory. Where is it? Sigh...

 

With the branch of https://github.com/sagemath/sage/pull/38219, also the new binary wheel files (such as rpds_py-0.18.1-cp310-cp310-macosx_11_0_arm64.whl and rpds_py-0.18.1-cp310-cp310-musllinux_1_2_i686.whl) will appear in this directory.

Would all they be included in (upstream/ directory of) the release tarball say sage-10.5.tar.gz? Then I fear that platform-dependent standard package would be a sure way to bloat the release tarballs. A user on a specific platform is forced to download many unnecessary wheel files via sage-10.x.tar.gz. To avoid this, we are forced to provide different tarballs for different platforms. I am back to my original question...

 

[Matthias Koeppe]

On Wednesday, August 14, 2024 at 3:08:39 PM UTC-7 Kwankyu Lee wrote:

On Thursday, August 15, 2024 at 4:11:50 AM UTC+9 Matthias Koeppe wrote:

On the other hand, the "sage-10.4.tar.gz" tarball that is built by the Release Manager and put on the Sage mirrors (as well as the "sage-10.4.tar.gz" tarball that is built using the same procedure and put on GitHub Releases - https://github.com/sagemath/sage/releases/tag/10.4) does contain an upstream/ directory that is populated with the upstream tarballs/wheel files of all standard normal/wheel packages.

Do you mean this https://github.com/sagemath/sage/archive/refs/tags/10.4.tar.gz ? This one also does not have "upstream/" directory. 

No, that one is also automatically generated by GitHub and contains the same files as the .zip file.

This one: https://github.com/sagemath/sage/releases/download/10.4/sage-10.4.tar.gz (hint: the list of Release Assets is sorted alphabetically)

The installation guide does not tell me where I can download sage-10.4.tar.gz that contains upstream/ directory. Where is it?

See https://doc.sagemath.org/html/en/installation/source.html#installation-steps

You may recall that we de-emphasized the use of self-contained tarballs in favor of pointing users to "git clone" earlier this year.

 

With the branch of https://github.com/sagemath/sage/pull/38219, also the new binary wheel files (such as rpds_py-0.18.1-cp310-cp310-macosx_11_0_arm64.whl and rpds_py-0.18.1-cp310-cp310-musllinux_1_2_i686.whl) will appear in this directory.

Would all they be included in (upstream/ directory of) the release tarball say sage-10.5.tar.gz?

Yes.

 

Then I fear that platform-dependent standard package would be a sure way to bloat the release tarballs. A user on a specific platform is forced to download many unnecessary wheel files via sage-10.x.tar.gz. To avoid this, we are forced to provide different tarballs for different platforms. I am back to my original question...

You may recall that we discussed the size of these wheel files previously in the thread "Current status of possibility of integrating libraries written in Rust into Sage" (https://groups.google.com/g/sage-devel/c/JiSryImYGe0/m/LeNyLi6dFAAJ). 

- Each of the rpds_py wheel files is a few hundred kilobytes (https://pypi.org/project/rpds-py/#files). The total is a few megabytes. 

- Each of the (fewer) clarabel wheel files is under 2 megabytes (https://pypi.org/project/clarabel/#files), the total is under 10 megabytes.

[Kwankyu Lee]

The installation guide does not tell me where I can download sage-10.4.tar.gz that contains upstream/ directory. Where is it?

See https://doc.sagemath.org/html/en/installation/source.html#installation-steps

Yes, there it is though quite buried among many files. Thanks.

 

You may recall that we de-emphasized the use of self-contained tarballs in favor of pointing users to "git clone" earlier this year.

Yes. But we may be kinder.to git phobias.

... we discussed the size of these wheel files previously in the thread "Current status of possibility of integrating libraries written in Rust into Sage" (https://groups.google.com/g/sage-devel/c/JiSryImYGe0/m/LeNyLi6dFAAJ). 

- Each of the rpds_py wheel files is a few hundred kilobytes (https://pypi.org/project/rpds-py/#files). The total is a few megabytes. 

- Each of the (fewer) clarabel wheel files is under 2 megabytes (https://pypi.org/project/clarabel/#files), the total is under 10 megabytes.

OK. Thanks for the clarification. I don't know how large is too large. We will know when users complain.

 

[Matthias Koeppe]

On Wednesday, August 14, 2024 at 7:33:49 PM UTC-7 Kwankyu Lee wrote:

... we discussed the size of these wheel files previously in the thread "Current status of possibility of integrating libraries written in Rust into Sage" (https://groups.google.com/g/sage-devel/c/JiSryImYGe0/m/LeNyLi6dFAAJ). 

- Each of the rpds_py wheel files is a few hundred kilobytes (https://pypi.org/project/rpds-py/#files). The total is a few megabytes. 

- Each of the (fewer) clarabel wheel files is under 2 megabytes (https://pypi.org/project/clarabel/#files), the total is under 10 megabytes.

OK. Thanks for the clarification. I don't know how large is too large. We will know when users complain.

For comparison: The current total size is 1.4 gigabytes. 

For the sizes of the release tarballs since 9.0, see https://mirrors.mit.edu/sage/src/index.html

[Kwankyu Lee]

For comparison: The current total size is 1.4 gigabytes. 

For the sizes of the release tarballs since 9.0, see https://mirrors.mit.edu/sage/src/index.html

To watch continuously and for non-developer users, those tarballs should be reachable by a click from the installation guide. 

[Kwankyu Lee]

Do you mean this https://github.com/sagemath/sage/archive/refs/tags/10.4.tar.gz ? This one also does not have "upstream/" directory. 

No, that one is also automatically generated by GitHub and contains the same files as the .zip file.

This one: https://github.com/sagemath/sage/releases/download/10.4/sage-10.4.tar.gz (hint: the list of Release Assets is sorted alphabetically)

Why don't we upload only sage-10.x.tar.gz as assets? It contains all other files uploaded there.

Platform-dependent wheels will increase the number of uploaded assets, and the main asset sage-10.x.tar.gz will be buried more deeply.

 

[Kwankyu Lee]

For comparison: The current total size is 1.4 gigabytes. 

For the sizes of the release tarballs since 9.0, see https://mirrors.mit.edu/sage/src/index.html

According to https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases, there is 2GB limit on the size of asset files. We may use that as a restraint on the size of our sage-10.x.tar.gz files.

[Matthias Koeppe]

On Thursday, August 15, 2024 at 4:54:29 PM UTC-7 Kwankyu Lee wrote:

Why don't we upload only sage-10.x.tar.gz as assets? It contains all other files uploaded there.

Our download procedure for upstream tarballs likes to download individual files.

[Matthias Koeppe]

On Thursday, August 15, 2024 at 4:10:27 PM UTC-7 Kwankyu Lee wrote:

> those tarballs should be reachable by a click from the installation guide.

That's exactly where the existing link in the installation guide leads, after the mirror selection.

Note that the decision to de-emphasize use of the self-contained tarballs was made deliberately. I see no reason to reverse it.

[Kwankyu Lee]

What download procedure do you mean? Where are those assets used?

[Kwankyu Lee]

On Friday, August 16, 2024 at 10:50:38 AM UTC+9 Matthias Koeppe wrote:

On Thursday, August 15, 2024 at 4:10:27 PM UTC-7 Kwankyu Lee wrote:

> those tarballs should be reachable by a click from the installation guide.

That's exactly where the existing link in the installation guide leads, after the mirror selection.

Ah, I see. That is the last line of the section 2 of the Installation Steps. 

 

Note that the decision to de-emphasize use of the self-contained tarballs was made deliberately. I see no reason to reverse it.

OK.  I just want those self-contained tarballs to be easily found for those who want them.

[Kwankyu Lee]

I now understand. You mean the installation procedure downloading upstream tarballs. OK. 

[Matthias Koeppe]

The Sage distribution attempts to download the files that are not already in upstream/ from various sources, configured in the directory .upstream.d (see https://github.com/sagemath/sage/tree/develop/.upstream.d). 

Downloading from the GitHub Release Assets is tried before the Sage mirrors are contacted.

[Matthias Koeppe]

On Sunday, August 4, 2024 at 11:03:38 AM UTC-7 Matthias Koeppe wrote:

Currently all "standard" packages of the Sage distribution, by policy (https://doc.sagemath.org/html/en/developer/packaging.html; recommended reading), 

- either can be installed from source ("normal" packages);

- or they are Python packages that can be installed from platform-independent wheels. [...]

In https://github.com/sagemath/sage/pull/38219 (needs review), I propose the following mild policy change:

- as a third option, a "standard" package is allowed to be a Python package that can be installed from platform-dependent (binary) wheels. In this case, a "configure" option must be provided that disables the package. [...]

I consider this approved. 

The implementation of the binary wheel infrastructure in https://github.com/sagemath/sage/pull/38219 is ready for review.

[julian...@fsfe.org]

I consider this approved.

While I don't think I would be opposed to the proposal here, I believe that our policy says that for a disputed PR to be "approved," a vote needs to happen here or on the PR.

 

The implementation of the binary wheel infrastructure in https://github.com/sagemath/sage/pull/38219 is ready for review.

julian

[Matthias Koeppe]

On Tuesday, August 20, 2024 at 9:02:14 AM UTC-7 julian...@fsfe.org wrote:

I consider this approved.

While I don't think I would be opposed to the proposal here, I believe that our policy says that for a disputed PR to be "approved," a vote needs to happen here or on the PR.

Julian, you snipped away what "this" is what I consider approved -- namely the proposed policy (not the PR).

[julian...@fsfe.org]

You also mentioned the PR in your quote. I just wanted to clarify that the PR itself is not approved. I am glad that we agree on this :)

julian

[Nils Bruin]

Thank you for the clarification. I certainly read the message as the presumed approval pertaining to both the policy and the PR, because they were tied together and discussed together from the start.

In any case, I would think that in this case it is really the policy change that needs the vote anyway; not the implementing PR (there review should take care of technical issues).

We have seen several examples where we really want functionality from software that is developed in languages that we don't want to support the build environment for (I think we now have examples of javascript and rust for this), so I think we're going to need a policy along the lines of what you and/or Dima have proposed. What is currently not clear to me is *where* those binary wheels are going to come from and how we're going to test/ensure that they are available for the platforms that we commit to supporting. It would also be good to think about what we're going to do if at some point we find that for one of our supported platforms such wheels are not available. Are we going to just drop that platform? We might have to. Or perhaps it's more advantageous to then "gracefully" reduce functionality and still provide sagemath for that platform; just not with the functionality that relies on the unavailable binary. But then it becomes more like an optional prerequisite for enhancing functionality.

I think the discussion is the important part here: by the looks of it reality is forcing the something like the policy on us, and we can't really vote that away.

[Matthias Koeppe]

On Tuesday, August 20, 2024 at 10:54:22 AM UTC-7 Nils Bruin wrote:

What is currently not clear to me is *where* those binary wheels are going to come from

The upstream project is building them and depositing them on PyPI.

 

and how we're going to test/ensure that they are available for the platforms that we commit to supporting.

In the same way that we test everything else. https://doc.sagemath.org/html/en/developer/portability_testing.html

It would also be good to think about what we're going to do if at some point we find that for one of our supported platforms such wheels are not available. Are we going to just drop that platform? We might have to.

No. The very point of the policy proposed here is that only the *feature* that depends on the binary wheel can be disabled by a configure flag. So the portability of the whole project is not compromised.

 

Or perhaps it's more advantageous to then "gracefully" reduce functionality and still provide sagemath for that platform; just not with the functionality that relies on the unavailable binary.

Yes, that's exactly the point of the policy that I proposed.

 

But then it becomes more like an optional prerequisite for enhancing functionality.

Yes, the only difference between "standard packages that can be disabled" and "optional packages" is what the default is.

 

I think the discussion is the important part here: by the looks of it reality is forcing the something like the policy on us, and we can't really vote that away.

I'll note that the word "we" is doing a lot of the heavy lifting here.

[Dima Pasechnik]

I don't consider this approved, as my complaint about the previous discussions and related proposals and credits due is not addressed, and the technical discussion on how we implement this is not finished.

Dima

[Matthias Koeppe]

On Wednesday, August 21, 2024 at 7:33:26 AM UTC-7 Dima Pasechnik wrote:

I don't consider this approved, as my complaint about the previous discussions and related proposals and credits due is not addressed

I'll note that extraordinary claims such as those that Dima Pasechnik made above in https://groups.google.com/g/sage-devel/c/vFsNl87zLxo/m/YjRMdBUQAQAJ and  https://groups.google.com/g/sage-devel/c/vFsNl87zLxo/m/JxIf6CAQAQAJ, as well as https://github.com/sagemath/sage/issues/38475, need at least a hint of substantiation. Such substantiation has not been provided.

[Nils Bruin]

On the google groups comment linked to by the second reference you give, Dima does link to a very explicit proposal that does share substantial features with the proposed policy change discussed here. So substantiation is definitely provided. In Nathan's response you can actually read his take on the difference between the proposals (just the version pinning, as far as I can tell).

[Dima Pasechnik]

This is just offensive.

Are you suffering from an extraordinary bout of forgetfulness?
Some time ago you proposed to promote pytest and build to standard packages.
I counter-proposed to turn them into pip packages, so that their dependencies don't litter our already overcrowded build/pkgs/

We had a long discussion on the dangers of binary wheels.
You have resurrected my proposal with an addition of hash checking.

Dima

[Matthias Koeppe]

On Wednesday, August 21, 2024 at 1:48:39 PM UTC-7 Nils Bruin wrote:

On the google groups comment linked to by the second reference you give, Dima does link to a very explicit proposal that does share substantial features with the proposed policy change discussed here.

Nils, what are these "substantial features"?

[Dima Pasechnik]

Using binary wheels hosted at PyPI.




>

[Matthias Koeppe]

On Tuesday, August 6, 2024 at 6:17:14 PM UTC-7 Dima Pasechnik wrote:

On Tue, Aug 6, 2024 at 9:45 PM Nathan Dunfield <nat...@dunfield.info> wrote:
> As I understand it, while both proposals result in binary wheels being pulled off PyPI for certain standard packages, they differ in that:
>
> a) The current proposal requires the version of the package to be pinned (with checksums).
Does this mean using `--require-hashes` while calling pip, i.e.
https://pip.pypa.io/en/stable/topics/secure-installs/#hash-checking-mode ? [...]

If anyone is confused about what these thoughts about "--require-hashes" may have to do with my proposal --- that's because there's no relation whatsoever.

[Matthias Koeppe]

Well, Nils? Please be specific.

[Nils Bruin]

Dear Matthias,

Dima has already pointed out a substantial similarity between the proposals.

Kind regards,

Nils

[Matthias Koeppe]

No, Nils, that's not good enough. 

On Friday, August 23, 2024 at 8:50:24 AM UTC-7 Nils Bruin wrote:

Dima has already pointed out a substantial similarity between the proposals.