Package upgrade PRs waiting for review

[Matthias Koeppe]

build/pkgs/numpy 2.0.1, scipy 1.13.1, cmake 3.30.2, gsl 2.8, meson 1.5.1, meson_python 0.16.0, ninja_build 1.12.1, openblas 0.3.28, pybind11 2.13.1, pythran 0.16.1

build/pkgs/tachyon: Fix rpath

build/pkgs/libpng: Update to 1.6.43

build/pkgs/cypari 2.2.0

build/pkgs/jupyterlab: Update to 4.2.4

build/pkgs/cvxpy 1.5.x

[Tobias Diez]

I've set the first one to needs work since it changes some versions in the pyproject.toml (which specifies the depenendies of sagelib). Since the PR doesn't include any other changes in sagelib, I don't think these stricter version constraints for sagelib make sense and should be reverted.

[Matthias Koeppe]

Since Tobias Diez's PR https://github.com/sagemath/sage/pull/36982 ("Make pyproject.toml the source for build dependencies"; TW: PR comments), merged in Sage 10.4, the version constraints of the packages that happen to be build dependencies of the Sage library (enumerated in https://github.com/sagemath/sage/blob/develop/bootstrap#L36) are set in https://github.com/sagemath/sage/blob/develop/src/pyproject.toml#L3 (and no longer in build/pkgs/*/version_constraints.txt).

The tighter version constraints are needed by the Sage distribution for the reasons that are documented by the comments added in my PR in src/pyproject.toml (https://github.com/sagemath/sage/pull/38227/files).

[Tobia...@gmx.de]

> the version constraints of the packages that happen to be build dependencies of the Sage library (enumerated in https://github.com/sagemath/sage/blob/develop/bootstrap#L36) are set in https://github.com/sagemath/sage/blob/develop/src/pyproject.toml#L3

Right (adhering to the standard of modern python libs)

> The tighter version constraints are needed by the Sage distribution

Then please specify these version constraints in sage distribution and not in sage-lib (eg by providing a version_constraint.txt file that is compatible with the version constraints of sage-lib and the tighter constraints set by other packages).

[Matthias Koeppe]

On Wednesday, August 21, 2024 at 2:12:30 PM UTC-7 Tobia...@gmx.de wrote:

> the version constraints of the packages that happen to be build dependencies of the Sage library (enumerated in https://github.com/sagemath/sage/blob/develop/bootstrap#L36) are set in https://github.com/sagemath/sage/blob/develop/src/pyproject.toml#L3

Right (adhering to the standard of modern python libs)

> The tighter version constraints are needed by the Sage distribution

Then please specify these version constraints in sage distribution and not in sage-lib (eg by providing a version_constraint.txt file that is compatible with the version constraints of sage-lib and the tighter constraints set by other packages).

No, I will not start maintaining two sets of constraints.

[Nils Bruin]

I don't think *you* would need to maintain that set. It would just be something to maintained. If Tobias argues convincingly that there is benefit to maintaining a different set of constraints, he'll probably be able to find people (himself maybe?) who are willing to maintain such constraints. If at all possible, no maintenance task should fall to any single person.

[Kwankyu Lee]

> The tighter version constraints are needed by the Sage distribution

Then please specify these version constraints in sage distribution and not in sage-lib (eg by providing a version_constraint.txt file that is compatible with the version constraints of sage-lib and the tighter constraints set by other packages).

Simply recovering the files build/pkgs/*/version_constraints.txt will do it? 

[Matthias Koeppe]

No. The list of packages involved is hardcoded in several places by the changes made in  https://github.com/sagemath/sage/pull/36982.

(I'll note that my revision of the implementation of this mechanism from https://github.com/sagemath/sage/pull/36982, which fixes this and other flaws, is waiting for review in https://github.com/sagemath/sage/pull/37902)

[Tobia...@gmx.de]

People interested in maintaining sagelib should only care about the version constraint in the pyproject.toml file (e.g. if they decide that a new feature of numpy is needed, then the version constraint of numpy should be changed accordingly). People interested in maintaining sage-the-distro should maintain sage-the-distro. If they (for some reason) need a tighter version constraint than sagelib, then it of course falls in their responsibility to maintain this constraint. This is similar to other distributions that are maintaining a consistent set of package versions (e.g. the conda lock files in the main repo, or linux distros). But obviously these are activities downstream of sage-the-library.

> Simply recovering the files build/pkgs/*/version_constraints.txt will do it? 

Yes, that should work with some trivial changes in the bootstrap script (where currently the corresponding version constraint file is generated).

[Matthias Koeppe]

On Thursday, August 22, 2024 at 2:41:57 AM UTC-7 Tobia...@gmx.de wrote:

People interested in maintaining sagelib should only care about the version constraint in the pyproject.toml file (e.g. if they decide that a new feature of numpy is needed, then the version constraint of numpy should be changed accordingly).

That's all correct, but it does not depend on the pyproject.toml file declaring the widest possible version range.

[Matthias Koeppe]

On Wednesday, August 21, 2024 at 3:35:57 PM UTC-7 Nils Bruin wrote:

I don't think *you* would need to maintain that set. It would just be something to maintained.

:)

 

[Tobia...@gmx.de]

Why would one want to specify a more narrow version range? That would only impose unnecessary constraints on downstream packaging efforts.

Another example: https://github.com/sagemath/sage/pull/38556

[Matthias Koeppe]

On Friday, August 23, 2024 at 1:21:47 PM UTC-7 Tobia...@gmx.de wrote:

Why would one want to specify a more narrow version range?

It's not a goal, it's a tradeoff.

[Dima Pasechnik]

No, it is Sage the distro getting in the way of sagelib.

[Tobia...@gmx.de]

@Kwankyu, replying here to your comments.

The cython constraint in scipy is for build-time. Sagelib only cares about the fact that scipy is installed in the same venv, not how you install or build it. In fact, pip uses build isolation by default, or you may install scipy using the prebuilt wheel from pypi, or you use conda. In neither of these cases, would sagelib ever even see the cython used to build scipy. It's a sage-the-distro specific problem, since this distribution insists that you need to be able to compile sagelib with the same cython version as scipy.

Moreover, there is actually a very simple way to specify a different constraint for sage-the-distro: just remove the auto-gen of the version-requirements file, and provide an own one for sage-the-distro.

[Kwankyu Lee]

On Friday, September 6, 2024 at 3:00:47 PM UTC+9 Tobia...@gmx.de wrote:

The cython constraint in scipy is for build-time. Sagelib only cares about the fact that scipy is installed in the same venv, not how you install or build it. In fact, pip uses build isolation by default, or you may install scipy using the prebuilt wheel from pypi, or you use conda. In neither of these cases, would sagelib ever even see the cython used to build scipy. It's a sage-the-distro specific problem, since this distribution insists that you need to be able to compile sagelib with the same cython version as scipy.

I see. Thanks.

 

Moreover, there is actually a very simple way to specify a different constraint for sage-the-distro: just remove the auto-gen of the version-requirements file, and provide an own one for sage-the-distro.

That way was discussed in https://github.com/sagemath/sage/pull/37902#discussion_r1744898150.

[Tobia...@gmx.de]

Moreover, there is actually a very simple way to specify a different constraint for sage-the-distro: just remove the auto-gen of the version-requirements file, and provide an own one for sage-the-distro.

That way was discussed in https://github.com/sagemath/sage/pull/37902#discussion_r1744898150.

I agree with your viewpoint. How to specify version constraints for certain modularized distros, however, is another topic that I don't want to open.

I've set the PR back to needs work since the version constraints in pyproject.toml for cython and numpy are not needed in my testing, definitely not for sage-the-lib but apparently not even for sage-the-distro (at least I didn't get any build errors without them).

[Matthias Koeppe]

On Friday, September 6, 2024 at 1:43:32 AM UTC-7 Tobia...@gmx.de wrote:

I've set the PR back to needs work since the version constraints in pyproject.toml for cython and numpy are not needed in my testing, definitely not for sage-the-lib but apparently not even for sage-the-distro (at least I didn't get any build errors without them).

The only place where these version constraints are used in sage-the-distribution is in the experimental "configure --enable-system-site-packages" mode. 

You don't say what you tested, and in any case such vague indications are not sufficient basis for holding up an upgrade PR.

[Tobia...@gmx.de]

The PR does not include any explanation as to why the changes to pyproject.toml are necessary, nor does it provide any guidance on how to test them. I used the standard configure process (now also with the system-site-packages switch) followed by make, both with and without these changes, and observed no visible differences. Could you please clarify how to observe the impact of the 'numpy >=1.22.4' constraint? Additionally, the comment preceding it is quite confusing, especially since this file actually lists "numpy>=2.0.0rc1,<2.3".

If the only apparent effect of these constraints is related to the system-site-package feature, I would appreciate feedback from the developers working on this feature (Dima?, Michael?) regarding how to specify these constraints now that sage-lib defines its own constraints in pyproject.toml.

To avoid further delaying the package upgrades in this PR, I am okay with reverting the changes to pyproject.toml for now and revisiting them once the discussion is concluded.

[Matthias Koeppe]

On Monday, September 9, 2024 at 6:08:36 AM UTC-7 Tobia...@gmx.de wrote:

The PR does not include any explanation as to why the changes to pyproject.toml are necessary, nor does it provide any guidance on how to test them.

Yes, it does not, because it's the same for every package upgrade PR. We update version constraints when needed.

I used the standard configure process (now also with the system-site-packages switch) followed by make, both with and without these changes, and observed no visible differences. Could you please clarify how to observe the impact of the 'numpy >=1.22.4' constraint? Additionally, the comment preceding it is quite confusing, especially since this file actually lists "numpy>=2.0.0rc1,<2.3".

If the only apparent effect of these constraints is related to the system-site-package feature,

Yes, that's exactly what they do. (If one ignores the modularized distributions.)

And it's a deliberate design decision that we made when the system-site-package feature was created: Because of the scarcity of developers working on it, we want to maintain only a single set of version constraints for synergy.

But note that the system-site-package feature is still experimental, and very little effort has been put in to making it work on distributions other than perhaps two developers' personal systems. I built the CI infrastructure to test it systematically. As you can see in the CI run for the latest beta  https://github.com/sagemath/sage/actions/runs/10873724362 (section "standard-sitepackages"), there are numerous failures. If this was in better shape, we could simply test upgrade PRs using the CI. 

I would appreciate feedback from the developers working on this feature (Dima?, Michael?) regarding how to specify these constraints now that sage-lib defines its own constraints in pyproject.toml.

The change is purely syntactic -- that's all that your PR (and the subsequent fixups of it) did. There has been no semantic change whatsoever. Previously all version constraints were set centrally in the build/pkgs/*/install-requires.txt files (recently renamed to version_requirements.txt), now some are set in src/pyproject.toml. That's all.

To avoid further delaying the package upgrades in this PR, I am okay with reverting the changes to pyproject.toml for now and revisiting them once the discussion is concluded.

No, that's not how it works. Scope matters! This discussion is clearly out of scope of the upgrade PR.

[Michael Orlitzky]

On Wed, 2024-09-18 at 16:23 -0700, Matthias Koeppe wrote:
>
> I would appreciate feedback from the developers working on this feature
> (Dima?, Michael?) regarding how to specify these constraints now that
> sage-lib defines its own constraints in pyproject.toml.
>
> The change is purely syntactic -- that's all that your PR (and the
> subsequent fixups of it) did. There has been no semantic change whatsoever.
> Previously all version constraints were set centrally in the
> build/pkgs/*/install-requires.txt files (recently renamed to
> version_requirements.txt), now some are set in src/pyproject.toml. That's
> all.

I don't know what happened, but a lot of the messages that I receive
from sage-devel now have no quoting. (One of the paragraphs above is
NOT mkoeppe). The quoting is there in HTML, but not in the plain-text
version that I read.

If I am failing to respond to questions, it's because I secretly have
no idea what's going on in 75% of these conversations any more.

[Dima Pasechnik]

Yes, I see this too. Another "improvement" of groups by GOOG.

[Tobia...@gmx.de]

Michael, to make things short: The discussion is around these changes to the `pyproject.toml` file https://github.com/sagemath/sage/pull/38227/files#diff-0acbedc0b174ebec342997fdca9442a5486917ab04644da866213e97d2543604 which he claims harmonize the constraints with scipy (although scipy actually declares a different numpy constraint). I'd argue that these are incorrect as build-time requirements of scipy should not influence what build-time requirements we declare for sage-the-lib. Matthias claims these are needed for the system-site-package feature.

The claim by Matthias that such version constraint updates are standard procedure seems to be invalid as well. Currently, the constraints for cython and numpy are not synced with the built-time requirements of scipy 1.12 and the last upgrade PR of scipy (only a couple of months ago) also didn't touch any of these versions https://github.com/sagemath/sage/pull/37266.

[Michael Orlitzky]

On Sun, 2024-09-22 at 20:05 -0700, 'Tobia...@gmx.de' via sage-devel
wrote:

> Michael, to make things short: The discussion is around these changes to
> the `pyproject.toml` file
> https://github.com/sagemath/sage/pull/38227/files#diff-0acbedc0b174ebec342997fdca9442a5486917ab04644da866213e97d2543604
> which he claims harmonize the constraints with scipy (although scipy
> actually declares a different numpy constraint). I'd argue that these are
> incorrect as build-time requirements of scipy should not influence what
> build-time requirements we declare for sage-the-lib. Matthias claims these
> are needed for the system-site-package feature.
>

--enable-system-site-packages doesn't care about pyproject.toml

As I probably argued when it was introduced, I think the pyproject.toml
for sagelib should contain the actual dependencies for sagelib, because
that's what it's for and that's what everyone expects and it makes life
easier for end users and packagers.

[Tobia...@gmx.de]

Thanks, that's helpful!

[Matthias Koeppe]

On Monday, September 23, 2024 at 5:34:26 AM UTC-7 Michael Orlitzky wrote:

On Sun, 2024-09-22 at 20:05 -0700, 'Tobia...@gmx.de' via sage-devel
wrote:
> Matthias claims these
> are needed for the system-site-package feature.

--enable-system-site-packages doesn't care about pyproject.toml

No, Michael, that's no longer true.

Since Tobias's PR https://github.com/sagemath/sage/pull/36982, at bootstrapping time, some of the version_requirements.txt files are generated from the src/pyproject.toml file. 

(This is also documented in https://doc.sagemath.org/html/en/developer/packaging.html#abstract-requirements-the-version-requirements-txt-file)

[Michael Orlitzky]

On 2024-09-25 11:55:36, Matthias Koeppe wrote:
> On Monday, September 23, 2024 at 5:34:26 AM UTC-7 Michael Orlitzky wrote:
>
> On Sun, 2024-09-22 at 20:05 -0700, 'Tobia...@gmx.de' via sage-devel
> wrote:
> > Matthias claims these
> > are needed for the system-site-package feature.
>
> --enable-system-site-packages doesn't care about pyproject.toml
>
> No, Michael, that's no longer true.
>
> Since Tobias's PR https://github.com/sagemath/sage/pull/36982, at
> bootstrapping time, some of the version_requirements.txt files are
> generated from the src/pyproject.toml file.

It still doesn't care what's in pyproject.toml. If pyproject.toml is
used to put inaccurate constraints in version_requirements.txt, that's
a separate issue. My opinion on this will be pretty consistent: the
version_requirements.txt for the sage distribution should contain the
version requirements for the sage distribution. That's what they're
for, and having accurate dependency information makes life better for
end users.

[Matthias Koeppe]

On Wednesday, September 25, 2024 at 7:01:54 PM UTC-7 Michael Orlitzky wrote:

the
version_requirements.txt for the sage distribution should contain the
version requirements for the sage distribution. That's what they're
for

That's right, and that's indeed what the update PR https://github.com/sagemath/sage/pull/38227 does (via bootstrapping-time generation from src/pyproject.toml).