> What is the modularization project? The Sage developer community has long been aware of the severe problems that the monolithic design of Sage has brought. See in particular the lively 2016 sage-devel thread "How we develop Sage" (https://groups.google.com/g/sage-devel/c/29ndCD8z94k) initiated by William.I looked at a few of the messages in this thread (there are something like hundred messages, I feel unable to read them all). I would like to make two remarks* 26 persons participated in this thread, roughly 10 of these are still active* many of the messages were quite critical regarding modularization of the math library in sage* it seems to me that at least a big part of the thread is about modularization of the *infrastructure* of sage, not the math library.In any case, I do not see consensus in this thread.
2.) If this is about dependencies on other software, why aren't the distributions named after these dependencies?
On Friday, April 19, 2024 at 5:08:02 AM UTC-7 Martin R wrote:2.) If this is about dependencies on other software, why aren't the distributions named after these dependencies?Martin, I have answered this already when you asked it in the PR: Some are.
Yes in a perfect world, but then you don't get a gold star for satisfying some purity test. We should just do the minimal amount of work to get us where we want to be. Lets focus on the direction to go and not too much on the process.
On Friday, April 19, 2024 at 7:18:03 PM UTC+2 Michael Orlitzky wrote:On Fri, 2024-04-19 at 09:46 -0700, Matthias Koeppe wrote:
>
> Michael, note that in my message I asked for a vote on that dependency
> https://github.com/sagemath/sage/pull/36676.
>
Even if 36676 gets approval, 36964 must be reverted. It was not
meaningfully voted upon.
--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/b9d41da8-57ec-4542-a5d8-7f2690849a49n%40googlegroups.com.
why do you introduce distributions sage-graphs, sage-combinat, sage-categories etc.
do I understand correctly that common lisp (via maxima) is the main dependency that prevents sagemath from being pip-installable?
By choosing to be an exception in the Python world,
Sage obviously does something quite wrong.
"normal Python" is not necessarily as relevant for those who would *only* want Sage, or at least mostly so. Having just another Python package might lead us to implementing powers as ** instead of ^, which would be a regression, or needing namespaces for almost everything, which again limits the value of Sage qua Sage.
in its current incarnation, the modularization relies
heavily on the sage distribution vendoring. Conflict arises because the
modularization is cited as a blocker whenever someone wants to pare
down or disentangle some aspect of the sage distribution. It's not the
modularization per se that we object to. Personally, I would like it to
succeed, but not at the expense of everything else
Can someone who is not Dima or Matthias explain to us how it is possible that they both are claiming to represent the normal Python way of doing things? There have been numerous statements by both of them about this, which makes it sound like there are two pieces to it (modularization but also "de-vendoring"), and I can only assume that it would be possible for both to occur simultaneously. It would be helpful for this to be clarified, though, so that one knows precisely what *piece* of their proposals represent the "normal Python ecosystem.
For the statements in this thread, I don't see any contradictions about the definition of the "normal Python way of doing things". My understanding of that term is to post self-contained binary wheels to PyPI for all supported platforms that install in a minute or two with no compilation (as Dima wrote), as well as a source-code only package that serves as a rarely-used backup if no suitable binary is available. (The self-contained bit is important; for example, on Linux here are the only external libraries a binary wheel is allowed to link to.)
I can imagine that it would make sense to make as much as possible into runtime dependencies - you wrote below that building the dependencies takes a lot of time. Maybe that's the core problem, I don't know.
Anyway, a normal Python pypi-installable package comes with binary wheels, i e. things are pre-built, and it's merely matter of downloading these to get a functional package. Few minutes on a fast network, not hours.
By choosing to be an exception in the Python world,
Sage obviously does something quite wrong.
Why would you separate mathematics into packages that have no more external dependencies from others, which at the same time may grow internal dependencies over time?
On 20 April 2024 19:34:49 BST, Matthias Koeppe <matthia...@gmail.com> wrote:
SageMath is already pip-installable.
>That was one of the first deliverables of the modularization project,
>completed in 2021.
>See https://wiki.sagemath.org/ReleaseTours/sage-9.4#Alternative_installation_methods_using_pip
>
>It's documented in our README:
>https://github.com/sagemath/sage/blob/develop/README.md#alternative-installation-using-pypi
>
>However, installing Sage in this way is equivalent to installing the full
>Sage distribution from source in the normal way.
No, it is not equivalent, far from it. One can read on
<https://pypi.org/project/sagemath-standard/#description>
Building sagemath-standard has a large number of system packages as prerequisites. See https://doc.sagemath.org/html/en/installation/source.html#linux-recommended-installation for partial lists for various systems.
That is, many packages are not built. What's being built is not a Sage distribution, it is sagelib.
>If you do allow me another example from discrete math: Sage has a lot of
>very good code in "sage.graphs" that provides functionality that is not
>available from other Python libraries.
>
>But to potential projects that would need this functionality, the
>proposition to have to depend on the monolithic project SageMath, with
>hours of compilation time and/or dependencies on system packages that are
>obviously unrelated to the needed functionality (boost, brial, ecl, eclib,
>ecm, fflas_ffpack, flint, libgd, gap, giac, givaro, glpk, gmp, gsl, iml,
>lcalc, libbraiding, libhomfly, libpng, linbox, m4ri, m4rie, mpc, mpfi,
>mpfr, ntl, pari, rw, singular, symmetrica), is simply a showstopper.
This is not true, either. First, a lot of these packages that you list are available on systems, and thus there is no need to build them.
(To "but macOS?" I have a reply: "ask Apple to provide you a package manager, it's the best OS, right, you pay for it a lot of money, or learn to use Homebrew like Mac users usually do...")
Anyway, a normal Python pypi-installable package comes with binary wheels, i e. things are pre-built, and it's merely matter of downloading these to get a functional package. Few minutes on a fast network, not hours.
By choosing to be an exception in the Python world,
Sage obviously does something quite wrong.
By choosing to be an exception in the Python world,
Sage obviously does something quite wrong.
Can someone who is not Dima or Matthias explain to us how it is possible that they both are claiming to represent the normal Python way of doing things? There have been numerous statements by both of them about this, which makes it sound like there are two pieces to it (modularization but also "de-vendoring"), ...
>That should not happen. Modularization is downstream to the sage library. Yes, we are restructuring some parts of the sage library to fit with modularization. But modularization should never be an obstacle in developing the sage library. If it ever be, the sage community might drop the modularization project.
>> My fear would be that at some point there is a request not to use symbolics in some module, because Lisp is hard to install on some system.>That should not happen. Modularization is downstream to the sage library. Yes, we are restructuring some parts of the sage library to fit with modularization. But modularization should never be an obstacle in developing the sage library. If it ever be, the sage community might drop the modularization project.
We already witness multiple instances where the modularization project is cited as a reason not to merge certain PRs that only touch sage-the-library. How does this reality fit this view?
On Sunday, April 21, 2024 at 2:30:15 AM UTC-7 Martin R wrote:Why would you separate mathematics into packages that have no more external dependencies from others, which at the same time may grow internal dependencies over time?Let's just go through the list of distribution packages and their dependencies for concreteness. (All depend on sagemath-categories and thus on the basic arithmetic libraries gmp, mpc, mpfr, gmpy2)
sagemath-combinat- non-Python dependency: symmetrica (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-combinat/pyproject.toml.m4#L65)
sagemath-graphs
sagemath-modules- non-Python dependency: gsl (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-modules/pyproject.toml.m4#L109)- Python build requirement: numpy
sagemath-groups- non-Python dependency (via sagemath-gap): gap (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-gap/pyproject.toml.m4#L52)- non-Python dependency (via sagemath-modules): gsl (see above)sagemath-polyhedra- non-Python dependency (via sagemath-glpk): glpk- non-Python dependency (via sagemath-modules): gsl (see above)- Python runtime dependency: pplpysagemath-schemes- non-Python dependency (via sagemath-modules): gsl (see above)- non-Python dependencies (via sagemath-singular, sagemath-flint, sagemath-ntl, sagemath-pari): singular, flint, ntl, pari (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-pari/pyproject.toml.m4#L61)- Python dependency (via sagemath-singular, sagemath-flint, sagemath-pari): cypari2- Python dependency: scipysagemath-symbolics- non-Python dependencies: ecl, maxima, singular https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-symbolics/pyproject.toml.m4#L89- non-Python dependencies (via sagemath-flint, sagemath-ntl, sagemath-modules): flint, ntl, pari, gsl- Python dependencies: mpmath, sympy, cypari2, numpy
--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/41d11d3f-d5e5-4bf5-9629-2ef17ce4d6b1n%40googlegroups.com.
In reply to the comment (https://github.com/sagemath/sage/pull/36676#issuecomment-2067371677)
once the modularization is in place, it will impose wide-ranging constraints on what functions from other modules you are able to use. Currently, you can use `sage.x` in `sage.y` for an arbitrary combination of `x` and `y`. The modularization project restricts this severely and you will only be able to use modules `x` that are declared as a dependency of `y` (the arrows in the picture in https://github.com/sagemath/sage/pull/35095).
On Sun, Apr 21, 2024 at 10:42 PM Matthias Koeppe <matthia...@gmail.com> wrote:Let's just go through the list of distribution packages and their dependencies for concreteness. (All depend on sagemath-categories and thus on the basic arithmetic libraries gmp, mpc, mpfr, gmpy2)these are seemingly incomplete:
sagemath-combinat- non-Python dependency: symmetrica (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-combinat/pyproject.toml.m4#L65)non-python: depends on GAP, givaro, too
sagemath-graphsnon-python: depends on GAP, givaro, too
sagemath-modules- non-Python dependency: gsl (https://github.com/mkoeppe/sage/blob/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-modules/pyproject.toml.m4#L109)- Python build requirement: numpynon-python: linbox, flas_ffpac, too ?
I meant the sage library as a collection of mathematical modules. If a certain module did not but somehow would develop to rely on the mathematical functionality of another module, then the design of the modularization should embrace the development. The splitting of the sage library in the present modularization reflects the present reality of the separation of different mathematical parts of the sage library. But of course the reality may change in the future as we develop the sage library. Then the modularization should reflect the change, not block the change.
I still don't see why you would name these distributions as you do, and why you collect them as you do.
For example, as far as I know, symmetrica is currently essentially only used by symmetric functions, Schubert polynomials. So, if symmetrica is such a burden to install, why would you want to install it if you don't need them?
On the other hand, combinatorial species will, after this summer, heavily depend on gap. Does this mean that you would want to move this part into sage-groups?
Apart from the strange naming, I don't think that it will make anybody happy if dependencies change frequently.
If I understand correctly, the current proposal does not mind if some things don't work or could be replaced without too much effort. For example, Dima might have referred to the fact that OrderedPartitions.cardinality uses gap, even though it is in sagemath-combinat.The gap dependency in `designs.database` (which is in sagemath-graphs) and `matrices.latin` (which is in sagemath-combinat) might have been overlooked.
in
src/sage/combinat//designs/block_design.py
you can see
lazy_import('sage.libs.gap.libgap', 'libgap')
lazy_import('sage.rings.finite_rings.finite_field_constructor', 'FiniteField')
fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 65) lazy_import('sage.libs.gap.libgap', 'libgap')
fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 66) lazy_import('sage.matrix.matrix_space', 'MatrixSpace')
fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 67) lazy_import('sage.modules.free_module', 'VectorSpace')
fdbe7f7e3348 (Matthias Koeppe 2023-07-12 10:53:08 -0700 68) lazy_import('sage.rings.finite_rings.finite_field_constructor', 'FiniteField')
What you see there is the result of work in the modularization project, using one of the techniques documented in https://doc.sagemath.org/html/en/developer/packaging_sage_library.html#dependencies-and-distribution-packages ("Reducing module-level run-time dependencies:") to reduce dependencies or to make dependencies milder.
Wouldn't people in the python world who need a serious amount of math know of sage anyway, and then, if they cannot rely on all of sage because that is too large, use, for example, `citation.get_systems` to see whether they can do without some dependencies?
--
You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/mqgtkLr2gXY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAHVvXxQUanPY%3D1svhf7Q8xDFD5BroD9wTLRc1-wmFC3vQhBMRg%40mail.gmail.com.
Presumably though a hypothetical person who wants CyPari2 but not all of Sage can already just use CyPari so that person is already well served.
I am wondering how representative the CyPari case is compared to other parts of Sage.
Thanks Marc. This seems like a good example of a useful part of Sage
that can be extracted to something much smaller than Sage.
Presumably though a hypothetical person who wants CyPari2 but not all
of Sage can already just use CyPari so that person is already well
served.
Is the plan that CyPari2 would effectively replace CyPari? Then the benefit is not needing to maintain CyPari separately from
sagelib's CyPari2 dependency?
On Wed, Apr 24, 2024 at 9:29 PM Nathan Dunfield wrote:
> On a related note, the reason that CyPari2 and CyPari are still separate relates to what Marc mentioned earlier about tension between two models of installing software: the "Linux distro way" using a system-level package manager (where there is typically only one version of any given library on the system) and the "Python pip" way (where all needed libraries are statically linked into the wheel). CyPari2 follows the first, CyPari the second. (This story is further complicated by the fact that, from a user's perspective, conda is like "Python pip" in that it is orthogonal to any system libraries, but developing packages for conda is akin to building them for a Linux distro.)
There isn't a big problem to set up a GitHub wheel builder for
CyPari2, so there is not really a sea of difference here in this
sense.
Also, probably it should be mentioned that linux distro way/pip way
can be very nicely combined using a python venv.
So one can use system packages, but add up more packages, and, if
needed, override system packages with other versions.
You mentioned several times, that discoverability is an important aspect. Do you have any evidence to support that?
Wouldn't people in the python world who need a serious amount of math know of sage anyway,
and then, if they cannot rely on all of sage because that is too large, use, for example, `citation.get_systems` to see whether they can do without some dependencies?
[...] I admit, however, that I cannot really think of any serious use of any of the functionality of sage outside of math, where people would know that what they need is, say combinatorics or graph theory.
I guess it would be quite insane to install sage or parts of sage to get an implementation of a particular algorithm in a non-math environment.
Is the benefit in this case mainly about reduced disk/network usage?
I could imagine other theoretical benefits like maybe some parts could
be installed natively on Windows or some parts might be easier to
provide binaries for etc.
On Wednesday, April 24, 2024 at 1:07:44 AM UTC-7 Martin R wrote:You mentioned several times, that discoverability is an important aspect. Do you have any evidence to support that?I mentioned "discoverability" in the context of how I have named the distributions.
Wouldn't people in the python world who need a serious amount of math know of sage anyway,What is a "serious" amount of math?
and then, if they cannot rely on all of sage because that is too large, use, for example, `citation.get_systems` to see whether they can do without some dependencies?What do you mean by, "whether they can do without some dependencies"?
That's exactly the point of the modularization:- To enable people to use parts of Sage without some [actually, most!] of the dependencies.
[...] I admit, however, that I cannot really think of any serious use of any of the functionality of sage outside of math, where people would know that what they need is, say combinatorics or graph theory.
What do you mean by "outside of math"?
I guess it would be quite insane to install sage or parts of sage to get an implementation of a particular algorithm in a non-math environment.Yes, "quite insane" would be a good description of doing this (reusing what is implemented in Sage) with the status quo, the monolithic Sage.
That's exactly the point.- From the viewpoint of users: It just makes no sense to use a small part of Sage: Because of space and time and friction (and limited portability).
- From the viewpoint of contributors of new code that could be reused: It just makes no sense to trap it in Sage, where it cannot be reused sanely.
Modularization makes it at least "less insane", or perhaps "kind of acceptable", or sometimes even "reasonable".Any of these degrees of improvement of "sanity" would already be a win.
Yes, native Windows would clearly be a very important target.
In another direction: I have started a port of Sage to pyodide, the distribution of Python for WebAssembly (WASM), which makes Python runnable directly in the browser. I can already run and test the modularized distributions **sagemath-objects**, **sagemath-categories** there.
I agree that my terminology is not good. I tried to make a distinction between research involving math and the - to me unknown - rest. I find it hard to imagine that any mathematician would bother installing anything else but all of sage.
Another example is large-scale pure math computation on clusters. Because of Sage's size and the nature of distributive file systems, the time to startup Sage can be 30 seconds or more, which complicates things if you want to do 100,000 calculations that are only 10 seconds each.
On Thu, Apr 25, 2024 at 1:28 PM Nathan Dunfield <nat...@dunfield.info> wrote:
>
> On Wednesday, April 24, 2024 at 10:14:09 PM UTC-5 Matthias Koeppe wrote:
>
> Yes, native Windows would clearly be a very important target.
>
>
> As a data point, downloads of our stand-alone SnapPy app, which is about as high level pure math as it gets, are 60% higher for Windows than macOS.
That's not for native Windows, that's for WSL, right?
Essential components of sagelib such as GAP, Singular, don't run on
native Windows (on Cygwin, yes, but
we know by now, Cygwin is too flaky for Sage to work) and I don't
think anyone is keen on
doing the hard work to port it.
Essential components of sagelib such as GAP, Singular, don't run on
native Windows
In another direction: I have started a port of Sage to pyodide, the distribution of Python for WebAssembly (WASM), which makes Python runnable directly in the browser. I can already run and test the modularized distributions **sagemath-objects**, **sagemath-categories** there.
It would be amazing if a decent portion of Sage could be run in the browser this way, e.g. to have the occasional HW assignment that needs Sage without the overhead of using something like CoCalc.
Although SageMathCell
does not run locally, it does run in the browser. There are
examples of Sage exercises in this book and
more on the about
page of SageMathCell. Having a completely offline version of
parts of Sage that can run in the browser with WASM will be
wonderful indeed.
Regards,
TB
On Thu, Apr 25, 2024 at 4:36 PM <dim...@gmail.com> wrote:
you might also like to know that in 2000 I asked whether we can have libgap :P
https://www.gap-system.org/ForumArchive/Pasechni.1/Dmitrii.1/using_GA.1/1.htmlIt
> On Wednesday, April 24, 2024 at 10:14:09 PM UTC-5 Matthias Koeppe wrote:
> Yes, native Windows would clearly be a very important target.
Essential components of sagelib such as GAP, Singular, don't run on native Windows (on Cygwin, yes [...]) and I don't think anyone is keen on doing the hard work to port it. This puts native Windows support into the area of wishful thinking.
On 25/04/2024 15:28, Nathan Dunfield wrote:
In another direction: I have started a port of Sage to pyodide, the distribution of Python for WebAssembly (WASM), which makes Python runnable directly in the browser. I can already run and test the modularized distributions **sagemath-objects**, **sagemath-categories** there.It would be amazing if a decent portion of Sage could be run in the browser this way, e.g. to have the occasional HW assignment that needs Sage without the overhead of using something like CoCalc.
Although SageMathCell does not run locally, it does run in the browser. There are examples of Sage exercises in this book and more on the about page of SageMathCell. Having a completely offline version of parts of Sage that can run in the browser with WASM will be wonderful indeed.
Another example is large-scale pure math computation on clusters. Because of Sage's size and the nature of distributive file systems, the time to startup Sage can be 30 seconds or more, which complicates things if you want to do 100,000 calculations that are only 10 seconds each. I was recently at a workshop on computational topology, and several researchers there regarded using Sage in this context as a non-starter, in one case they were completely changing their approach to avoid using Sage.
On Thursday 25 April 2024 at 05:13:37 UTC+2 Matthias Koeppe wrote:On Wednesday, April 24, 2024 at 1:07:44 AM UTC-7 Martin R wrote:You mentioned several times, that discoverability is an important aspect. Do you have any evidence to support that?I mentioned "discoverability" in the context of how I have named the distributions.Sorry that my question was not clear enough. Do you have evidence, that this naming enhances discoverability, and that this enhanced discoverability would be worthwhile, since it comes with a cost (as outlined above)?
Wouldn't people in the python world who need a serious amount of math know of sage anyway,What is a "serious" amount of math?You know it when you see it.
What I mean is, roughly, that it certainly does not make sense to use sage as a package if you need a few graph algorithms (like shortest paths or some such), because then you'd be better served with using a specialised library
or perhaps copy the code and adapt it.
You might want to use sage as a package, if you want to do serious enumeration of arbitrary combinatorial objects. But then you will need gap, symmetrica, nauty and maxima or fricas anyway.
and then, if they cannot rely on all of sage because that is too large, use, for example, `citation.get_systems` to see whether they can do without some dependencies?What do you mean by, "whether they can do without some dependencies"?That's exactly the point of the modularization:- To enable people to use parts of Sage without some [actually, most!] of the dependencies.The only point I am critical of is the splitting of the math library into arbitrary pseudo-mathematical parts (i.e., sage-combinat ... sage-symbolics, as listed above), which has nothing to do with dependencies.
[...] I admit, however, that I cannot really think of any serious use of any of the functionality of sage outside of math, where people would know that what they need is, say combinatorics or graph theory.What do you mean by "outside of math"?
I agree that my terminology is not good. I tried to make a distinction between research involving math and the - to me unknown - rest. I find it hard to imagine that any mathematician would bother installing anything else but all of sage.
So, maybe I should have written: "I cannot think of .... sage in an environment, where dependencies are show-stopper, ...".