Modularization project: II. The tools

165 views
Skip to first unread message

Matthias Koeppe

unread,
Jun 12, 2023, 2:13:26 PM6/12/23
to sage-devel
(previous post: I. The goals)

The monorepo. Sage development will continue to use the same one repository sagemath/sage for development of the Sage distribution, the Sage library, and the documentation. And the structure of the source tree (src/sage) will remain unchanged.

Integration testing. The entire Sage library will continue to be tested as a whole, both on Sage developers' machines (make ptest, ./sage -t --all) and on GitHub Actions.

The modularization linter. One of the modularization principles: "Don't import more than you need."  For example, the package sage.misc is split over 5 pip-installable distributions, so importing from sage.misc.all is not a good practice in the Sage libary. The linter ./sage --tox -e relint helps by flagging imports from sage.*.all. 

The import fixer. You can still type "from sage.all import this, that, something_else" when you add code to the Sage library. Just run ./sage --fix-imports on the source file afterwards; it rewrites it using more specific import statements that make the modularization linter happy.

Doctesting against a baseline. When preparing a modularized distribution, at the beginning, when testing it separately, there will be lots and lots of doctest failures. Using ./sage -t --baseline-stats-path known-test-failures.json, we can monitor modularization regressions.

Modularized testing in venvs. We test the modularized distributions using automatically created virtual environments. The technical tool for this is tox, but to run the tests in the Sage distribution, we can just run "make SAGE_CHECK=yes pypi-wheels". 

Automatic modularized testing on GitHub Actions. The test of the modularized distributions that are already defined in Sage is already running on every PR as part of the Build&Test workflow (example). Build failures are flagged as errors, but test failures are currently ignored. (#35095 changes this to using known-test-failures.json.)

The doctest dataflow consistency checker. It checks that "# long time" and "# optional" annotations are used consistently. This is added in #35620.

Wishlist item: A "# optional" re-alignment tool. Currently I am using a crude set of Emacs macros to mass-add "# optional" annotations to doctests and to align them according to the style guide added in #35749. But it would be nice to have a command-line tool that can do this type of reformatting. Also, if we decide to introduce codeblock-scoped "# optional" annotations (#35750), this tool could also rewrite line-level annotations repeated throughout a codeblock to codeblock-scoped annotations.

Information system for the features provided by modularized distributions. Our reference manual already contains sections for each of the SPKGs that are part of Sage-the-distribution. Each modularized distribution (defined in pkgs/) has a corresponding SPKG (defined in build/pkgs/) and thus a section in the reference manual. #35734 improves this by putting these section in relation to Features; and #35749 adds cross-references.

Automatic deployment to PyPI. The modularized distributions are automatically built  and deployed to PyPI on every release.

Matthias Koeppe

unread,
Jun 17, 2023, 12:51:05 PM6/17/23
to sage-devel
Some new Issues / PRs to improve the tools:

Matthias Koeppe

unread,
Jun 23, 2023, 6:42:32 PM6/23/23
to sage-devel
On Monday, June 12, 2023 at 11:13:26 AM UTC-7 Matthias Koeppe wrote:
The monorepo. Sage development will continue to use the same one repository sagemath/sage for development of the Sage distribution, the Sage library, and the documentation. And the structure of the source tree (src/sage) will remain unchanged.

Integration testing. The entire Sage library will continue to be tested as a whole, both on Sage developers' machines (make ptest, ./sage -t --all) and on GitHub Actions.

I'll expand a bit on the role of the monorepo.

The monorepo contains: 
- Configuration and build system of Sage-the-distribution (top-level directories build/, m4/ and files bootstrap, configure.ac, Makefile).
    - as part of it, metadata and scripts for packages (directory build/pkgs/)
- Monolithic source tree (top-level directory src/)
- Python distribution packages (top-level directory pkgs/). (Many more are added in the modularization project.)

Our top-level bootstrap script is in charge of synchronizing these parts. The metadata in build/pkgs/ contains version constraints for dependencies on Python packages. bootstrap fills the version information into templates in pkgs/*/. After running bootstrap, each of the directories in pkgs/ contains the source of a complete Python distribution package, which can be built and installed using standard Python tools. There is no reference to Sage-the-distribution anywhere in these distribution packages.

Among the Python distribution packages, there is a special distribution pkgs/sage-conf, which provides the configuration variable values determined by the top-level configure script. A separate version of this distribution package, pkgs/sage-conf_pypi is published on PyPI; this version runs the configure script by itself and can build the non-Python bits of the Sage distribution.

Message has been deleted

Matthias Koeppe

unread,
Jul 8, 2023, 12:59:50 AM7/8/23
to sage-devel
On Saturday, June 17, 2023 at 9:51:05 AM UTC-7 Matthias Koeppe wrote:
Some new Issues / PRs to improve the tools:

This is now implemented in https://github.com/sagemath/sage/pull/35749:
"Add style guide / reference for # optional - sage.... doctest tags, extend sage -t and sage -fixdoctests for modularization tasks"

Ready for review.
 

Matthias Koeppe

unread,
Aug 1, 2023, 4:24:04 PM8/1/23
to sage-devel
This new tool is available in the latest beta.
Reply all
Reply to author
Forward
0 new messages