Question about the sage venv.

[Marc Culler]

I have a very narrow techical question.  I am hoping for a narrow technical answer, preferably from someone who was involved in designing the Sage venv.

QUESTION: Why doesn't the Sage venv contain a pyvenv.cfg file?

When you read PEP 405 you learn that the pyvenv.cfg file is what makes a directory be a venv.  For example:

"Thus, a Python virtual environment in its simplest form would consist of nothing more than a copy or symlink of the Python binary accompanied by a pyvenv.cfg file and a site-packages directory."

In addition to identifying the directory as being a venv, the pyvenv.cfg file controls how the sys.prefix is configured when the python binary in the venv starts up.

- Marc

[Matthias Koeppe]

On Thursday, December 7, 2023 at 10:04:14 AM UTC-8 Marc Culler wrote:

I have a very narrow techical question.  I am hoping for a narrow technical answer, preferably from someone who was involved in designing the Sage venv.

QUESTION: Why doesn't the Sage venv contain a pyvenv.cfg file?

It does whenever it's actually a venv in the technical sense (which you explain correctly).

If system python gets used, then the Sage distribution creates an actual venv over it. So there's a pyvenv.cfg.

But if a python is built by the Sage distribution, then it is merely an installation tree of that python. In this case, there's no pyvenv.cfg.

 

[Marc Culler]

Thank you, Matthias.

Now I have a followup question, which is the tip of an iceberg.  The iceberg consists of my thoughts about how to simplify and improve the organization of files in Sage.  I am not asking for a yes or no answer, or a list of possible problems. I am just trying to find out whether people would be willing to consider an idea.

Would it be possible for the sage build process to use sage/local as the unique "system location"?

The sage/local directory would be built by one stage of the sage build process, preceding the stage that builds the venv.  Inside sage/local one would find either files built by the sage build system, or symlinks to files in other locations on the host system, such as /usr/local, or /opt/local, or a conda directory.  The question of whether to use a symlink for X would be the same as the question of whether to use "a system-X".  The venv would always contain a pyvenv.cfg file.  Within the venv every python extension module which uses other libraries would have its rpath pointing inside of sage/local, possibly to a real library or possibly to a symlink. Binary packages assembled by delocate would be exceptions because they already set the rpaths, to a subdirectory of the python package named .dylibs.

- Marc

[Matthias Koeppe]

On Thursday, December 7, 2023 at 11:06:51 AM UTC-8 Marc Culler wrote:

Would it be possible for the sage build process to use sage/local as the unique "system location"?

The sage/local directory would be built by one stage of the sage build process, preceding the stage that builds the venv.

That's what `make all-build-local` (or `make all-sage-local`) do.

We already use this separation of phases in our portability CI (CI Linux, CI macOS).

For Linux, the workflow jobs named "standard-pre" and "minimal-pre" build it and create the Docker images with suffix "-with-targets-pre". See for example https://github.com/sagemath/sage/actions/runs/7108217427/job/19351107432#step:11:4936

For macOS, the workflow jobs named "stage-1" do this. 

See for example https://github.com/sagemath/sage/actions/runs/7108217428/job/19351098262#step:13:1

The separation into SAGE_LOCAL (which does not contain any Python) and SAGE_VENV makes it possible to build and test different versions or configurations (e.g., --enable-system-site-packages) of Python over the same SAGE_LOCAL.

Inside sage/local one would find either files built by the sage build system, or symlinks to files in other locations on the host system, such as /usr/local, or /opt/local, or a conda directory.  The question of whether to use a symlink for X would be the same as the question of whether to use "a system-X".  

That's an interesting proposal. 

We do have a number of defects in the build system of the Sage distribution, for which something like what you propose could be part of the solution:

- the problem that unattended system package updates can break an existing from-source installation of Sage

- the problem that switching between from using an SPKG to using a system package is not supported very well (https://github.com/sagemath/sage/issues/29387, https://github.com/sagemath/sage/issues/29708)

- the problem that all system packages are collectively represented by the single file SAGE_LOCAL/var/lib/sage/installed/.dummy in our build system

- the problem that our own packages do not track dependencies on system packages (https://github.com/sagemath/sage/issues/29711)

However, what is not clear to me is whether we can rely on the build systems of the various packages not to resolve the symlinks to the actual installation location. Also pkg-config files (or other package-provided discovery mechanisms) will contain the actual installation location.

> The venv would always contain a pyvenv.cfg file.

Could you explain why this matters for what you have in mind?

[Marc Culler]

On Thursday, December 7, 2023 at 2:31:24 PM UTC-6 Matthias Koeppe wrote:

> The venv would always contain a pyvenv.cfg file.

Could you explain why this matters for what you have in mind?

It is not essential.  It is just cleaner and more uniform.  I think it is misleading to have something called a venv which is missing one of the key required features of a venv.  And in the past I was misled by that.

 Also, it suggests something which I think would be an organizational improvement.  It could stop being the case that sage/local *never* contains any python.  Instead sage/local would be the prefix for Sage's installation of Python in the case where a system python is not being used.  The rule could be that if sage/local/lib/pythonX.Y exists then its site-packages subdirectory should be empty, with all Python packages being installed in the venv.  The venv would be a true venv and its symbolic link to an installed python would either be to the python installation in sage/local or to a python installation elsewhere on the system.

I would also suggest putting the venv at the top level, adjacent to local, rather than have a symlink adjacent to local which points into sage/local/var/sage.  I would find that to be a much more transparent layout.

- Marc

[Matthias Koeppe]

On Thursday, December 7, 2023 at 1:06:50 PM UTC-8 Marc Culler wrote:

 It could stop being the case that sage/local *never* contains any python.  Instead sage/local would be the prefix for Sage's installation of Python in the case where a system python is not being used.

Sure, but that would preclude using the same SAGE_LOCAL for an installation with venv over a system python and an installation of SPKG python.

But if it's important, we can put the SPKG python in its own prefix that is only used for creating a venv over it. Very easy change.

I would also suggest putting the venv at the top level, adjacent to local, rather than have a symlink adjacent to local which points into sage/local/var/sage.  I would find that to be a much more transparent layout.

The symlink is there because it also covers the case that a user used "configure --with-sage-venv=/some/directory".

But I wouldn't be opposed to changing the default physical location of the venvs from SAGE_LOCAL/var/lib/sage/venv-python3.11 to SAGE_ROOT/venv-python3.11; there would still be the convenience symlink "venv".

 

[Marc Culler]

On Thursday, December 7, 2023 at 5:04:25 PM UTC-6 Matthias Koeppe wrote:

Sure, but that would preclude using the same SAGE_LOCAL for an installation with venv over a system python and an installation of SPKG python.

 

I don't understand.  The bash script sage/venv/bin/sage calls sage/venv/bin/python which is a symlink to a python executable.  It doesn't matter where that python executable is located.  It could be in sage/local/bin or it could be somewhere else.  (Of course, for a notarized macOS application it must be inside the application bundle, but that is a different story.)

- Marc