Split into interpreter and python package to facilitate development

[Tobias Diez]

Hi everybody,

(disclaimer: everything I say comes from a few days of playing around with sage's code - so I might make some wrong statements that are obvious to more experienced sage devs)

currently `./sage -b` rebuilds the python interpreter part of sage (i.e. special syntax support etc) as well as the actual python package (e.g. sage.manifolds). This makes development hard as small changes to a package (e.g. sage.manifolds.chart.py) need a lengthy recompilation of everything. Moreover, the build copies the content of `/src/sage` to `site-packages/sage` which makes IntelliSense of vs code go crazy (It seems the whole site-packages folder is reindexed after a build, which takes really long).

In view of these problems, I think it would be worthwhile to split both aspects of sage (interpreter vs library), and make it possible to propagate changes to package files without a complete recompilation of sage.

For now I came up with the following workaround, which works but feels like a huge hack: Create a file `/src/sage/test.py` with the following content. You can then run this file using the local python (e.g. ./local/bin/python3). Once you change the content of the package under consideration (here manifolds), rerun the "Reload manifold packages..." block, which reloads the files so that the changes take effect. This takes about one second in contrast to the 5 min of `sage -b`.

I hope this is helpful for someone (I doubt I'm the first one facing the problem).

Best regards

Tobias

Ps: one complication I faced was that all imports in sage are absolute (e.g. sage.manifolds.differentiable.metric) instead of relative (e.g. `.metric`). What's the reason for this?

# This script loads packages from /src/sage instead of site-packages/sage to make development easier

# For this to work, the following preparation is necessary:

# - Compile sage using `./sage -b`

# - Remove the imports of the packages from site-packages/sage/all.py

# - Delete (or rename) the packages from site-packages/sage

# Moreover, this file has to reside in `src/sage`, e.g. `src/sage/test.py`

# %% Load sage

import sys

import importlib

# Import sage from site-packages

import sage.all

# Load manifolds and tensor packages from src folder, and put them at "sage.manifolds" and "sage.tensor"

spec = importlib.util.find_spec("manifolds")

print(spec) # This should show that the module is loaded from /sage/src/

module = importlib.util.module_from_spec(spec)

sys.modules["manifolds"] = module

sys.modules["sage.manifolds"] = module

spec = importlib.util.find_spec("tensor")

print(spec) # This should show that the module is loaded from /sage/src/

module = importlib.util.module_from_spec(spec)

sys.modules["sage.tensor"] = module

# %% 

# Reload manifold packages (actually not needed on first run, use this if you change code in /src/manifolds)

for k,v in sys.modules.items():

    if k.startswith('sage.manifolds'):

        print(k)

        importlib.reload(v)

# Actual imports

from sage.manifolds.differentiable.tensorfield import TensorField

from sage.manifolds.manifold import Manifold

# %% Do what you want here

M = Manifold(2, 'M'); M

XM = M.vector_field_module()

[Eric Gourgoulhon]

Hi Tobias,

Le dimanche 31 mai 2020 14:49:46 UTC+2, Tobias Diez a écrit :

For now I came up with the following workaround, which works but feels like a huge hack: Create a file `/src/sage/test.py` with the following content. You can then run this file using the local python (e.g. ./local/bin/python3). Once you change the content of the package under consideration (here manifolds), rerun the "Reload manifold packages..." block, which reloads the files so that the changes take effect. This takes about one second in contrast to the 5 min of `sage -b`.

I'm surprised about this: usually, when I modify some source file in src/sage/manifolds and I run sage -b afterwards, it takes only about 1 second to rebuild Sage.

Eric.

[Dima Pasechnik]

indeed, if you only touch a *.py file, it will be copied and
byte-compiled by ./sage -b, but this would be the only work done by
./sage -b

if you see more, it could either be that your setup touches a lot of
files, or you did actually touch *.pyx files yourself - and they take
more time to rebuild.


> Eric.
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/b7b265b0-a3c4-48f4-9343-778c3504d424%40googlegroups.com.

[Tobias Diez]

Thanks for the reply.

Then something is wrong with my setup. Even without changing anything a run of sage -b takes more than 4 min for me. I'd now run multiple builds `sage -b` directly without any changes in between and all apps closed etc. The output is attached below. Maybe the reason is that I'm using WSL 2 on Windows?

Regardless of these problems (which are apparently specific to my situation), don't you think it would be worthwhile to extract the library part of sage? I think this would reduce the overhead needed to develop code for sage.

Best

Tobias

tobias@DESKTOP-OMSMBL2:/mnt/d/Programming/Projects/sage$ ./sage -b
cd . && export                                    \
    SAGE_ROOT=/doesnotexist                               \
    SAGE_SRC=/doesnotexist                                \
    SAGE_SRC_ROOT=/doesnotexist                           \
    SAGE_DOC_SRC=/doesnotexist                            \
    SAGE_BUILD_DIR=/doesnotexist                          \
    SAGE_PKGS=/mnt/d/Programming/Projects/sage/build/pkgs                \
&& sage-python -u setup.py --no-user-cfg build install
/mnt/d/Programming/Projects/sage/src/bin/sage-env: line 130: cd: /doesnotexist: No such file or directory
Warning: overwriting SAGE_ROOT environment variable:
Old SAGE_ROOT=/doesnotexist
New SAGE_ROOT=
Discovering Python/Cython source code....
Discovered Python/Cython sources, time: 1.53 seconds.
running build
Generating auto-generated sources
Building interpreters for fast_callable
running build_cython
Enabling Cython debugging support
Updating Cython code....
Finished Cythonizing, time: 76.81 seconds.
running build_py
running build_ext
Executing 0 commands (using 1 thread)
Time to execute 0 commands: 0.24 seconds.
Total time spent compiling C/C++ extensions: 31.99 seconds.
running install
running install_lib
byte-compiling /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages/sage/ext_data/nbconvert/postprocess.py to postprocess.cpython-37.pyc
running install_egg_info
Removing /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages/sage-9.1-py3.7.egg-info
Writing /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages/sage-9.1-py3.7.egg-info
Cleaning up stale installed files....
- cleaning /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages
Cleaning up stale file: /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages/sage/ext_data/nbconvert/__pycache__/postprocess.cpython-37.pyc
- cleaning build/lib.linux-x86_64-3.7
Finished cleaning, time: 15.45 seconds.
if [ "$UNAME" = "CYGWIN" ]; then                         \
    sage-rebase.sh "$SAGE_LOCAL" 2>/dev/null;            \
fi

real    6m1.625s
user    0m23.031s
sys     1m1.448s
tobias@DESKTOP-OMSMBL2:/mnt/d/Programming/Projects/sage$ ./sage -b
cd . && export                                    \
    SAGE_ROOT=/doesnotexist                               \
    SAGE_SRC=/doesnotexist                                \
    SAGE_SRC_ROOT=/doesnotexist                           \
    SAGE_DOC_SRC=/doesnotexist                            \
    SAGE_BUILD_DIR=/doesnotexist                          \
    SAGE_PKGS=/mnt/d/Programming/Projects/sage/build/pkgs                \
&& sage-python -u setup.py --no-user-cfg build install
/mnt/d/Programming/Projects/sage/src/bin/sage-env: line 130: cd: /doesnotexist: No such file or directory
Warning: overwriting SAGE_ROOT environment variable:
Old SAGE_ROOT=/doesnotexist
New SAGE_ROOT=
Discovering Python/Cython source code....
Discovered Python/Cython sources, time: 0.90 seconds.
running build
Generating auto-generated sources
Building interpreters for fast_callable
running build_cython
Enabling Cython debugging support
Updating Cython code....
Finished Cythonizing, time: 50.40 seconds.
running build_py
running build_ext
Executing 0 commands (using 1 thread)
Time to execute 0 commands: 0.21 seconds.
Total time spent compiling C/C++ extensions: 21.32 seconds.
running install
running install_lib
byte-compiling /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages/sage/ext_data/nbconvert/postprocess.py to postprocess.cpython-37.pyc
running install_egg_info
Removing /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages/sage-9.1-py3.7.egg-info
Writing /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages/sage-9.1-py3.7.egg-info
Cleaning up stale installed files....
- cleaning /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages
Cleaning up stale file: /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages/sage/ext_data/nbconvert/__pycache__/postprocess.cpython-37.pyc
- cleaning build/lib.linux-x86_64-3.7
Finished cleaning, time: 15.17 seconds.
if [ "$UNAME" = "CYGWIN" ]; then                         \
    sage-rebase.sh "$SAGE_LOCAL" 2>/dev/null;            \
fi

real    4m27.670s
user    0m24.173s
sys     0m36.523s
tobias@DESKTOP-OMSMBL2:/mnt/d/Programming/Projects/sage$ ./sage -b
cd . && export                                    \
    SAGE_ROOT=/doesnotexist                               \
    SAGE_SRC=/doesnotexist                                \
    SAGE_SRC_ROOT=/doesnotexist                           \
    SAGE_DOC_SRC=/doesnotexist                            \
    SAGE_BUILD_DIR=/doesnotexist                          \
    SAGE_PKGS=/mnt/d/Programming/Projects/sage/build/pkgs                \
&& sage-python -u setup.py --no-user-cfg build install
/mnt/d/Programming/Projects/sage/src/bin/sage-env: line 130: cd: /doesnotexist: No such file or directory
Warning: overwriting SAGE_ROOT environment variable:
Old SAGE_ROOT=/doesnotexist
New SAGE_ROOT=
Discovering Python/Cython source code....
Discovered Python/Cython sources, time: 0.91 seconds.
running build
Generating auto-generated sources
Building interpreters for fast_callable
running build_cython
Enabling Cython debugging support
Updating Cython code....
Finished Cythonizing, time: 50.39 seconds.
running build_py
running build_ext
Executing 0 commands (using 1 thread)
Time to execute 0 commands: 0.22 seconds.
Total time spent compiling C/C++ extensions: 21.22 seconds.
running install
running install_lib
byte-compiling /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages/sage/ext_data/nbconvert/postprocess.py to postprocess.cpython-37.pyc
running install_egg_info
Removing /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages/sage-9.1-py3.7.egg-info
Writing /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages/sage-9.1-py3.7.egg-info
Cleaning up stale installed files....
- cleaning /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages
Cleaning up stale file: /mnt/d/Programming/Projects/sage/local/lib/python3.7/site-packages/sage/ext_data/nbconvert/__pycache__/postprocess.cpython-37.pyc
- cleaning build/lib.linux-x86_64-3.7
Finished cleaning, time: 15.04 seconds.
if [ "$UNAME" = "CYGWIN" ]; then                         \
    sage-rebase.sh "$SAGE_LOCAL" 2>/dev/null;            \
fi

real    4m25.825s
user    0m24.257s
sys     0m36.241s
tobias@DESKTOP-OMSMBL2:/mnt/d/Programming/Projects/sage$

On Sunday, May 31, 2020 at 3:07:40 PM UTC+2, Dima Pasechnik wrote:

On Sun, May 31, 2020 at 1:59 PM Eric Gourgoulhon <egourg...@gmail.com> wrote:
>
> Hi Tobias,
>
> Le dimanche 31 mai 2020 14:49:46 UTC+2, Tobias Diez a écrit :
>>
>>
>> For now I came up with the following workaround, which works but feels like a huge hack: Create a file `/src/sage/test.py` with the following content. You can then run this file using the local python (e.g. ./local/bin/python3). Once you change the content of the package under consideration (here manifolds), rerun the "Reload manifold packages..." block, which reloads the files so that the changes take effect. This takes about one second in contrast to the 5 min of `sage -b`.
>
>
> I'm surprised about this: usually, when I modify some source file in src/sage/manifolds and I run sage -b afterwards, it takes only about 1 second to rebuild Sage.
>
indeed, if you only touch a *.py file, it will be copied and
byte-compiled by ./sage -b, but this would be the only work done by
./sage -b

if you see more, it could either be that your setup touches a lot of
files, or you did actually touch *.pyx files yourself - and they take
more time to rebuild.


> Eric.
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.

[Matthias Koeppe]

On Sunday, May 31, 2020 at 5:49:46 AM UTC-7, Tobias Diez wrote:

currently `./sage -b` rebuilds the python interpreter part of sage (i.e. special syntax support etc)

There's no such thing, actually. "sage -b" just builds and installs "sagelib" (the "actual python package"). 

If this were done using just "pip install .", then it would indeed have to build all of sagelib, and this would take very long because of the size of sage.*

Because of this, our src/setup.py actually makes a lot of efforts to "propagate changes to package files without a complete recompilation of sage".

There are multiple steps of copying, including the last step of copying into site-packages.

It is possible that parts of this do not play well with VS code -- this is something that we should investigate and improve.

The logs that you included showed a 4 min of wall clock time, but "only" 24 seconds of user time for an incremental build in which effectively nothing was built. It would be interesting to profile this to see where the time is spent. Likely this is due to slow file I/O on WSL during distutils/Cython dependency tracking.

In a slightly different direction, we may also want to check whether "in-place" builds, such as those done with "pip install --editable", could be supported.

If you are interested in helping improve the build system of sagelib, you may be interested in the following meta-tickets:

https://trac.sagemath.org/ticket/21507

  Meta-ticket: Make sagelib a pip-installable Python source package, listed on PyPI

https://trac.sagemath.org/ticket/21508

  Meta-ticket: Clean up src/setup.py to bring it to standard distutils behavior

https://trac.sagemath.org/ticket/29705

  Meta-ticket: Modularize sagelib into separate distributions (distutils packages)

[Tobias Diez]

Thanks for the explanation, this makes a few things clearer to me!

The "in-place" builds sounds very promising indeed. I would like to help in this direction, but feel like I don't have the knowledge to contribute directly. However, if there is something to try-out / play around with, please let me know!