Current status of possibility of integrating libraries written in Rust into Sage

[Jing Guo]

Dear all,

Recently we released a library for counting graph homomorphisms [0] in Sage. Due to performance and parallelism reasons, I was considering the possibility of re-writing some/all of the algorithms in Rust. I found a Rust library called `pyo3` [1] seems to be good for Python-Rust interop.

The latest post I could found is from one year ago [2], so I was wondering what the current status of possibility of integrating libraries written in Rust into Sage? Is the recommended approach still to make it an optional package? For instance, what I have something in mind is like `addcombq` [3] -- a Rust library which is callable from Sage/Python.

Many thanks!

Jing

[0]: https://github.com/guojing0/count-graph-homs

[1]: https://github.com/PyO3/pyo3

[2]: https://groups.google.com/g/sage-devel/c/OpBIfmbOlPA/m/hFKTdyE4CgAJ

[3]: https://github.com/Torrencem/addcombq

[Jing Guo]

It seems that Dima Pasechnik's replies are not displaying here, so I will add them:

First message:

Hi,

pyo3 seems to be for calling Python from Rust. You need the opposite, Rust from Python, e.g. as described in https://bheisler.github.io/post/calling-rust-in-python/.

Making a PyPI (pip) package seems to be the most obvious option.

With the current (outdated, I think) policies it will not be possible to make such a package standard part of Sage (but I am gathering support for relaxing these policies - see the thread on standard pip packages - that's what would be needed).

HTH, Dima

Second message:

PS. I realised that pyo3 is an option too.

[Dima Pasechnik]

Hi,

pyo3 seems to be for calling Python from Rust. You need the opposite, Rust from Python, e.g. as described in
<https://bheisler.github.io/post/calling-rust-in-python/>

Making a PyPI (pip) package seems to be the most obvious option.
With the current (outdated, I think) policies it will not be possible to make such a package standard part of Sage (but I am gathering support for relaxing these policies - see the thread on standard pip packages - that's what would be needed).

HTH,
Dima

[Dima Pasechnik]

PS. I realised that pyo3 is an option too.

[Matthias Koeppe]

Unlikely that we would add a package to the Sage distribution that builds a Rust library from source. 

Not so long ago we added support for installing Python packages from platform-independent wheels. We did this to sidestep the concern of shipping more and more of Javascript (Node.js) infrastructure that is needed for building JupyterLab components.

Likewise, we will soon add support for installing Python packages from platform-dependent wheels. This is needed for updating some Jupyter components that have started to use Rust (https://github.com/crate-py/rpds, a dependency of jsonschema).

 

(In the Sage distribution, they won't be "pip" packages -- which are an underdeveloped mechanism in the build system of the Sage distribution -- but rather a variant of the existing "wheel" packages.)

As the upstream developer, you would publish binary wheels to PyPI, likely in the same way as is done in https://github.com/crate-py/rpds/blob/main/.github/workflows/CI.yml

[Michael Orlitzky]

On Sat, 2024-06-01 at 10:02 -0700, Matthias Koeppe wrote:
> Unlikely that we would add a package to the Sage distribution that builds a
> Rust library from source.
>
> Not so long ago we added support for installing Python packages from
> platform-independent wheels. We did this to sidestep the concern of
> shipping more and more of Javascript (Node.js) infrastructure that is
> needed for building JupyterLab components.
>
> Likewise, we will soon add support for installing Python packages from
> platform-dependent wheels. This is needed for updating some Jupyter
> components that have started to use Rust (https://github.com/crate-py/rpds,
> a dependency of jsonschema).

The jsonschema package was being replaced by fastjsonschema in its
consumers the last time we ran in to this problem.

Rust is not nearly as portable as C, and has an unstable ABI that makes
shipping compatible versions of packages from multiple sources nearly
impossible. Will Sage's wheel packages be compatible with the wheels
built by the Gentoo package manager? Even for a single source, more
labor is involved -- instead of rebuilding each package once when it
changes, you wind up rebuilding all of its consumers, and then all of
their consumers, and... so on recursively.

We're already drowning in this regard and incorporating rust in any way
will make it a lot worse.

[Matthias Koeppe]

On Monday, June 3, 2024 at 12:41:41 PM UTC-7 Michael Orlitzky wrote:

On Sat, 2024-06-01 at 10:02 -0700, Matthias Koeppe wrote:
> we will soon add support for installing Python packages from
> platform-dependent wheels. This is needed for updating some Jupyter
> components that have started to use Rust (https://github.com/crate-py/rpds,
> a dependency of jsonschema).

The jsonschema package was being replaced by fastjsonschema in its
consumers the last time we ran in to this problem.

Could you share details regarding this? I'm not sure who "we" is in what you write, but in the last Jupyter PR that I prepared, I had to use some older versions of some packages to avoid pulling in the Rust dependency at this point.

[Matthias Koeppe]

On Monday, June 3, 2024 at 12:41:41 PM UTC-7 Michael Orlitzky wrote:

Rust is not nearly as portable as C, and has an unstable ABI that makes
shipping compatible versions of packages from multiple sources nearly
impossible.

I share this concern about ABI compatibility, and would therefore for now like a policy that restricts the use of such binary packages to

- optional packages 

- semi-standard, i.e., "disablable", packages such as the Jupyter frontend.

[Michael Orlitzky]

On Mon, 2024-06-03 at 12:54 -0700, Matthias Koeppe wrote:
>
> Could you share details regarding this? I'm not sure who "we" is in what
> you write, but in the last Jupyter PR that I prepared, I had to use some
> older versions of some packages to avoid pulling in the Rust dependency at
> this point.

The one I have in mind is: I am peacefully using borgbackup, a python
program, to back up our servers.

Borg depends on pkgconfig (python). Pkgconfig uses poetry (python) as
its build system. But poetry has a dependency on jsonschema (python),
and one day, the jsonschema package adds a dependency on rpds (rust).
Now I need to rebuild rust packages every time I update my system in
order to use my python backup program. This requires a rust compiler,
ABI unstable, so not only do I have to update the packages that have
changed but pretty much every rust package on the system during every
upgrade.

Poetry is a popular build system so this impacted many casual users of
python programs. It was a major PITA on Gentoo in particular because
python runs on more architectures than rust. Lots of end user packages
"broke" when the rust dependency was introduced because on certain
arches those dependencies "disappeared."

Anyway, poetry eventually replaced jsonschema with fastjsonschema to
avoid cursing its own users:

https://github.com/python-poetry/poetry-core/pull/642

[Kwankyu Lee]

Likewise, we will soon add support for installing Python packages from platform-dependent wheels. This is needed for updating some Jupyter components that have started to use Rust (https://github.com/crate-py/rpds, a dependency of jsonschema).

First sorry for an off-topic comment. If we need a feature (installing Python packages from platform-dependent wheels) just to maintain jupyter, then I think we should consider first removing jupyter from sage packages. As far as I know, there is no technical reason why we have to ship jupyter, we do it just because sage aims for "batteries included" assuming no internet connection. But jupyter is too big to carry as a battery. 

 

[Dima Pasechnik]

There is no need to remove it - it suffices to convert it to a pip package. (yes, for this we need to allow standard pip packages - as I have been proposing).

Removing won't be very easy, as some parts of Jupyter are used by Sphinx. (doable, yes).

Dima

[Matthias Koeppe]

Yes, this is well worth discussing. 

But note that there is one non-trivial interaction of Sage with Jupyter via the recently added Live Documentation feature (jupyter-sphinx). 

Already "pip install jupyter-sphinx" pulls in the Rust-based package rpds_py.

[Dima Pasechnik]

How about not inventing more "semi-", "quasi-", etc. here, and just go on with my proposal to allow standard pip packages?

Dima

[Kwankyu Lee]

There is no need to remove it - it suffices to convert it to a pip package. (yes, for this we need to allow standard pip packages - as I have been proposing).

Yes. I am all for removing "no internet connection" assumption in installing the sage distribution from source.

 

[kcrisman]

Yes. I am all for removing "no internet connection" assumption in installing the sage distribution from source.

Since we don't ship binaries, I'd like to hear from those who have had to install Sage in circumstances with slow/no internet situations about this (e.g. several of our French developers have had to do so in Francophone Africa, if I recall correctly).  Perhaps this is no longer an issue, but it would be nice to have confirmation.  I do agree that the issue of "certain customers" needing no-internet installation should no longer be a consideration.

[Dima Pasechnik]

There is absolutely no need to create huge tarfiles if you have to work offline with a collection of Pyhton packages (e.g. the collection of Sage packages); there are tools out there to creates offline mirrors of a collection of PyPI packages, such as Morgan:
<https://github.com/ido50/morgan>

Dima

[Dima Pasechnik]

On Monday, June 3, 2024 at 8:16:30 PM UTC+1 Matthias Koeppe wrote:

Unlikely that we would add a package to the Sage distribution that builds a Rust library from source. 

Not so long ago we added support for installing Python packages from platform-independent wheels. We did this to sidestep the concern of shipping more and more of Javascript (Node.js) infrastructure that is needed for building JupyterLab components.

Likewise, we will soon add support for installing Python packages from platform-dependent wheels. This is needed for updating some Jupyter components that have started to use Rust (https://github.com/crate-py/rpds, a dependency of jsonschema).

Could you be more specific on this, please? There are different way this can be interpreted. Where would these wheels come from? From PyPI?

[Matthias Koeppe]

On Wednesday, June 5, 2024 at 9:46:05 AM UTC-7 Dima Pasechnik wrote:

On Monday, June 3, 2024 at 8:16:30 PM UTC+1 Matthias Koeppe wrote:

Unlikely that we would add a package to the Sage distribution that builds a Rust library from source. 

Not so long ago we added support for installing Python packages from platform-independent wheels. We did this to sidestep the concern of shipping more and more of Javascript (Node.js) infrastructure that is needed for building JupyterLab components.

Likewise, we will soon add support for installing Python packages from platform-dependent wheels. This is needed for updating some Jupyter components that have started to use Rust (https://github.com/crate-py/rpds, a dependency of jsonschema).

Could you be more specific on this, please? [...] Where would these wheels come from? From PyPI?

(In the Sage distribution, they won't be "pip" packages -- which are an underdeveloped mechanism in the build system of the Sage distribution -- but rather a variant of the existing "wheel" packages.)

As the upstream developer, you would publish binary wheels to PyPI, likely in the same way as is done in https://github.com/crate-py/rpds/blob/main/.github/workflows/CI.yml

- Like I wrote below the fold: The upstream project makes the binary wheels and uploads them to PyPI as usual.

- When we make a package update in the Sage distribution, we download the wheels from PyPI (the upstream_url of the package) and upload them to the mirrors (and to the GitHub Release Assets).

- When the package is installed, the Sage distribution downloads the wheels from GitHub Release Assets, falling back to mirrors, then falling back to the upstream_url.

In other words, just like what we do for any other normal/wheel package in the Sage distribution. 

The only technical change: There's more than one "tarball" associated with these packages.

[Dima Pasechnik]

And what is the point of this? Why do we need to mirror PyPI?

Is there any Python project which resorts to mirroring binary PyPI wheels?

I think it's a very drastic step, and it needs to be discussed, brought up for a vote, etc.

Dima

 

[Matthias Koeppe]

On Sunday, June 9, 2024 at 5:25:54 PM UTC-7 Dima Pasechnik wrote:

Is there any Python project which resorts to mirroring binary PyPI wheels?

Sage-the-Python-project does not do this.

Sage-the-distribution does. Distributions do distribution stuff. 

More questions?

[Dima Pasechnik]

You've pulled this line ("Is there any Python project which resorts to mirroring binary PyPI wheels?") out of context.

Let me re-iterate.

> And what is the point of this? Why do we need to mirror PyPI?

> Is there any Python project which resorts to mirroring binary PyPI wheels?

So, why do we need to mirror PyPI ? The only "distro" that does it is Sage itself, AFAIK.

What are the reasons to believe that it will improve anything? The burden of such a move

is quite considerable, on the other hand.

> I think it's a very drastic step, and it needs to be discussed, brought up for a vote, etc.

So you don't think this needs any further discussion?

Dima

 

[Kwankyu Lee]

To add  variation to this boring discussion, let me intervene:

 

So, why do we need to mirror PyPI ? 

From this discussion, I guess that the answer is 

1. We want to pin the version of standard package

2. We do not want to assume internet connection

Is there other reason, Matthias?

Dima, which of (1) and (2) do you think we can abolish?

 

[Kwankyu Lee]

So, why do we need to mirror PyPI ?  

I understand "mirroring PyPI"  as what we do with "wheel" packages, that is, delivering the wheel (downloaded from PyPI) of the specified version in the sage tarball.

[Kwankyu Lee]

This is Dima's response:

> None are relevant:
> (1) can be trivially achieved without mirroring.
> (2) is irrelevant here, as creating a tarball with all these binary
> wheels pushes us into multi-Gigabyte  territory.
> That is, you'd need prefetch the set of binary wheels for your machine
> somehow - and again, for this you don't need to mirror PyPI.

In my own words, he seems to be saying that "we don't need to download the wheel and put it into our tarball; we can just call pip to fetch the specified version from internet at install time". To do what he is advocating, all we have to do is to throw away the "no internet for installing from tarball" assumption.

Did I understand you correctly, Dima? 

[Dima Pasechnik]

None are relevant:

(1) can be trivially achieved without mirroring.

(2) is irrelevant here, as creating a tarball with all these binary
wheels pushes us into multi-Gigabyte territory.
That is, you'd need prefetch the set of binary wheels for your machine
somehow - and again, for this you don't need to mirror PyPI.

Dima

>
>
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/b47a3e26-31e6-4755-8a7d-20e6e471aa9en%40googlegroups.com.

[Dima Pasechnik]

On Monday, June 10, 2024 at 3:37:48 PM UTC+1 Kwankyu Lee wrote:

So, why do we need to mirror PyPI ?  

I understand "mirroring PyPI"  as what we do with "wheel" packages, that is, delivering the wheel (downloaded from PyPI) of the specified version in the sage tarball.

This is not mirroring. Mirroring is setting up a site B which in part copies content from another site A. Then one says "B is mirroring A".

See examples in upstream/mirror_list.

With binary wheels, creating a reasonable size tarball containing what's needed to install Sage on a particular platform easily gets into Gigabytes.

It's also not clear whether the existing mirror structure would allow for much more content and traffic - but that's another story.

[Dima Pasechnik]

Yes and no: namely, using the proposed mirrored binary wheels will throw away
"no internet for installing from tarball" assumption. just as well.
So there is no difference at this point.

Dima
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/f5ff66fd-6164-496b-9b4b-39663ce23eb5n%40googlegroups.com.

[Matthias Koeppe]

On Monday, June 10, 2024 at 7:37:48 AM UTC-7 Kwankyu Lee wrote:

So, why do we need to mirror PyPI ?  

I understand "mirroring PyPI"  as what we do with "wheel" packages, that is, delivering the wheel (downloaded from PyPI) of the specified version in the sage tarball.

Exactly.

[Matthias Koeppe]

On Monday, June 10, 2024 at 7:52:09 AM UTC-7 Kwankyu Lee wrote:

This is Dima's response:

> [...] multi-Gigabyte  territory.

For a fact-based discussion, people may want to look at the actual size of the wheels of the only rust-based package that we are discussing. 

https://pypi.org/project/rpds-py/#files (look for files *-cp*-.whl)

It's just a few megabytes total.

[Dima Pasechnik]

Exactly, what?

I explained to Kwankyu that I mean "mirroring" in the usual sense of this word, not the one he thought.

Dima


>

[Dima Pasechnik]

rpds-py is rather small.

We are discussing your (general) proposal to provide binary wheels for packages by mirroring PyPI.
That is where gigabytes come from.
E.g. the binary wheels for scipy (with one version of Python) add up to about 200Mb.

Dima
>

[Matthias Koeppe]

On Monday, June 10, 2024 at 11:10:34 AM UTC-7 Dima Pasechnik wrote:

We are discussing your (general) proposal to provide binary wheels for packages by mirroring PyPI.
That is where gigabytes come from.
E.g. the binary wheels for scipy (with one version of Python) add up to about 200Mb.

Nobody proposed to make a change to how we install scipy.

I will not continue to respond in this thread.

[Kwankyu Lee]

I explained to Kwankyu that I mean "mirroring" in the usual sense of this word, not the one he thought.

OK. Sage tarballs and the wheels are uploaded to our mirror sites. So wheel packages end up in the mirror sites. This is what you say "mirroring PyPI".

You object to "mirroring PyPI", that is, wheel packages. For its replacement, you propose to fetch wheels directly from PyPI by switching to "pip" packages.

For users installing sage from sage tarball, this (switching to "pip" packages) means (again) to assume internet connection at install time. Right?

After some thought, now 

(A) "assuming internet connection for users installing from sage tarball" 

sounds a sort of oxymoron, because (A) defeats the purpose of providing tarball in the first place. Why tarball? It is for users with no internet connection. Otherwise, you can simply download sage source from github by git clone and the wheels from PyPI.

[Kwankyu Lee]

OK. Sage tarballs and the wheels are uploaded to our mirror sites. So wheel packages end up in the mirror sites. This is what you say "mirroring PyPI".

You object to "mirroring PyPI", that is, wheel packages. For its replacement, you propose to fetch wheels directly from PyPI by switching to "pip" packages.

For users installing sage from sage tarball, this (switching to "pip" packages) means (again) to assume internet connection at install time. Right?

This is Dima's reponse to this:

> You still need to get the tarball from the internet, no? 

Once a user gets the tarball either from the internet or by usb stick, he can install sage from the tarball later without internet connection, say in flight.

If we keep this installation form as the standard way to install sage from source and do not give up tarball, then I think wheel packages and "mirroring PyPI" is unavoidable.

[Kwankyu Lee]

Please Dima, let me stop this quoting business from your private messages.

Dima's response:

> It is perfectly avoidable, as I explained in more details, which you have chosen not to copy here.

OK. Then which one of these quotes from you is the details explaining how to install sage from source without internet connection by "pip" packages?

 

(1) > You can specify a collection of pip packages to download by pip (cf. "pip download") for installing along with the Sage tarball, if you must.

(2) > ... actually, with pip-installable Sage, you can simply tell pip to install from a github repo. Conditional on all the libraries already installed, that's how it all can work for development purposes - and for installation of a stable sagemath, one can just use PyPI.