Thoughts on adding Python as a build dependency?

[Jeff Squyres]

Open MPI packagers —

There’s active work going on to (slowly) convert the Perl scripts used in Open MPI’s build process to Python >= v3.7 for the Open MPI 6.0.x series.  That being said, even with this work, I’m sure we won’t eliminate the Perl build requirement before v6.0.0.  

Depending on how packagers build Open MPI, you may have effectively already had a Python 3 build requirement.  But these changes would require Python 3 for *all* build scenarios — even for build-from-tarball-source end users.

Does anyone have any opinions on Open MPI v6.x adding Python >=v3.7 as a build requirement?  Possibilities include:

• “No problem / I don’t care"
• “This is terrible!  Let me tell you why…”
• …something else

I’d love to hear your thoughts here.

Thanks!

-- 

{+} Jeff Squyres

[Mitchell Dorrell]

On Thu, May 15, 2025, 10:13 Jeff Squyres <je...@squyres.com> wrote:

Does anyone have any opinions on Open MPI v6.x adding Python >=v3.7 as a build requirement?

Just Python? Or would some modules (e.g. NumPy) also become build requirements for Open MPI?

-MD

[Jeff Squyres]

Good question: I was not specific.

This is a build requirement, not a run-time requirement.  We’re using core Python just to generate some C/Fortran source code files based on the results of running “configure”.  As far as we can tell, no additional pip packages are needed — it’s simple read-config-files-and-do-some-string-manipulation stuff.

This is stuff we’ve done with (core) Perl since Open MPI v1.x.  We’re just converting this Per to Python because, frankly, many developers don’t know Perl these days.

To unsubscribe from this group and stop receiving emails from it, send an email to ompi-packager...@lists.open-mpi.org.

-- 

{+} Jeff Squyres

[Jeff Squyres]

Somehow Alastair’s response got eaten by Google Groups, so I include it below.

Good points — thanks Alastair.  FWIW:

• In the Perl scripts that we’ve converted to Python so far, we’ve managed to do it with just core Python — no pip installs needed.
• Open MPI has required Perl as part of its build process for years (decades); this is just updating those Perl scripts to Python.
• We require >= Python 3.7, but try to stay pretty “vanilla” Python so that it works across a bunch of different versions.  
• These Perl-now-Python scripts are simply generating some C and Fortran source code (reacting to results determined by the configure script tests) — we’re not talking about an MPI Python package, for example.  Hence, there’s no pyproject.toml, Messon, … etc.
• There is one exception: when you git clone Open MPI and need to build the docs + man pages, you do need to pip install Sphinx (per docs/requirements.txt).  To be clear: Sphinx is not required when building Open MPI from an official distribution tarball.  Open MPI tarballs are bootstrapped with pre-built HTML docs and nroff man pages so that users don’t have to have Sphinx installed.

We haven’t ever supported building the Fortran bindings with multiple different Fortran compilers in a single build; that’s unfortunately never been a requirements, and would also unfortunately take quite a bit of work (e.g., we’d have to split out a whole chunk of configury into a standalone thing that could effectively govern a subset of the C + Fortran code base).

-------- Forwarded Message --------

Subject:	Re: [OMPI packagers] Thoughts on adding Python as a build dependency?
Date:	Thu, 15 May 2025 15:47:38 +0100
From:	Alastair McKinstry <alas...@mckinstry.ie>
To:	ompi-pa...@lists.open-mpi.org

From the Debian/Ubuntu side, no problem, but it would be good to pay attention to any Python dependencies that get sucked in. Is it Just moving to Python3 or a larger project to something like pyproject.toml support, Meson, etc?

When working on large and complex builds (eg ones that have a mixture of C,C++ and Fortran interfaces and libraries, multiple Pythons,etc) its useful to have a build system that (1) has some form of program language structure or classes etc (2) breaks out builds.

For example while in Debian we try to mostly support one (latest) Python instance (in the current Debian 13 release we're working on, python 3.13) we allow for (especially in development) multiple Pythons. So libraries/extensions get built for multiple python versions. In a good build system this might mean calling build in a ./wrappers/python directory multiple times for each supported Python. For a bad one, rebuilding everything (all the C and other libraries) for each Python.

Similarly, in upcoming releases expect that the Fortran libraries may be rebuilt with different Fortran compilers (see my upcoming talk at Debconf on multiple parallel Fortran stacks) for gfortran, flang, etc.

Thanks

Alastair

To unsubscribe from this group and stop receiving emails from it, send an email to ompi-packager...@lists.open-mpi.org.

-- 
Alastair McKinstry,
GPG: 82383CE9165B347C787081A2CBE6BB4E5D9AD3A5
e: alas...@mckinstry.ie, im: @alastair:mckinstry.ie @amcki...@mastodon.ie

Commander Vimes didn’t like the phrase “The innocent have nothing to fear,”
 believing the innocent had everything to fear, mostly from the guilty but in the longer term
 even more from those who say things like “The innocent have nothing to fear.”
 - T. Pratchett, Snuff

-- 

{+} Jeff Squyres

[Kenneth Hoste]

Hi Jeff,

Out of personal experience since Python 2.4 (with EasyBuild, which is
fully implemented in Python), a couple of attention points:

- Stick to the Python standard library, don't require additional Python
packages (from PyPI).
If you can, avoid packaging tools (incl. setuptools).

- Have CI in place that tests the scripts with all Python versions you
support.
Sooner or later you will catch something that surprises you.
Make sure you also test with new Python releases, which often include
breaking changes.

- Requiring Python >= 3.7 seems somewhat arbitrary, any particular
reason for that specific version requirement?

If you can, I strongly recommend you to also support Python 3.6,
which is the default Python version in RHEL8. That's an important Linux
distribution for HPC, since RHEL8 + derivatives thereof are likely the
most popular Linux distribution for HPC systems.


I fully understand why there's an interest in switching away from Perl,
and I think it's a good idea, but there may be unexpected dragons when
you tread into the magic wonderland of Python, so come in prepared and
tread carefully. :)


regards,

Kenneth

On 15/05/2025 16:11, Jeff Squyres wrote:
> Open MPI packagers —
>
> There’s active work going on to (slowly) convert the Perl scripts used
> in Open MPI’s build process to Python >= v3.7 for the Open MPI 6.0.x
> series.  That being said, even with this work, I’m sure we won’t
> eliminate the Perl build requirement before v6.0.0.
>
> Depending on how packagers build Open MPI, you may have effectively
> already had a Python 3 build requirement.  But these changes would
> require Python 3 for *all* build scenarios — even for build-from-
> tarball-source end users.
>
> Does anyone have any opinions on Open MPI v6.x adding Python >=v3.7 as a
> build requirement?  Possibilities include:
>

> * “No problem / I don’t care"
> * “This is terrible!  Let me tell you why…”
> * …something else

[Jeff Squyres]

Kenneth —

Thanks for the info.  See my replies on this thread for some additional information.

I did not know about Python 3.6 in RHEL 8, though — that’s good new information for me.  There is a reason that we require Python >= v3.7, but it’s a nuanced issue:

• The C/Fortran-generating Python scripts we use definitely work with core Python v3.6 (I didn’t bother testing with Python < v3.6) and require no additional pip modules.
• They’re pretty simplistic Python scripts.
• These scripts are run in all Open MPI builds — packager builds, end-user builds, git clone builds, … etc.

• Building the documentation — specifically, rendering the MPI API function signatures — uses the MPI Forum pympistandard module (https://github.com/mpi-forum/pympistandard)
• This module requires Python v3.7.  I don’t remember the specific issue offhand, but I remember specifically testing this and coming to the conclusion that that module uses some feature that was introduced in Python v3.7.
• However, the pympistandard module is only used if you are building the Open MPI documentation, which is not required (by default) for end-user (and many packager?) tarball builds because Open MPI official distribution tarballs include pre-built HTML and man page documentation.

However, for simplicity, we just made Open MPI’s configure script require Python >= v3.7 (as opposed to always requiring Python >= v3.6, and only requiring Python >= v3.7 if you’re building the documentation).

I’ll go back and revisit this.

-- 

{+} Jeff Squyres