python3-3.9.2 failed to build while running 'sage -i pynormaliz' (ModuleNotFoundError: No module named 'binascii')

[Max Alekseyev]

Hi!

I'm trying to install PyNormaliz within freshly installed Sage 9.3 within freshly installed Ubuntu 20.04.02 LTS.It breaks giving the following error:

===

LD_LIBRARY_PATH=/usr/local/SageMath.93/local/var/tmp/sage/build/python3-3.9.2/src CC='gcc -pthread' LDSHARED='gcc -pthread -shared -Wl,-rpath-link,/usr/local/SageMath.93/local/lib -L/usr/local/SageMath.93/local/lib -Wl,-rpath,/usr/local/SageMath.93/local/lib  -L. -Wl,-rpath-link,/usr/local/SageMath.93/local/lib -L/usr/local/SageMath.93/local/lib -Wl,-rpath,/usr/local/SageMath.93/local/lib   ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall'         _TCLTK_INCLUDES='' _TCLTK_LIBS=''       ./python -E ./setup.py  build

Traceback (most recent call last):

  File "/usr/local/SageMath.93/local/var/tmp/sage/build/python3-3.9.2/src/./setup.py", line 33, in <module>

    from distutils import log

  File "/usr/local/SageMath.93/local/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 83, in create_module

    return importlib.import_module('setuptools._distutils')

  File "/usr/local/SageMath.93/local/var/tmp/sage/build/python3-3.9.2/src/Lib/importlib/__init__.py", line 127, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "/usr/local/SageMath.93/local/lib/python3.9/site-packages/setuptools/__init__.py", line 16, in <module>

    import setuptools.version

  File "/usr/local/SageMath.93/local/lib/python3.9/site-packages/setuptools/version.py", line 1, in <module>

    import pkg_resources

  File "/usr/local/SageMath.93/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 23, in <module>

    import zipfile

  File "/usr/local/SageMath.93/local/var/tmp/sage/build/python3-3.9.2/src/Lib/zipfile.py", line 6, in <module>

    import binascii

ModuleNotFoundError: No module named 'binascii'

make[3]: *** [Makefile:638: sharedmods] Error 1

********************************************************************************

Error building python3-3.9.2

********************************************************************************

===

A full log is attached.

Please advise.

Thanks,

Max

[Dima Pasechnik]

don't build Python, use one from the system.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/cce1a457-2345-433c-8b25-75e40b5590d0n%40googlegroups.com.

[Max Alekseyev]

I'd be happy to, but Sage somehow decides that it needs to build its own Python from scratch.

I simply sun 'sage -i pynormaliz' and get a broken sage as a result.

What's wrong?

Regards,

Max

[Dima Pasechnik]

please post top-level config.log

Do you have python3 excecutable, version 3.7 or newer?

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/d976804d-b1f3-4e1e-adc9-4a5bf9af1b81n%40googlegroups.com.

[Dima Pasechnik]

On Fri, Jun 18, 2021 at 9:46 PM Max Alekseyev <max...@gmail.com> wrote:
>

> I'd be happy to, but Sage somehow decides that it needs to build its own Python from scratch.
> I simply sun 'sage -i pynormaliz' and get a broken sage as a result.
> What's wrong?
>

we are trying to switch to a sane building scheme:

./bootstrap
./configure
make build
make pynormaliz

sage -i is an old hack noone should use...

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/d976804d-b1f3-4e1e-adc9-4a5bf9af1b81n%40googlegroups.com.

[Max Alekseyev]

Please find config.log attached. The system has Python 3.8.5 installed.

Max

[Max Alekseyev]

I've just followed your suggestion and noticed that 'configure' produced the following warning:

===

    notice: the following SPKGs did not find equivalent system packages:

        python3   _recommended boost cbc cmake coxeter3 gp2c graphviz igraph isl libsemigroups libxml2 lrslib ninja_build pandoc pari_elldata pari_galpol pari_nftables pari_seadata perl_cpan_polymake_prereq perl_term_readline_gnu tox

        

checking for the package system in use... debian

configure:

    hint: installing the following system packages, if not

    already present, is recommended and may avoid having to

    build them (though some may have to be built anyway):

      $ sudo apt-get update 

      $ sudo apt-get install  python3 libpython3-dev python3-distutils

===

However, these particular packaged DO present in the system. Running apt-get as suggested gives:

===

$ sudo apt-get install  python3 libpython3-dev python3-distutils

[sudo] password for maxal: 

Reading package lists... Done

Building dependency tree       

Reading state information... Done

libpython3-dev is already the newest version (3.8.2-0ubuntu2).

python3 is already the newest version (3.8.2-0ubuntu2).

python3 set to manually installed.

python3-distutils is already the newest version (3.8.10-0ubuntu1~20.04).

0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

===

Why Sage does not detect these packages as installed?

Regards,

Max

[Dima Pasechnik]

On Fri, Jun 18, 2021 at 10:20 PM Max Alekseyev <max...@gmail.com> wrote:
>
> Please find config.log attached. The system has Python 3.8.5 installed.

## Checking whether SageMath should install SPKG python3... ##
## -------------------------------------------------------- ##
configure:32260: python3 has already been installed by SageMath

here is why it does not use system python3 - because you already had
python3 built by Sage from source.

(and the same for the host of other packages.)

You can uninstall packages by doing

make <package>-clean

e.g.

make python3-clean

if you don't want to build from scratch.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/e8ad5341-915e-44bd-93c3-0b7f40caaa38n%40googlegroups.com.

[Max Alekseyev]

I installed Sage from sage-9.3-Ubuntu_20.04-x86_64.tar.bz2

I've just run

make python3-clean

./bootstrap

./configure

but it still complaints that python3 is not installed in the system.

Why?

Regards,

Max

[Matthias Koeppe]

We would need to see config.log again to know why.

[Dima Pasechnik]

On Fri, Jun 18, 2021 at 10:37 PM Max Alekseyev <max...@gmail.com> wrote:
>
> I installed Sage from sage-9.3-Ubuntu_20.04-x86_64.tar.bz2

this is a binary distribution. It doesn't really allow for building
optional packages.
Please get the source distribution instead.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/a0f6e252-8458-473b-b5ae-fac743fde7efn%40googlegroups.com.

[Max Alekseyev]

Here is config.log 

[Max Alekseyev]

I'll try the source distribution, but I'd like to mention that building optional packages from a binary distribution worked well in earlier Sage versions (e.g., 9.1 and 9.2)

Regards,

Max

[Dima Pasechnik]

On Fri, Jun 18, 2021 at 11:02 PM Max Alekseyev <max...@gmail.com> wrote:
>
> I'll try the source distribution, but I'd like to mention that building optional packages from a binary distribution worked well in earlier Sage versions (e.g., 9.1 and 9.2)

please bear with us. We aim at

pip install sagemath -u

to be the normal, Pythonic way to deal with Sage...

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/454fa616-c0cd-4654-9024-0d229ce112d9n%40googlegroups.com.

[Max Alekseyev]

A source distribution has many more points to break (I'll send my issues in a new thread), and it takes ages to compile.

For now, it's still unresolved why 'configure' in the binary distribution fails to detect python3 in the system.

Regards,

Max

[Dima Pasechnik]

On Sat, Jun 19, 2021 at 12:41 PM Max Alekseyev <max...@gmail.com> wrote:
>
> A source distribution has many more points to break (I'll send my issues in a new thread), and it takes ages to compile.
> For now, it's still unresolved why 'configure' in the binary distribution fails to detect python3 in the system.

A binary distribution normally speaking is not something that has or
needs configure.

By the way, conda distribution might be something you'd prefer, this
is a configurable binary distribution
as you might know.
https://wiki.sagemath.org/Conda
They have normaliz, as far as I know.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/ea9dfef8-a81e-4c09-8a31-5524440875f6n%40googlegroups.com.

[Max Alekseyev]

I've tried to use Conda distribution, but it says "FeatureNotPresentError: PyNormaliz is not available."

So, I'm stuck -- I cannot install Normaliz within the binary distribution of Sage 9.3, and I cannot compile Sage 9.3 with Normaliz from the sources, getting all kinds of errors here and there. It's very frustrating ;(

I was able to install Normaliz from the binary distribution of Sage 9.2 without an issue.

Max

[Dima Pasechnik]

Conda's package is called normaliz, as far as I know it includes python interface

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/5eff97ec-1ff0-4117-bed0-60dfadc2be67n%40googlegroups.com.

[Isuru Fernando]

conda doesn't have pynormaliz yet. You can install normaliz through conda and pynormaliz through pip

conda create -n sage sage=9.3 python=3.9

conda activate sage

conda install normaliz

export NORMALIZ_LOCAL_DIR=$CONDA_PREFIX

pip install pynormaliz

Isuru

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq3bhmaZc5408wFnK8FqwP9TdcGKOet9pfn%3DUr5XT9kRag%40mail.gmail.com.

[Marc Culler]

I am seeing the same error with Python 3.9.5 when I attempt to build Sage 9.4.beta3 from source on macOS 11.4.  The build fails when it attempts to import distutils, which depends on the binascii module.  Searching through the python build directory with find indicates that binascii.o does not get built.  If I run make in the python build directory using my normal environment then binascii.o is built, as is the extension module binascii.cpython-39-darwin.so, and the python build succeeds.  But If I run make in a sage-buildsh subshell then binascii.o does not get built and the make command fails in the same way as with the build of sage.  So evidently something in the sage build environment is preventing python from compiling its binascii module.

I am attaching the log file.

- Marc

[Dima Pasechnik]

On Thu, Jun 24, 2021 at 5:46 PM Marc Culler <marc....@gmail.com> wrote:
>
> I am seeing the same error with Python 3.9.5 when I attempt to build Sage 9.4.beta3 from source on macOS 11.4.

There is no need to build Python, you can either use one from Homebrew
or from the official Python distribution.

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/f53afd48-abd2-440b-a996-f9950e16ad58n%40googlegroups.com.

[Marc Culler]

On Thu, Jun 24, 2021 at 11:50 AM Dima Pasechnik <dim...@gmail.com> wrote:
>
> There is no need to build Python, you can either use one from Homebrew
> or from the official Python distribution.
>

That is an extremely unhelpful answer, especially given what I am
trying to do. It is also ridiculous. If people are following the
build instructions in the README file and the build is failing then
the build is broken and it should be fixed.

Fortunately, I was able to solve my problem in spite of this unhelpful
help. The build is not broken. But the instructions in the README
file are incomplete. The answer which would have helped me is this:

"If you switch branches in the git repository then you need to run
make distclean before rebuilding sage."

And that really should be added to the README file.

- Marc

PS If you are going to tell people that they do not have to build
python to build sage then you should also provide instructions for how
to build sage without building python.

> You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/euE9MJnfjAo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq1Z%2BJWoupqNMrqx19w8kBh3LctrREVjsCdBnyBbwpXKng%40mail.gmail.com.

[Matthias Koeppe]

I have opened https://trac.sagemath.org/ticket/32056 for the mysterious issue with 'binascii'.

[Matthias Koeppe]

On Thursday, June 24, 2021 at 10:56:37 AM UTC-7 marc....@gmail.com wrote:

If people are following the
build instructions in the README file and the build is failing then
the build is broken and it should be fixed.

I agree 

 

[...] The answer which would have helped me is this:

"If you switch branches in the git repository then you need to run
make distclean before rebuilding sage."

I agree that we need to give better instructions for incremental builds. There are long-standing issues in our build system, tracked in  https://trac.sagemath.org/ticket/29708 and https://trac.sagemath.org/ticket/31277, that make a "make distclean" necesssary.

If "make distclean" helped resolve the issue with "binascii" and the python3 spkg was afterwards successfully built, it suggests that the python3 installation picks up site packages from the previous installation in SAGE_LOCAL. This is a very specific problem that should definitely be fixed separately from the above build system issues.

 

[Dima Pasechnik]

On Thu, Jun 24, 2021 at 6:56 PM Marc Culler <marc....@gmail.com> wrote:
>
> On Thu, Jun 24, 2021 at 11:50 AM Dima Pasechnik <dim...@gmail.com> wrote:
> >
> > There is no need to build Python, you can either use one from Homebrew
> > or from the official Python distribution.
> >
>
> That is an extremely unhelpful answer, especially given what I am
> trying to do. It is also ridiculous. If people are following the
> build instructions in the README file and the build is failing then
> the build is broken and it should be fixed.

We have 10 people per day asking what to do with broken Python build, you know,
it is tiring...
Please see
https://doc.sagemath.org/html/en/installation/source.html#linux-recommended-installation

>
> Fortunately, I was able to solve my problem in spite of this unhelpful
> help. The build is not broken. But the instructions in the README
> file are incomplete. The answer which would have helped me is this:
>
> "If you switch branches in the git repository then you need to run
> make distclean before rebuilding sage."
>
> And that really should be added to the README file.
>
> - Marc
>
> PS If you are going to tell people that they do not have to build
> python to build sage then you should also provide instructions for how
> to build sage without building python.

Please see
https://doc.sagemath.org/html/en/installation/source.html#linux-recommended-installation
as well as read the last half a page of the output of ./configure

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CALcZXRFcfY9jS5HZY65AAT-ZWc5nt520toeJ%2Bv7c42Vmk0LMkQ%40mail.gmail.com.

[Matthias Koeppe]

There's now a proposed solution in the branch of https://trac.sagemath.org/ticket/32056, needs review