Unable to `load` or `import/cimport` Cython files into Sage while developing

[Jing Guo]

Dear all,

I have implemented an algorithm related to graph homomorphism counting, and recently I am thinking about improving its performance, so I think of writing some of the modules/codes in Cython.

For instance, I wrote helper functions in `helper_functions.pyx`, which can be imported sucessfully in Python 3, via `from helper_functions import *`.

However, when I run the above command in Sage notebook or Sage, the output reads

sage: import helper_functions
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In [1], line 1
----> 1 import helper_functions

ModuleNotFoundError: No module named 'helper_functions'

When I try to run `load("helper_functions.pyx")`, it will output a long line of errors (see end of the post).

I searched online for relevant keywords but couldn't find good solutions to these problems. I am on macOS 10.13.6

Thank you for your time!

Jing

sage: load("helper_functions.pyx")
Compiling ./helper_functions.pyx...
---------------------------------------------------------------------------
DistutilsExecError                        Traceback (most recent call last)
File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/unixccompiler.py:266, in UnixCCompiler.link(self, target_desc, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    264     linker = compiler_fixup(linker, ld_args)
--> 266     self.spawn(linker + ld_args)
    267 except DistutilsExecError as msg:

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/ccompiler.py:1041, in CCompiler.spawn(self, cmd, **kwargs)
   1040 def spawn(self, cmd, **kwargs):
-> 1041     spawn(cmd, dry_run=self.dry_run, **kwargs)

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/spawn.py:70, in spawn(cmd, search_path, verbose, dry_run, env)
     69     cmd = cmd[0]
---> 70 raise DistutilsExecError(
     71     "command {!r} failed with exit code {}".format(cmd, exitcode)
     72 )

DistutilsExecError: command '/opt/local/bin/gcc' failed with exit code 1

During handling of the above exception, another exception occurred:

LinkError                                 Traceback (most recent call last)
File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/sage/misc/cython.py:454, in cython(filename, verbose, compile_message, use_cache, create_local_c_file, annotate, sage_namespace, create_local_so_file)
    453     with redirection(2, errfile, close=False):
--> 454         dist.run_command("build")
    455 finally:

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/dist.py:989, in Distribution.run_command(self, command)
    986 # Postpone defaults until all explicit configuration is considered
    987 # (setup() args, config files, command line and plugins)
--> 989 super().run_command(command)

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/dist.py:988, in Distribution.run_command(self, command)
    987 cmd_obj.ensure_finalized()
--> 988 cmd_obj.run()
    989 self.have_run[command] = 1

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/command/build.py:131, in build.run(self)
    130 for cmd_name in self.get_sub_commands():
--> 131     self.run_command(cmd_name)

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/cmd.py:318, in Command.run_command(self, command)
    314 """Run some other command: uses the 'run_command()' method of
    315 Distribution, which creates and finalizes the command object if
    316 necessary and then invokes its 'run()' method.
    317 """
--> 318 self.distribution.run_command(command)

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/dist.py:989, in Distribution.run_command(self, command)
    986 # Postpone defaults until all explicit configuration is considered
    987 # (setup() args, config files, command line and plugins)
--> 989 super().run_command(command)

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/dist.py:988, in Distribution.run_command(self, command)
    987 cmd_obj.ensure_finalized()
--> 988 cmd_obj.run()
    989 self.have_run[command] = 1

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/command/build_ext.py:88, in build_ext.run(self)
     87 old_inplace, self.inplace = self.inplace, 0
---> 88 _build_ext.run(self)
     89 self.inplace = old_inplace

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py:345, in build_ext.run(self)
    344 # Now actually compile and link everything.
--> 345 self.build_extensions()

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py:467, in build_ext.build_extensions(self)
    466 else:
--> 467     self._build_extensions_serial()

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py:493, in build_ext._build_extensions_serial(self)
    492 with self._filter_build_errors(ext):
--> 493     self.build_extension(ext)

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/command/build_ext.py:249, in build_ext.build_extension(self, ext)
    248     self.compiler = self.shlib_compiler
--> 249 _build_ext.build_extension(self, ext)
    250 if ext._needs_stub:

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/Cython/Distutils/build_ext.py:135, in build_ext.build_extension(self, ext)
    134 ext.sources = new_ext.sources
--> 135 super(build_ext, self).build_extension(ext)

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py:572, in build_ext.build_extension(self, ext)
    570 language = ext.language or self.compiler.detect_language(sources)
--> 572 self.compiler.link_shared_object(
    573     objects,
    574     ext_path,
    575     libraries=self.get_libraries(ext),
    576     library_dirs=ext.library_dirs,
    577     runtime_library_dirs=ext.runtime_library_dirs,
    578     extra_postargs=extra_args,
    579     export_symbols=self.get_export_symbols(ext),
    580     debug=self.debug,
    581     build_temp=self.build_temp,
    582     target_lang=language,
    583 )

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/ccompiler.py:752, in CCompiler.link_shared_object(self, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    737 def link_shared_object(
    738     self,
    739     objects,
   (...)
    750     target_lang=None,
    751 ):
--> 752     self.link(
    753         CCompiler.SHARED_OBJECT,
    754         objects,
    755         output_filename,
    756         output_dir,
    757         libraries,
    758         library_dirs,
    759         runtime_library_dirs,
    760         export_symbols,
    761         debug,
    762         extra_preargs,
    763         extra_postargs,
    764         build_temp,
    765         target_lang,
    766     )

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/unixccompiler.py:268, in UnixCCompiler.link(self, target_desc, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    267     except DistutilsExecError as msg:
--> 268         raise LinkError(msg)
    269 else:

LinkError: command '/opt/local/bin/gcc' failed with exit code 1

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
Cell In [2], line 1
----> 1 load("helper_functions.pyx")

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/sage/misc/persist.pyx:175, in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:4520)()
    173
    174     if sage.repl.load.is_loadable_filename(filename):
--> 175         sage.repl.load.load(filename, globals())
    176         return
    177

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/sage/repl/load.py:277, in load(filename, globals, attach)
    275     if attach:
    276         add_attached_file(fpath)
--> 277     exec(load_cython(fpath), globals)
    278 elif ext == '.f' or ext == '.f90':
    279     from sage.misc.inline_fortran import fortran

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/sage/repl/load.py:68, in load_cython(name)
     55 """
     56 Helper function to load a Cython file.
     57
   (...)
     65   module.
     66 """
     67 from sage.misc.cython import cython
---> 68 mod, dir = cython(name, compile_message=True, use_cache=True)
     69 import sys
     70 sys.path.append(dir)

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/sage/misc/cython.py:460, in cython(filename, verbose, compile_message, use_cache, create_local_c_file, annotate, sage_namespace, create_local_so_file)
    458 except Exception as msg:
    459     msg = str(msg) + "\n" + distutils_messages
--> 460     raise RuntimeError(msg.strip())
    462 if verbose >= 0:
    463     sys.stderr.write(distutils_messages)

RuntimeError: command '/opt/local/bin/gcc' failed with exit code 1
ld: unknown option: -platform_version
collect2: error: ld returned 1 exit status

[Dima Pasechnik]

it seems you have gcc in /opt/local - what is it?
Not very usual. Normally speaking gcc on macOS is Apple's clang. How have you built (?) Sage?

[Jing Guo]

Sorry for the confusion. I used Sage downloaded from the website.

Gcc was installed with MacPorts.

Jing

--
You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/7jq8wyLfvrU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/829275C4-EB26-4203-904B-77D8A75201B5%40gmail.com.

[Dima Pasechnik]

On 27 December 2023 01:25:30 WET, Jing Guo <dev....@gmail.com> wrote:
>Sorry for the confusion. I used Sage downloaded from the website.

it's not clear what you mean.
We don't distribute Sage binaries, you need to build them.
Please provide more details.

>
>Gcc was installed with MacPorts.

gcc (gcc.gnu.org, not Apple's "gcc")
cannot be used to build Sage on macOS.

[Jing Guo]

I downloaded from here https://github.com/3-manifolds/Sage_macOS/releases

I was not and am not building Sage on Mac. To clarify, I’m simply “developing” this algorithm locally, though with the intention of submitting it to Sage source code.

Jing

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/654DF95C-F50F-42DC-8A98-66E58B427750%40gmail.com.

[Dima Pasechnik]

On 27 December 2023 01:45:58 WET, Jing Guo <dev....@gmail.com> wrote:
>I downloaded from here https://github.com/3-manifolds/Sage_macOS/releases
>
>I was not and am not building Sage on Mac. To clarify, I’m simply “developing” this algorithm locally, though with the intention of submitting it to Sage source code.

It's most probably a bug (or a feature) of this specific app.

It could be that compilers from MacPorts interfere here.
Try temporarily moving MacPorts away or uninstall it.

[Jing Guo]

I just did. The following is the new error.

➜  local-hom-count git:(main) ✗ sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 10.2, Release Date: 2023-12-03                    │
│ Using Python 3.11.1. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘

sage: load("helper_functions.pyx")
Compiling ./helper_functions.pyx...
---------------------------------------------------------------------------
DistutilsExecError                        Traceback (most recent call last)
File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/unixccompiler.py:266, in UnixCCompiler.link(self, target_desc, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
    264     linker = compiler_fixup(linker, ld_args)
--> 266     self.spawn(linker + ld_args)
    267 except DistutilsExecError as msg:

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/ccompiler.py:1041, in CCompiler.spawn(self, cmd, **kwargs)
   1040 def spawn(self, cmd, **kwargs):
-> 1041     spawn(cmd, dry_run=self.dry_run, **kwargs)

File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/spawn.py:70, in spawn(cmd, search_path, verbose, dry_run, env)
     69     cmd = cmd[0]
---> 70 raise DistutilsExecError(
     71     "command {!r} failed with exit code {}".format(cmd, exitcode)
     72 )

DistutilsExecError: command '/usr/bin/gcc' failed with exit code 1

LinkError: command '/usr/bin/gcc' failed with exit code 1

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)

Cell In [1], line 1

RuntimeError: command '/usr/bin/gcc' failed with exit code 1
ld: unknown option: -platform_version
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[Dima Pasechnik]

On Wed, Dec 27, 2023 at 12:23 PM Jing Guo <dev....@gmail.com> wrote:
>
> I just did. The following is the new error.
>
> ➜ local-hom-count git:(main) ✗ sage
> ┌────────────────────────────────────────────────────────────────────┐
> │ SageMath version 10.2, Release Date: 2023-12-03 │
> │ Using Python 3.11.1. Type "help()" for help. │
> └────────────────────────────────────────────────────────────────────┘
> sage: load("helper_functions.pyx")
> Compiling ./helper_functions.pyx...
> ---------------------------------------------------------------------------
> DistutilsExecError Traceback (most recent call last)
> File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/unixccompiler.py:266, in UnixCCompiler.link(self, target_desc, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang)
> 264 linker = compiler_fixup(linker, ld_args)
> --> 266 self.spawn(linker + ld_args)
> 267 except DistutilsExecError as msg:
>
> File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/ccompiler.py:1041, in CCompiler.spawn(self, cmd, **kwargs)
> 1040 def spawn(self, cmd, **kwargs):
> -> 1041 spawn(cmd, dry_run=self.dry_run, **kwargs)
>
> File /private/var/tmp/sage-10.2-current/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/setuptools/_distutils/spawn.py:70, in spawn(cmd, search_path, verbose, dry_run, env)
> 69 cmd = cmd[0]
> ---> 70 raise DistutilsExecError(
> 71 "command {!r} failed with exit code {}".format(cmd, exitcode)
> 72 )
>
> DistutilsExecError: command '/usr/bin/gcc' failed with exit code 1

Please check https://doc.sagemath.org/html/en/installation/index.html#macos
and note that you re doing development. I.e.
you are trying to do Sage development in an unsupported environment.
The app you are using does not come with all the tools needed to
create Python extensions from Cython (this needs Apple software, which
one cannot just bundle for copyright reasons).

Having said that, your errors probably indicate that you have an
outdated XCode/command line tools installed. But I don't know whether
this can be made to work at all, or you really need a different
install of Sage for your tasks.

> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/7e37ce49-be9c-4a24-a130-f9fbfb21a781n%40googlegroups.com.

[Vincent Delecroix]

Dear Jing,

Not being able to use Cython from this sage app is indeed problematic.
However as Dima said this is more a problem of the app rather than a
problem with sage itself. You would better open an issue at
https://github.com/3-manifolds/Sage_macOS. If you do, please post the
link of the issue in this thread.

Best
Vincent

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq0K7Ye7pEwvZN5JJddJvMcQc6LXDoOFKcHcGjwe4wOHHA%40mail.gmail.com.

[Jing Guo]

Thank you for all your helps. I asked in that GitHub repo and here is the issue: https://github.com/3-manifolds/Sage_macOS/issues/62

The reason it does not work is probably caused by my macOS being too old (10.13.6), so `ld` does not recognize `-platform_version` flag. MacPorts does not seem to support `ld` as well. I guess I will try to find other workarounds.

[Dima Pasechnik]

On 31 December 2023 23:05:42 WET, Jing Guo <dev....@gmail.com> wrote:
>Thank you for all your helps. I asked in that GitHub repo and here is the
>issue: https://github.com/3-manifolds/Sage_macOS/issues/62
>
>The reason it does not work is probably caused by my macOS being too old
>(10.13.6), so `ld` does not recognize `-platform_version` flag. MacPorts
>does not seem to support `ld` as well. I guess I will try to find other
>workarounds.

Building Sage with Homebrew is well-supported.
With MacPorts, no, noone I think managed to build Sage from source using a nontrivial number of MacPorts packages.
So it's probably a MacPorts problem too.

>> https://groups.google.com/d/msgid/sage-devel/654DF95C-F50F-42DC-8A98-66E58B427750%40gmail.com

>> .
>> > >> >
>> > >
>> > > --
>> > > You received this message because you are subscribed to the Google
>> Groups "sage-devel" group.
>> > > To unsubscribe from this group and stop receiving emails from it, send
>> an email to sage-devel+...@googlegroups.com.
>> > > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/sage-devel/7e37ce49-be9c-4a24-a130-f9fbfb21a781n%40googlegroups.com
>> .
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "sage-devel" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to sage-devel+...@googlegroups.com.
>> > To view this discussion on the web visit

>> https://groups.google.com/d/msgid/sage-devel/CAAWYfq0K7Ye7pEwvZN5JJddJvMcQc6LXDoOFKcHcGjwe4wOHHA%40mail.gmail.com
>> .
>>
>

[Jing Guo]

It’s fine, I’m not building Sage on Mac anyway.

I used homebrew in the past but it doesn’t support old macOS versions any more. Also I find MacPorts to be easier to manage and keep the versions consistent.

Jing

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/242E160C-04C1-42A7-A4E6-3D30FF3AB7A1%40gmail.com.

[Dima Pasechnik]

On 31 December 2023 23:20:28 WET, Jing Guo <dev....@gmail.com> wrote:
>It’s fine, I’m not building Sage on Mac anyway.

You're doing a complicated enough thing to put you into "building Sage" ballpark.

>
>I used homebrew in the past but it doesn’t support old macOS versions any more. Also I find MacPorts to be easier to manage and keep the versions consistent.

Oops, sorry, only now I realised how old macOS 10.13 actually is (it's well past EOL). I think you are seriously out of luck here, due to outdated Apple's tools for this platform.

[Isuru Fernando]

With conda, you can use sage on macOS 10.13. We supply our own linker built from Apple's sources.

Isuru

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/B905236D-1EE5-4586-81FA-D4807F496C24%40gmail.com.