Cython as toplevel script (python -m my_cython_module)?

[Jon Olav Vik]

Regular Python modules can be run as a "top-level script" by using
"python -m modulename".
http://docs.python.org/library/__main__.html
Python will look up the module wherever it is on the PYTHONPATH, and
its __name__ will be set to "__main__", a convention indicating that
the module is being run as a script rather than imported.

This does not work for my compiled Cython modules; it fails with "No
code object available for my_cython_module".

Is this maybe what "cython --embed" is for?
http://wiki.cython.org/EmbeddingCython
Will it allow "python -m my_cython_module"?

I'm not familiar with manual compilation (I'm working on Windows 7 and
Linux) and usually just copy an existing setup.py file whenever I
write a new Cython module. Is there a way to specify the --embed
option using setup.py or the "python setup.py ..." command? Or to ask
it not to re-generate an existing .c file? (I can run "cython --embed
pyxtest.pyx" just fine, and obtain a .c file, but I have no clue where
to go from there.)

Thanks in advance for any help.

Regards,
Jon Olav Vik

[Stefan Behnel]

Jon Olav Vik, 21.07.2011 14:52:

The latest Cython sources have a "cythonrun" script that does what you
want. We considered merging it into the main "cython" script, but that
hasn't been done yet.

https://github.com/cython/cython/blob/master/bin/cythonrun

Stefan

[Jon Olav Vik]

Thanks for the suggestion. It doesn't quite do what I desired, though.

I upgraded the Cython on a VirtualBox Ubuntu using
cd ~/git/cython
git branch master
git pull
sudo python setup.py install
(Disclaimer: I know next to nothing about git either.)

Cython seems to be working fine. However, it seems "cythonrun" isn't
installed anywhere on the PATH or PYTHONPATH.
$ cythonrun
cythonrun: command not found
$ python -m cythonrun
/usr/bin/python: No module called cythonrun

The docstring for cythonrun says that basic usage is "python cythonrun
somefile.py [ARGS]". Hence I tried:

jon@jon-VirtualBox:~$ ~/git/cython/bin/cythonrun ~/git/cgp-devel/
pyxtest/pyxtest.pyx
Using Cython 0.14.1+ to compile /home/jon/git/cgp-devel/pyxtest/
pyxtest.pyx
gcc -pthread -c -o /home/jon/git/cgp-devel/pyxtest/pyxtest.o /home/jon/
git/cgp-devel/pyxtest/pyxtest.c -I/usr/include/python2.6 -fno-strict-
aliasing -g -O2 -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes
gcc -pthread -o /home/jon/git/cgp-devel/pyxtest/pyxtest /home/jon/git/
cgp-devel/pyxtest/pyxtest.o -L/usr/lib -L/usr/lib/python2.6/config -
lpython2.6 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -
Wl,-Bsymbolic-functions
/home/jon/git/cgp-devel/pyxtest/pyxtest
Hello Cython!

This differs in some respects from the "python -m pyxtest"
functionality that I would like:
* It recompiles the module every time, rather than using whatever
pyxtest.so or pyxtest.pyd already exists.
* It requires me to know the exact path (and extension) of the module,
rather than looking it up on the PYTHONPATH.
* Installing Cython doesn't copy "cythonrun" to a bin directory on the
PATH. Of course, I could do that manually; I'm merely suggesting that
cythonrun be treated like e.g. pydoc, which can be invoked by either
"pydoc ..." (on Linux) or "python -m pydoc ...".
* It seems to ignore the setup.py file (?); then how would I specify
e.g. include_dirs and library_dirs?

This isn't a big deal for me at the moment; the above is just feedback
to developers in case they're interested. Thank you for your help.

Best regards,
Jon Olav

[Stefan Behnel]

Jon Olav Vik, 21.07.2011 23:15:

> This differs in some respects from the "python -m pyxtest"
> functionality that I would like:
> * It recompiles the module every time, rather than using whatever
> pyxtest.so or pyxtest.pyd already exists.
> * It requires me to know the exact path (and extension) of the module,
> rather than looking it up on the PYTHONPATH.
> * Installing Cython doesn't copy "cythonrun" to a bin directory on the
> PATH. Of course, I could do that manually; I'm merely suggesting that
> cythonrun be treated like e.g. pydoc, which can be invoked by either
> "pydoc ..." (on Linux) or "python -m pydoc ...".

Sure. We take patches.

> * It seems to ignore the setup.py file (?);

Using a "setup.py" doesn't make sense for a module on the PYTHONPATH, so
that's a contradicting goal.

> then how would I specify
> e.g. include_dirs and library_dirs?

According to the source code (hint, hint), you can at least pass CFLAGS.
Don't think there's currently a way to pass external libraries.

Stefan

[Jon Olav Vik]

On Jul 21, 11:28 pm, Stefan Behnel <stefan...@behnel.de> wrote:
> Jon Olav Vik, 21.07.2011 23:15:
>
> > This differs in some respects from the "python -m pyxtest"
> > functionality that I would like:
> > * It recompiles the module every time, rather than using whatever
> > pyxtest.so or pyxtest.pyd already exists.
> > * It requires me to know the exact path (and extension) of the module,
> > rather than looking it up on the PYTHONPATH.
> > * Installing Cython doesn't copy "cythonrun" to a bin directory on the
> > PATH. Of course, I could do that manually; I'm merely suggesting that
> > cythonrun be treated like e.g. pydoc, which can be invoked by either
> > "pydoc ..." (on Linux) or "python -m pydoc ...".
>
> Sure. We take patches.

I'm aware of that. However, makefiles and distutils, compiler issues
and the internal workings of Cython are all somewhat beyond me, so all
I can offer for now is to articulate my suggestions as an appreciative
end-user.

> > * It seems to ignore the setup.py file (?);
>
> Using a "setup.py" doesn't make sense for a module on the PYTHONPATH, so
> that's a contradicting goal.

Sorry if I was being unclear, but what I've always done is "compile
once, then place on pythonpath". Doesn't Cython itself use a setup.py
to install itself on the pythonpath? (If I were more familiar with
distutils, I guess I could make an "install" option in addition to
"build_ext".) Currently, the actual way I compile my modules and place
them on the pythonpath is as follows.

1. Write my .pyx file.
2. Hack together a setup.py file by modifying an existing one.
3. python setup.py build_ext --inplace.
4. Somehow place the compiled module on the PYTHONPATH:
4a. Manually copy the .so or .pyd file to somewhere on the pythonpath,
or
4b. Have the .pyx and something_setup.py file already on the
pythonpath, so the .so or .pyd file also ends up there (this requires
giving each something_setup.py a unique name, and removing the "build"
subdirectory between compilations).
4c. Make subdirectories, within the pythonpath, with something.pyx and
setup.py, so I can "import something.something" (this is what I do for
cythonization of autogenerated code).

I may be missing the nuances between using a setup.py (using
distutils, if I understand correctly) and using Cython.Build. Also,
there is pyximport, which I've only got to work when I didn't need to
specify include_dirs or library_dirs. Is there some documentation I
ought to read about this?

> > then how would I specify
> > e.g. include_dirs and library_dirs?
>
> According to the source code (hint, hint),

Please understand that I do put in quite a lot of googling, reading
docstrings, and also looking at source code before bothering the list
with questions. (Digression: My only experience with CFLAGS comes from
following other people's installation instructions, and both CFLAGS,
include_dirs and library_dirs look like voodoo from my point of
vantage.) I represent the user who is quite skilled in pure Python and
high-level scientific computing, and is using Cython to minimize my
dealings with the intricacies of C. I think this is an important
market segment for Cython, but users are likely to be alienated by
admonishments for not knowing how to debug C compilation.

> you can at least pass CFLAGS.
> Don't think there's currently a way to pass external libraries.

Okay, thank you. In either case, it seems cythonrun is solving a
different issue than mine.

Jon Olav

[Robert Bradshaw]

If you're not doing something too complicated,

from distutils.core import setup
from Cython.Build import cythonize

setup(
name = 'MyProject',
ext_modules = cythonize(["*.pyx"]),
)

could be the only setup.py file you'd ever need.

>> > then how would I specify
>> > e.g. include_dirs and library_dirs?

In your .pyx file you can do

# distutils: include_dirs = /opt/food/include

if needed. See http://wiki.cython.org/enhancements/distutils_preprocessing

>> According to the source code (hint, hint),
>
> Please understand that I do put in quite a lot of googling, reading
> docstrings, and also looking at source code before bothering the list
> with questions. (Digression: My only experience with CFLAGS comes from
> following other people's installation instructions, and both CFLAGS,
> include_dirs and library_dirs look like voodoo from my point of
> vantage.) I represent the user who is quite skilled in pure Python and
> high-level scientific computing, and is using Cython to minimize my
> dealings with the intricacies of C. I think this is an important
> market segment for Cython, but users are likely to be alienated by
> admonishments for not knowing how to debug C compilation.

If you're just using Cython, you don't need to worry about details
like this. It's only when trying to interface with C libraries that
you have to know a bit about C compilation, and there's not much we
can do to avoid that.

- Robert

[Brent Pedersen]

That would be nice. How about this:

diff --git a/Cython/Compiler/Main.py b/Cython/Compiler/Main.py
index db765e3..ac21f74 100644
--- a/Cython/Compiler/Main.py
+++ b/Cython/Compiler/Main.py
@@ -647,6 +647,11 @@ def setuptools_main():
def main(command_line = 0):
args = sys.argv[1:]
any_failures = 0
+
+ if sys.argv[1] == "run":
+ from Cython.Build.BuildExecutable import build_and_run
+ return build_and_run(sys.argv[2:])
+
if command_line:
from CmdLine import parse_command_line
options, sources = parse_command_line(args)

[Stefan Behnel]

Brent Pedersen, 26.07.2011 22:36:

> On Thu, Jul 21, 2011 at 7:17 AM, Stefan Behnel wrote:
>> Jon Olav Vik, 21.07.2011 14:52:
>>>
>>> Regular Python modules can be run as a "top-level script" by using
>>> "python -m modulename".
>>> http://docs.python.org/library/__main__.html
>>> Python will look up the module wherever it is on the PYTHONPATH, and
>>> its __name__ will be set to "__main__", a convention indicating that
>>> the module is being run as a script rather than imported.
>>>
>>> This does not work for my compiled Cython modules; it fails with "No
>>> code object available for my_cython_module".
>>>
>>> Is this maybe what "cython --embed" is for?
>>> http://wiki.cython.org/EmbeddingCython
>>> Will it allow "python -m my_cython_module"?
>>>
>>> I'm not familiar with manual compilation (I'm working on Windows 7 and
>>> Linux) and usually just copy an existing setup.py file whenever I
>>> write a new Cython module. Is there a way to specify the --embed
>>> option using setup.py or the "python setup.py ..." command? Or to ask
>>> it not to re-generate an existing .c file? (I can run "cython --embed
>>> pyxtest.pyx" just fine, and obtain a .c file, but I have no clue where
>>> to go from there.)
>>
>> The latest Cython sources have a "cythonrun" script that does what you want.
>> We considered merging it into the main "cython" script, but that hasn't been
>> done yet.
>>
>> https://github.com/cython/cython/blob/master/bin/cythonrun
>

> That would be nice. How about this:
>
>
> diff --git a/Cython/Compiler/Main.py b/Cython/Compiler/Main.py
> index db765e3..ac21f74 100644
> --- a/Cython/Compiler/Main.py
> +++ b/Cython/Compiler/Main.py
> @@ -647,6 +647,11 @@ def setuptools_main():
> def main(command_line = 0):
> args = sys.argv[1:]
> any_failures = 0
> +
> + if sys.argv[1] == "run":
> + from Cython.Build.BuildExecutable import build_and_run
> + return build_and_run(sys.argv[2:])
> +
> if command_line:
> from CmdLine import parse_command_line
> options, sources = parse_command_line(args)

That doesn't add anything to the current interface, though. The idea is
rather to support the "-m" switch in the same way CPython does, so that you
can just say "cython -m somemodule", and have it find the module in the
PYTHONPATH, etc.

Stefan

[Matan]

A group search still leads here for this topic. Running a cythonized python file as a python (-m) module still yields the same error, I assume it is simply not a supported use case. Adding a plain python source that calls the cythonized module side-steps the issue, which can be helpful when e.g. building one's first cython function and seeing it run.