correct way to compile cython project with icc?

1,223 views
Skip to first unread message

Zak Stone

unread,
Mar 5, 2011, 11:20:30 PM3/5/11
to cython...@googlegroups.com
Hello everyone,

I would like to compile a Cython project with Intel's compiler icc to
benchmark the resulting binary against one produced by gcc. It appears
that calling "python setup.py build_ext" with CC="icc" is enough to
switch the Cython compilation step to icc, but it is difficult to
prevent distutils from passing the wrong flags to icc, and distutils
insists on performing the linking step with gcc.

It is possible to follow NumPy's lead and patch distutils to support
icc -- that only requires a minor change to ccompiler.py and the
addition of a file like intelccompiler.py -- but I'd rather not
maintain a patch on my system-wide Python installation, and it appears
that virtualenv doesn't make it easy to patch Python itself in an
isolated local environment. It might be possible to use a modified
local copy of distutils only as NumPy seems to, but then I would also
have to modify Cython to interface with the modified copy of
distutils, and the resulting toolchain would be rather fragile.

Of course, there could be deeper problems still, considering that the
system Python installation was compiled with gcc, as were NumPy and
SciPy; I don't know whether gcc and icc play well together.

Is the "right solution" to use icc to compile a completely separate
installation of Python, NumPy, SciPy, and Cython and then to develop
my project in that environment? If so, could anyone recommend tools to
build and maintain such an environment? (pythonbrew appears to be a
step in the right direction.) Or is there a simpler route? For
example, the binary produced with CC="icc" in my current test case
appears to run properly despite the various gcc confounds, but I don't
know whether to trust it.

Thanks,
Zak

Robert Bradshaw

unread,
Mar 7, 2011, 5:18:44 PM3/7/11
to cython...@googlegroups.com

Distutils pulls its compiler and flag information from the Python
build it's executed with, so it'd be non-trivial to trick it into
using the correct icc flags. The easiest would be to either see if
there's some kind of compatibility mode or write a shell script that
does the appropriate flag mutation/dropping before passing things
along to icc. Fortunately, C, as opposed to C++, has a fairly well
defined ABI so you are much freer to mix compilers and it should just
work (though I'm speaking more from theory than practice here).
Someone on the distutils/Python lists would probably know better, as
once the .c file is produced Cython's job is over and it just uses the
regular Python C extension steps that it would had the .c file been
written by hand.

I'm pleasantly surprised that Cython emits icc compatible code (I
don't know that anyone's ever tested this before, but we do try to
stay pretty close to standard C). Cool.

- Robert

Dag Sverre Seljebotn

unread,
Mar 8, 2011, 2:25:39 AM3/8/11
to cython...@googlegroups.com

If you want this kind of flexibility I'd switch to a real build tool,
such as 'waf'. Here's one example; it does more than you need since it
also builds and links with Fortran code:

https://github.com/dagss/healpix4py/blob/9a730266a15ca8986165cfdc9077a8537ed595e9/wscript
https://github.com/dagss/healpix4py/blob/9a730266a15ca8986165cfdc9077a8537ed595e9/healpix/wscript

The downside is you can't that easily distribute your package in
standard ways, but "bento" is here to help with that:

http://cournape.wordpress.com/2011/03/06/adding-a-distutils-compatibility-layer-to-bento/

(I didn't try this yet, as I don't need to easy_install my packages.)
So, it's outside of the beaten path, but in the end it may be better
than wasting your time with distutils.

Dag Sverre

Dag Sverre Seljebotn

unread,
Mar 8, 2011, 2:29:57 AM3/8/11
to cython...@googlegroups.com

Just realized that waf may or may not work for your usecase, depending
on how good the Python tool in waf is.

At least if you're on Linux, if you get a module to compile and load
using 'icc' I think you're pretty much safe, even if 'gcc' is used to
link etc. IIRC 'icc' is written to be a "drop-in replacement" for gcc,
Intel very much understands that gcc is the de facto standard on Linux.

Dag Sverre

Dag Sverre Seljebotn

unread,
Mar 8, 2011, 2:32:37 AM3/8/11
to cython...@googlegroups.com


... and finally, I'd recommend that you get things working with a simple
shell script or Makefile first, and then look at using a build tool for
this.

The "python-config" program can help you get the flags you need (+ add
-fno-strict-aliasing if it's not there!).

Dag Sverre

Zak Stone

unread,
Mar 8, 2011, 4:23:09 PM3/8/11
to cython...@googlegroups.com, Dag Sverre Seljebotn
>>> If you want this kind of flexibility I'd switch to a real build tool,
>>> such as 'waf'. Here's one example; it does more than you need since it also
>>> builds and links with Fortran code:
>>>
>>> https://github.com/dagss/healpix4py/blob/9a730266a15ca8986165cfdc9077a8537ed595e9/wscript
>>>
>>> https://github.com/dagss/healpix4py/blob/9a730266a15ca8986165cfdc9077a8537ed595e9/healpix/wscript
>>
>> Just realized that waf may or may not work for your usecase, depending on
>> how good the Python tool in waf is.
>>
>> At least if you're on Linux, if you get a module to compile and load using
>> 'icc' I think you're pretty much safe, even if 'gcc' is used to link etc.
>> IIRC 'icc' is written to be a "drop-in replacement" for gcc, Intel very much
>> understands that gcc is the de facto standard on Linux.
>
> ... and finally, I'd recommend that you get things working with a simple
> shell script or Makefile first, and then look at using a build tool for
> this.
>
> The "python-config" program can help you get the flags you need (+ add
> -fno-strict-aliasing if it's not there!).

Thanks for all of the feedback, everyone! I hadn't realized how easy
it is to escape distutils and setup.py files entirely and compile a
simple Cython project manually. That makes it really easy to use icc,
which has produced significantly faster binaries in my informal tests
so far.

The main problem I encountered along the way was the somewhat
confusing message "ImportError: dynamic module does not define init
function". It turns out that I didn't understand the naming scheme for
C extensions; without a setup.py file, it seems necessary for the .pyx
file and the ultimate .so file to have the same name. Please feel free
to correct me if I still have this wrong -- I didn't see an option to
customize the module name for a .pyx file in either the cython binary
options or in the cythonize code in Cython/Build/Dependencies.py.

Thanks again,
Zak

Lisandro Dalcin

unread,
Mar 8, 2011, 5:02:17 PM3/8/11
to cython...@googlegroups.com

Yes, the names should match.

> I didn't see an option to
> customize the module name for a .pyx file in either the cython binary
> options or in the cythonize code in Cython/Build/Dependencies.py.
>

supose you have foo.pyx, but you are going to get bar.so from it. Then
you could try to pass -Dinitfoo=initbar to the C compiler. However, I
agree Cython should have an option to tweak the module name.


--
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

Robert Bradshaw

unread,
Mar 8, 2011, 6:49:35 PM3/8/11
to cython...@googlegroups.com

I wouldn't oppose such an addition, but I never understood why people
are so set on naming their .pyx, etc. files differently than the
eventual module names (and one is required to do for .py files).

- Robert

Zak Stone

unread,
Mar 9, 2011, 12:29:06 AM3/9/11
to cython...@googlegroups.com, Robert Bradshaw
> I wouldn't oppose such an addition, but I never understood why people
> are so set on naming their .pyx, etc. files differently than the
> eventual module names (and one is required to do for .py files).

In my case, I'm still looking for the clearest naming scheme for the
several related files in a Cython project. Suppose I start with
reference implementations of a few algorithms in algs.py and pure-C
implementations of the same algorithms in algs.c. I then need to write
a Cython wrapper to manage the preparation of NumPy arrays and other
variables to feed to the C implementations, but I can't call that file
algs.pyx, since the generated C filename would be algs.c, which is
already in use. So I name the file something like algs_wrapper.pyx
instead.

In the end, given the rule that the .pyx filename determines the
ultimate module name, I end up importing my C algorithm
implementations from "algs_wrapper" in Python, but I would find it
clearer to import them from a module called something like "c_algs". A
small point, perhaps.

I'd be delighted to hear suggestions from others about readable Cython
naming schemes.

Thanks,
Zak

Lisandro Dalcin

unread,
Mar 9, 2011, 8:49:11 AM3/9/11
to cython...@googlegroups.com
On 9 March 2011 02:29, Zak Stone <zst...@gmail.com> wrote:
>> I wouldn't oppose such an addition, but I never understood why people
>> are so set on naming their .pyx, etc. files differently than the
>> eventual module names (and one is required to do for .py files).
>
> In my case, I'm still looking for the clearest naming scheme for the
> several related files in a Cython project. Suppose I start with
> reference implementations of a few algorithms in algs.py and pure-C
> implementations of the same algorithms in algs.c. I then need to write
> a Cython wrapper to manage the preparation of NumPy arrays and other
> variables to feed to the C implementations, but I can't call that file
> algs.pyx, since the generated C filename would be algs.c, which is
> already in use. So I name the file something like algs_wrapper.pyx
> instead.
>

$ touch algs.pyx
$ cython algs.pyx -o algs_wrapper.c
$ ls algs*
algs.pyx algs_wrapper.c

Zak Stone

unread,
Mar 9, 2011, 11:38:07 AM3/9/11
to cython...@googlegroups.com
>>> I wouldn't oppose such an addition, but I never understood why people
>>> are so set on naming their .pyx, etc. files differently than the
>>> eventual module names (and one is required to do for .py files).
>>
>> In my case, I'm still looking for the clearest naming scheme for the
>> several related files in a Cython project. Suppose I start with
>> reference implementations of a few algorithms in algs.py and pure-C
>> implementations of the same algorithms in algs.c. I then need to write
>> a Cython wrapper to manage the preparation of NumPy arrays and other
>> variables to feed to the C implementations, but I can't call that file
>> algs.pyx, since the generated C filename would be algs.c, which is
>> already in use. So I name the file something like algs_wrapper.pyx
>> instead.
>>
>
> $ touch algs.pyx
> $ cython algs.pyx -o algs_wrapper.c
> $ ls algs*
> algs.pyx  algs_wrapper.c

Ah, of course! And then the ultimate module name will be algs instead
of algs_wrapper?

Thanks,
Zak

Christopher Barker

unread,
Mar 9, 2011, 11:47:52 AM3/9/11
to cython...@googlegroups.com
On 3/8/11 9:29 PM, Zak Stone wrote:
>> I never understood why people
>> are so set on naming their .pyx, etc. files differently than the
>> eventual module names (and one is required to do for .py files).

Keeping this in mind: a python (*.py) file name root is the name of the
module

So it makes sense that a cython (*.pyx) filename root should also be
the name of the module
...

> Suppose I start with
> reference implementations of a few algorithms in algs.py and pure-C
> implementations of the same algorithms in algs.c.

why does the *.c file need the same name? C doesn't have the same
concept of filename<=>module name -- indeed doesn't ahve modules at all
-- so why not call that c_algs (or something more descriptive)?

> a Cython wrapper to manage the preparation of NumPy arrays and other
> variables to feed to the C implementations, but I can't call that file
> algs.pyx,

Does this Cython code duplicate all the functionaility of the algs.py?
If so, then algs.pyx is perfect, if not, perhaps it should have another
name as well.

IF you ignore the multiple languages here, it seems to me you have:

1) a module with assorted implimentations of algorithms
2) a sub-module that is that impliments some of these
3) a collection of routimes used by (2)

If these were all written in the same language, you wouldn't give them
all the same names!

Things get a bit tricky when you duplicate functionality in Python and
Cython -- in that case, you might want them to have the same names -- at
least individual functions should. In this case, there are a couple options:

1) maybe you can use Cython's pure python mode to keep it all in one
file, rather than a python and a cython version.

2) give the cython module a different name: cy_algs.pyx maybe.
Fortunately, with Python symantics for importing and name binding, it's
easy to do:

import cy_algs as algs

and you're done.

But this also lets you import the two independently in test code that
compares the two.

> So I name the file something like algs_wrapper.pyx

I like the _wrapper convention for code that is wrapping an existing,
stand-alone C lib. It sounds like that isn't the case here -- this code
is written specifically to provide stuff for the Cython code. That's
really a nit-pick, though.

> In the end, given the rule that the .pyx filename determines the
> ultimate module name, I end up importing my C algorithm
> implementations from "algs_wrapper" in Python, but I would find it
> clearer to import them from a module called something like "c_algs". A
> small point, perhaps.

now I'm really confused -- why not call your cython file "c_algs.py"
then? you'd get exactly what you say you want!

-Chris


--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov

Zak Stone

unread,
Mar 10, 2011, 9:15:06 PM3/10/11
to cython...@googlegroups.com
> Keeping this in mind: a python (*.py) file name root is the name of the
> module
>
> So it makes sense that a cython  (*.pyx) filename root should also be the
> name of the module

Agreed -- the naming complications arise in my case because the Cython
.pyx file is merely a thin wrapper, the C implementations have to be
kept separate in a pure-C file, and I'm trying to make the project
layout as usable as possible for a C expert who is unfamiliar with
Cython.

Given the suggestions in this thread, it would probably work well to
keep the pure-Python implementation in "algs.py", write the Cython
wrapper in "c_algs.pyx", use "cython c_algs.pyx -o c_algs_wrapper.c"
to name the Cython output file more descriptively (and make clear it
isn't the first thing to read), and to keep the pure-C implementations
in "algs.c". Then the Python module name for the C library would be
c_algs, which makes sense.

> why does the *.c file need the same name? C doesn't have the same concept of
> filename<=>module name -- indeed doesn't ahve modules at all -- so why not
> call that c_algs (or something more descriptive)?

Since the overall goal is to compare Python implementations with
pure-C implementations, I liked the idea of a parallel "algs.py" /
"algs.c" naming scheme for the most salient parts of the code. This is
merely a matter of taste, and I'm still learning which project layouts
are most readable.

> Does this Cython code duplicate all the functionaility of the algs.py? If
> so, then algs.pyx is perfect, if not, perhaps it should have another name as
> well.

The Cython code is a thin wrapper at present, but I may eventually add
one or more intermediate Cython implementations to the benchmark,
which will make clear naming even more complicated.

> Things get a bit tricky when you duplicate functionality in Python and
> Cython -- in that case, you might want them to have the same names -- at
> least individual functions should.

Exactly. This is my benchmarking situation.

> In this case, there are a couple options:
>
> 1) maybe you can use Cython's pure python mode to keep it all in one file,
> rather than a python and a cython version.

In this case, I need a pure-C implementation in a separate file.

> 2) give the cython module a different name: cy_algs.pyx maybe. Fortunately,
> with Python symantics for importing and name binding, it's easy to do:

Good idea.

> I like the _wrapper convention for code that is wrapping an existing,
> stand-alone C lib.

That's exactly the case here; sorry that wasn't clear. I am using
Cython (in this project) purely as a wrapper for algorithm
implementations that must be written in pure C, and I am comparing the
performance of a pure-Python implementation of each algorithm with the
corresponding pure-C implementation accessed via Cython.

> now I'm really confused -- why not call your cython file "c_algs.py" then?
> you'd get exactly what you say you want!

This turned out to be part of the solution above!

Thanks to all,
Zak

ashwinD12

unread,
Jun 18, 2016, 4:31:06 AM6/18/16
to cython-users
I am trying to build cython using icc. Did you ever get this to work ?
Reply all
Reply to author
Forward
0 new messages