Python to Cython conversion

[Waqas Muhi]

Hello,

I'm currently working on a project for my Master's which involves optimizing a fairly big simulation system originally written in Python. I've read about and researched Cython and I think it would give me the best performance improvements. My software system basically is a bunch of Python .py files and the top-level module makes use of functions defined in them as appropriate.

Based on some profiling, I've been able to identify a few of the .py files which are good candidates for the conversion. In particular, there are a few functions in one particular file - let's say xyz.py - which I'm looking to convert. My questions are:

1) Is it possible to convert a few of the functions to Cython and leave the rest with the original Python code?

2) Would I have to rename the file to use the .pyx extension if the above is possible?

3) If 1) is not possible, would file xyz.py still work properly with the other .py files in the software system if I do a full conversion to Cython? In other words, would anything change from the other files' perspective?

Any feedback would be highly appreciated. Thanks a lot!

Waqas

[mark florisson]

On 5 February 2012 23:19, Waqas Muhi <waqas...@gmail.com> wrote:
> Hello,
>
> I'm currently working on a project for my Master's which involves optimizing
> a fairly big simulation system originally written in Python. I've read about
> and researched Cython and I think it would give me the best performance
> improvements. My software system basically is a bunch of Python .py files
> and the top-level module makes use of functions defined in them as
> appropriate.
>
> Based on some profiling, I've been able to identify a few of the .py files
> which are good candidates for the conversion. In particular, there are a few
> functions in one particular file - let's say xyz.py - which I'm looking to
> convert. My questions are:
>
> 1) Is it possible to convert a few of the functions to Cython and leave the
> rest with the original Python code?

Sure.

> 2) Would I have to rename the file to use the .pyx extension if the above is
> possible?

Yes, I believe that if you use the .py extension Cython expects a
pure-mode Python file (possibly annotated with Cython types etc).
That's probably not what you want.

> 3) If 1) is not possible, would file xyz.py still work properly with the
> other .py files in the software system if I do a full conversion to Cython?
> In other words, would anything change from the other files' perspective?

You can keep xyz.py and have it import performance-critical functions
from _xyz.pyx.

> Any feedback would be highly appreciated. Thanks a lot!

Just give the Cython tutorials a go and it will be a lot clearer to you :)

> Waqas

[Henry Gomersall]

On Mon, 2012-02-06 at 21:02 +0000, mark florisson wrote:
> > 2) Would I have to rename the file to use the .pyx extension if the
> above is
> > possible?
>
> Yes, I believe that if you use the .py extension Cython expects a
> pure-mode Python file (possibly annotated with Cython types etc).
> That's probably not what you want.

If you didn't want to put the whole thing in a cython module, you could
refactor your code so only the bits that needed to be made faster were
in that module.

Cheers,

Henry

[Stefan Behnel]

mark florisson, 06.02.2012 22:02:

> On 5 February 2012 23:19, Waqas Muhi wrote:
>> 2) Would I have to rename the file to use the .pyx extension if the above is
>> possible?
>
> Yes, I believe that if you use the .py extension Cython expects a
> pure-mode Python file (possibly annotated with Cython types etc).
> That's probably not what you want.

Well, "probably" is too strong a word here. It may or may not be what the
OP wants. Pure mode works pretty well for many things, although it has its
limits. The OP should start with that, and only switch to .pyx files in
Cython syntax where it turns out to be truly necessary.

Pure mode keeps having the major advantage of being pure Python, and thus
working in anything that can run, analyse or debug Python code.

Stefan

[Waqas Muhi]

Thanks a lot for all the replies so far. I'll read them properly and post again if I have any more questions!

[Waqas Muhi]

Hello again,

@Stefan, all the files are already written in Python so what do you mean when you say "pure-mode"?

@Mark, when you say "annotated with Cython types", do you mean a pure Python file with Cython static typing as described on this page here: http://docs.cython.org/src/quickstart/cythonize.html?

Another point I would like to add is that my simulation system (including this xyz.py file) makes heavy use of numpy. I read in an old thread on this group that I could run into some problems during my Cython conversion because of this. What kind of problems can I run into?

Thanks for all the help. I'm a Cython newbie so please don't mind some silly questions :)

Waqas

[Stefan Behnel]

Waqas Muhi, 09.02.2012 02:59:

> @Stefan, all the files are already written in Python so what do you mean
> when you say "pure-mode"?

http://docs.cython.org/src/tutorial/pure.html

> @Mark, when you say "annotated with Cython types", do you mean a pure
> Python file with Cython static typing as described on this page here:
> http://docs.cython.org/src/quickstart/cythonize.html?

He meant the same thing that I referenced above.

> Another point I would like to add is that my simulation system (including
> this xyz.py file) makes heavy use of numpy.

Ah, in that case, you'd rather want to move the respective code into a .pyx
file, because that would allow you to interface with NumPy arrays directly
at the C level. Pure mode doesn't currently give you that.

Here are some examples:

http://docs.cython.org/src/tutorial/numpy.html

> I read in an old thread on this
> group that I could run into some problems during my Cython conversion
> because of this. What kind of problems can I run into?

You should give us a link to the thread, but off the top of my head, I
wouldn't know of any problems. Cython is very well integrated with NumPy,
simply because there is such a large user base that combine both.

Stefan

[Aronne Merrelli]

On Wed, Feb 8, 2012 at 9:56 PM, Stefan Behnel <stef...@behnel.de> wrote:

> I read in an old thread on this
> group that I could run into some problems during my Cython conversion
> because of this. What kind of problems can I run into?

You should give us a link to the thread, but off the top of my head, I
wouldn't know of any problems. Cython is very well integrated with NumPy,
simply because there is such a large user base that combine both.

Stefan

I think this might have been related to a comment I made recently. What I meant was if you are starting from a piece of python code that contains a lot of calls to scipy/numpy functions, it may not be trivial to "cythonize" it. A relatively simple example would be an algorithm that is inverting a matrix or solving a linear system in a nested loop. If you are doing that with calls to a function in numpy.linalg, a significant bottleneck will remain at that numpy.linalg call even if you convert everything else to static types, because there will be a lot of overhead reaching back to the python-level module. (At least I am pretty certain this is true, though I have not tried that specific example). You'd need to replace it with a direct call into the BLAS library (or equivalent), I think, in order to get the speed up you might expect. That may or may not be "simple" for one to implement, depending on your level of knowledge with the C-level of things.

So by "problems" I was mainly thinking about complications and unanticipated complexity during cythonization. Is that clearer?

Cheers,
Aronne

[Robert Bradshaw]

That is true, though fortunately "cythonize" is not an all or none
thing. Depending on your problem and the size of your data, the Python
call overhead in numpy.linalg may be negligible compared to the amount
of work done in numpy.linalg. (Indeed, in the extreem case where this
call dominates your runtime, there's little advantage to Cythonizing
at all). On the other hand Cython particularly shines if you're doing
any element-wise manipulations of primitive types. Most code falls
somewhere in the middle.

- Robert

[Henry Gomersall]

On Wed, 2012-02-08 at 20:29 -0800, Robert Bradshaw wrote:
> That is true, though fortunately "cythonize" is not an all or none
> thing. Depending on your problem and the size of your data, the Python
> call overhead in numpy.linalg may be negligible compared to the amount
> of work done in numpy.linalg. (Indeed, in the extreem case where this
> call dominates your runtime, there's little advantage to Cythonizing
> at all). On the other hand Cython particularly shines if you're doing
> any element-wise manipulations of primitive types. Most code falls
> somewhere in the middle

The other big advantage of course is that you *can* call C libs that
might be faster.

Henry

[Sturla Molden]

On 09.02.2012 05:29, Robert Bradshaw wrote:

> That is true, though fortunately "cythonize" is not an all or none
> thing. Depending on your problem and the size of your data, the Python
> call overhead in numpy.linalg may be negligible compared to the amount
> of work done in numpy.linalg. (Indeed, in the extreem case where this
> call dominates your runtime, there's little advantage to Cythonizing
> at all). On the other hand Cython particularly shines if you're doing
> any element-wise manipulations of primitive types. Most code falls
> somewhere in the middle.

One thing to remember here is that NumPy code tends to be memory bound,
whereas Cython, C, or Fortran code tend to be CPU bound. Also, NumPy and
SciPy do not always release the GIL when they should, e.g. they do not
know if the LAPACK library is reentrant.

Sturla

[Waqas Muhi]

Thanks for all the replies once again. Really appreciated. Things are definitely getting clearer now. I'll post if I've any more questions.

[Waqas Muhi]

One thing I forgot: is there any support in Cython for Python dictionaries?

[Sturla Molden]

Den 13.02.2012 00:40, skrev Waqas Muhi:
> One thing I forgot: is there any support in Cython for Python dictionaries?
>
>

Just use them as you would in Python. But remember that keys and values
are always of type object. If you need something similar for C or C++
types, consider the "unordered associative containers" in the STL for
the most recent C++ standard (C++11). They correspond closely to
Python's set and dict.

http://en.wikipedia.org/wiki/Unordered_associative_containers_%28C%2B%2B%29

(These are a reason I actually consider to use C++ again.)

Sturla

[Waqas Muhi]

Hello once again,

How do I import my performance-critical functions from _xyz.pyx to xyz.py? I'm currently doing this in xyz.py:

from _xyz import _performance_critical_function_1

But when I try to run my top-level module using Python, it complains saying "No module named _xyz". What am I doing wrong here?

Thanks,

Waqas

[Aronne Merrelli]

On Sun, Feb 19, 2012 at 4:25 PM, Waqas Muhi <waqas...@gmail.com> wrote:

Hello once again,

How do I import my performance-critical functions from _xyz.pyx to xyz.py? I'm currently doing this in xyz.py:

from _xyz import _performance_critical_function_1

But when I try to run my top-level module using Python, it complains saying "No module named _xyz". What am I doing wrong here?

Thanks,

Waqas

See the documentation here:

http://docs.cython.org/src/quickstart/build.html

You need to compile the code into a shared object file (manually or with distutils), or use pyximport. If you are just writing/running code yourself and not distributing to others, then pyximport is the easiest. I suggest this:

import pyximport
pyximport.install(reload_support=True)

After that runs, commands like "import stuff" will do the conversion/compilation from stuff.pyx to the shared object file. By setting reload_support, if you need to reload the cython module after changing any code, it will then be properly recompiled and reloaded into the current session.

Also, note that using pyximport means the C translation and compilation to *.so will happen in subdirectories from ~/.pyxbld - in rare cases you might get this out of sync; in which case you might need to delete those files manually to get the compilation to start over from scratch. I've had to do this a couple times - I can't remember exactly what I did, but I think it was something dumb like having two versions of a .pyx file with the same name.

HTH,
Aronne

[Waqas Muhi]

Hi,

I didn't get a chance to test this out until today. Aronne, I did what you suggested i.e. using import pyximport, however, now I get an error message saying "ImportError: No module names pyximport". I do have Cython installed so I'm not sure what's going wrong.

Any help would be appreciated.

Thanks,
Waqas

[Aronne Merrelli]

On Sun, Apr 1, 2012 at 3:30 PM, Waqas Muhi <waqas...@gmail.com> wrote:
> Hi,
>
> I didn't get a chance to test this out until today. Aronne, I did what you
> suggested i.e. using import pyximport, however, now I get an error message
> saying "ImportError: No module names pyximport". I do have Cython installed
> so I'm not sure what's going wrong.
>
> Any help would be appreciated.

I'm not sure - for me, the pyximport module has always been installed
simultaneously with cython, with no problems. I would guess that means
the cython installation did not work, or perhaps it is not in your
python path... Does "import cython" also fail?

Aronne

[Waqas Muhi]

Hi,

Yes, import cython also fails. Following is the cmd window output when I type "python" and "cython":

C:\>python
Python 2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

C:\>cython
'cython' is not recognized as an internal or external command,
operable program or batch file.

This does seem to indicate that either cython is not in my python path or the cython installation didn't work. I modified both the 'Path' environment variable and the "PYTHONPATH" environment variable with C:\Cython-0.15.1 but I still get the same messages as above.

Where am I going wrong?

Thanks,
Waqas

[Chris Barker]

> Yes, import cython also fails. Following is the cmd window output when I
> type "python" and "cython":
>
> C:\>python
> Python 2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)]
> on
> win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> exit()
>
> C:\>cython
> 'cython' is not recognized as an internal or external command,
> operable program or batch file.

that's not an import. Try:

import cython

once at the python prompt

> This does seem to indicate that either cython is not in my python path or
> the cython installation didn't work. I modified both the 'Path' environment
> variable and the "PYTHONPATH" environment variable with C:\Cython-0.15.1 but
> I still get the same messages as above.
>
> Where am I going wrong?

PATH shoujld be pointed to somethign like:

C:\Python27\Scripts

That's where the main Cython script lies (and other scripts installed
with packages).

And depending on how Windows is set up and Cython was installed, you may need:

cython.py at the DOS prompt.

You could also try:

C:\python27\Scripts\cython.py

(I may have that a bit wrong -- no Windows box handy right now)

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Waqas Muhi]

Yes, I did try putting in the "import cython" statement in my code. I didn't mean to imply with the cmd window output that that was an "import cython". Sorry for the confusion. I'll give the other suggestions a try and see how it goes.

[Waqas Muhi]

There was some problem with the Cython installation. I reinstalled it and yes, the main Cython script was installed in the Python26\scripts folder.

Thanks for all the help. I'll post if I have any more questions during this conversion :)

[Waqas Muhi]

Hello again,

There's a particular file I seem to be missing. I tried both the "import pyximport" as well as the "Building a Cython module using the DistUtils" approach mentioned here: http://docs.cython.org/src/quickstart/build.html

In both cases, I get an error message about vcvarsall.bat:

1) "import pyxmport" approach

C:\Source\src_Python_sim\PNT_Cython_001>python run_pnt.py
Traceback (most recent call last):
  File "run_pnt.py", line 3, in <module>
    from presynaptic_nerve_terminal_DEVS import presynaptic_nerve_terminal_DEVS
  File "C:\Source\src_Python_sim\PNT_Cython_001\presynaptic_nerve_terminal_DEVS.
py", line 3, in <module>
    from tethered_particle_system_DEVS import tethered_particle_system
  File "C:\Source\src_Python\tethered_particle_system_DEVS.py", line 4, in <modu
le>
    from tethered_particle_system import tethered_particle_system
  File "C:\Source\src_Python\tethered_particle_system.py", line 5, in <module>
    from _tethered_particle_system import _calculate_position
  File "C:\Python26\lib\site-packages\pyximport\pyximport.py", line 335, in load
_module
    self.pyxbuild_dir)
  File "C:\Python26\lib\site-packages\pyximport\pyximport.py", line 183, in load
_module
    so_path = build_module(name, pyxfilename, pyxbuild_dir)
  File "C:\Python26\lib\site-packages\pyximport\pyximport.py", line 167, in buil
d_module
    reload_support=pyxargs.reload_support)
  File "C:\Python26\lib\site-packages\pyximport\pyxbuild.py", line 85, in pyx_to
_dll
    dist.run_commands()
  File "C:\Python26\lib\distutils\dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "C:\Python26\lib\distutils\dist.py", line 995, in run_command
    cmd_obj.run()
  File "C:\Python26\lib\site-packages\Cython\Distutils\build_ext.py", line 135,
in run
    _build_ext.build_ext.run(self)
  File "C:\Python26\lib\distutils\command\build_ext.py", line 340, in run
    self.build_extensions()
  File "C:\Python26\lib\site-packages\Cython\Distutils\build_ext.py", line 143,
in build_extensions
    self.build_extension(ext)
  File "C:\Python26\lib\distutils\command\build_ext.py", line 499, in build_exte
nsion
    depends=ext.depends)
  File "C:\Python26\lib\distutils\msvc9compiler.py", line 458, in compile
    self.initialize()
  File "C:\Python26\lib\distutils\msvc9compiler.py", line 368, in initialize
    vc_env = query_vcvarsall(VERSION, plat_spec)
  File "C:\Python26\lib\distutils\msvc9compiler.py", line 260, in query_vcvarsal
l
    raise DistutilsPlatformError("Unable to find vcvarsall.bat")
ImportError: Building module failed: ['DistutilsPlatformError: Unable to find vc
varsall.bat\n']


2) "DistUtils approach"
C:\Source\src_Python>python setup.py build_ext --inplace
running build_ext
cythoning _tethered_particle_system.pyx to _tethered_particle_system.c
building '_tethered_particle_system' extension
error: Unable to find vcvarsall.bat

I installed Cython on my machine using one of the binaries here: http://www.lfd.uci.edu/~gohlke/pythonlibs/#cython and this, as I mentioned previously, installed Cython in the correct location and helped me get rid of the problems I was seeing before ("import cython" failing and all that...).

Please let me know where and how I can get this missing file - vcvarsall.bat.

Thanks,
Waqas

[Chris Barker]

On Sun, Apr 8, 2012 at 7:02 AM, Waqas Muhi

> There's a particular file I seem to be missing. I tried both the "import
> pyximport" as well as the "Building a Cython module using the DistUtils"
> approach mentioned here: http://docs.cython.org/src/quickstart/build.html

> In both cases, I get an error message about vcvarsall.bat:

vcvarsall.bat comes with Microsoft Visual Studio (Visual C, whatever
the heck else they call it). It is used to ser of the compiler
environment when invoked.

Cython, of course, requires a compiler. distutils (and I presume
pyximport) is set up (by default) to build extensions with the same
complier and settings that Python was build with (that's one of its
core nifty features).

The Python 2.6 and 2.7 binaries distributed by python.org are built
with VS2008 (I'm pretty sure). The easiest way to get set up for that
is to install the Microsoft Visual Studio 2008 express addition (It's
free on the MS web site -- it's a bit hard to find - MS tries to get
you to use the 2010 version, but that may not work with the python
builds). I've found installing VS 2008 express then "just works" with
distutils (and thus Cython).

If you want 64 bit -- it theoretically can be done, but its takes some
hacking, and I haven't gotten it to work yet (I've given up twice
after messing around or an hour or two). But maybe I just messed
something up -- there is a page in the Cython wiki about how to do it.

[Waqas Muhi]

Hello,

I installed MS Visual Studio 2008 Express Edition from this link: http://www.microsoft.com/en-us/download/details.aspx?id=14597 and I'm no longer seeing the problem I was seeing before. Thanks Chris for pointing that out.

Now, finally getting to my actual Python-to-Cython conversion :), below is the Cython .pyx file I've started writing. Since this file involves usage of numpy, I'm using this link as a guide: http://docs.cython.org/src/tutorial/numpy.html

import numpy as np
# "cimport" is used to import special compile-time information
# about the numpy module (this is stored in a file numpy.pxd which is
# currently part of the Cython distribution).
cimport numpy as np
# We now need to fix a datatype for our arrays. I've used the variable
# DTYPE for this, which is assigned to the usual NumPy runtime
# type info object.
DTYPE = np.float
# "ctypedef" assigns a corresponding compile-time type to DTYPE_t. For
# every type in the numpy module there's a corresponding compile-time
# type with a _t-suffix.
ctypedef np.float_t DTYPE_t

def calculate_position(np.ndarray[DTYPE_t, ndim=1] u_A, np.ndarray[DTYPE_t, ndim=1] v_A, float t, float t_A):
   
    assert u_A.dtype == DTYPE and v_A.dtype == DTYPE
    cdef float told = t - t_A
    return (u_A + v_A)*told.tolist()

However, when I run my top-level .py module, I see that the above .pyx file is compiled ok but I get the following error output:

C:\Source\src_Python_sim\PNT_Cython_001>python run_pnt.py

_tethered_particle_system.c
C:\Users\Waqas/.pyxbld\temp.win32-2.6\Release\pyrex\_tethered_particle_system.c(
237) : fatal error C1083: Cannot open include file: 'numpy/arrayobject.h': No su
ch file or directory

Traceback (most recent call last):
  File "run_pnt.py", line 3, in <module>
    from presynaptic_nerve_terminal_DEVS import presynaptic_nerve_terminal_DEVS
  File "C:\Source\src_Python_sim\PNT_Cython_001\presynaptic_nerve_terminal_DEVS.
py", line 3, in <module>
    from tethered_particle_system_DEVS import tethered_particle_system
  File "C:\Source\src_Python\tethered_particle_system_DEVS.py", line 4, in <modu
le>
    from tethered_particle_system import tethered_particle_system
  File "C:\Source\src_Python\tethered_particle_system.py", line 5, in <module>

    from _tethered_particle_system import calculate_position

  File "C:\Python26\lib\distutils\msvc9compiler.py", line 533, in compile
    raise CompileError(msg)
ImportError: Building module failed: ['CompileError: command \'"C:\\Program File
s (x86)\\Microsoft Visual Studio 9.0\\VC\\BIN\\cl.exe"\' failed with exit status
 2\n']

If I remove the numpy stuff from my file, then I don't see the above error output. So it is clearly something to do with how I'm using numpy. Can anyone please guide me where I'm going wrong?

Thanks,
Waqas

[Ian Bell]

Waqas,

Cython can't find your numpy headers.  You need to tell Cython where to look.  In your distutils setup.py file, you need to

a) import numpy
b) into the setup() function, you need to add the numpy include folder, something like

setup(
  name = 'MyProject',
  cmdclass={'build_ext': build_ext},
  ext_modules = [My_module],
  include_dirs = [numpy.get_include()]
)

Hope this helps.

Ian

[Waqas Muhi]

Ok thanks. So that basically means I can't use the straightforward "import pyximport" approach pointed out by a few others earlier?

[Chris Barker]

On Wed, May 2, 2012 at 7:02 AM, Waqas Muhi <waqas...@gmail.com> wrote:
> Ok thanks. So that basically means I can't use the straightforward "import
> pyximport" approach pointed out by a few others earlier?

yup -- pyximport is nifty, but quite limited. From the Wiki:

"""
This allows you to automatically run Cython on every .pyx that Python
is trying to import. You should use this for simple Cython builds only
where no extra C libraries and no special building setup is needed.
"""

-Chris

[Stefan Behnel]

Chris Barker, 02.05.2012 18:06:

> On Wed, May 2, 2012 at 7:02 AM, Waqas Muhi wrote:
>> Ok thanks. So that basically means I can't use the straightforward "import
>> pyximport" approach pointed out by a few others earlier?
>
> yup -- pyximport is nifty, but quite limited. From the Wiki:
>
> """
> This allows you to automatically run Cython on every .pyx that Python
> is trying to import. You should use this for simple Cython builds only
> where no extra C libraries and no special building setup is needed.
> """

Yes, pyximport is pretty old and patchy. It would hugely benefit from at
least a partial rewrite based on Cython.Build.Dependencies.cythonize().
That would allow you to declare the distutils setup in the module file
(including C libraries), amongst other things like automatic rebuilds of
(and based on) dependencies.

Any volunteering on this is warmly appreciated. Shouldn't be more than a
day's work.

Stefan

[Aronne Merrelli]

On Wed, May 2, 2012 at 9:02 AM, Waqas Muhi <waqas...@gmail.com> wrote:
> Ok thanks. So that basically means I can't use the straightforward "import
> pyximport" approach pointed out by a few others earlier?
>

Well, if I just add a pyxdep and pyxbld file (see bottom of message),
pyximport works fine. I just did that with the code you pasted and I
can import it at my end. I can't remember exactly where I saw this in
the documentation - part of it is here:
http://docs.cython.org/src/userguide/source_files_and_compilation.html#dependency-handling

I guess at some point I should look at a better way to do this (?)


cython_code.pyxdep:
--------------------------

/path_to_numpy_includes/arrayobject.h
/path_to_numpy_includes/ufuncobject.h

cython_code.pyxbld:
---------------------------
def make_ext(modname, pyxfilename):
from distutils.extension import Extension
return Extension(name = modname,
sources = [pyxfilename,],
include_dirs=['/path_to_numpy_includes/'] )

[Chris Barker]

On Wed, May 2, 2012 at 8:09 PM, Aronne Merrelli

> Well, if I just add a pyxdep and pyxbld file (see bottom of message),
> pyximport works fine.

> I guess at some point I should look at a better way to do this (?)
>
>
> cython_code.pyxdep:
> --------------------------
>
> /path_to_numpy_includes/arrayobject.h
> /path_to_numpy_includes/ufuncobject.h
>
> cython_code.pyxbld:
> ---------------------------
> def make_ext(modname, pyxfilename):
>    from distutils.extension import Extension
>    return Extension(name = modname,
>                     sources = [pyxfilename,],
>                     include_dirs=['/path_to_numpy_includes/'] )

cool! I hadn't seen that. However, it looks like this is static.
Ideally, this kind of stuff coud be specified at run-time -- i.e. be
able to use "numpy.get_include()", so your code could run (and
dynamically build) on an installation.

I think that's more or less what Stefan was suggesting be added.

[Waqas Muhi]

Thanks a lot guys. I was able to compile my .pyx file successfully :)

[Waqas Muhi]

Hello once again,

Moving on to my actual conversion (finally), I basically have 2 versions of code (for my performance-critical functions) to play around with. One is the original Python (slow) and the second is its equivalent written for the weave.inline() function (optimized and faster). I'm wondering which one would be the most appropriate for my Cython conversion.

As an example, here's the code for a function "position" which I'm working on right now. Note that the original function makes use of arrays while the optimized version replaces that with for loops. The parts of interest are in bold.

Original Python:

def position(Psi, t, id_A):
    """
    Given the dictionary of particle data "Psi", the current time "t", a
    particle ID "id_A", and "position" results in the current position of the
    identified particle at the current time.
    """
    psi_A = Psi[id_A]
    t_A = psi_A["t"]
    u_A = psi_A["u"]
    v_A = psi_A["v"]
    u_A_ = (array(u_A) + array(v_A)*(t - t_A)).tolist()
    return u_A_

Optimized version for Weave.inline()

def position(Psi, t, id_A):
    """
    Given the dictionary of particle data "Psi", the current time "t", a
    particle ID "id_A", and "position" results in the current position of the
    identified particle at the current time.
    """
    psi_A = Psi[id_A]
    t_A = float(psi_A["t"])
    u_A = psi_A["u"]
    v_A = psi_A["v"]
    n= u_A.__len__()
    t = float(t)
   
    code = """
    float *u_AC, *v_AC, told = (t-t_A);
    py::list resu(n);
    u_AC = new float[n];
     v_AC = new float[n];
   
        for(int x = 0; x < n; x++) {
           u_AC[x] = u_A[x];             
               v_AC[x] = v_A[x];   
   
           resu[x] = u_AC[x] + v_AC[x]*(told);  
    }

   
    return_val = resu;

        delete[] u_AC;  
        delete[] v_AC;

       """
    return weave.inline(code,['t_A','t', 'u_A','v_A','n'],type_converters=converters.blitz, compiler = 'gcc')

Thanks in advance,

Waqas

[Robert Bradshaw]

On Sun, May 6, 2012 at 9:15 AM, Waqas Muhi <waqas...@gmail.com> wrote:
> Hello once again,
>
> Moving on to my actual conversion (finally), I basically have 2 versions of
> code (for my performance-critical functions) to play around with. One is the
> original Python (slow) and the second is its equivalent written for the
> weave.inline() function (optimized and faster). I'm wondering which one
> would be the most appropriate for my Cython conversion.
>
> As an example, here's the code for a function "position" which I'm working
> on right now. Note that the original function makes use of arrays while the
> optimized version replaces that with for loops. The parts of interest are in
> bold.

I'd start with the Python version. You're biggest speedup is going to
be removing all the conversions to and from lists, you could probably
get away with a pure Python implementation using NumPy arrays that
would be *way* faster.

- Robert

[Waqas Muhi]

Thanks for the reply. By pure Python, do you mean this: http://docs.cython.org/src/tutorial/pure.html ?

Also, the original (terribly slow) Python code that I have already makes use of NumPy arrays so I'm guessing you mean converting the code to pure Python but sticking with those arrays?

[Robert Bradshaw]

On Thu, May 10, 2012 at 6:53 AM, Waqas Muhi <waqas...@gmail.com> wrote:
> Thanks for the reply. By pure Python, do you mean this:
> http://docs.cython.org/src/tutorial/pure.html ?

No, I meant just Python.

> Also, the original (terribly slow) Python code that I have already makes use
> of NumPy arrays so I'm guessing you mean converting the code to pure Python
> but sticking with those arrays?

I am assuming your arrays are of moderate size. In that case, your
slowness is entirely the array(...) and .tolist() calls. Get rid of
those and you should be much faster. Otherwise, yes, use Cython
directly by unrolling the loop as you do in your weave inline version
(though you can use memory views rather than pointers, again making
care to avoid the expensive to/from list conversions).

[Chris Barker]

On Thu, May 10, 2012 at 9:46 AM, Robert Bradshaw <robe...@gmail.com> wrote:

> I am assuming your arrays are of moderate size. In that case, your
> slowness is entirely the array(...) and .tolist() calls. Get rid of
> those and you should be much faster.

Exactly -- to quote Travis Oliphant in a talk about pyton performance
for numerical computing:

"If you are going to use numpy arrays, use numpy"

What he means is that simply using numpy arrays as container buys you
nothing, and may result in slower performance -- numpy arrays are
fairly slow to create, and slow to access individual elements of. They
are fast to do array=processing with -- so look for code that creates
new arrays and/or works with individual elements, and see if you can
remove that.

Interestingly, you've done the opposite -- store with list, compute
with numpy arrays, but the principle is the same -- use numpy wherever
you can.


In your case:

psi_A = Psi[id_A]
t_A = psi_A["t"]
u_A = psi_A["u"]
v_A = psi_A["v"]
u_A_ = (array(u_A) + array(v_A)*(t - t_A)).tolist()
return u_A_

store t_A as numpy arrays in the first place, and then you can change
that key line to just:

u_A_ = u_A + v_A*(t - t_A)

which is the kind of thing that numpy does well.

for an addition tweak (at the expense of readability), you may be able
to save some temporaries by doing something like:

t_A *= -1
t_A += t
a_A += v_A * t_A


(the in-place operators change the array in plave without cretin a new
one -- this can help performance depending on tehoperation and size of
the array)

[Waqas Muhi]

Hello,

A question somewhat related to the one I asked before, is there any concept of lists in Cython? Like the ones we have in Python? Or is there only the NumPy array that we're supposed to use in place of lists?

Here's an example piece of Cython code from one of my functions to understand this better:

for y in range(n):
        u_B_[y]= (u_B[y] + v_B[y]*(told_B_2))
        u_A_[y]= (u_A[y] + v_A[y]*(told_A_2))    
        v_AB[y] = v_B[y] - v_A[y]
       
        if(u_A_[y] != u_B_[y]):
            same = 0
           
        u[y] = (u_B_[y] - u_A_[y]);
        sumval1 = sumval1 + (u_B_[y] - u_A_[y])*(u_B_[y] - u_A_[y]) 

Here, u_B, v_B, u_A, and v_A are lists which are passed into the Cython function from my calling Python function. I'm just treating them as normal lists inside the Cython function. However, how should I assign a data type to the resulting (new) list elements i.e. u_B_, u_A_, v_AB and u?

I do know that the values contained in the lists will be of type float. But I don't know if there's a way to declare lists of type float of a certain length in Cython?

[Chris Barker]

On Sun, Jun 3, 2012 at 4:14 PM, Waqas Muhi <waqas...@gmail.com> wrote:

> A question somewhat related to the one I asked before, is there any concept
> of lists in Cython? Like the ones we have in Python?

well, what exactly is a pyton list? It is an implimentation of the
python sequence API that supports dynamic sizing and storing any
python data type. If that's what you want, you might as well use a
Python list in Cython.

However, I don't think that's what you want -- for performance in
Cython, you really want something like a C array (or maybe a Fortran
array) -- simple homogenous data of a known-at-compile-time type.

numpy arrays are essentially really handy wrappers around C arrays,
and Cython has good support for them, so yes, that's probably what you
want. Why not use them? If for some odd reason you really don't want
to use them in your Python code, you can simply put something like:

cdef a_func(input_list):
cdef np.ndarray[np.float64_t, ndim=2] arr

arr = np.array(input_list, dtype=np.float64).reshape((-1,))


at the top of your function -- numpy has a bunch of nifty code for
converting arbitrarty python sequences to numpy arrays -- why not use
it?

> Or is there only the
> NumPy array that we're supposed to use in place of lists?

There is also the built-in python array.array -- it is a wrapper
around a buffer of basic C data types. I think support for it has been
added to Cython recently (probably not in a released version yet) --
it may give you the advanatge of not havin gthe numpy dependency, but
otherwise, I'd stick with numpy arrays.

> Here's an example piece of Cython code from one of my functions to
> understand this better:
>
> for y in range(n):
>         u_B_[y]= (u_B[y] + v_B[y]*(told_B_2))
>         u_A_[y]= (u_A[y] + v_A[y]*(told_A_2))
>         v_AB[y] = v_B[y] - v_A[y]
>
>         if(u_A_[y] != u_B_[y]):
>             same = 0
>
>         u[y] = (u_B_[y] - u_A_[y]);
>         sumval1 = sumval1 + (u_B_[y] - u_A_[y])*(u_B_[y] - u_A_[y])
>
> Here, u_B, v_B, u_A, and v_A are lists which are passed into the Cython
> function from my calling Python function. I'm just treating them as normal
> lists inside the Cython function. However, how should I assign a data type
> to the resulting (new) list elements i.e. u_B_, u_A_, v_AB and u?
>
> I do know that the values contained in the lists will be of type float.

careful -- they will be a python float -- which is a C double -- if
you want a C float, you can't store it in a list (at least not
easily!)

> But
> I don't know if there's a way to declare lists of type float of a certain
> length in Cython?

nope -- that's not what lists are.

Note that you can assign a data type to the element you pull out of the list:


cdef float uB, vB, toldB

# then:

for y in range(n):
uB = u_B[y]
vB = v_B[y[
toldB = told_B_2[y]

    u_B_[y]= ( uB + vB*toldB )

That will save you some type checkign and unpacking, but not a lot --
you really want a numpy array!

What's the resisitance?

I think if it this way:

doing number crunching in python without numpy arrays would be like
doing text processing without the string object -- i.e. painfully
difficult and inefficient!

[Robert Bradshaw]

You can use cython.view.array and memory views [1] if you really want
to avoid the NumPy dependence, but it sounds like NumPy really is a
good fit here (and this isn't specific to Cython).

[1] http://docs.cython.org/src/userguide/memoryviews.html

[Chris Barker]

On Mon, Jun 4, 2012 at 11:23 AM, Robert Bradshaw <robe...@gmail.com> wrote:
> You can use cython.view.array and memory views [1] if you really want
> to avoid the NumPy dependence,

I was thining of recommending that, but it seems the only way to
intilize a memoryview is with a buffer object (or setting the elements
directly of course), so if you wanted to use on in Cython code that
you call from python, then you'd need to either:

Convert your list or whatever in Python to an appropriate buffer
object -- maybe an array.array.

or

write the code to parse out your list and set the elements of the
memoryview in Cython -- not too hard, I suppose, but who wants to
write tha code.

On the other hand:

np.array(a_list)

does all that for you.


Did I miss a way to initialize a memoryview from an arbitrary python object?

-Chris

[Waqas Muhi]

Thanks for the detailed replies Chris. I spent a good few hours today typing all my Python lists to Cython Numpy arrays. In doing so, I basically did the following:

## Python code

def position(Psi, t, id_A):

    psi_A = Psi[id_A]

    t_A = float(psi_A["t"])

    u_A = array(psi_A["u"]) #This used to be a list but I'm now converting it to a numpy array

    v_A = array(psi_A["v"])

    t = float(t)

    

    return calculate_position(u_A, v_A, t, t_A) #Function call to Cython implementation (see below)

##Cython implementation

def calculate_position(np.ndarray[DTYPE_t, ndim=1] u_A, np.ndarray[DTYPE_t, ndim=1] v_A, float t, float t_A):

    

    assert u_A.dtype == DTYPE and v_A.dtype == DTYPE

    cdef float told = t - t_A

    cdef int i

    cdef int n = u_A.__len__()

    cdef np.ndarray[DTYPE_t, ndim=1] result = np.zeros(1, dtype=DTYPE) #Initialization.

    

    for i in range(n):

        result.append(u_A[i] + v_A[i]*(told))

        

    return result

A few things:

1) When I try and run this, I get a "ValueError: Buffer has wrong number of dimensions (expected:2, got:1)" message. I'm passing in a list-converted-to-array to the Cython function. Where am I going wrong?

2) Why do I need the  assert u_A.dtype == DTYPE and v_A.dtype == DTYPE statement?

3) Is my initialization of my result array correct?

Thanks for all the help.

Regards,

Waqas

[Chris Barker]

Hi,

It's always a good idea to post a complete-as-possible, but small
example -- i.e. some test code with data. IN fact, had you posted the
Cython, a setup.py to build it, and a little sample script to test it,
I may have actually built and run it myself, so I could test my
suggestions.

see below for specific comments:

On Sun, Jun 10, 2012 at 4:18 PM, Waqas Muhi <waqas...@gmail.com> wrote:
> Thanks for the detailed replies Chris. I spent a good few hours today typing
> all my Python lists to Cython Numpy arrays. In doing so, I basically did the
> following:
>
> ## Python code
>
> def position(Psi, t, id_A):
>
>     psi_A = Psi[id_A]
>     t_A = float(psi_A["t"])
>     u_A = array(psi_A["u"]) #This used to be a list but I'm now converting
> it to a numpy array
>     v_A = array(psi_A["v"])
>     t = float(t)
>
>     return calculate_position(u_A, v_A, t, t_A) #Function call to Cython
> implementation (see below)

note that you could do the conversion to numpy arrays in the Cython
function -- that would create a Cython function that could take any
compatible sequence, and remove the numpy dependency in the Python
code. HOwever, I suspect that you'd benefit from using numpy arrays in
your pyton code anyway...

But in Cython or python:

> u_A = array(psi_A["u"]) #This used to be a list but I'm now converting it to a numpy array

I like to be more specific, particularly when i know I want to pass it
off to compiled code:

u_A = np.array(psi_A["u"], dtype = np.float32).reshape((-1,))

- I use np.* -- because "namepsaces are one honking great idea" and
"import numpy as np" is more or less the convention these days.

- I used np.float32 as the dtype, becasue you are using floats in
Cyhton (rather than doubles), so you should be consistent here.

- the reshape call makes sure you get a 1-d array -- np.array() parses
out what's passed in and figures out what shape it thinks it should be
-- adding a reshape call enforces that you do get what you expect.
granted, if you've passed something othere than a simple 1-d list,
then it's re-shape, and probably be an error anyway.

> ##Cython implementation
>
> def calculate_position(np.ndarray[DTYPE_t, ndim=1] u_A, np.ndarray[DTYPE_t,
> ndim=1] v_A, float t, float t_A):

I wouldn't use DTYPE_t there -- in this case, you want a specific dtype:

def calculate_position(np.ndarray[np.float32_t, ndim=1] u_A,
np.ndarray[np.float32_t, ndim=1] v_A,

float t,
float t_A):

>     assert u_A.dtype == DTYPE and v_A.dtype == DTYPE
>     cdef float told = t - t_A
>     cdef int i

I'd use unsigned int here -- Cython can be a bit smarted it it knows
it isn't going to get negative indexes:

cdef unsigned int i

>     cdef int n = u_A.__len__()

and why not len() here? or, maybe better, an array attribute:

cdef unsigned int n = u_A.shape[0]

>     cdef np.ndarray[DTYPE_t, ndim=1] result = np.zeros(1, dtype=DTYPE)
> #Initialization.
>
>     for i in range(n):
>         result.append(u_A[i] + v_A[i]*(told))

you can't append to a numpy array -- but you'be already crated it at
the right size:

    for i in range(n):
        result[i] = u_A[i] + v_A[i]*told

> 1) When I try and run this, I get a "ValueError: Buffer has wrong number of
> dimensions (expected:2, got:1)" message. I'm passing in a
> list-converted-to-array to the Cython function. Where am I going wrong?

I suspect the list passed in to the np.,array() call isn't a simple
list -- but with my reshape call above, this shouldn't happen. (you
may not get the right result though!)

> 2) Why do I need the  assert u_A.dtype == DTYPE and v_A.dtype == DTYPE
> statement?

because in this case, you've used a generic DTYPE -- with my
re-factor, you won't need that.

(NOTE: we may want to edit the Wiki a bit here...)

> 3) Is my initialization of my result array correct?

except for the dtype, yes (or did you typedef DTYPE somewhere?

HTH,

[Waqas Muhi]

Hi again,

Thanks so much for taking the time out to write such a super useful reply! I'll improve my code further as per your recommendations and let you know how it goes but so far, I've been able to get rid of the "ValueError" problem - that was a result of non-compilation of some modified code.

And yes, I've this ctypedef np.float_t DTYPE_t right at the top of my Cython file - as per the example given on one of the Cython Wiki pages.

Thanks once again,

Waqas

[Waqas Muhi]

Hello again,

I've a simple question: how do I declare/initialize a Cython np.ndarray with a fixed set of values based on the discussion so far in this thread? 

This doesn't seem to work:

cdef np.ndarray[DTYPE_t, ndim=1] values = [-3, -2, -1, 1, 2, 3]

Thanks,

Waqas

[Chris Barker]

On Mon, Jul 2, 2012 at 4:29 PM, Waqas Muhi <waqas...@gmail.com> wrote:
> I've a simple question: how do I declare/initialize a Cython np.ndarray with
> a fixed set of values based on the discussion so far in this thread?

It should work the same way you do it in Python:

cdef np.ndarray[DTYPE_t, ndim=1] values = np.array( [-3, -2, -1, 1, 2,
3], dtype=DTYPE_t )

I don't know that there is a more efficient short cut than going
through a python literal, but for the most part, you'd only be doing
this for small arrays anyway -- big ones would be populated some other
way.

[Waqas Muhi]

Thanks once again. That solves it :)

Btw, what's the Cython equivalent of "None" for an int variable type? E.g. I've a variable "id_root" in Python which is normally supposed to have an integer value but in some cases, it could be "None". 

If I try do this in Cython, it doesn't accept "None" as a valid int value. Should I be using 0 in such cases instead?

[Chris Barker]

On Sun, Jul 8, 2012 at 12:55 PM, Waqas Muhi <waqas...@gmail.com> wrote:

> Btw, what's the Cython equivalent of "None" for an int variable type? E.g.
> I've a variable "id_root" in Python which is normally supposed to have an
> integer value but in some cases, it could be "None".

> If I try do this in Cython, it doesn't accept "None" as a valid int value.
> Should I be using 0 in such cases instead?

well, Cython compiles down to C, and C is statically typed -- so if
it's an int, it can only be an int (not null, None, or anything else).

Where do you need the None? one option is to keep your variable a
python object, so it can be None or an integer, and only cast it to an
integer at the last minute -- where you really need the speed.

If you really need a None value in the C code, then you need to select
a "special value" that means None in your case. It conventional to use
0 as FAlse, and other integers as non-flase values, but in many cases,
0 is perfectly reasonable value. In that case you can use a special
value that isn't reasonable in your case. There is an old tradition of
using 99999 and the like in text files, or another option is something
like sys.maxint -- i.e the largest integer value -- that's not taking
much our of your domain anyway.

HTH,
-Chris

[Robert Bradshaw]

On Mon, Jul 9, 2012 at 9:00 AM, Chris Barker <chris....@noaa.gov> wrote:
> On Sun, Jul 8, 2012 at 12:55 PM, Waqas Muhi <waqas...@gmail.com> wrote:
>
>> Btw, what's the Cython equivalent of "None" for an int variable type? E.g.
>> I've a variable "id_root" in Python which is normally supposed to have an
>> integer value but in some cases, it could be "None".
>
>> If I try do this in Cython, it doesn't accept "None" as a valid int value.
>> Should I be using 0 in such cases instead?
>
> well, Cython compiles down to C, and C is statically typed -- so if
> it's an int, it can only be an int (not null, None, or anything else).
>
> Where do you need the None? one option is to keep your variable a
> python object, so it can be None or an integer, and only cast it to an
> integer at the last minute -- where you really need the speed.
>
> If you really need a None value in the C code, then you need to select
> a "special value" that means None in your case. It conventional to use
> 0 as FAlse, and other integers as non-flase values, but in many cases,
> 0 is perfectly reasonable value. In that case you can use a special
> value that isn't reasonable in your case. There is an old tradition of
> using 99999 and the like in text files, or another option is something
> like sys.maxint -- i.e the largest integer value -- that's not taking
> much our of your domain anyway.

Note that sys.maxint might not fit in an int, it's the maximum size of
a long int.

If negative values aren't a valid value -1 or -2 make good candidates
as well. It'd be good to name this value, e.g.

cdef enum Xxx:
NONE_INT_VALUE = -2

- Robert

[Chris Barker]

On Mon, Jul 9, 2012 at 9:08 AM, Robert Bradshaw <robe...@gmail.com> wrote:
> Note that sys.maxint might not fit in an int, it's the maximum size of
> a long int.

as in a C long?

I've lost track of what an "int" in C is anyway -- a integer at least
as big as a pointer?

It's always seemed to me to be bad practice to use "int" anyway --
short or long makes more sense (although long isn't defined the same
on different 64 bit platforms, either) -- can you use np.int32, etc.
in Cython code and be sure of what you're getting?

-Chris

> If negative values aren't a valid value -1 or -2 make good candidates
> as well. It'd be good to name this value, e.g.
> cdef enum Xxx:
> NONE_INT_VALUE = -2
>
> - Robert

[Robert Bradshaw]

On Mon, Jul 9, 2012 at 10:17 AM, Chris Barker <chris....@noaa.gov> wrote:
> On Mon, Jul 9, 2012 at 9:08 AM, Robert Bradshaw <robe...@gmail.com> wrote:
>> Note that sys.maxint might not fit in an int, it's the maximum size of
>> a long int.
>
> as in a C long?

Yep.

> I've lost track of what an "int" in C is anyway -- a integer at least
> as big as a pointer?

No, in fact this is often not the case (especially on 64-bit
machines). An int is at least 16 bits, and supposed to be the "most
efficient native integer type" for a given platform.

> It's always seemed to me to be bad practice to use "int" anyway --
> short or long makes more sense (although long isn't defined the same
> on different 64 bit platforms, either) --

The C long type is what's used to back Python (2.x) int objects, which
is where maxint comes in (and does vary from platform to platform).

> can you use np.int32, etc.
> in Cython code and be sure of what you're getting?

Yes, or you can use stdint.h.

- Robert

[Waqas Muhi]

Thanks guys for the valuable tips once again! I'll try using a negative value (-1, -2 etc.) and see how it goes.