Re: [cython-users] 'nd_array' is not a type identifier

[Warren Weckesser]

On 5/22/14, Julien Delafontaine <mura...@gmail.com> wrote:
> Hello,
>
> I have followed the "Working with Numpy" tutorial and tried inserting the
> same exact code in my .pxd to speed it up, but the compilation fails with
> `'*nd_array'
> is not a type identifier*`.
> I have no "numpy.pxd" in my filesystem, and trying to add the file I found
> here <http://cython.org/release/Cython-0.13/Cython/Includes/numpy.pxd> in
> Cython/Includes/numpy/ did not help. I installed Cython with "pip install"
> only a few days ago. I used `include_dirs=[np.get_include()]` in my
> Extension in "setup.py".
> Here is a very simplified version of my code, can you please tell me what
> is wrong?
>
> script.pyx:
>
> from numpy import zeros
> def myfunction():
> A = zeros((3,4))
>
> script.pxd:
>
> import numpy as np
> cimport numpy as np
> DTYPE = np.double
> ctypedef np.double_t DTYPE_t
>
> cpdef inline myfunction():
> cdef np.nd_array[DTYPE_t, ndim=2] A
>


Use `np.ndarray`, not `np.nd_array`.

Warren


> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Chris Barker]

I think Warren identified your problem, but a note about how one usually uses Python, Cython, and py, pyx and pyd files:

script.pyx:

    from numpy import zeros
    def myfunction():
        A = zeros((3,4))

This is pure python -- you usually want to keep your pure python in python files, which call teh Cython stuff. Particualry test calling coed like this.

 

script.pxd:

    import numpy as np
    cimport numpy as np
    DTYPE = np.double
    ctypedef np.double_t DTYPE_t
 
    cpdef inline myfunction():
        cdef np.nd_array[DTYPE_t, ndim=2] A

the pxd files are intended to be declarations. so you'd normally put the function definition on a pyx file that cimports the pxd file. For something this simple, you don't really need the pxd file at all -- you'd use that if you have a bunch of declarations that you want to share between multiple cython modules. Of course, this is likely the starting point for something more complex, so you may want to keep the pxd file in that case.

Note: your python and cython models need different names.

HTH,

  -Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Robert Bradshaw]

On Thu, May 22, 2014 at 10:25 AM, Chris Barker <chris....@noaa.gov> wrote:
> I think Warren identified your problem, but a note about how one usually
> uses Python, Cython, and py, pyx and pyd files:
>
>> script.pyx:
>>
>> from numpy import zeros
>> def myfunction():
>> A = zeros((3,4))
>
>
> This is pure python -- you usually want to keep your pure python in python
> files,

Of course you don't need to have two separate modules; essentially
anything that can be written in pure python can be put into the pyx
file as well. There are pros and cons, but the cons are not near as
bad as when writing a C extension against the Python C API directly in
which case you would want to pull absolutely as much as possible out.

>> script.pxd:
>>
>> import numpy as np
>> cimport numpy as np
>> DTYPE = np.double
>> ctypedef np.double_t DTYPE_t
>>
>> cpdef inline myfunction():
>> cdef np.nd_array[DTYPE_t, ndim=2] A
>>
>
> the pxd files are intended to be declarations. so you'd normally put the
> function definition on a pyx file that cimports the pxd file. For something
> this simple, you don't really need the pxd file at all -- you'd use that if
> you have a bunch of declarations that you want to share between multiple
> cython modules. Of course, this is likely the starting point for something
> more complex, so you may want to keep the pxd file in that case.
>
> Note: your python and cython models need different names.
>
> HTH,
> -Chris
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>
> Chris....@noaa.gov
>

[Julien Delafontaine]

Thank you. Indeed I was so ashamed when I realized I mistyped 'ndarray' that I tried to delete my post...

Anyway, your advice is very precious here! Eventually I will replace some of my code by C++ functions, but I always want to keep a version that I can run with the Python compiler only (for diplomatic reasons). So I cannot put any "cdef" in it, and if I don't call it ".pyx", it will never import the .pxd.

Usually my workflow is
- Edit a A.py version
- Copy A.py to B.pyx
- Compile B.pyx with B.pxd

The only thing I want to do is to replace my functions by a C version with fixed types at compilation time, by overriding them in the .pxd. I have no idea it this works and actually I don't see any difference in running time yet.

[Chris Barker]

On Fri, May 23, 2014 at 1:27 AM, Julien Delafontaine <mura...@gmail.com> wrote:

 

Anyway, your advice is very precious here! Eventually I will replace some of my code by C++ functions,

do you mean hand-written C++, or do you mean C++ generated by Cython? Ideally, there should be no need to hand-write C++ unless you need it for other purposes -- or you are familiar with C++, and prefer to use it.

but I always want to keep a version that I can run with the Python compiler only (for diplomatic reasons). So I cannot put any "cdef" in it, and if I don't call it ".pyx", it will never import the .pxd.

Usually my workflow is
- Edit a A.py version
- Copy A.py to B.pyx
- Compile B.pyx with B.pxd

The only thing I want to do is to replace my functions by a C version with fixed types at compilation time, by overriding them in the .pxd.

This is a bit of an add structure / work flow. Maybe it works for you, but:

Robert is quite right, of course you can put pure python in *.pyx files and compile them. But then they are harder to change, test, debug, etc, and pure python doesn't buy you much when cythonized. I'll stay away for calling this "usual" or "accepted", but my recommended work flow for something like this is:

 - Prototype in pure python: A.py

 - If parts of that code need optimizing with Cython for some reason or another:

   - copy the parts of A.py that need Cythonizing to B.pyx (maybe one function, etc. at a time)

   - keep the high level calling code,  test code, etc in A.py

   - import the cythonized pieces from B into A (so the A namespace looks the same)

   - add the needed cdefs, etc to B.py (and maybe to B.pxd) -- probably a bit at a time, profiling and testing as you go.

  - when you've got the performance you need (or are going to get...) you're done.

B.pxd is mostly useful if you need to share declarations with other Cython modules -- that's what they are for.

my $0.2 -- hope it's helpful.

-Chris

 

I have no idea it this works and actually I don't see any difference in running time yet.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Bradshaw]

On Fri, May 23, 2014 at 8:28 AM, Chris Barker <chris....@noaa.gov> wrote:
> On Fri, May 23, 2014 at 1:27 AM, Julien Delafontaine <mura...@gmail.com>
> wrote:
>
>>
>> Anyway, your advice is very precious here! Eventually I will replace some
>> of my code by C++ functions,
>
>
> do you mean hand-written C++, or do you mean C++ generated by Cython?
> Ideally, there should be no need to hand-write C++ unless you need it for
> other purposes -- or you are familiar with C++, and prefer to use it.
>
>> but I always want to keep a version that I can run with the Python
>> compiler only (for diplomatic reasons). So I cannot put any "cdef" in it,
>> and if I don't call it ".pyx", it will never import the .pxd.
>>
>> Usually my workflow is
>> - Edit a A.py version
>> - Copy A.py to B.pyx
>> - Compile B.pyx with B.pxd
>>
>> The only thing I want to do is to replace my functions by a C version with
>> fixed types at compilation time, by overriding them in the .pxd.
>
>
> This is a bit of an add structure / work flow. Maybe it works for you, but:

You might also want to look at pure mode:
http://docs.cython.org/src/tutorial/pure.html

> Robert is quite right, of course you can put pure python in *.pyx files and
> compile them. But then they are harder to change, test, debug, etc,

Hopefully not by much, and we're looking at closing this gap.

> and pure python doesn't buy you much when cythonized.

That depends on the code.

> I'll stay away for calling this
> "usual" or "accepted", but my recommended work flow for something like this
> is:
>
> - Prototype in pure python: A.py
> - If parts of that code need optimizing with Cython for some reason or
> another:
> - copy the parts of A.py that need Cythonizing to B.pyx (maybe one
> function, etc. at a time)
> - keep the high level calling code, test code, etc in A.py
> - import the cythonized pieces from B into A (so the A namespace looks
> the same)
> - add the needed cdefs, etc to B.py (and maybe to B.pxd) -- probably a
> bit at a time, profiling and testing as you go.
> - when you've got the performance you need (or are going to get...) you're
> done.

I do the same, but module-by-module rather than function-by-function,
just renaming A.py to A.pyx. This avoids issues with circular
dependencies between A and B, and chopping splitting up a single
logical module into two separate files.

> B.pxd is mostly useful if you need to share declarations with other Cython
> modules -- that's what they are for.
>
> my $0.2

Woah, 20 cents :)

> -- hope it's helpful.

Yes, I think typical workflows of experienced developers is useful. Thanks.

- Robert

[Chris Barker]

On Fri, May 23, 2014 at 8:39 AM, Robert Bradshaw <robe...@gmail.com> wrote:

> Robert is quite right, of course you can put pure python in *.pyx files and

> compile them. But then they are harder to change, test, debug, etc,

Hopefully not by much, and we're looking at closing this gap.

cool! 

Though you're always going to need to  cythonize and compile when you make a change, and the interpreter can't re-load compiled modules, so you can't just stay in a an ipython session and reload.. (pyximport does address this, I know...)

To some extent it depend on how big your module is -- if you keep it nicely self contained, then all at once may make sense. You calling code and test code should probably be somewhere else anyway.

> my $0.2

Woah, 20 cents :)

inflation?

-Chris 

 

[Stefan Behnel]

Robert Bradshaw, 23.05.2014 17:39:

> On Fri, May 23, 2014 at 8:28 AM, Chris Barker wrote:
>> and pure python doesn't buy you much when cythonized.
> That depends on the code.

Yep, the speedup for plain, unmodified Python code can be somewhere between
20% and a factor of 2.x according to the Python benchmark suite.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-pybenchmarks-py27/lastSuccessfulBuild/artifact/bench_chart.html


Adding an external .pxd file can then buy you a factor of 3-50 or so,
potentially even higher for more low-levelish code.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-cybenchmarks-py3k/lastStableBuild/artifact/bench_chart.html

Stefan