pure Python mode: .pxd and @locals

[Julien Delafontaine]

Hi all,

I have a Python program that I want to speed up using Cython, while keeping the source readable by the Python interpreter ("Pure Python mode"). I get a 20% speed increase just by compiling it as it is. Then I created a .pxd file and tried to add some "@cython.locals(...)" throughout my .pyx script to fix the types of variables. But I see no difference in running time before and after the addition of these two features.
So I have three similar, very simple questions:

First, I added in my .pyx :

class C(object):
    @cython.locals(start=cython.int, end=cython.int)
    def __init__(self, start, end)
        self.start = start
        self.end = end

Is this even doing anything? (I eventually create a million instances.)

Second, if I have in my .pyx a class

class Counter(object):
    def __init__(self):
        self.n = 0
    def __call__(self, x):
        self.n += 1

is there anything I can do to fix the type of "n" (as would a "cdef int n" after the "cdef class")?
(I create a single instance of this one but __call__ it a million times)

Finally, if I have in my .pyx a function :

def count():
    <do something with variables a,b,c>

does it change anything to write the following in the .pxd? :

cpdef inline count():
    cdef:
        int a
        int b
        int c

Thanks in advance for your help.

This has already been asked here but unfortunately no one seems to really know well...

[Stefan Behnel]

Julien Delafontaine, 23.05.2014 10:41:

> I have a Python program that I want to speed up using Cython, while keeping
> the source readable by the Python interpreter ("Pure Python mode"). I get a
> 20% speed increase just by compiling it as it is. Then I created a .pxd
> file and tried to add some "@cython.locals(...)" throughout my .pyx script
> to fix the types of variables. But I see no difference in running time
> before and after the addition of these two features.
> So I have three similar, very simple questions:
>
> First, I added in my .pyx :
>
> class C(object):
> @cython.locals(start=cython.int, end=cython.int)
> def __init__(self, start, end)
> self.start = start
> self.end = end
>
> Is this even doing anything?

What it does is: convert the start/end object arguments into C int values
on entry, then convert them back into Python objects and assign them to the
Python object attributes self.start/end. Not what you want, I guess.

Run "cython -a yourmodule.py" and take a look at the HTML file it
generates. This is particularly helpful with pure Python syntax, where it's
not always immediately obvious if all types were properly applied (typos,
misplaced .pxd files, etc.).

http://docs.cython.org/src/quickstart/cythonize.html?highlight=html#determining-where-to-add-types

The latest Cython master has a couple of beautifications for the HTML
annotator, including syntax highlighting with Pygments (if it's installed).


> I eventually create a million instances.

In that case, you should declare it as an extension type in the .pxd file
(and drop the @locals()):

cdef class C:
cdef int start
cdef int end

Also: make sure "int" size is enough. If it's only bound by memory size,
you may want to use Py_ssize_t, which is 64 bits on a 64bit platform.

If you need to access them from Python code, use "cdef readonly int" or
"cdef public int" to create properties.

> Second, if I have in my .pyx a class
>
> class Counter(object):
> def __init__(self):
> self.n = 0
> def __call__(self, x):
> self.n += 1
>
> is there anything I can do to fix the type of "n" (as would a "cdef int n"
> after the "cdef class")?

Same as above.

> (I create a single instance of this one but __call__ it a million times)

If the class is really as simple as above, don't call it, just use direct
attribute access instead.

> Finally, if I have in my .pyx a function :
>
> def count():
> <do something with variables a,b,c>
>
> does it change anything to write the following in the .pxd? :
>
> cpdef inline count():
> cdef:
> int a
> int b
> int c

This specific example will definitely not work as expected, but providing
inline functions in .pxd files generally does work. However, I'd rather use
the @cython.inline and @cython.ccall decorators inside of the .py file.


> This has already been asked here<http://stackoverflow.com/questions/23784564/cython-pure-python-mode-pxd-and-locals>but unfortunately no one seems to really know well...

Yes, stackoverflow isn't a particularly good source for Cython related
help. This list is.

BTW, the current "pure Python mode" docs aren't particularly well
accessible. They are more like a reference than a howto guide. If someone
feels like writing up a little smoother document here, I'm sure it would
make a lot of people out there happy.

Stefan

[Julien Delafontaine]

Thanks, it helps a lot!

> Finally, if I have in my .pyx a function :
>
> def count():
>     <do something with variables a,b,c>
>
> does it change anything to write the following in the .pxd? :
>
> cpdef inline count():
>     cdef:
>         int a
>         int b
>         int c

This specific example will definitely not work as expected, but providing
inline functions in .pxd files generally does work. However, I'd rather use
the @cython.inline and @cython.ccall decorators inside of the .py file.

Could you explain to me why this specific example does not work ?

(I could not find any occurrence of the @cython.inline decorator in the doc, it should probably be added at the bottom of the Pure Python mode page together with the others).

BTW, the current "pure Python mode" docs aren't particularly well
accessible. They are more like a reference than a howto guide. If someone
feels like writing up a little smoother document here, I'm sure it would
make a lot of people out there happy.

I totally agree. Unless the pure Python mode itself is discouraged... but it makes things much easier to test and maintain, and more understandable by other participants to the project.

[Julien Delafontaine]

In that case, you should declare it as an extension type in the .pxd file
(and drop the @locals()):

    cdef class C:
        cdef int start
        cdef int end

This results in an error "Assignment to non-lvalue 'C'" (even without the definitions for 'start' and 'end').

[Stefan Behnel]

Julien Delafontaine, 23.05.2014 13:24:

> Thanks, it helps a lot!
>
>>> Finally, if I have in my .pyx a function :
>>>
>>> def count():
>>> <do something with variables a,b,c>
>>>
>>> does it change anything to write the following in the .pxd? :
>>>
>>> cpdef inline count():
>>> cdef:
>>> int a
>>> int b
>>> int c
>>
>> This specific example will definitely not work as expected, but providing
>> inline functions in .pxd files generally does work. However, I'd rather
>> use
>> the @cython.inline and @cython.ccall decorators inside of the .py file.
>
> Could you explain to me why this specific example does not work ?

While you can override a def function with a cpdef function *signature* in
a .pxd file, the above actually defines two different *functions*: a def
function in the .py file and an inline function in the .pxd file. I'm not
sure which one Cython eventually chooses, but in any case, I'm sure that
duplicating the implementation is not what you want. The reason why inline
functions in .pxd files are special is that they are a nice way to
duplicate them across module boundaries (that's clearly what you want when
you say "inline"). I don't think there's a way to do that for .py modules yet.

> (I could not find any occurrence of the @cython.inline decorator in the
> doc, it should probably be added at the bottom of the Pure Python mode page
> together with the others).

Hmm, looks like I was mistaken and there actually is no @cython.inline.
There should be.

https://github.com/cython/cython/commit/fea7ad6e4d7e6db8ff39b0d5ce86ed8dbc37b23a

(Well, there was something with that name already, but it does something
different.)

>> BTW, the current "pure Python mode" docs aren't particularly well
>> accessible. They are more like a reference than a howto guide. If someone
>> feels like writing up a little smoother document here, I'm sure it would
>> make a lot of people out there happy.
>
> I totally agree. Unless the pure Python mode itself is discouraged... but
> it makes things much easier to test and maintain, and more understandable
> by other participants to the project.

It's not discouraged. It's just not developed very actively, nor is it well
documented. It could certainly use some more loving.

Stefan

[Stefan Behnel]

Julien Delafontaine, 23.05.2014 14:11:

Could you provide a complete example?

Stefan

[Julien Delafontaine]

draft.pyx:

class Counter(object):
    def __init__(self):
        self.n = 0

    def __call__(self, alignment):
        self.n += 1

draft.pxd:

cdef class Counter:
    cpdef int n     # or 'cdef int n', or 'pass', or whatever

$ python setup.py build_ext --inplace :
"Assignment to non-lvalue 'Counter'"


(btw. I need the __call__ to increase n because it is a callback)

[Chris Barker]

On Fri, May 23, 2014 at 4:24 AM, Julien Delafontaine <mura...@gmail.com> wrote:

BTW, the current "pure Python mode" docs aren't particularly well
accessible. They are more like a reference than a howto guide. If someone
feels like writing up a little smoother document here, I'm sure it would
make a lot of people out there happy.

I totally agree. Unless the pure Python mode itself is discouraged... but it makes things much easier to test and maintain, and more understandable by other participants to the project.

I think this is a place where "newbies" can help a lot -- you know better than the core developers what initial explanation is needed. Perhaps you could write up a tutorial following your process in "cythonizing" your code in pure-python mode.

Of course, it would require review by the "experts", but I find we're  a lot more likley to get helpful comments and review than getting one of them to write the whole thing from scratch!

-CHB

 

 

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Robert Bradshaw]

On Fri, May 23, 2014 at 7:37 AM, Julien Delafontaine
<mura...@gmail.com> wrote:
> draft.pyx:
>
> class Counter(object):
> def __init__(self):
> self.n = 0
> def __call__(self, alignment):
> self.n += 1
>
> draft.pxd:
>
> cdef class Counter:
> cpdef int n # or 'cdef int n', or 'pass', or whatever
>
> $ python setup.py build_ext --inplace :
> "Assignment to non-lvalue 'Counter'"

OK, that error could be better. The problem is that the declaration
of Counter in the .pyx file needs to match that of the .pxd file (i.e.
be "cdef class") If it was draft.py we would add the cdef
automatically.

> On Friday, 23 May 2014 14:28:42 UTC+2, Stefan Behnel wrote:
>>
>> Julien Delafontaine, 23.05.2014 14:11:
>> >> In that case, you should declare it as an extension type in the .pxd
>> >> file
>> >> (and drop the @locals()):
>> >>
>> >> cdef class C:
>> >> cdef int start
>> >> cdef int end
>> >
>> > This results in an error "Assignment to non-lvalue 'C'" (even without
>> > the
>> > definitions for 'start' and 'end').
>>
>> Could you provide a complete example?
>>
>> Stefan
>>

[Stefan Behnel]

Julien Delafontaine, 23.05.2014 16:37:

> draft.pyx:
>
> class Counter(object):
> def __init__(self):
> self.n = 0
> def __call__(self, alignment):
> self.n += 1
>
> draft.pxd:
>
> cdef class Counter:
> cpdef int n # or 'cdef int n', or 'pass', or whatever
>
> $ python setup.py build_ext --inplace :
> "Assignment to non-lvalue 'Counter'"

Rename draft.pyx to draft.py. The .pxd type overriding is only enabled when
compiling Python modules. In .pyx files (i.e. Cython files), the
declarations in the .pyx file and the .pxd file must be in agreement.

Stefan

[Julien Delafontaine]

I think this is a place where "newbies" can help a lot -- you know better than the core developers what initial explanation is needed. Perhaps you could write up a tutorial following your process in "cythonizing" your code in pure-python mode.

Of course, it would require review by the "experts", but I find we're  a lot more likley to get helpful comments and review than getting one of them to write the whole thing from scratch!

I actually wanted to do so anyway, because I will report my work on a bioinformatics blog (in french), as I often do.
At the moment though, I can't go really far following the official doc. What I would write already looks like that, and maybe it is half wrong:

* From A.py, how to create the setupA.py, compile and see how you gain 20% speed.
--[No idea why there is a "cythonize" function but other examples with "Extension", doing the same while "cythonize" is more limited]--
* Then two methods:

a) Create A.pxd, that is not meant for it but can be used nonetheless to override python methods by cython ones. It has limitations - to be listed - over method b).
Do not rename to A.pyx because it would ignore A.pxd at compilation time.
To override functions, one must use "cpdef inline".
To override classes, one must use

cdef class <name>:
    cpdef <type> <future attribute>

For instance, the python class

class C(object):
    def __init__(self, a=0, b=0):
        self.a = a
        self.b = b

can be overriden by specifying the following in the .pxd:

cdef class C:
    cpdef int a,b

so that it is in the end equivalent to the folowing Cython code:

cdef class C(object):
    cdef int a,b
    def  __init__(self, int a=0, int b=0):
        self.a = a
        self.b = b

--[Btw. no idea if it is necessary to specify the type at "cdef int a,b" as well as in the __init__ arguments, in the Cython code]--

b) [Not really full Python but useful if some parts are not subject to changes]
Write B.pyx. Move some functions that need optimization from A.py to B.pyx. Rewrite functions in B.pyx using Cython's syntax. Also, "to make them accessible from Python code, you need to declare them as "public" or "readonly" (source) [uh? I think "cpdef" does that]. Compile B.pyx with a separate setupB.py. Then in A.py import (not cimport because it is only when there is a .pxd) functions and classes from B. Finally, compile A.py with setupA.py.

* How to debug (still trying).


To my opinion the doc lacks an example of a full class with __init__, __cinit__, arguments to __init__ and their default values.

[Robert Bradshaw]

On Wed, May 28, 2014 at 2:34 AM, Julien Delafontaine
<mura...@gmail.com> wrote:
>> I think this is a place where "newbies" can help a lot -- you know better
>> than the core developers what initial explanation is needed. Perhaps you
>> could write up a tutorial following your process in "cythonizing" your code
>> in pure-python mode.
>>
>> Of course, it would require review by the "experts", but I find we're a
>> lot more likley to get helpful comments and review than getting one of them
>> to write the whole thing from scratch!
>
>
> I actually wanted to do so anyway, because I will report my work on a
> bioinformatics blog (in french), as I often do.
> At the moment though, I can't go really far following the official doc. What
> I would write already looks like that, and maybe it is half wrong:
>
> * From A.py, how to create the setupA.py, compile and see how you gain 20%
> speed.
> --[No idea why there is a "cythonize" function but other examples with
> "Extension", doing the same while "cythonize" is more limited]--

How is cythonize more limited? The primary reason the docs do
otherwise is that it didn't exist before.

> * Then two methods:
>
> a) Create A.pxd, that is not meant for it but can be used nonetheless to
> override python methods by cython ones. It has limitations - to be listed -
> over method b).
> Do not rename to A.pyx because it would ignore A.pxd at compilation time.
> To override functions, one must use "cpdef inline".

Huh?

> To override classes, one must use
> cdef class <name>:
> cpdef <type> <future attribute>
>
> For instance, the python class
>
> class C(object):
> def __init__(self, a=0, b=0):
> self.a = a
> self.b = b
>
> can be overriden by specifying the following in the .pxd:
>
> cdef class C:
> cpdef int a,b
>
> so that it is in the end equivalent to the folowing Cython code:
>
> cdef class C(object):
> cdef int a,b
> def __init__(self, int a=0, int b=0):
> self.a = a
> self.b = b
>
> --[Btw. no idea if it is necessary to specify the type at "cdef int a,b" as
> well as in the __init__ arguments, in the Cython code]--

No need to specify the type in the __init__ method.

> b) [Not really full Python but useful if some parts are not subject to
> changes]
> Write B.pyx. Move some functions that need optimization from A.py to B.pyx.
> Rewrite functions in B.pyx using Cython's syntax. Also, "to make them
> accessible from Python code, you need to declare them as "public" or
> "readonly" (source) [uh? I think "cpdef" does that].

public/readonly is about class attributes. public has a completely
different meaning elsewhere (namely, public to C == don't mangle the
name)

> Compile B.pyx with a
> separate setupB.py. Then in A.py import (not cimport because it is only when
> there is a .pxd) functions and classes from B. Finally, compile A.py with
> setupA.py.

An alternate (b) is to rename a.py to a.pyx and start hacking on it
right there. You may not even need a.pxd at this point.

> * How to debug (still trying).
>
>
> To my opinion the doc lacks an example of a full class with __init__,
> __cinit__, arguments to __init__ and their default values.

Please contribute!

- Robert

[Julien Delafontaine]

How is cythonize more limited? The primary reason the docs do
otherwise is that it didn't exist before.

Actually I was just confused. When a newbie wants to compile the first time, he goes to the doc and sees chapters:
- Getting started: Building Cython code
- Users guide: Source files and compilation
- Reference guide: Compilation
and it is confusing. Now I got it: Extensions have nothing to do with Cython and must be cythonized. How it is possible that without cythonizing (because I copied/pasted the wrong example) I still get .c and .so files and all working is a total mystery. So what does "cythonize" do exactly?

> * Then two methods:
>
> a) Create A.pxd, that is not meant for it but can be used nonetheless to
> override python methods by cython ones. It has limitations - to be listed -
> over method b).
> Do not rename to A.pyx because it would ignore A.pxd at compilation time.
> To override functions, one must use "cpdef inline".

Huh?

Everybody says that .pxd files serve as interface between Cython modules and it looks like a hack to use them to augment .py files - with which apparently they act very differently than with .pyx. Still, if the examples I gave works, it magically allows to get a much faster executable when compiling the raw, independent python script, by overriding/augmenting python functions by a closer-to-C version.
To make this work, the .pxd can contain only "cdef inline" functions, otherwise the compile complains.
About limitations... not sure yet. No __cinit__ for instance (?).
If it works, then it is just like compiling pure Cython. In the first case one has to maintain the .py and the .pxd in parallel, but has an interpretable pure Python version of the code; in the second one has only 1 file but needs compilation every time.
I finally managed to make it work. Because before I tried to "cpdef" the variables in the class constructor instead of "cdef public" in the .pxd !

Actually method b) is indeed the same as the full Cython mode. It is much easier for new users.

 

> To my opinion the doc lacks an example of a full class with __init__,
> __cinit__, arguments to __init__ and their default values.

Please contribute!

- Robert

A full example of method a) that I would give, please correct me:

A.py:

def count(a, b=0, c=''):
    if c=='hello':
        return  a+b

class Counter(object):
    def __init__(self, b=0, c=''):
        self.a = 5
        self.b = b
        self.c = c
    def mymethod(self, x):
        return (self.a + x)/self.b

A.pxd:

cpdef inline int count(int a,int b,str c)

cdef class Counter:
    cdef public double a,b
    cdef public str c
    cdef inline double mymethod(self,double x)

Not sure if one can do more (__cinit__?).
As I understood @locals does not allow to define types of class attributes, so @cclass adds very little. And why would we need to call @locals in the .pxd, as the "Pure Python mode" chapter suggests? It seems to me that

cpdef int dostuff(int n)
    cdef int t,i

does the same as

import cython
@cython.locals(t=cython.int, i=cython.int)
cpdef int dostuff(int n)

At my level of understanding I would not dare to push any documentation to GitHub, but at least you have the newbie's point of view to fix it.

P.S. And as a final note, having a lambda declared in a function is considered as a closure and so it is impossible to "cdef" it, right?

[Chris Barker - NOAA Federal]

> How is cythonize more limited?

The cythonize API is confusing, because it can be used with just a
.pyx file name to generate an extension object, but if you need
additional features, you need to create an Extension object, and then
pass that into cythonize. I'm a little confused as to why cythonize
can't take all the keyword arguments and pass them along to Extension
under the hood -- or can it? But the documentation could be more clear
in any case.

Perhaps a pull request that updates all the docs to use cythonize is in order.

And maybe the docs should skip over the "just pass in the pyx file
name" option, while that's a nice easy API for the simple case, it's
does add to the confusion as to what cythonize really is/does. I.e. It
is not a replacement for Extension.

-CHB

[Chris Barker - NOAA Federal]

Sorry for the top- post..it's a pain to do it right on a phone...

It seems there is a bit of confusion here about pure-python mode vs. "regular" cython. Which is the goal for this tutorial ? Is this in fact intended as a draft tutorial ?

I'll help edit a bit once I understand the goal.

-Chris

--

[Julien Delafontaine]

Yes, as a draft tutorial for the pure Python mode, which documentation is not very clear atm. I struggled a lot with it  as a usual Python developer. That is, how to not touch the Python script but create a corresponding .pxd file that adds the highest speed boost after compilation.

[Chris Barker]

On Thu, May 29, 2014 at 8:43 AM, Julien Delafontaine <mura...@gmail.com> wrote:

Yes, as a draft tutorial for the pure Python mode, which documentation is not very clear atm. I struggled a lot with it  as a usual Python developer. That is, how to not touch the Python script but create a corresponding .pxd file that adds the highest speed boost after compilation.

OK -- I think this is a worthwhile contribution, and it does change a bit the workflow I'd suggest.

Perhaps best would be for you to fork the repo, add this page, then we can all work on it in that repo, and you can submit a pull request when we think it's ready. But in the meantime:

 * From A.py, how to create the setupA.py, compile and see how you gain 20% speed.

OK, so start with a pretty simple A.py, and another file (test_A.py?) that calls it does some timing perhaps.

--[No idea why there is a "cythonize" function but other examples with "Extension", doing the same while "cythonize" is more limited]--

I'd suggest a simple setup.py example, and I'd have it build an Extension object, then use cythonize on that extension object - it's a bit more clear that way. Though now that I think about it -- I'm not sure how to build an extension in pure python mode -- have you got that working?

a) Create A.pxd, that is not meant for it but can be used nonetheless to override python methods by cython ones.

you can do pure python mode with the  special functions and decorators. It may be better to start off that way, rather than the *.pxd approach. OR  jump straight ot the *.pxd -- perhaps someone else can suggest the best approach for newbies, and/or provide a little advice on why one might use one method or the other.

 

It has limitations - to be listed - over method b).
Do not rename to A.pyx because it would ignore A.pxd at compilation time.

I'm not sure I understand this -- though I haven't used pure python mode beyond a little experimenting, ...

 

To override functions, one must use "cpdef inline".
To override classes, one must use

cdef class <name>:
    cpdef <type> <future attribute>

For instance, the python class

class C(object):
    def __init__(self, a=0, b=0):
        self.a = a
        self.b = b

can be overriden by specifying the following in the .pxd:

cdef class C:
    cpdef int a,b

so that it is in the end equivalent to the folowing Cython code:

cdef class C(object):
    cdef int a,b
    def  __init__(self, int a=0, int b=0):
        self.a = a
        self.b = b

--[Btw. no idea if it is necessary to specify the type at "cdef int a,b" as well as in the __init__ arguments, in the Cython code]--

b) [Not really full Python but useful if some parts are not subject to changes]
Write B.pyx. Move some functions that need optimization from A.py to B.pyx. Rewrite functions in B.pyx using Cython's syntax.

now this is getting out of pure python mode. I'd suggest that in this case, you want to keep it all in A.py and work to make it fully cythonized in pure python mode.

And at each step, run the test_A.py file and make sure it works the same and see the performance changes.

 

Also, "to make them accessible from Python code, you need to declare them as "public" or "readonly" (source) [uh? I think "cpdef" does that]. Compile B.pyx with a separate setupB.py. Then in A.py import (not cimport because it is only when there is a .pxd) functions and classes from B. Finally, compile A.py with setupA.py.

again, I don't think I'd bother with the B.pyx, that may be been my suggestion to start with, but that's what I'd do if you weren't targeting pure python mode. With pure python mode, the idea is to leave the python file essentially the same.

* How to debug (still trying).

well, debugging is a challenge!

But one stage you should highlight is calling cython -a to get an html annotated version, then looking for the yellow to see where a lot of python calls are being made -- that will tell you what still needs to be augmented for full Cython performance.

 

To my opinion the doc lacks an example of a full class with __init__, __cinit__, arguments to __init__ and their default values.

yes, that would be a good examples to have. Though off the top of my head, I can't think of where you'd need a __cinit__ in pure python mode. Do you have an example?

-Chris

[Julien Delafontaine]

On Thursday, 29 May 2014 20:49:38 UTC+2, Chris Barker wrote:

On Thu, May 29, 2014 at 8:43 AM, Julien Delafontaine <mura...@gmail.com> wrote:

Yes, as a draft tutorial for the pure Python mode, which documentation is not very clear atm. I struggled a lot with it  as a usual Python developer. That is, how to not touch the Python script but create a corresponding .pxd file that adds the highest speed boost after compilation.

OK -- I think this is a worthwhile contribution, and it does change a bit the workflow I'd suggest.

Perhaps best would be for you to fork the repo, add this page, then we can all work on it in that repo, and you can submit a pull request when we think it's ready.

Ok, I will do that.

But in the meantime:

 * From A.py, how to create the setupA.py, compile and see how you gain 20% speed.

I'd suggest a simple setup.py example, and I'd have it build an Extension object, then use cythonize on that extension object - it's a bit more clear that way. Though now that I think about it -- I'm not sure how to build an extension in pure python mode -- have you got that working?

At first I created and extension and did "ext_modules = [extensions]", ran it and it worked... Now I do the same with "cythonize([extensions])" and it works, too, although I have no idea why and what it does.
 

you can do pure python mode with the  special functions and decorators. It may be better to start off that way, rather than the *.pxd approach. OR  jump straight ot the *.pxd -- perhaps someone else can suggest the best approach for newbies, and/or provide a little advice on why one might use one method or the other.

Here I can say already, that @locals has very limited use because it does not allow to type class attributes. Because of that @cclass is almost useless. Only @locals can speed up functions that are not class methods. Plus why add the "import cython" dependency to the python script if one can obtain the same from a .pxd file?
 

Do not rename to A.pyx because it would ignore A.pxd at compilation time.

I'm not sure I understand this -- though I haven't used pure python mode beyond a little experimenting, ...

I understand it better now, but as one can augment python functions declared in a .py with a .pxd, if one renames the .py to .pyx it behaves completely differently with the .pxd and does not work as expected. Newbie mistake.
 

* How to debug (still trying).

well, debugging is a challenge!

But one stage you should highlight is calling cython -a to get an html annotated version, then looking for the yellow to see where a lot of python calls are being made -- that will tell you what still needs to be augmented for full Cython performance.

 Yes, this one is very interesting and I am often surprised by the result! Definitely to include.

[Chris Barker]

On Thu, May 29, 2014 at 12:35 PM, Julien Delafontaine <mura...@gmail.com> wrote:

Perhaps best would be for you to fork the repo, add this page, then we can all work on it in that repo, and you can submit a pull request when we think it's ready.

Ok, I will do that.

post here when you've  got that going...

 

At first I created and extension and did "ext_modules = [extensions]", ran it and it worked... Now I do the same with "cythonize([extensions])" and it works, too, although I have no idea why and what it does.

I think cythonize adds some smarts so that distutils knows when it needs to re-run cython , rebuild, etc. It might make "setup.py clean" work better as well.

 

Here I can say already, that @locals has very limited use because it does not allow to type class attributes. Because of that @cclass is almost useless. Only @locals can speed up functions that are not class methods. Plus why add the "import cython" dependency to the python script if one can obtain the same from a .pxd file?

sounds like the *.py + *.pxd is the way to go for this tutorial then.

But one stage you should highlight is calling cython -a to get an html annotated version, then looking for the yellow to see where a lot of python calls are being made -- that will tell you what still needs to be augmented for full Cython performance.

 Yes, this one is very interesting and I am often surprised by the result! Definitely to include.

thanks for taking the time to put this together.

[Julien Delafontaine]

post here when you've  got that going...

I sent a pull request (#298).

[Stefan Behnel]

Chris Barker - NOAA Federal, 29.05.2014 17:17:

> Perhaps a pull request that updates all the docs to use cythonize is in order.

I've updated them a while ago.

https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/

> And maybe the docs should skip over the "just pass in the pyx file
> name" option, while that's a nice easy API for the simple case, it's
> does add to the confusion as to what cythonize really is/does. I.e. It
> is not a replacement for Extension.

Agreed that passing Extensions is the more important option.

https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/src/reference/compilation.html#configuring-the-c-build

Stefan

[Stefan Behnel]

Julien Delafontaine, 29.05.2014 21:35:

> At first I created and extension and did "ext_modules = [extensions]", ran
> it and it worked...

I guess you were monkey patching the "build_ext" command, though, right?

> Now I do the same with "cythonize([extensions])" and it
> works, too, although I have no idea why and what it does.

It translates your Cython modules to .c files and fixes up the Extensions
instances to refer to them rather than the .pyx files.

> Here I can say already, that @locals has very limited use because it does
> not allow to type class attributes. Because of that @cclass is almost
> useless.

You would use declare() for cdef class attributes. They are not "locals",
neither technically nor semantically.

> Plus why add the "import cython" dependency to the python script if one can
> obtain the same from a .pxd file?

Your choice. Some people prefer keeping everything in one place instead of
letting you look at code that does something different than what it says.

> I understand it better now, but as one can augment python functions
> declared in a .py with a .pxd, if one renames the .py to .pyx it behaves
> completely differently with the .pxd and does not work as expected.

Depends on what you expect and which you try first, but yes, it's either
.pyx, or .py plus overrides in .pxd. For .pyx files, the corresponding .pxd
file declarations must be in line, simply because there is no reason to
have (and definitely not allow) diverting declarations in both files. For
.py, allowing more restrictive declarations in the corresponding .pxd file
is not only allowed and meaningful but a real feature.

Stefan

[Stefan Behnel]

Stefan Behnel, 30.05.2014 15:37:

> Julien Delafontaine, 29.05.2014 21:35:

>> I understand it better now, but as one can augment python functions
>> declared in a .py with a .pxd, if one renames the .py to .pyx it behaves
>> completely differently with the .pxd and does not work as expected.
>
> Depends on what you expect and which you try first, but yes, it's either
> .pyx, or .py plus overrides in .pxd. For .pyx files, the corresponding .pxd
> file declarations must be in line, simply because there is no reason to
> have (and definitely not allow) diverting declarations in both files. For
> .py, allowing more restrictive declarations in the corresponding .pxd file
> is not only allowed and meaningful but a real feature.

In other words, the difference is not between the usage of .pxd files in
both cases but between .py files and .pyx files. They behave differently,
because one has static type declarations and the other does not. The .pxd
file behaves the same in both cases.

Stefan

[Julien Delafontaine]

Before I send the new pull request with corrections from "scoder", why is it that

cpdef int myfunction(int x,int y=*)

in the .pxd works, but

cpdef int myfunction(int x,int y=*):
    pass

says: "function definition in pxd file must be declared 'cdef inline'" ?
And why

cpdef inline int myfunction(int x,int y=*):
    cpdef int a

fails with "TypeError: 'int' object is not iterable"?
The .py contains

def myfunction(x, y=2):
    a = x-y
    return a + x * y

[Stefan Behnel]

Julien Delafontaine, 30.05.2014 17:11:

> Before I send the new pull request with corrections from "scoder", why is
> it that
>
> cpdef int myfunction(int x,int y=*)
>
> in the .pxd works, but
>
> cpdef int myfunction(int x,int y=*):
> pass
>
> says: "function definition in pxd file must be declared 'cdef inline'" ?

Because in the second case, you're not only *declaring* a function
*signature*, you are implementing an actual *function* in a .pxd file,
which is meant for declarations (.px*d*). That can normally only be done in
code modules (.py or .pyx), but there is an explicit exception that allows
you to share functions through .pxd files if (and only if) they are
"inline" functions.

> And why
>
> cpdef inline int myfunction(int x,int y=*):
> cpdef int a
>
> fails with "TypeError: 'int' object is not iterable"?
> The .py contains
>
> def myfunction(x, y=2):
> a = x-y
> return a + x * y

Hm? That sounds like a bug.

In any case, "cpdef int a" should be rejected by the compiler. There is no
such thing as a Python wrapper around a local variable.

I also wonder if

> cpdef inline int myfunction(int x,int y=*):
> cpdef int a

is the right thing to support at all. It looks like a function
implementation but isn't?

Stefan

[Stefan Behnel]

Julien Delafontaine, 29.05.2014 13:07:

> Everybody says that .pxd files serve as interface between Cython modules
> and it looks like a hack to use them to augment .py files - with which
> apparently they act very differently than with .pyx.

A better way to think about the two use cases is that Cython is more
relaxed about declaration mismatches between a .py files and its .pxd file
than between a .pyx file and its .pxd file, simply because .py files do not
allow specifying these declarations in their syntax due to their "weaker"
or "less explicit" type system. Thus, Cython trusts .pxd files over .py
files, but it requires correct correspondence between .pyx files and .pxd
files, which can both express the same type system.

> About limitations... not sure yet. No __cinit__ for instance (?).

That method simply doesn't exist in pure mode, regardless if you use .pxd
files or .py-only declarations. The semantics of __cinit__() do not exist
in Python code.

> If it works, then it is just like compiling pure Cython. In the first case
> one has to maintain the .py and the .pxd in parallel, but has an
> interpretable pure Python version of the code; in the second one has only 1
> file but needs compilation every time.

In pure mode, you are more or less restricted to code that can be expressed
(or at least emulated) in Python, plus static type declarations. Anything
beyond that can only be done in .pyx files with extended language syntax,
because it depends on compilation. Using it in a pure Python .py file is
useless.

> I finally managed to make it work. Because before I tried to "cpdef" the
> variables in the class constructor instead of "cdef public" in the .pxd !

Right. "cpdef variable" should be rejected by the compiler.

(I think the rest of your post is answered in other replies already.)

Stefan

[Robert Bradshaw]

Thanks for helping clean up our docs! I think one thing to mention
that hasn't come up yet is that pure mode (at least in its current
form) is definitely a more advanced use than just going with a
straight .pyx file. In particular, I wouldn't recommend unless there's
a very compelling need using it as the way to make a .py file faster.
Definitely easier to first write normal Cython code then to figure out
how to write the Cython you need without violating any Python syntax.

- Robert

[Julien Delafontaine]

I mentioned that it was not recommended unless there are good reasons to do so. Indeed I fell back to writing pure Cython instead, in the end.
For a Python developer with few experience in C, the compilation step is the frightening part and I for instance was afraid of losing any possibility of running my code for reasons like "Apple changed gcc for clang and Xcode5.1 is bugged so be patient until the next update"... So in one sense the Pure Python mode is very reassuring and seems even more intuitive.
The ideal mode would be not to use .pxd since it is confusing, but provide a similar way of augmenting Python code from an external file allowing to override/complement any part of the Python by Cython commands. If possible, without a mix of "cdef" and decorators (which seem at start to be two different ways of obtaining the result but actually need to be used together).

At this point I think I should let someone experienced complete/correct (or remove) my commits. Hope it was useful.