.py and .pyx versions side-by-side?

[Skip Montanaro]

I'm brand new to Cython. Have barely made it through the basic tutorial, so go easy on me. :-)

I have an existing Python code base. I know I can use pyximport and tell it to try and Cythonize everything it encounters. That would seem to give only modest performance boost in the common case, as my Python code won't have any type declarations. Can I copy foo.py to foo.pyx and start annotating it, safe in the realization that pyximport will grab the pyx version first? Or do I need to delete the py version?

Thx,

Skip Montanaro

[Björn Dahlgren]

On Tuesday, 29 April 2014 16:07:48 UTC+2, Skip Montanaro wrote:

I'm brand new to Cython. Have barely made it through the basic tutorial, so go easy on me. :-)

I have an existing Python code base. I know I can use pyximport and tell it to try and Cythonize everything it encounters. That would seem to give only modest performance boost in the common case, as my Python code won't have any type declarations. Can I copy foo.py to foo.pyx and start annotating it, safe in the realization that pyximport will grab the pyx version first? Or do I need to delete the py version?

I seem to need to rename it:

mytest.py:

    import cython                                                              
                                                                               
    def callback():                                                            
        if cython.compiled:                                                    
            return "I am a compiled callback."                                 
        else:                                                                  
            return "I am being interpreted."


main.py:

    import pyximport
    pyximport.install()
    from mytest import callback
    print(callback())

$ ls
main.py  mytest.py
$ python main.py
I am being interpreted.
$ cp mytest.py mytest.pyx
$ python main.py
I am being interpreted.
$ rm mytest.py
$ python main.py
I am being interpreted.
$ rm mytest.pyc
$ python main.py
I am a compiled callback.

I usually keep the original lying around and give the cython a prepended _ to itsname, e.g. _mytest.pyx
 

Thx,

Skip Montanaro

You may also want to look at:

http://docs.cython.org/src/tutorial/pure.html

But in my experience pure mode is a bit too limited.

Best regards,
/Björn

[Skip Montanaro]

> I seem to need to rename it:

...
> I usually keep the original lying around and give the cython a prepended _ to its name...

Renaming files, removing .pyc files, etc seems counterintuitive. If your .py and .pyx files can't have the same base name and .pyx can't take precedence over .pyc, how would

pyximport.install(pyimport=True)

ever work? (It doesn't work for me, at least at the moment, but that's a different story, I think.)

Thx,

Skip

[Robert Bradshaw]

pyximport compiles .py files. If both the pyx and py file exist,
there's ambiguity over which one should be used

>>> import this
...
In the face of ambiguity, refuse the temptation to guess.

I suppose we could change the precedence and put foo.pyx right next to foo.so at

http://stackoverflow.com/questions/6319379/python-shared-object-module-naming-convention

though generally I think it's preferable to handle this decision in
your setup.py file rather than using pyximport at runtime.

- Robert

[Skip Montanaro]

On Tue, Apr 29, 2014 at 4:17 PM, Robert Bradshaw <robe...@gmail.com> wrote:
> pyximport compiles .py files.

That wasn't clear from my reading of the description of this module.

Thx,

Skip

[Skip Montanaro]

Also, the name of the module strongly suggests its function is to import pyx files.

S

[Skip Montanaro]

I suppose we could change the precedence and put foo.pyx right next to foo.so at

http://stackoverflow.com/questions/6319379/python-shared-object-module-naming-convention

though generally I think it's preferable to handle this decision in
your setup.py file rather than using pyximport at runtime.

Perhaps my use case is atypical. I have a large Python code base I want to speed up by targeting selected modules with Cython. (Simply using pyximport without touching any files failed miserably. Don't know why.) We run on two different platforms (openSUSE and Solaris 10), both of which have Cython available. If I can distribute .pyx files and compile them on-the-fly, then I can continue to deliver one platform-independent package for both Linux and Solaris. Also, if compilation/import of a pyx file fails for some reason, it seems that falling back to normal Python imports (thus requiring the presence of the .py or .pyc files) makes some sense.

Skip

[Stefan Behnel]

Skip Montanaro, 30.04.2014 00:35:

> On Tue, Apr 29, 2014 at 4:17 PM, Robert Bradshaw wrote:
>> pyximport compiles .py files.
>
> That wasn't clear from my reading of the description of this module.

It is meant to compile .pyx files (thus the name) but can be configured to
also intercept the import of .py files and compile them. However, that will
include all sorts of externally installed modules and the standard library,
some of which will just fail to compile or run. Therefore, it's not a good
idea to use it for what you intend.

Stefan

[Stefan Behnel]

Skip Montanaro, 30.04.2014 02:51:

That's not how it's done. Instead, compile the .pyx modules to .c files,
properly test them on your side, and then ship them with the source
release. That removes the need to have (some specific version of) Cython
installed on user side in order to get results that are reproducible on
your side (if things go wrong). In fact, disable Cython compilation at
install time and make it an explicit option, so that users who happen to
have a different version of Cython installed than you don't accidentally
end up generating the C sources.

All that's needed then on user side is a C compiler to install the compiled
modules (plain distutils build), and failing that, you can fall back to
Python code as usual.

Specifically, pyximport or any other form of runtime compilation is not
what you want here.

Stefan

[Skip Montanaro]

On Wed, Apr 30, 2014 at 12:41 AM, Stefan Behnel <stef...@behnel.de> wrote:
> That's not how it's done. Instead, compile the .pyx modules to .c files,
> properly test them on your side, and then ship them with the source
> release.

Sorry, I wasn't clear. Our packages (we use encap) are all installed
locally, but we have hundreds of machines. All run Python 2.7 and
Cython 0.17.1. All packages are also ready-to-run out of the box. I've
bitten the bullet at this point and now just build Linux- and
Solaris-specific binary packages.

Skip

[Chris Barker - NOAA Federal]

> On Apr 30, 2014, at 6:14 AM, Skip Montanaro <sk...@pobox.com> wrote:
>
>
> Sorry, I wasn't clear. Our packages (we use encap) are all installed
> locally, but we have hundreds of machines. All run Python 2.7 and
> Cython 0.17.1. All packages are also ready-to-run out of the box.

Not even a setup.py install? (Or pip, or conda, or?) Usually, packages
are installed, even if pure python. In which case the compilation can
happen at install time.

This is usually an issue with compiled modules when users don 't have
a properly set up compiler, which it sounds like is not your case.

However, if you have a lot of similar machines, all that compiling and
installing can get annoying. In this case, setting up a wheelhouse
that you put binary wheels is can be a nice distribution method.

Chris

> I've
> bitten the bullet at this point and now just build Linux- and
> Solaris-specific binary packages.
>
> Skip
>

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Skip Montanaro]

On Wed, Apr 30, 2014 at 10:26 AM, Chris Barker - NOAA Federal
<chris....@noaa.gov> wrote:
> Not even a setup.py install? (Or pip, or conda, or?)

Nope. I use the term "package" here in a much-broader-than-Python
sense. If, for example, our system didn't have a valgrind package, I
could build it from source and package it up. Installation is
generally little more than unpacking a compressed tar file into
/opt/local/encap and fixing up a bunch of symlinks, which Encap takes
care of (the epkg command).

Back into the Python world, if I find something I want on PyPI, I
download it, build and install it (which might include executing a
setup.py file or running pip) in /var/tmp/$USER, then follow a
reasonably straightforward recipe of about three commands to build an
Encap package. I toss that onto our NFS package server from where it
can be installed manually or in an automated fashion on a daily basis
using a set of "tags". Recent PyPI offerings installed this way
include statsmodels, patsy, pylint, scikit, and Theano.

Skip

[Chris Barker]

On Wed, Apr 30, 2014 at 8:40 AM, Skip Montanaro <sk...@pobox.com> wrote:

Nope. I use the term "package" here in a much-broader-than-Python

sense. If, for example, our system didn't have a valgrind package, I
could build it from source and package it up.

fair enough.

 Recent PyPI offerings installed this way
include statsmodels, patsy, pylint, scikit, and Theano.

If you're doing all that, then building some of your own cython based stuff should be straightforward.

It sounds like you've got a fine working system, but if you do feel the need to improve it, you might look at conda -- while primarily about Python, it is explicitly designed to support packages that are anything -- very much like your use-case as a matter of fact.

-CHB

 

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Robert Bradshaw]

So why not package your files similar to scikit, theano, etc. with a
setup.py and shipped .c files generated by Cython ahead of time? The c
files are platform independent. pyximport doesn't sound like the right
tool here.

- Robert

[Skip Montanaro]

On Wed, Apr 30, 2014 at 1:08 PM, Robert Bradshaw <robe...@gmail.com> wrote:
> So why not package your files similar to scikit, theano, etc. with a
> setup.py and shipped .c files generated by Cython ahead of time? The c
> files are platform independent. pyximport doesn't sound like the right
> tool here.

While we have hundreds of hosts, we only have a couple "certified"
build hosts. Client machines are not guaranteed to have all the
necessary "devel" packages installed.

The convenience of pyximport was probably only going to ever be a
transient help. My bad.

Skip

[Stefan Behnel]

Skip Montanaro, 30.04.2014 20:30:

> On Wed, Apr 30, 2014 at 1:08 PM, Robert Bradshaw wrote:
>> So why not package your files similar to scikit, theano, etc. with a
>> setup.py and shipped .c files generated by Cython ahead of time? The c
>> files are platform independent. pyximport doesn't sound like the right
>> tool here.
>
> While we have hundreds of hosts, we only have a couple "certified"
> build hosts. Client machines are not guaranteed to have all the
> necessary "devel" packages installed.

You don't need devel packages on a client, just on the build hosts. Once
Cython is done with translating the .pyx file into a .c file, the remaining
package build is exactly the same as for other binary packages in Python,
such as the ones you named before, numpy, scikit, ...

If you can manage to package those libraries for your set of machines, I
can't see what the problem with a Cython module should be. Just build an
sdist with distutils that includes the generated .c file(s), and then pass
it into your packaging toolchain.

Stefan

[Skip Montanaro]

On Wed, Apr 30, 2014 at 1:57 PM, Stefan Behnel <stef...@behnel.de> wrote:
> You don't need devel packages on a client, just on the build hosts. Once
> Cython is done with translating the .pyx file into a .c file, the remaining
> package build is exactly the same as for other binary packages in Python,
> such as the ones you named before, numpy, scikit, ...

Right, on the official build hosts. The include files (and perhaps
some libraries) necessary to compile and link C files may not be
available everywhere.

S