speed up python routine writing C extension: can Cython be useful?

[GB]

Hi,

I have a program that analyze 3000000 data and for each of them it makes 200 calculations (an interpolation with interp1d from scipy.interpolate).

I would like to speed up the process because now it takes near 12 hours...

I know that the bottleneck is this interp1d because I have profiled the program for different (small) numebers of data and this is the process that increase in time.

Do you think that using a C extension could speed up the process? How can I do that (it's the first time I try something like this)?

thanks

GB

[Robert Bradshaw]

I would guess that scipy.interpolate is already in a C extension;
calling it from a C extension won't make it faster.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Chris Barker - NOAA Federal]

On Apr 11, 2014, at 2:16 PM, Robert Bradshaw <robe...@gmail.com> wrote:

> I would guess that scipy.interpolate is already in a C extension;
> calling it from a C extension won't make it faster.

Though re-writing a simple one for you special case might. But I'm
guessing you've got another bottleneck.

Do make sure that you are creating the interpolation function once,
then re-using it. Rather than recreating it.

-Chris

[Sturla Molden]

Robert Bradshaw <robe...@gmail.com> wrote:
> I would guess that scipy.interpolate is already in a C extension;
> calling it from a C extension won't make it faster.

Calling it from a C extension will not make it faster, regardless of how it
is implemented.

It seems scipy.interpolate.interp1 is mostly implemented in Python, except
for spline interpolation which is written in Fortran 77.


Sturla

[Björn Dahlgren]

On Friday, 11 April 2014 19:46:17 UTC+2, GB wrote:

Hi,

I have a program that analyze 3000000 data and for each of them it makes 200 calculations (an interpolation with interp1d from scipy.interpolate).

I would like to speed up the process because now it takes near 12 hours...

For one project I had PiecewisePolynomial from scipy.interpolate being the bottle neck. This C/Cython implementation
I wrote should cut that computation down to ~1 second if you are ok with linear interpolation:

https://github.com/bjodah/cinterpol