Call "typed python" from nopython

[carlosj...@gmail.com]

Hi all,

suppose I've written a jitted loop that has to call a python function f every once in a while. f takes two integers and returns another integer, but it's body executes some logic that is not currently jittable by numba. Nevertheless, it's not difficult to see that f can be wrapped as some typed_f: i8(i8, i8) that just converts its arguments to python integers, calls f and converts the result back to i8. AFAICS this can be currently achieved using @overload, but I find its generality a bit cumbersome for such a simple case (it's more of an extensibility mechanism). What I would like is something similar to @cfunc but which doesn't force nopython code generation, for example:

@typed(i8(i8,i8))
def f(x, y):
     ....
     return z

This would be very handy in order to circumvent nopython mode limitations when loop-lifting is not enough (for example, when calling python from inside a loop).

Is @overload the closer numba currently gets to the above or am I missing anything else?

Cheers
--
Carlos 

[Tem Pl]

Carlos,

You might be interested in some of the ideas floated to be discussed in this workshop: http://python-compilers-workshop.github.io/ 

[Carlos Pita]

You might be interested in some of the ideas floated to be discussed in this workshop: http://python-compilers-workshop.github.io/ 

This is the closest match I could find to my topic there: 

If I wrap a C function using Cython/CFFI/SWIG/…, could there somehow be a way to expose the original C function to Numba/PyPy/Pyston/etc. to cut out the wrapper overhead? What about vice-versa: if I have a Python function that’s been JIT-compiled and I pass it to some native code like scipy.optimize.fmin, could there be some way for the native code in scipy to call the JIT-compiled function directly without going through tuple packing/unpacking? Can Numba and Pyston benefit from PyPy’s work on CFFI?

But my question is not about calling c from jitted or jitted from c, but python from jitted. Specifically, I find the separation between python and nopython modes too drastical and loop-lifting is no silver-bullet.

Some time ago I had and interesting OT discussion about this with Antoine Pitrou and Stan Seibert:

https://github.com/numba/numba/issues/1544

Cheers
--
Carlos

[Antoine Pitrou]

On Thu, 30 Jun 2016 11:17:14 -0700 (PDT)
Carlos Pita <carlosj...@gmail.com>
wrote:

>
> If I wrap a C function using Cython/CFFI/SWIG/…, could there somehow be a
> > way to expose the original C function to Numba/PyPy/Pyston/etc. to cut out
> > the wrapper overhead? What about vice-versa: if I have a Python function
> > that’s been JIT-compiled and I pass it to some native code like
> > scipy.optimize.fmin

> > <https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin.html#scipy.optimize.fmin>,

> > could there be some way for the native code in scipy to call the
> > JIT-compiled function directly without going through tuple
> > packing/unpacking? Can Numba and Pyston benefit from PyPy’s work on CFFI?
> >
>
> But my question is not about calling c from jitted or jitted from c, but
> python from jitted. Specifically, I find the separation between python and
> nopython modes too drastical and loop-lifting is no silver-bullet.

Yes, the current separation is very inflexible (but was easy for us to
develop, which is why it was done like that :-)). One long-term goal is
to make it more flexible, or perhaps obsolete it altogether.

Regards

Antoine.

[Tem Pl]

Carlos, 

It seems the following paragraph describes a solution to your case, no?

 Could there be some way to write a library like numpy so that a single codebase could simultaneously target CPython and the newer compilers, while achieving competitive speed in all cases? If so, what would it take to make that happen? If not, then what’s the next-best alternative?

Antoine,

The idea of expanding Numba to general python code  is very intriguing  and would be quite helpful.

.. but isn't python a bit too dynamic and variable for that? Pypy spends years working on the problem and ends up with a often only  marginally beneficial  (compared to numba) implementation with slow startup and other costs. 

Is there a timeline and strategy in mind?