Re: Bringing Cython and PyPy closer together
Stefan Behnel
Stefan Behnel, 15.02.2012 12:32:
> The current state of the discussion seems to be that PyPy provides ways to
> talk to C code, but nothing as complete as CPython's C-API in the sense
> that it allows efficient two-way communication between C code and Python
> objects. Thus, we need to either improve this or look for alternatives.
>
> In order to get us more focussed on what can be done and what the
> implications are, so that we may eventually be able to decide what should
> be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement
> Proposal).
>
> http://wiki.cython.org/enhancements/pypy
The discussion so far makes me rather certain that the most promising
short-term solution is to make Cython generate C code that PyPy's cpyext
can handle. This should get us a rather broad set of running code somewhat
quickly, while requiring the least design-from-scratch type of work in a
direction that does not yet allow us to see if it will really make existing
code work or not.
On top of the basic cpyext interface, it should then be easy to implement
obvious optimisations like native C level calls to Cython wrapped functions
from PyPy (and potentially also the other direction) and otherwise avoid
boxing/unboxing where unnecessary, e.g. for builtins. After all, it all
boils down to native code at some point and I'm sure there are various ways
to exploit that.
Also, going this route will help both projects to get to know each other
better. I think that's a required basis if we really aim for designing a
more high-level interface at some point.
The first steps I see are:
- get Cython's test suite to run on PyPy
- analyse the failing tests and decide how to fix them
- adapt the Cython generated C code accordingly, special casing for PyPy
where required
Here is a "getting started" guide that tells you how testing works in Cython:
http://wiki.cython.org/HackerGuide
Once we have the test suite runnable, we can set up a PyPy instance on our
CI server to get feed-back on any advances.
https://sage.math.washington.edu:8091/hudson/
So, any volunteers or otherwise interested parties to help in getting this
to work? Anyone in for financial support?
Stefan
Stefan Behnel
> Stefan Behnel, 15.02.2012 12:32:
>> The current state of the discussion seems to be that PyPy provides ways to
>> talk to C code, but nothing as complete as CPython's C-API in the sense
>> that it allows efficient two-way communication between C code and Python
>> objects. Thus, we need to either improve this or look for alternatives.
>>
>> In order to get us more focussed on what can be done and what the
>> implications are, so that we may eventually be able to decide what should
>> be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement
>> Proposal).
>>
>> http://wiki.cython.org/enhancements/pypy
>
> The discussion so far makes me rather certain that the most promising
> short-term solution is to make Cython generate C code that PyPy's cpyext
> can handle. This should get us a rather broad set of running code somewhat
> quickly, while requiring the least design-from-scratch type of work in a
> direction that does not yet allow us to see if it will really make existing
> code work or not.
Update:
Amaury Forgeot d'Arc fiddled out a couple of fixes and hacks to make it run
(although with some clear bugs in the exception handling code). There is a
Jenkins job now to (try to) run the test suite of my own branch in the
latest PyPy nightly build:
https://sage.math.washington.edu:8091/hudson/view/dev-scoder/job/cython-scoder-pypy-nightly/
It currently crashes rather badly at some point, but at least it looks like
it's actually getting somewhere.
Stefan
Lisandro Dalcin
such as PYPY_VERSION_HEX in such a way us we can properly use
conditional compilation. For example, a few days ago I was pushing
PyPy fixes to Cython. I tried to use _PyLong_Sign in my patch, but the
interpreter broke at runtime. This issue will be eventually fixed, I
hope. Unce that happens, how can we know it is save to use the call
for that pypy version and upwards? I mean, Cython should still support
previous PyPy releases...
--
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/
4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa
Office Phone: +966 12 808-0459