Compiling Django with Cython
Stefan Behnel
I've been hanging my head out of the window on python-dev regarding the
usage of pyximport for speed.(pypy|python).org, so I actually wanted to
back that with a couple of facts.
I enabled pyximport's pyimport feature and started playing around with
Django. There are some modules that fail to compile with the latest github
cython, but also several modules that fail to run after compilation. The
main reason for the failures is that Cython compiled Python functions are
PyCFunctions and do not accept attributes (e.g. to set their __name__ after
wrapping them). They are also not weak-referencible, as I had to find out.
After blacklisting 33 failing-at-runtime modules, I managed to compile 162
modules to start up a little Django application. That's 30 modules from the
stdlib, 125 from Django and a couple of my own ones from the test app. Not
that bad for a start. I couldn't run any benchmarks yet (and I actually
think it's still, erm, a bit too fragile for that), but at least it tells
us that we are really getting a lot closer, especially once we get the
function compatibility bug fixed.
Stefan
Anime Lovin
Stefan Behnel
> On Sat, Apr 9, 2011 at 4:21 PM, Stefan Behnel wrote:
>> I've been hanging my head out of the window on python-dev regarding the
>> usage of pyximport for speed.(pypy|python).org, so I actually wanted to back
>> that with a couple of facts.
>>
>> I enabled pyximport's pyimport feature and started playing around with
>> Django. There are some modules that fail to compile with the latest github
>> cython, but also several modules that fail to run after compilation. The
>> main reason for the failures is that Cython compiled Python functions are
>> PyCFunctions and do not accept attributes (e.g. to set their __name__ after
>> wrapping them). They are also not weak-referencible, as I had to find out.
>>
>> After blacklisting 33 failing-at-runtime modules, I managed to compile 162
>> modules to start up a little Django application. That's 30 modules from the
>> stdlib, 125 from Django and a couple of my own ones from the test app. Not
>> that bad for a start. I couldn't run any benchmarks yet (and I actually
>> think it's still, erm, a bit too fragile for that), but at least it tells us
>> that we are really getting a lot closer, especially once we get the function
>> compatibility bug fixed.
>
Here's the complete blacklist I found so far:
'email.mime.text',
'email.mime.nonmultipart',
'django.utils.log',
'django.utils.http',
'django.utils.text',
'django.utils.html',
'django.utils.dictconfig',
'django.utils.functional',
'django.core.urlresolvers',
'django.core.mail.message',
'django.template.defaultfilters',
'django.contrib.staticfiles.finders',
'django.forms.fields',
'django.forms.models',
'django.db.models.base',
'django.db.models.query_utils',
'django.db.models.query',
'django.db.models.sql.query',
'django.db.models.deletion',
'django.db.models.manager',
'django.db.models.fields.subclassing',
'django.db.models.fields.related',
'django.db.models.fields.files',
'django.db.transaction',
'django.views.decorators.cache',
'django.contrib.admin.options',
'django.contrib.admin.actions',
'django.contrib.admin.admin',
'django.contrib.auth.models',
'django.contrib.contenttypes.generic',
There may be others, depending on what you do with it. As I said, most of
them fail because they wrap functions and try to override their name (see
django.utils.functional). I didn't look too closely into the few other
failure reasons.
> i've been compiling custom Django applications in cpython 2.7 and Cython
> 0.14.1 without any compilation issues for a while now..
I assume you mean "compiling your own code in a Django application", rather
than "compiling Django itself", right? You sure can use compiled modules
*in* Django.
Stefan
Robert Bradshaw
Cool. On this note, for fun the other day I tried seeing how much of
the Python (2.7) standard library would compile with Cython. Of the
1538 .py files, 1219 of them compile just fine. The vast majority of
them are due to either
- Relative imports, I'm hoping to merge that soon
- Byte literals, i.e. "Decoding unprefixed string literal from
'UTF-8' failed. Consider usinga byte string or unicode string
explicitly, or adjust the source code encoding." We'll somehow need to
support this in pure python mode.
Haven't tried running any of it yet...
- Robert