Django and Cython

[Daniel Anechitoaie]

I'm looking at frameworks like https://vibora.io/ that make use of Cython to greatly improve the performance and I'm just wondering if something like this would work with Django?

Was there any discussion (taken into consideration) about using Cython to boost the performance of certain aspects of the framework?

Would it help make it faster?

I really love Python and Django for me is the best web framework out there, but you have to notice how fast other frameworks from other programming languages are.

Would using Cython help boost the performance?

[Jason Johns]

There was a discussion a while back about this https://groups.google.com/forum/#!searchin/django-developers/cython/django-developers/Fi4U602GxHA/mE50LOPkBgAJ

tl;dr not sure what benefits Django would get from it, since the bottlenecks you experience are most likely non-Django/Python parts of your project, such as networking latency, db queries, connection initiation, etc.  In addition, pypy is an alternative interpreter that you can drop in for up to 3.5.x Python versions.

[Daniel Anechitoaie]

So the fact that frameworks like Vibora and webapps developed in other faster programming languages like GO/NodeJS/etc. support so many more req/s than Django it's relevant only in benchmarks?

And in real life/live web apps the difference in performance won't be noticeable?

I have to mention again that I really don't try to start any flame war and I'm legit interested in this.

As a huge Python fan and unfortunately lone Python dev among my peers they all flame me when I bring Python up and show me all kind of benchmarks how fast NodeJS (as this is what they use) is vs Python. 

I'm just trying to make sure I have ground to stand upon. 

[Jason Johns]

Not at all, just that Cython itself is probably not the approach to take.  Andrew Godwin recently posted a proposal for a Django ASGI roadmap and there's a good amount of work to be done there.

Benchmarks have their place.  But if they're not matching real world problems, then I don't see much use of them other than stroking egos of their developers.  And my personal experience says that most of the problems with Django's slowness have most of the bottlenecks with db connections, networking latency and other things outside of Python.  And those problems inside Python might not be the best things to leverage Cython for.

I get the issue of wanting to show up to your peers with node, but personally, I just don't find it worth it.  It has its use cases, but more and more I see it being squashed into a use case its really not suited for simply because the creator or team are very one-dimensional as far as languages..  Use the right tool for the job, and just ignore the fanbois.

[Tom Forbes]

The people showing you benchmarks comparing Django to ‘NodeJS’ are comparing apples to oranges. Django is not an asynchronous framework (yet!) so you cannot fairly compare the number of raw requests per second the two handle and present that as evidence one is better than the other. Each one has it’s strengths and weaknesses but Django wins hands down in a number of places. Plus it’s not JavaScript, so that’s one thing Django has going for it!

I’ve had immense success with uvloop and asyncio which are much more comparable to NodeJS. Here are some synthetic benchmarks showing it handing over 100,000 requests per second: https://magic.io/blog/uvloop-blazing-fast-python-networking/. I would recommend playing around with asyncio and aiohttp to build some small services and benchmark them yourself, you might be surprised at the speed.

Regarding Cython: while some very specific parts of Django could potentially benefit from the speedup Cython would bring we would need to maintain two separate versions: A Cython one and a fallback Python one. This would add a maintenance burden for an uncertain gain and even if you implemented this the raw requests per second Django would reach would not be close to that of an asynchronous framework with the same resources. Typically Django apps are not CPU bound by the Django code itself, and a lot of time is spent waiting on the network for the database or other services.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/24d4d03d-d9ed-4d7f-9a69-8066846f090e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Curtis Maloney]

There was a discussion some time ago about perhaps cythonising some of
the template engine and other select parts of Django. Someone did some
eexcellent independant work on this, and you should be able to find
their results in the mailing list archives.

However, I think you'll find the common wisdom on speeding up Django is
to reach for tools like PyPy, which can provide hot-spot JIT
compilation, thus focusing performance work on the parts where your app
needs it.

Further to that, my own experiences of optimising and scaling django
have shown often the performance bottlenecks are not where you expect.

As anyone in the industry will tell you, beware of benchmarks not
testing what you _actually_ do.

One time we got a dramatic improvement in site resource usage by moving
a user setting (like preferred language] into a cookie instead of the
user session. This allowed many requests to never access the session
store at all, avoiding a round-trip.

Another time, a site trebled in throughput (on average) by simply
upgrading Django. As is often the case, a smarter algorithm (in this
case, in how QuerySets made copies) will yield improvements orders of
magnitude more than minor optimisations.

As has been mentioned in other posts, async IO is often viewed as a
solution to performance problems. This is a furphy; asyncio can help
improve resource utilisation when you have an IO-bound workload.

It can allow fewer processes to scale more readily to consume more CPU
by avoiding blocking on IO. It's trues that the majority of web sites
are IO bound [waiting for DB, disk, or even other web sites].

However, this doesn't necessarily make each individual request any
faster - in many cases, it can actually be slower.

As is always the case with improving performance, measurement and
careful analysis are requried.

--
Curtis