Native JIT compiler for Django?

215 views
Skip to first unread message

Etienne Robillard

unread,
Jan 27, 2018, 2:04:44 PM1/27/18
to django...@googlegroups.com, django-...@googlegroups.com
Hi,

I would like to know if it's possible to compile Django code with a JIT
compiler to a C file using LLVM?

What do you think?

Etienne


Etienne Robillard
tka...@yandex.com
https://www.isotopesoftware.ca/

Jani Tiainen

unread,
Jan 28, 2018, 3:12:54 AM1/28/18
to django...@googlegroups.com
Hi

Why you even want to add such a complexity?

Have you measured what part in your apps are really slow?

Have you tried to optimize your python code before trying such an extreme actions?

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/ec8bf736-c316-8e9a-01ec-16383c78e746%40yandex.com.
For more options, visit https://groups.google.com/d/optout.

Etienne Robillard

unread,
Jan 28, 2018, 3:54:18 AM1/28/18
to Jani Tiainen, django...@googlegroups.com, django-...@googlegroups.com

Hi Jani,

I don't want to optimize my code manually except with machine-compiled instructions.

Cheers,

Etienne

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

For more options, visit https://groups.google.com/d/optout.

Jani Tiainen

unread,
Jan 28, 2018, 5:02:11 AM1/28/18
to django...@googlegroups.com

Hi,

So what is your motivation to even try to make Python(and thus Django to convert it to C code (or run it on top of something like LLVM)?

Is there some part which is too slow in your application? Are you even sure that doing things that Django does would even be faster when compiling it to something like C-code?

In my projects I have few really complex calculation systems and I have resolved them by writing the slow part in C(++) and using C-types to bind C code to my Python (Django) app.

If you're just seeking generic speedup, there exists PyPy project [1] which can run Django and relevant database backends at least successfully. PyPy is JIT for python, works quite well without any modifications to Python code.

PyPy can run Django library just fine, I have seen reports about 15 times faster execution in most operations. Never used it myself though since I haven't really found out Python be too slow in my apps.





--
Jani Tiainen

- Well planned is half done, and a half done has been sufficient before...

Etienne Robillard

unread,
Jan 28, 2018, 5:17:16 AM1/28/18
to Jani Tiainen, django...@googlegroups.com

The reason I want to use a JIT compiler for running embedded Django applications is this:


"More recently, developers began pioneering a new field in dynamic compilation called trace compiling, or tracing. During compilation into the intermediate language, the compiler can mark the branches within the code, for example, a label in front of a return statement, a jmp operation at a break statement, or call or return. The reasoning is simple, if you strip away all of the jumps and function calls, no matter how long the program is, as long as its written by a human, then from an engineer's perspective, the program will always run in negligible time. Where programs begins to eat up memory however, are the various loops, recursive calls, and whatnot. Essentially, the introduction of branches makes the complexity of the program unpredictable, hence by optimizing the frequently used loops and function calls, we can effectively balance the efficiency of the program with the resources required to compile the code (which is still not trivial by any means). It also solves another major problem. Whenever the program counter (or instruction pointer) hits one of the branch markers, its entire execution flow is traced and stored (and will eventually be compiled if that route is taken frequently enough). Because the program flow of a string concatenation differs from those of an integer addition or even an overloaded function, each of these flow generates unique traces with their unique signatures (the input). For this very reason, if we encounter a string concatenation before an integer addition, the program will not only be able to identify the integer case as a completely different route, but will also be able to break down the complexity of the environment so that further optimizations during the compilation of the integer route can safely assume that its inputs are strictly integers. Neat huh?" [1]


https://gist.github.com/leegao/1073233

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

For more options, visit https://groups.google.com/d/optout.

James Bennett

unread,
Jan 28, 2018, 5:25:41 AM1/28/18
to django...@googlegroups.com, Jani Tiainen
You have not really provided an argument, though.

Programmers generally do not just do a thing for the sake of doing the thing, even if it's a "neat" thing. They do a thing because they have a need for that thing. We turn to things that can speed up our programs once we have determined that some part of our program is not running fast enough to meet some specific goal. Until we know that we have a problem, we don't try to reach for a solution.

Django can, as the other reply mentioned, run on PyPy, which uses a nice JIT compiler. If you have no specific goal in mind other than "I heard about JIT and want to use it", then run Django on PyPy. If you want to have web applications written in C, write your web applications in C.

Jason

unread,
Jan 28, 2018, 7:55:42 AM1/28/18
to Django users
What I get from you, Etienne, is that you think this is a cool area to look into, but don't have anything else aside from that suspect coolness factor to apply to your proposal.  Besides, this thing you said above

I don't want to optimize my code manually except with machine-compiled instructions.


makes me extremely concerned that you're in the middle of an X-Y Problem and are avoiding the core issue because you don't feel its your responsibility to practice good design, execute profiling to benchmark performance and implement what optimizations are available for the bottlenecks identified.
 

Etienne Robillard

unread,
Jan 28, 2018, 8:10:54 AM1/28/18
to Jason, django...@googlegroups.com

Hi Jason,

Please don't read me wrong. I want to allow my users to optimize the efficiency of Python/Django web applications using JIT.  

Cheers,

Etienne

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

For more options, visit https://groups.google.com/d/optout.

Jani Tiainen

unread,
Jan 28, 2018, 8:31:18 AM1/28/18
to django...@googlegroups.com
Hi.


28.1.2018 3.10 ip. "Etienne Robillard" <tka...@yandex.com> kirjoitti:

Hi Jason,

Please don't read me wrong. I want to allow my users to optimize the efficiency of Python/Django web applications using JIT.  


And thats the catch. You can easily write code that is slow and no language, no JIT or tool can fix.

For example many times problem with Django is poorly written ORM code. By doing unnecessary database queries. 

Also to back up your claims how you can be sure that using tool X over normal Python is faster?

What if your tool actally is slower?

Also you have to note that Django is only one part of whole stack of tools used to run Django web app.

Cheers,

Etienne


Le 2018-01-28 à 07:55, Jason a écrit :
What I get from you, Etienne, is that you think this is a cool area to look into, but don't have anything else aside from that suspect coolness factor to apply to your proposal.  Besides, this thing you said above

I don't want to optimize my code manually except with machine-compiled instructions.


makes me extremely concerned that you're in the middle of an X-Y Problem and are avoiding the core issue because you don't feel its your responsibility to practice good design, execute profiling to benchmark performance and implement what optimizations are available for the bottlenecks identified.
 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

Etienne Robillard

unread,
Jan 28, 2018, 8:47:38 AM1/28/18
to Jani Tiainen, django...@googlegroups.com

Hi Jani,

I agree.

I'm really not interesting in "fixing" bad code. I work with open-source grade only softwares like Django. :)

Cheers,

Etienne

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

For more options, visit https://groups.google.com/d/optout.

Etienne Robillard

unread,
Jan 28, 2018, 1:07:03 PM1/28/18
to Jani Tiainen, django...@googlegroups.com, django-...@googlegroups.com

Django-hotsauce is a high-performance toolkit for Django. Using Cython under the hood is really a good method to optimize standard Django applications, but I want to investigate how JIT and PyPy can further improve performances beyond Cython.

 

Le 2018-01-28 à 11:14, Jani Tiainen a écrit :
So whats the whole fuzz about JIT and all that if you're happy with Django as is and how it works with Python..? :-o

--
Jani Tiainen

- Well planned is half done, and a half done has been sufficient before...
Reply all
Reply to author
Forward
0 new messages